<?xml version="1.0" encoding="utf-8"?>
<comments>
  <row Id="1" PostId="3" Score="5" Text="Could be a poster child fo argumentative and subjective.  At the least, need to define 'valuable'." CreationDate="2010-07-19T19:15:52.517" UserId="13" />
  <row Id="2" PostId="5" Text="Yes, R is nice- but WHY is it 'valuable'." CreationDate="2010-07-19T19:16:14.980" UserId="13" />
  <row Id="3" PostId="9" Text="Again- why?  How would I convince my boss to use this over, say, Excel." CreationDate="2010-07-19T19:18:54.617" UserId="13" />
  <row Id="4" PostId="5" Score="7" Text="It's mature, well supported, and a standard within certain scientific communities (popular in our AI department, for example)" CreationDate="2010-07-19T19:19:56.657" UserId="37" />
  <row Id="5" PostId="3" Score="1" Text="Define &quot;valuable&quot;..." CreationDate="2010-07-19T19:20:28.013" UserId="5" />
  <row Id="6" PostId="14" Score="8" Text="why ask the question here?  All are community-wiki, why not just fix the canonical answer?&#xA;" CreationDate="2010-07-19T19:22:27.947" UserId="23" />
  <row Id="7" PostId="18" Score="1" Text="also the US census data http://www.census.gov/main/www/access.html" CreationDate="2010-07-19T19:25:47.877" UserId="36" />
  <row Id="9" PostId="16" Text="Andrew Gelman has a nice R library that links Bugs to R." CreationDate="2010-07-19T19:30:24.590" UserId="78" />
  <row Id="10" PostId="23" Score="3" Text="I am not sure I understand the difficulty. If the functional form is known just take the derivative otherwise take differences. Am I missing something here?" CreationDate="2010-07-19T19:31:18.657" />
  <row Id="11" PostId="43" Score="4" Text="There are many R GUI's available for Windows, so I'm not following your point." CreationDate="2010-07-19T19:34:20.383" UserId="5" />
  <row Id="12" PostId="38" Text="That's just an example - it might have a median that is much smaller, on the order of 200 (it depends on how I partition the data). That would preclude using a normal distribution, right?" CreationDate="2010-07-19T19:37:33.823" UserId="54" />
  <row Id="13" PostId="20" Score="1" Text="What levels of kurtosis and skewdness are acceptable to meet the assumption of normality?" CreationDate="2010-07-19T19:38:01.450" UserId="24" />
  <row Id="14" PostId="46" Score="5" Text="this is an incredibly unclear response. Please try to write in English. " CreationDate="2010-07-19T19:38:30.340" UserId="74" />
  <row Id="15" PostId="3" Score="2" Text="Maybe the focus shouldn't be on &quot;valuable&quot; but rather &quot;pros&quot; and &quot;cons&quot; of each project?" CreationDate="2010-07-19T19:44:47.753" UserId="24" />
  <row Id="16" PostId="8" Score="2" Text="Probably better asked on meta.  People feel like they have to downvote it because it is an off-topic question.  But then the downvotes make it look like staticians have no sense of humor :(" CreationDate="2010-07-19T19:46:19.480" UserId="13" />
  <row Id="18" PostId="36" Score="4" Text="http://xkcd.com/552/" CreationDate="2010-07-19T19:48:32.580" UserId="68" />
  <row Id="19" PostId="8" Text="@Jason Punyon in particular gets a humorless downvote for removing my &quot;verboten&quot; tag! ;-)" CreationDate="2010-07-19T19:50:40.047" UserId="6" />
  <row Id="20" PostId="54" Score="3" Text="I am not sure if characterizing one or the other as the 'wrong' formula is the way to understand the issue. It is just that the second one is 'better' in in the sense that it is an unbiased estimator of the true standard deviation. So, if you care about unbiased estimates then the second one is 'better'/'correct'." CreationDate="2010-07-19T19:51:06.567" />
  <row Id="21" PostId="77" Text="I like the first example you give. That will certainly get the students talking ;)" CreationDate="2010-07-19T19:56:18.490" UserId="8" />
  <row Id="22" PostId="56" Text="I like the analogy. I would find it very useful if there were a defined question (based on a dataset) in which an answer was derived using frequentist reasoning and an answer was derived using Bayesian - preferably with R script to handle both reasonings. Am I asking too much?" CreationDate="2010-07-19T19:56:21.640" UserId="104" />
  <row Id="23" PostId="57" Score="2" Text="-1 for wikipedia cut and paste" CreationDate="2010-07-19T19:56:28.490" UserId="74" />
  <row Id="24" PostId="73" Text="Very subjective question: this question cannot be answered, and is not suitable for a QA site." CreationDate="2010-07-19T19:58:20.090" UserId="107" />
  <row Id="25" PostId="76" Score="1" Text="+1 for ggplot2 - by far best graphs you can get" CreationDate="2010-07-19T19:59:04.910" UserId="22" />
  <row Id="27" PostId="77" Text="There's an interesting discussion by Steve Steinberg on his blog here: http://blog.steinberg.org/?p=11 about some of the implications of 1 and where it might lead in terms of Weak AI." CreationDate="2010-07-19T20:01:15.373" UserId="55" />
  <row Id="28" PostId="57" Score="4" Text="@el chief: On Stackexchange sites, it is allowed to copy and paste answers from other resources. I give the source link along with the answers. It's pointless to give -1 to a right answer. See meta for what is allowed here and what is not; and a big -1 from me to your behavior!" CreationDate="2010-07-19T20:02:25.197" UserId="69" />
  <row Id="29" PostId="79" Text="I wasn't suggesting it was, I was just curious as to why such a difference might have arisen, what sort of level of error following the wrong advice might give and whether there was a decent explanation of the difference I could give to my students." CreationDate="2010-07-19T20:03:30.203" UserId="55" />
  <row Id="30" PostId="57" Score="3" Text="Wikipedia cut and paste doesn't bother me when cited, as it is  here [is it compatible with the cc-sa license?]. However, these explanations are complicated for a beginner" CreationDate="2010-07-19T20:03:31.450" UserId="87" />
  <row Id="31" PostId="54" Text="I was characterising the formula as &quot;wrong&quot; purely in the sense that in an exam if you use the formula which isn't proscribed by the syllabus you'll end up with the &quot;wrong&quot; answer. Plus if the values are not a sample of population per se then surely the first formula gives the more accurate value." CreationDate="2010-07-19T20:05:22.803" UserId="55" />
  <row Id="32" PostId="73" Score="2" Text="Should probably be community wiki; useful question here but doesn't have definitive answer." CreationDate="2010-07-19T20:05:57.903" UserId="5" />
  <row Id="33" PostId="73" Score="1" Text="@Shane: good point. moved.&#xA;@ Egon: subjective indeed. but if the answers come from knowledgeable people i don't mind dose of subjectivity.&#xA;i've started learning R quite recently and have couple of dozens installed to explore, however i notice that there are tools that I use much more often irrespectively of the task at hand." CreationDate="2010-07-19T20:06:56.373" UserId="22" />
  <row Id="34" PostId="56" Score="1" Text="The simplest thing that I can think of that tossing a coin n times and estimating the probability of a heads (denote by p). Suppose, we observe k heads. Then the probability of getting k heads is:&#xA;&#xA;P (k heads in n trials) = (n, k) p^k (1-p)^(n-k)&#xA;&#xA;Frequentist inference would maximize the above to arrive at an estimate of p = k / n.&#xA;&#xA;Bayesian would say: Hey, I know that p ~ Beta(1,1) (which is equivalent to assuming that p is uniform on [0,1]). So, the updated inference would be:&#xA;&#xA;p ~ Beta(1+k,1+n-k) and thus the bayesian estimate of p would be &#xA;&#xA;p = 1+k / (2+n)&#xA;&#xA;I do not know R, sorry." CreationDate="2010-07-19T20:11:13.040" />
  <row Id="35" PostId="43" Text="I wasn't aware of rapidminer. That looks nice, thanks!" CreationDate="2010-07-19T20:11:28.967" UserId="33" />
  <row Id="36" PostId="3" Text="Or maybe even &quot;How X will help you get Y done faster/cheaper and kill the germs that cause bad breath.&quot;" CreationDate="2010-07-19T20:15:51.763" UserId="13" />
  <row Id="38" PostId="73" Text="It would be interesting if StackExchange could support some method of linking community wiki posts across sites.  Because I will bet this question has been asked on Stackoverflow and I also think that Statistical Analysis may attract some people that wouldn't usually visit SO." CreationDate="2010-07-19T20:19:28.543" UserId="13" />
  <row Id="39" PostId="75" Score="1" Text="This is really a Stackoverflow question as it has to do with learning the R programming language.  With the current wording the question is only associated with statistics by virtue of R's focus on statistical analysis." CreationDate="2010-07-19T20:23:01.700" UserId="13" />
  <row Id="41" PostId="16" Score="2" Text="@Colin: I'd rephrase that &quot;the most popular statistical tool in bioinformatics&quot;... Bioinformaticians doing microarray analysis use it extensively, yes. But bioinformatics is not limited to that ;)" CreationDate="2010-07-19T20:25:52.130" UserId="120" />
  <row Id="42" PostId="65" Score="2" Text="Colin, an unbiased estimator of the standard deviation does not have a closed form representation in the general case. What does exist is the unbiased estimator of the &lt;i&gt;variance&lt;/i&gt; (s&lt;sup&gt;2&lt;/sup&gt; in this case).&#xA;Noteworthy that both are consistent estimators of the population variance - and so by the continuous mapping theorem, are the two estimators of the standard deviations.&#xA;A related point is that s&lt;sub&gt;n&lt;/sub&gt;&lt;sup&gt;2&lt;/sup&gt; has a lower MSE than s&lt;sup&gt;2&lt;/sup&gt;. The additional advantage from imposing unbiasedness is arguable." CreationDate="2010-07-19T20:30:23.307" UserId="47" />
  <row Id="43" PostId="75" Text="I guess you're right. Voting to close my own question." CreationDate="2010-07-19T20:32:02.100" UserId="69" />
  <row Id="44" PostId="103" Score="3" Text="i suggest everyone put their favourite image from the blog, so it's not just a collection of links..." CreationDate="2010-07-19T20:37:20.480" UserId="74" />
  <row Id="45" PostId="73" Text="@Sharpie: there have been several interesting SO posts like http://stackoverflow.com/questions/1295955/what-is-the-most-useful-r-trick or http://stackoverflow.com/questions/1535021/whats-the-biggest-r-gotcha-youve-run-across however they are not focused on packages. and i agree, linkage of community wiki could be really useful." CreationDate="2010-07-19T20:37:27.563" UserId="22" />
  <row Id="46" PostId="44" Score="4" Text="This is an incredibly general question." CreationDate="2010-07-19T20:38:14.037" UserId="46" />
  <row Id="47" PostId="105" Text="+1 Beat me by 10 seconds with that one..." CreationDate="2010-07-19T20:39:12.240" UserId="5" />
  <row Id="48" PostId="103" Text="There is no single correct answer, should be wiki..." CreationDate="2010-07-19T20:39:39.180" UserId="122" />
  <row Id="49" PostId="103" Text="@mike...ah...it is wiki..." CreationDate="2010-07-19T20:40:02.767" UserId="5" />
  <row Id="50" PostId="16" Text="@Nicojo - Very good point!" CreationDate="2010-07-19T20:43:25.707" UserId="8" />
  <row Id="51" PostId="65" Text="@Tirthankar - very sloppy of me. I've altered the answer slightly. Thanks." CreationDate="2010-07-19T20:45:49.057" UserId="8" />
  <row Id="52" PostId="38" Text="The normal approximation to the Poisson distribution is pretty robust, the difference between the CDFs is bounded by something like 0.75/sqrt(lambda), if I recall correctly.  I wouldn't be too worried about using lambda=200, but if you're more risk-averse then definitely go with the negative binomial." CreationDate="2010-07-19T20:46:12.283" UserId="61" />
  <row Id="53" PostId="80" Text="thank you user93!" CreationDate="2010-07-19T20:46:46.277" UserId="58" />
  <row Id="54" PostId="80" Text="By &quot;January 1st&quot;, I presume you mean the cut-off is an entire year and not 6 months period that applies to your children." CreationDate="2010-07-19T20:48:04.963" UserId="58" />
  <row Id="55" PostId="59" Score="3" Text="Do you get bonus points for the first graph in the stats exchange?" CreationDate="2010-07-19T20:53:53.513" UserId="8" />
  <row Id="56" PostId="101" Score="1" Text="Take your time! I won't be thinking about selecting a &quot;Best Answer&quot; for a week or so." CreationDate="2010-07-19T20:54:37.757" UserId="13" />
  <row Id="57" PostId="75" Score="1" Text="I wouldn't vote to close it just yet- there could be a good question in there.  Perhaps something like &quot;Where can I find useful tutorials that focus on putting statistical concepts into practice using a tool such as R?&quot; or &quot;Where can I find useful tutorials that teach statistics by example using tools such as R?&quot;" CreationDate="2010-07-19T20:58:26.750" UserId="13" />
  <row Id="59" PostId="116" Text="It doesn't have an RSS feed though :(" CreationDate="2010-07-19T21:11:30.023" UserId="8" />
  <row Id="60" PostId="110" Text="Thanks for the great answer and for book advice. Also, do you know about relation of window function to generalized functions? It seems (from wikipedia article) they are suitable as domains for functionals." CreationDate="2010-07-19T21:18:07.720" UserId="117" />
  <row Id="62" PostId="116" Text="http://cscs.umich.edu/~crshalizi/weblog/index.rss" CreationDate="2010-07-19T21:20:53.977" UserId="61" />
  <row Id="64" PostId="118" Score="5" Text="In a way, the measurement you proposed is widely used in case of error (model quality) analysis -- then it is called MAE, &quot;mean absolute error&quot;." CreationDate="2010-07-19T21:30:23.000" UserId="88" />
  <row Id="65" PostId="130" Score="3" Text="If you would taste R, it is highly probable that you will resign from MATLAB (as in my case). " CreationDate="2010-07-19T21:33:33.003" UserId="88" />
  <row Id="66" PostId="116" Text="@Rich - Thanks. I wonder why FF didn't pick it up?" CreationDate="2010-07-19T21:34:21.303" UserId="8" />
  <row Id="67" PostId="130" Text="IMO, this should be community wiki (language &quot;versus&quot; type questions are pretty subjective)." CreationDate="2010-07-19T21:34:28.043" UserId="5" />
  <row Id="68" PostId="130" Text="This is definitely a question concerning programming languages and should be asked on Stack Overflow." CreationDate="2010-07-19T21:35:25.750" UserId="13" />
  <row Id="69" PostId="130" Text="I agree with Sharpie.  @Vivi: you should change the question title to be &quot;advantages and disadvantages for data munging&quot; or something along that line so that it's more on-topic." CreationDate="2010-07-19T21:37:16.187" UserId="5" />
  <row Id="70" PostId="134" Score="1" Text="I think this is the first candidate to be moved to Stack Overflow." CreationDate="2010-07-19T21:38:27.890" UserId="88" />
  <row Id="71" PostId="120" Score="3" Text="Nice analogy of euclidean space!" CreationDate="2010-07-19T21:38:48.373" UserId="83" />
  <row Id="73" PostId="138" Score="6" Text="Should be community wiki." CreationDate="2010-07-19T21:40:12.757" UserId="5" />
  <row Id="74" PostId="116" Text="@Colin: The author of the blog would need to put `&lt;link rel=&quot;alternate&quot; type=&quot;application/rss+xml&quot; title=&quot;RSS&quot; href=&quot;http://cscs.umich.edu/~crshalizi/weblog/index.rss&quot; /&gt;` in his the `&lt;head&gt;` section of the HTML document for Firefox to pick it up automatically :)" CreationDate="2010-07-19T21:42:04.757" UserId="66" />
  <row Id="75" PostId="138" Score="6" Text="Asked a purely programming question, should be on Stack Overflow. Also possibly a duplicate of http://stats.stackexchange.com/questions/75/where-can-i-find-useful-r-tutorials-with-various-implementations" CreationDate="2010-07-19T21:47:08.097" UserId="13" />
  <row Id="76" PostId="130" Text="I don't know what munging is  :(" CreationDate="2010-07-19T21:51:17.790" UserId="90" />
  <row Id="77" PostId="130" Score="3" Text="@Sharpie, @Shane IMO to this extent it is a question about tools, so it is acceptable. " CreationDate="2010-07-19T21:54:08.290" UserId="88" />
  <row Id="78" PostId="145" Text="similar to http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples/70#70" CreationDate="2010-07-19T21:54:47.917" UserId="22" />
  <row Id="79" PostId="145" Text="Voting to close as a duplicate." CreationDate="2010-07-19T21:56:44.593" UserId="13" />
  <row Id="80" PostId="130" Text="@mbq, we definitely need a set of community guidelines on these sorts of questions, help us decide on meta: http://meta.stats.stackexchange.com/questions/1/how-to-answer-r-questions" CreationDate="2010-07-19T21:59:52.943" UserId="13" />
  <row Id="81" PostId="130" Text="@Sharpie It is not only in contex of R -- to this end I've made a separate meta discussion for that, http://meta.stats.stackexchange.com/questions/35/how-much-programming-here" CreationDate="2010-07-19T22:18:16.563" UserId="88" />
  <row Id="82" PostId="20" Score="3" Text="Most statistical methods assume normality, not of the data, but rather of an assumed random variable, e.g. the error term in a linear regression. Checking involves looking at the residuals, not the original data!" CreationDate="2010-07-19T22:24:14.713" />
  <row Id="83" PostId="145" Text="I agree, first duplicate ;-)" CreationDate="2010-07-19T22:28:22.817" UserId="88" />
  <row Id="84" PostId="7" Text="This should be community-wiki." CreationDate="2010-07-19T22:31:13.873" UserId="88" />
  <row Id="85" PostId="75" Text="http://meta.stats.stackexchange.com/questions/35/how-much-programming-here" CreationDate="2010-07-19T22:39:45.467" UserId="88" />
  <row Id="86" PostId="110" Text="The full technical explanation is complicated.  The shorter, less technical explanation is that, since they are typically smooth, and either of compact support or they decay exponentially fast, many (but definitely not all) windowing functions make good 'test functions' to examine operator behavior with.  A good starting point might be the chapter on distributions in http://see.stanford.edu/materials/lsoftaee261/book-fall-07.pdf" CreationDate="2010-07-19T22:42:33.997" UserId="61" />
  <row Id="87" PostId="130" Text="@Sharpie : check this out: http://meta.stats.stackexchange.com/questions/1/how-to-answer-r-questions" CreationDate="2010-07-19T22:52:59.610" UserId="90" />
  <row Id="88" PostId="157" Text="+1 that problem twisted my brain when I first read and thought about it- and the solution is pretty simple but teaches a lot about probability." CreationDate="2010-07-19T23:01:30.547" UserId="13" />
  <row Id="89" PostId="86" Text="It's probably best not to use the term &quot;normal variable&quot; here when you do not mean a normally distributed random variable." CreationDate="2010-07-19T23:28:39.953" UserId="159" />
  <row Id="90" PostId="143" Text="Random Forest seems very interesting. Thanks. :)" CreationDate="2010-07-19T23:32:20.603" UserId="131" />
  <row Id="91" PostId="137" Text="This is a great answer! I'll look into your book suggestions, and the description of your process is also great. I especially like the suggestions for feature vectorizations." CreationDate="2010-07-19T23:34:20.443" UserId="131" />
  <row Id="92" PostId="137" Text="(If anyone would like to elaborate even further on the vectorization part, that'd be great.)" CreationDate="2010-07-19T23:35:23.343" UserId="131" />
  <row Id="93" PostId="150" Text="curious that that book has no reviews on Amazon" CreationDate="2010-07-19T23:54:17.283" UserId="74" />
  <row Id="94" PostId="86" Text="Agreed.  Although I personally would look at someone funny for a few seconds if they said &quot;normal variable&quot; and didn't throw the word &quot;random&quot; or &quot;distributed&quot; in there somewhere to cue me that that is what they were discussing.  But I am also an engineer and not a statistician so I don't use that much domain-specific notation." CreationDate="2010-07-19T23:56:45.997" UserId="13" />
  <row Id="95" PostId="145" Score="2" Text="Actually, I think the previous question is more a subset of this one. EAMann was asking for a datasets with some particular characteristics (although I'm not sure anybody is paying attention to those criteria); this question is wide-open.  For example, I feel comfortable voting up many of these answers, because they are, in fact, datasets, but not any of the answers on the previous question, because I haven't opened them up to see if they suit EAMann's requests." CreationDate="2010-07-20T00:10:26.297" UserId="71" />
  <row Id="96" PostId="101" Text="Now that I've had a chance to come back and read the whole answer- a big +1 for the student height example.  Very clear and well laid out." CreationDate="2010-07-20T00:12:40.647" UserId="13" />
  <row Id="97" PostId="43" Score="3" Text="&quot;Windows-based&quot; may mean that it works on the Windows operating system (which R does) rather than meaning that it is a heavily GUI-oriented tool. Note the capital letter!" CreationDate="2010-07-20T01:01:12.440" UserId="173" />
  <row Id="98" PostId="5" Score="8" Text="It's extensible and there's no statistical technique that can't be done in it." CreationDate="2010-07-20T01:22:43.947" UserId="1356" />
  <row Id="99" PostId="134" Text="Possibly, but it'd need a lot more explanation on SO." CreationDate="2010-07-20T01:29:42.683" UserId="174" />
  <row Id="100" PostId="86" Score="2" Text="Random variables may be classified as *discreet* if they don't draw attention to themselves. If they're merely countable we say *discrete* :-P Also, you mean prescribe rather than proscribe, but I think *describe* might be more appropriate. Nice answer, anyway -- hopefully +1 will help mitigate the nitpicking!" CreationDate="2010-07-20T01:43:31.557" UserId="174" />
  <row Id="101" PostId="46" Text="maybe so. is a person asking this question a person who walked in off the street, or a person who has at least opened a statistics book. &#xA;&#xA;Telling someone the standard deviation is just the square root of the variance is completely begging the question. " CreationDate="2010-07-20T01:59:53.420" UserId="62" />
  <row Id="104" PostId="188" Score="2" Text="That's a good explanation, but not for a nontechnical layperson. I suspect the OP wanted to know how to explain it to, say, the MBA who hired you to do some statistical analysis! How would you describe MCMC to someone who, at best, sorta understands the concept of a standard deviation (variance, though, may be too abstract)?" CreationDate="2010-07-20T02:18:02.333" UserId="6" />
  <row Id="105" PostId="196" Text="To give everyone a chance -- this should be wikified" CreationDate="2010-07-20T02:35:25.880" UserId="87" />
  <row Id="106" PostId="188" Score="2" Text="@Harlan: It's a hard line to straddle; if someone doesn't at least know what a random variable is, why we might want to estimate probabilities, and have some hazy idea of a density function, then I don't think it *is* possible to meaningfully explain the how or why of MCMC to them, only the &quot;what&quot;, which in this case would boil down to &quot;it's a way of numerically solving an otherwise impossible problem by simulation, like flipping a coin a lot to estimate the probability that it lands on heads&quot;." CreationDate="2010-07-20T02:39:24.543" UserId="61" />
  <row Id="108" PostId="25" Text="do you mean a GUI graphical tool that runs on Windows, or a command line based one that runs on Windows (or either)" CreationDate="2010-07-20T04:09:28.610" UserId="74" />
  <row Id="109" PostId="200" Text="@walkytalky, coming up with a persuasive theoretical framework that explains the patterns is very hard, especially when there are thousands of variables involved. Real world is a complicated business. " CreationDate="2010-07-20T04:33:45.423" UserId="175" />
  <row Id="110" PostId="101" Score="1" Text="Nice work ... but we need to add &#xA;&#xA;(C) our model (embodied in the formula/statistical routine) is wrong." CreationDate="2010-07-20T05:07:08.683" UserId="187" />
  <row Id="112" PostId="54" Score="6" Text="Srikant, I don't think that the second one is an unbiased estimator.  The square of it *is* an unbiased estimator of the true variance.  However, Jensen's Inequality establishes that the expectation of a curvilinear function of a random variable is not the same as the function of the expectation of the random variable.  Hence the second formula can't be an unbiased estimator of the true standard deviation." CreationDate="2010-07-20T05:28:20.490" UserId="187" />
  <row Id="113" PostId="200" Text="@Ngu Nobody promised it would be easy. Expecting meaningful patterns just to magically emerge from a sea of data is what gives data miners a bad name." CreationDate="2010-07-20T05:29:24.030" UserId="174" />
  <row Id="114" PostId="220" Score="2" Text="Answer can be checked from most textbooks / google. " CreationDate="2010-07-20T05:41:15.367" UserId="195" />
  <row Id="115" PostId="167" Text="Wouldn't that be FA then?" CreationDate="2010-07-20T05:52:50.417" UserId="144" />
  <row Id="116" PostId="222" Text="el chef has a condensed answer over here -&gt; http://stats.stackexchange.com/questions/146/pca-scores-in-multiple-regression. HTH" CreationDate="2010-07-20T05:53:59.137" UserId="144" />
  <row Id="117" PostId="224" Text="What do you mean, standalone application?" CreationDate="2010-07-20T06:05:54.420" UserId="5" />
  <row Id="118" PostId="225" Text="The title says 99th percentile, but obviously it should be 98th; sorry about that.  The question still holds." CreationDate="2010-07-20T06:08:48.517" UserId="196" />
  <row Id="119" PostId="167" Score="1" Text="No. FA is not regression. I am referring to a response variable regressed against the principal components computed from a large number of explanatory variables. The principal components themselves are closely related to the factors of FA." CreationDate="2010-07-20T06:14:32.780" UserId="159" />
  <row Id="120" PostId="223" Text="What's the significance of the fact that your friend is an MD?" CreationDate="2010-07-20T06:25:48.217" UserId="5" />
  <row Id="121" PostId="225" Score="1" Text="You should be able to edit the title (I did a similar thing myself earlier)." CreationDate="2010-07-20T06:26:28.887" UserId="173" />
  <row Id="122" PostId="224" Text="By standalone application I mean an executable program." CreationDate="2010-07-20T06:32:54.647" UserId="128" />
  <row Id="123" PostId="223" Text="I think the significance of the MD is the person in question is a busy clinician, and not a researcher. Therefore we are being guided towards your less weighty tomes as source recommendations. Well, that's from the MD's I come across anyway." CreationDate="2010-07-20T06:42:40.103" UserId="199" />
  <row Id="124" PostId="25" Text="I mean anything (GUI or command line based) that can run on Windows operating systems." CreationDate="2010-07-20T06:57:11.337" UserId="69" />
  <row Id="125" PostId="170" Score="1" Text="Consider making this Comm Wiki as there will be no best single answer." CreationDate="2010-07-20T07:02:17.910" UserId="69" />
  <row Id="126" PostId="167" Text="I'm sorry, I should have been more precise in my comment. Your writing that the explanatory variables can be reduced to a small number of PC rang me &quot;factor analysis&quot; bell." CreationDate="2010-07-20T07:32:47.977" UserId="144" />
  <row Id="127" PostId="223" Text="Exactly! The fact that he is an MD poses a few restriction on (a) the volume of the introductory material (b) on what to assume about the &quot;starting point&quot; and (c) on the time willing to spend to reintroduce himself with basic stuff." CreationDate="2010-07-20T07:48:49.047" UserId="79" />
  <row Id="128" PostId="118" Text="In accepting an answer it seems important to me that we pay attention to whether the answer is circular.  The normal distribution is based on these measurements of variance from squared error terms, but that isn't in and of itself a justification for using (X-M)^2 over |X-M|." CreationDate="2010-07-20T07:59:54.683" UserId="196" />
  <row Id="129" PostId="10" Score="1" Text="Technically Likert scales are the sum of Likert-type items and as such end up being a reasonable approximation (at least according to many psychometricians in Psychology) of an interval data point." CreationDate="2010-07-20T08:03:30.507" UserId="196" />
  <row Id="130" PostId="240" Text="Univariate boxplots are useful for spotting univariate outliers. But they can completely miss multivariate outliers.&#xA;&#xA;The regression idea is ok if I had a Y and a bunch of X variables. But as I said in the question, there is no Y so regression is inappropriate." CreationDate="2010-07-20T08:12:10.470" UserId="159" />
  <row Id="131" PostId="233" Score="4" Text="Residuals from what? You can't have residuals without a model." CreationDate="2010-07-20T08:12:30.407" UserId="159" />
  <row Id="132" PostId="241" Text="Yes, I could compute the Mahalanobis distance of each observation from the mean of the data. But the observations with the greatest distance from the mean are not necessarily multivariate outliers. Think of a bivariate scatterplot with high correlation. An outlier can be outside the main cloud of points but not that far from the mean." CreationDate="2010-07-20T08:14:21.200" UserId="159" />
  <row Id="136" PostId="229" Text="Is there a way to express in plain language what the difference is between a given percentile and the maximum of N values?  From a lay perspective, it is hard to see why a datapoint that comes from a given (Y) percentile wouldn't be expected to be (on average) the same as the top scorer from a group of 100/Y.  &#xA;&#xA;For example, if I found that your answers were ranked in the 90th percentile, I'd expect that your answer would usually be the top answer among any randomly selected group of 10 answers." CreationDate="2010-07-20T08:45:52.963" UserId="196" />
  <row Id="137" PostId="242" Text="could you edit the title to something like &quot;Using time series analysis to analyze/predict violent behavior&quot;? " CreationDate="2010-07-20T08:56:46.340" UserId="87" />
  <row Id="138" PostId="134" Score="2" Text="Most programmers know &quot;median&quot;.  (sort(array))[length/2] is a big enough hint for those who forgot.  Also at its most basic for each new point you only need to do a bisection/insert on one half of the array..." CreationDate="2010-07-20T09:04:20.020" UserId="87" />
  <row Id="139" PostId="250" Text="R is also lazy evaluated." CreationDate="2010-07-20T09:04:48.753" UserId="88" />
  <row Id="140" PostId="189" Text="Thats the same as taking the derivative, but just more inaccurate so why would you do it?" CreationDate="2010-07-20T09:39:03.803" UserId="214" />
  <row Id="141" PostId="261" Text="How about putting it in publication?" CreationDate="2010-07-20T09:59:57.197" UserId="217" />
  <row Id="142" PostId="263" Text="How to make a vector file usable in a publication using it?" CreationDate="2010-07-20T10:01:17.570" UserId="217" />
  <row Id="143" PostId="261" Score="4" Text="R produces some of the best quality graphics around. As an editor of an international research journal, I would love all our authors to use R." CreationDate="2010-07-20T10:02:05.183" UserId="159" />
  <row Id="144" PostId="263" Text="@Łukasz I added it to my answer" CreationDate="2010-07-20T10:11:50.243" UserId="190" />
  <row Id="145" PostId="263" Text="Could you put a code snippet for completness? It would be very useful for people in the future finding this page." CreationDate="2010-07-20T10:19:14.293" UserId="217" />
  <row Id="146" PostId="262" Text="How about saving to file?" CreationDate="2010-07-20T10:19:47.867" UserId="217" />
  <row Id="147" PostId="254" Text="Thank you. Would you please explain the two lines of R code for those of us (me included) who don't even understand R syntax." CreationDate="2010-07-20T10:33:59.177" UserId="213" />
  <row Id="148" PostId="250" Text="@mbq good to know, thanks!" CreationDate="2010-07-20T10:48:05.383" UserId="171" />
  <row Id="149" PostId="257" Score="2" Text="How do you define 'publication-quality'? Please elaborate on what aspects you like to see covered... e.g. color use, line widths, etc. Should answers focus on font size, instead?" CreationDate="2010-07-20T10:51:44.160" UserId="107" />
  <row Id="150" PostId="263" Text="@Łukasz Hmm, some suggestion how to upload an svg figure?" CreationDate="2010-07-20T10:52:57.933" UserId="190" />
  <row Id="151" PostId="261" Score="1" Text="@Rob... see my comment on the question... how do you define 'publication-quality', or 'best quality'... from a editor perspective?" CreationDate="2010-07-20T10:53:25.593" UserId="107" />
  <row Id="152" PostId="261" Score="8" Text="I like to see vector graphics (no jpegs), graphical design following the principles of Tufte &amp; Cleveland, readable fonts, uncluttered legends, no shaded backgrounds, sensible axis limits and tick intervals, labelled axes, no overlap of text and plotting characters or lines, etc. Most authors use the default settings of their software, so good software has good defaults. This is where Excel fails miserably and R does pretty well. But it is possible to produce lousy graphs in R and good graphs in Excel. It's just easier to produce high quality graphics in R." CreationDate="2010-07-20T11:07:42.903" UserId="159" />
  <row Id="154" PostId="267" Text="You should use Colin's answer, still your idea of making Monte Carlo simulation is also correct." CreationDate="2010-07-20T11:14:06.997" UserId="88" />
  <row Id="155" PostId="163" Text="It's a good example, but I'd rather see a proper definition (a function from a sample space to the real numbers, which you can make more general) followed up by an example. " CreationDate="2010-07-20T11:14:55.340" UserId="62" />
  <row Id="156" PostId="254" Text="Thank you again. Please see my edit to the original post that clarifies the problem even more" CreationDate="2010-07-20T11:21:08.607" UserId="213" />
  <row Id="157" PostId="274" Text="Good answer. I think you should go further into what you mean by &quot;do your inference based on that&quot;. That's kind of the second part of my question. " CreationDate="2010-07-20T11:36:16.100" UserId="62" />
  <row Id="158" PostId="274" Text="mmm... I didn't really understand what you meant by what common variables and statistics... Oh, do you mean like you use z distribution if you have the population variance and the t-distribution if you only have the sample variance and the sample size is small? Something along those lines?" CreationDate="2010-07-20T12:03:37.080" UserId="90" />
  <row Id="159" PostId="278" Score="1" Text="Your random reordering reminded me of this AI koan: In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. &quot;What are you doing?&quot;, asked Minsky. &quot;I am training a randomly wired neural net to play Tic-Tac-Toe.&quot; &quot;Why is the net wired randomly?&quot;, asked Minsky. &quot;I do not want it to have any preconceptions of how to play.&quot; Minsky shut his eyes. &quot;Why do you close your eyes?&quot;, Sussman asked his teacher. &quot;So the room will be empty.&quot; At that moment, Sussman was enlightened." CreationDate="2010-07-20T12:40:15.697" UserId="56" />
  <row Id="160" PostId="263" Score="2" Text="@Peter: You could have mentioned in your answer that matplotlib allows rendering of all typography in the plot with LaTeX so it perfectly integrates visually. " CreationDate="2010-07-20T12:49:37.267" UserId="56" />
  <row Id="162" PostId="213" Score="1" Text="If a scatterplot matrix won't catch it, you could try a 3D scatterplot.  That won't work out to 4D, of course, but then you could create a 4th dimension as time and make a movie.  :)" CreationDate="2010-07-20T13:06:49.603" UserId="5" />
  <row Id="163" PostId="298" Score="1" Text="Are you asking about how to reduce the effect of outliers or when to use the log of some variable?" CreationDate="2010-07-20T13:14:06.983" UserId="56" />
  <row Id="164" PostId="298" Score="2" Text="I think that the OP is saying &quot;I've heard of people using the log on input variables: why do they do that?&quot;" CreationDate="2010-07-20T13:24:23.870" UserId="5" />
  <row Id="165" PostId="274" Text="What I was getting at was that mean and standard deviation are parameters associated with the population, but they're estimated by the sample mean ((1/N)*\sum(x_i)) and the sample standard deviation ((1/(N-1))*\sum(x_i - x^bar)^2)." CreationDate="2010-07-20T13:42:11.253" UserId="62" />
  <row Id="168" PostId="249" Score="3" Text="The m_i's do not have to be equal&#xA;&#xA;Wikipedia has a simplified description of the model." CreationDate="2010-07-20T14:17:12.293" UserId="8" />
  <row Id="170" PostId="163" Score="1" Text="@Baltimark That's a valid comment which I generally agree with but would someone who is new to statistics be scared away by a formal definition and then not proceed on to the more intuitive example? I have edited my answer, but have still kept it simple :)" CreationDate="2010-07-20T14:38:40.323" UserId="81" />
  <row Id="171" PostId="262" Text="+1 for R and ggplot2" CreationDate="2010-07-20T14:38:52.787" UserId="22" />
  <row Id="172" PostId="119" Score="1" Text="I agree. Standard deviation is the *right* way to measure dispersion if you assume normal distribution. And a lot of distributions and real data are an approximately normal." CreationDate="2010-07-20T14:40:02.050" UserId="217" />
  <row Id="173" PostId="244" Text="+1 for comprehensive collection" CreationDate="2010-07-20T14:40:51.067" UserId="22" />
  <row Id="174" PostId="9" Score="1" Text="If moving from Excel is the issue, you could try:&#xA;&#xA;* http://www.coventry.ac.uk/ec/~nhunt/pottel.pdf &#xA;&#xA;* http://www.forecastingprinciples.com/files/McCullough.pdf &#xA;&#xA;* http://www.lomont.org/Math/Papers/2007/Excel2007/Excel2007Bug.pdf &#xA;&#xA;* http://www.csdassn.org/software_reports/gnumeric.pdf " CreationDate="2010-07-20T14:44:45.503" UserId="229" />
  <row Id="175" PostId="163" Text="You're right, but I'm still confused as to whether answers here are being addressed to rank beginners, or someone with a minimum of knowledge. Correct me if I'm wrong, but when I was in college, we usually took stats after a certain amount of other math classes (for me, it was after calculus, so I was familiar and comfortable with the definition of functions). " CreationDate="2010-07-20T14:44:49.387" UserId="62" />
  <row Id="176" PostId="301" Text="But still, using log changes the model -- for linear regression it is y~a*x+b, fo linear regression on log it is y~y0*exp(x/x0)." CreationDate="2010-07-20T14:50:05.523" UserId="88" />
  <row Id="177" PostId="309" Score="3" Text="Especially where students are concerned." CreationDate="2010-07-20T14:51:13.023" UserId="71" />
  <row Id="178" PostId="163" Text="@Baltimark I'm not too sure myself tbh, I've just been basing the level of my answers on how the questions are asked, so e.g. if someone asks a question showing that they have some knowledge of the field, then you can say things that won't scare them away. This question seemed basic so I assume a very simple reply is appropriate. I'm sure http://meta.stats.stackexchange.com/ has some more guidance." CreationDate="2010-07-20T14:51:44.100" UserId="81" />
  <row Id="179" PostId="232" Text="+1 for Mondrian - very useful toy, especially for large data" CreationDate="2010-07-20T14:54:06.230" UserId="22" />
  <row Id="180" PostId="310" Text="You're not missing something if all you're trying to do is estimate the parameter from a set of observations. That was definitely the main idea of the OP's question. However, she was also asking generally (if not rigorously) &quot;how to estimate poisson models&quot;. Perhaps she wants to know the value of the pdf at a specific point. In that case, the normal approx. is probably going to be better than scaling the parameter, and the observations by 100, or whatever, if the observations are large enough to make calculating the factorial impractical." CreationDate="2010-07-20T14:54:19.397" UserId="62" />
  <row Id="182" PostId="220" Text="I have voted to close this question for the reasons mentioned in this meta thread (http://meta.stats.stackexchange.com/questions/67/what-should-be-our-criteria-for-closing-questions). Please go to the meta thread if you think that we should not close questions like the one above." CreationDate="2010-07-20T15:01:21.203" />
  <row Id="183" PostId="10" Score="1" Text="@drknexus - So, multiple items serve as a measurement triangulation for construct scales?  If yes, what are the criteria for determining that a researcher has enough relevant data points (i.e., items) to use the scale as an interval measurement?" CreationDate="2010-07-20T15:06:18.870" UserId="24" />
  <row Id="185" PostId="313" Text="First of all, this is just a big example and doesn't really explain explain the concept of p-value and test-statistic. Second, you're just claiming that if you get fewer than 5 or more than 15 white marbles, you reject the null hypothesis. What's your distribution that you're calculating those probabilities from? This can be approximated with a normal dist. centered at 10, with a standard deviation of 3. Your rejection criteria is not nearly strict enough. " CreationDate="2010-07-20T15:21:27.113" UserId="62" />
  <row Id="186" PostId="224" Score="4" Text="This should probably be community wiki since there is no definitive answer." CreationDate="2010-07-20T15:30:58.523" UserId="5" />
  <row Id="188" PostId="304" Text="Answers will be reordered based on votes, so please try not to refer to other answers." CreationDate="2010-07-20T15:54:34.873" UserId="220" />
  <row Id="189" PostId="301" Text="I agree - taking log's changes your model. But if you have to transform your data, that implies that your model wasn't suitable in the first place." CreationDate="2010-07-20T16:00:09.427" UserId="8" />
  <row Id="190" PostId="99" Text="+1 for multicore - very useful indeed" CreationDate="2010-07-20T16:02:04.153" UserId="22" />
  <row Id="191" PostId="316" Text="+1 for RODBC and sqldf - i find them essential as well" CreationDate="2010-07-20T16:02:52.030" UserId="22" />
  <row Id="192" PostId="307" Score="1" Text="I seen examples where a model has ten data points and nine parameters. On pointing out that the model has too many parameters, I was told that the R^2 was 0.999 so the model must be correct!" CreationDate="2010-07-20T16:03:44.010" UserId="8" />
  <row Id="193" PostId="321" Text="What other variants?  It might be helpful to tighten up this question a little (more specificity)." CreationDate="2010-07-20T16:03:59.503" UserId="5" />
  <row Id="195" PostId="138" Score="3" Text="You should add your background. Programmers who came to R have different issues than people without a programming background." CreationDate="2010-07-20T16:06:54.697" UserId="237" />
  <row Id="196" PostId="323" Text="These were both listed in the original question..." CreationDate="2010-07-20T16:12:38.997" UserId="5" />
  <row Id="197" PostId="114" Text="I added the blogs in the question as answers to allow proper voting to find the most popular blogs." CreationDate="2010-07-20T16:13:35.627" UserId="237" />
  <row Id="198" PostId="285" Text="Thanks Jeromy. I explored several possible transformations of data using Stata's gladder function for the ladder of powers.&#xA;&#xA;Teoreically based solution will be one way to go, but before that I wanted to check for the data driven solution." CreationDate="2010-07-20T16:19:56.930" UserId="22" />
  <row Id="199" PostId="60" Text="Thanks Reed. k-means is definitely a way to go and I explored several solutions. However I am trying to go for a solution without subjective decision of the number of clusters.   Hierarchical approach on the other hand produced huge classification and again it was hard for me to specify where to stop." CreationDate="2010-07-20T16:22:43.323" UserId="22" />
  <row Id="200" PostId="323" Score="3" Text="By that reasoning another question which asks for the best blogs but lists other blogs in the starting question wouldn't be a duplicate of the question.&#xA;I think it makes sense to vote on all blogs to find out which are most popular including those already listed in the OP." CreationDate="2010-07-20T16:22:54.993" UserId="237" />
  <row Id="201" PostId="268" Text="Thanks Egon. SOM approach might be very interesting indeed. Will have a look at it." CreationDate="2010-07-20T16:23:45.883" UserId="22" />
  <row Id="202" PostId="254" Text="Colin, I tried to figure out how to expand the Wikipedia formulae to non-equal m_i's, but didn't have any success. Can you please help me with this? (sorry)" CreationDate="2010-07-20T16:38:50.220" UserId="213" />
  <row Id="205" PostId="307" Text="All worship the mighty R^2!" CreationDate="2010-07-20T17:00:03.297" UserId="229" />
  <row Id="206" PostId="43" Text="fine, you win :)" CreationDate="2010-07-20T17:06:12.040" UserId="74" />
  <row Id="208" PostId="325" Score="2" Text="Check out http://en.wikipedia.org/wiki/Statistical_hypothesis_testing and http://en.wikipedia.org/wiki/Bayesian_inference" CreationDate="2010-07-20T17:09:42.297" UserId="88" />
  <row Id="209" PostId="321" Text="Right; it is just one variant among others." CreationDate="2010-07-20T17:11:25.697" UserId="88" />
  <row Id="211" PostId="333" Score="1" Text="These are good references, but I disagree with your assessment of Ed Thrope as the founder of this field.  Statistical analysis of financial data and statistical arbitrage are not the same thing: one would perform statistical analysis for most financial analysis (e.g. modern portfolio theory)." CreationDate="2010-07-20T17:20:49.130" UserId="5" />
  <row Id="213" PostId="333" Score="1" Text="I agree, Markowitz definitely invented portfolio theory" CreationDate="2010-07-20T17:55:23.310" UserId="74" />
  <row Id="214" PostId="86" Text="@walkytalky Thanks for the corrections- I have made some fixes." CreationDate="2010-07-20T18:00:44.033" UserId="13" />
  <row Id="215" PostId="134" Text="@walkytalky I don't think it would require any more explanation than any other algorithm question on SO.  Probably less as the median is a relatively basic concept." CreationDate="2010-07-20T18:15:39.900" UserId="13" />
  <row Id="216" PostId="127" Score="1" Text="Agree strongly. Both great books. Start with Bayesian Computation With R, then get Gelman et al." CreationDate="2010-07-20T19:12:45.460" UserId="247" />
  <row Id="217" PostId="321" Text="OK, I'll ask for comparison to adaboost since that is perhaps the best known." CreationDate="2010-07-20T19:24:06.430" UserId="220" />
  <row Id="218" PostId="306" Text="Agreed - just to throw out some additional ideas on modeling: logistic to predict which patients will have 1+ violent outbursts, Poisson(esque) regression to predict which patients will have many outbursts,  multilevel to examine variations from room-to-room and/or ward-to-ward... " CreationDate="2010-07-20T20:13:46.233" UserId="71" />
  <row Id="219" PostId="7" Text="I didn't even see that checkbox.  Done :-)" CreationDate="2010-07-20T20:51:13.820" UserId="38" />
  <row Id="220" PostId="310" Text="@Srikant, you are right, to estimate the parameters the factorial is not an issue, but in general you will want the value of the likelihood for a given model, and you would have to use the factorial for that. Also, for hypothesis testing (e.g. likelihood ratio test) you will need the value of the likelihood. " CreationDate="2010-07-20T21:31:03.190" UserId="90" />
  <row Id="221" PostId="310" Text="@Baltimark: yes, I want to know in general, whether it is valid to change the unit of measurement of Poisson. I was asked this question and I didn't know what to say." CreationDate="2010-07-20T21:33:48.650" UserId="90" />
  <row Id="222" PostId="302" Text="the two answers look too different to me. One says 20 to 30, the other says 20 to 30 times slopes. So if you have 5 slopes, one rule tells you 20 to 30, the other 100 to 150 observations. That doesn't seem right to me...." CreationDate="2010-07-20T21:37:31.240" UserId="90" />
  <row Id="223" PostId="313" Text="I would agree that this is just an example, and I it is true I just picked the numbers 5 and 15 out of the air for illustrative purposes. When I have time I will post a second answer, which I hope will be more complete." CreationDate="2010-07-20T22:00:38.863" UserId="226" />
  <row Id="224" PostId="337" Text="I know, however that doesn't help me to decide whether using sample entropy or shannon entropy or some other kind of entropy is appropriate for the data that I'm working with." CreationDate="2010-07-20T22:31:17.717" UserId="237" />
  <row Id="226" PostId="337" Text="What I wrote in my post is just that for a certain type of data/process/system there is only one *true* entropy definition. Sample Entropy is *not* an entropy measure, it is just some statistic with a confusing name. Make a question where you define the data for which you want to calculate the entropy, and will get the formula." CreationDate="2010-07-20T23:17:28.040" UserId="88" />
  <row Id="227" PostId="302" Text="They are pretty different guidelines.  I suspect the disconnect is whether you think that the test of the overall model matters (the lower N guideline) or the test of the individual slopes that matter (the higher N guideline)." CreationDate="2010-07-20T23:48:55.603" UserId="196" />
  <row Id="228" PostId="358" Score="1" Text="The answer in that other thread don't really explain why 2 is a better value than other values that are very near to 2 but are no natural numbers." CreationDate="2010-07-21T00:04:53.277" UserId="237" />
  <row Id="229" PostId="10" Score="1" Text="I'm not sure; that might be a worthy question for the community in general.  I'd guess that it is probably in part a value judgement on the part of the researcher &amp; area.  Some areas are completely willing to treat a single Likert item as interval even though it clearly is ordinal.  A reasonable answer might be to use a different analysis method, e.g. a permutation or bootstrapped test.  Another answer might be to conduct a simple test of normality, so long as the aggregate doesn't significantly depart from normality you are probably okay." CreationDate="2010-07-21T00:07:20.837" UserId="196" />
  <row Id="230" PostId="360" Score="1" Text="Your point about month N's count not necessarily being correlated with N-1 is well-taken.  With a slow-growing disease like TB, that's something I'd have to look at carefully, but I'm pretty sure I could identify about how much lag there is between the time we report a source case and the time we report any secondary cases." CreationDate="2010-07-21T00:11:05.730" UserId="71" />
  <row Id="231" PostId="360" Score="1" Text="However, it's your point about analyzing the distribution of monthly counts that's at the heart of my question.  There is a definite decline in TB, both nationally in the US and in my district.  For example, when I compare 2009 to the previous years, there are decidedly fewer cases.  2010 is on track to have fewer still.  What I'm trying to identify (which I did a poor job of explaining in the question) is whether or not these declines are part of an ongoing downward trend, or just a downward wobble.  Thanks - you've gotten me to think much more carefully about the problem." CreationDate="2010-07-21T00:15:26.720" UserId="71" />
  <row Id="232" PostId="358" Text="I think it does; still I'll try to extend the answer." CreationDate="2010-07-21T00:21:35.973" UserId="88" />
  <row Id="234" PostId="135" Score="2" Text="For R ks.test in the default &quot;stats&quot; package can conduct the KS test without installing additional packages." CreationDate="2010-07-21T00:23:25.580" UserId="196" />
  <row Id="235" PostId="10" Score="1" Text="... but in general it seems like one could evoke the central limit theorem and suggest that 20 to 30 items should be sufficient to use the scale as an interval measurement." CreationDate="2010-07-21T00:26:48.193" UserId="196" />
  <row Id="236" PostId="310" Text="@Vivi: I am not sure why you would want to compute the likelihood with k_i! included as in most applications (e.g., likelihood ratio test, bayesian estimation) the constant will not matter. In any case, I do not think you can re-scale as you suggested. If I feel otherwise I will update my answer. " CreationDate="2010-07-21T00:28:39.063" />
  <row Id="237" PostId="310" Text="@Srikant, I see your point, but some softwares (Eviews, for example) include this by default, and large numbers are an issue you like it or not. I guess I was really after an explanation of why you can or can't do it rather than a way around it, but the discussion has been interesting and instructive nonetheless  :)" CreationDate="2010-07-21T00:42:20.127" UserId="90" />
  <row Id="238" PostId="363" Score="2" Text="Probably a community wiki question?" CreationDate="2010-07-21T00:48:03.580" UserId="196" />
  <row Id="239" PostId="217" Text="Protovis looks awesome but do you know what browser support it has? particularly IE?" CreationDate="2010-07-21T01:39:03.210" UserId="191" />
  <row Id="240" PostId="134" Text="@Sharpie Perhaps not more than other SO algo questions. But certainly more than what it actually says here!" CreationDate="2010-07-21T02:58:14.737" UserId="174" />
  <row Id="241" PostId="227" Text="Sorry, but what are loadings (c in your formula) and how do you determine them?" CreationDate="2010-07-21T03:17:32.773" UserId="191" />
  <row Id="242" PostId="234" Text="Would there be any benefit or could you plot them on a 3-d scatter plot? " CreationDate="2010-07-21T03:18:06.567" UserId="191" />
  <row Id="243" PostId="226" Text="How does a higher Y value &quot;explain&quot; a bigger chunk of the variance? Is it how the PCA is computed? If so I think I've got another question to post ;)" CreationDate="2010-07-21T03:31:11.207" UserId="191" />
  <row Id="244" PostId="287" Score="1" Text="See http://en.wikipedia.org/wiki/Method_of_moments_(statistics) and  http://en.wikipedia.org/wiki/Generalized_method_of_moments" CreationDate="2010-07-21T03:52:40.053" />
  <row Id="245" PostId="375" Text="Similar question on SO: http://stackoverflow.com/questions/56411/how-to-test-randomness-case-in-point-shuffling" CreationDate="2010-07-21T05:09:44.513" UserId="68" />
  <row Id="246" PostId="215" Text="I agree completely with what you're saying, but the reason for looking at tests here is to satisfy others. The situation is modelling possible extreme operational losses bassed on historical loss experience and the regulator needs to be convinced that the choice of distribution is supported by the data. What the regulator thinks is reasonable and what the business thinks is reasonable for the results can differ quite a bit! Using a (reasonably) standard statistical test may provide a somewhat independent approach to the justifying a particular choice." CreationDate="2010-07-21T05:40:47.770" UserId="173" />
  <row Id="247" PostId="372" Score="2" Text="I think this is a too broad question. Because almost all statistics can be used in data mining, I don't see any reason for this question to exist." CreationDate="2010-07-21T06:11:42.967" UserId="190" />
  <row Id="248" PostId="325" Text="I don't see what you mean with this question, nor can I imagine what kind of answer you search. Could you please elaborate more?" CreationDate="2010-07-21T06:14:34.550" UserId="190" />
  <row Id="249" PostId="375" Score="2" Text="Or this one http://stats.stackexchange.com/questions/30/" CreationDate="2010-07-21T06:16:52.393" UserId="56" />
  <row Id="250" PostId="368" Text="I don't have any requirements how correct my ASR should be. For example, when I change my model a bit my error goes down from 30% to 29.8%. For that change I want to now if it's significant" CreationDate="2010-07-21T06:17:05.280" UserId="190" />
  <row Id="251" PostId="325" Text="What do you mean by reasonable? I fully agree with Peter ! vote to close. " CreationDate="2010-07-21T06:30:28.000" UserId="223" />
  <row Id="252" PostId="215" Text="Then using a 'distance between distribution test' (like chi square or Kolmogorov-Smirnov or ... is a better idea because it is easely understood by the end user. " CreationDate="2010-07-21T06:57:31.783" UserId="223" />
  <row Id="253" PostId="372" Score="1" Text="All statistics are usable, but if your goal is to study the *most* important (or often used) parts of statistics when applied to Data Mining - a subset would be useful." CreationDate="2010-07-21T07:46:57.840" UserId="252" />
  <row Id="254" PostId="372" Score="1" Text="Suggestion: why not made this a community wiki ? (p.s I wasn't the one who voted to close this - but I understand why someone voted for it)" CreationDate="2010-07-21T07:56:08.037" UserId="253" />
  <row Id="255" PostId="372" Text="Good idea. Community wiki = on!" CreationDate="2010-07-21T08:05:44.053" UserId="252" />
  <row Id="256" PostId="113" Text="What is the difference between model selection http://www.modelselection.org/ (hot topic in statistic during the past 20 years) and method selection. " CreationDate="2010-07-21T08:30:20.000" UserId="223" />
  <row Id="257" PostId="385" Score="1" Text="I agree, but I was indeed worried about the k parameter -- if the unbalance will create some class-wise differences in the observation densities in the feature space, the same k will tend to a smaller sphere in feature space for an observation from denser class. Won't it influence the k parameter optimization then?" CreationDate="2010-07-21T08:45:38.653" UserId="88" />
  <row Id="258" PostId="372" Text="I agree with @Peter; way too vague.  " CreationDate="2010-07-21T09:40:14.487" UserId="5" />
  <row Id="261" PostId="368" Text="In that case you would use test 2. See the table for &quot;Two-proportion z-test, pooled for d0 = 0&quot; at the wiki: http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Common_test_statistics" CreationDate="2010-07-21T10:44:25.737" />
  <row Id="262" PostId="344" Text="Can you give a description a problem where RNB gave you good results?" CreationDate="2010-07-21T10:57:06.420" UserId="217" />
  <row Id="263" PostId="344" Text="No ;-) This was only to revive the pool." CreationDate="2010-07-21T11:05:37.750" UserId="88" />
  <row Id="264" PostId="395" Score="1" Text="Look at http://stats.stackexchange.com/questions/173/time-series-for-count-data-with-counts-20/ . It's not a duplicate, but the problem is similar at a first sight." CreationDate="2010-07-21T11:20:20.783" UserId="88" />
  <row Id="265" PostId="386" Text="How would this deal with cases where you don't know how many outliers you have, i.e. when the N-1 points still have a bias since they include outliers?" CreationDate="2010-07-21T11:58:10.653" UserId="56" />
  <row Id="266" PostId="386" Score="1" Text="if n is sufficiently large and the number of outlier is small then this bias is negligible. If there are a large number of outliers then, maibe it is not outliers and anyway, as I mentionned you can use leave k out strategy ... (in this case, you have to find out a strategy to avoid tracking all configurations which may be NP hard ... ) and if you don't know k, you can try many values for k and keep the most relevent. " CreationDate="2010-07-21T12:07:43.927" UserId="223" />
  <row Id="267" PostId="154" Score="1" Text="Cool! Please post a link here to your thesis once it's complete and/or published!" CreationDate="2010-07-21T12:20:54.163" UserId="6" />
  <row Id="268" PostId="293" Score="1" Text="Yes, rather than starting with the data, start with the question and the data generating process." CreationDate="2010-07-21T12:35:24.620" UserId="46" />
  <row Id="269" PostId="192" Score="2" Text="This question doesn't make a lot of sense - where is your statistical/scientific question?" CreationDate="2010-07-21T12:35:50.530" UserId="46" />
  <row Id="270" PostId="262" Score="1" Text="Or a little more succinctly with melt and qplot: `m &lt;- melt(d, id = &quot;x&quot;); qplot(variable, value, data = m, colour = variable)`&#xA;" CreationDate="2010-07-21T12:39:58.103" UserId="46" />
  <row Id="271" PostId="400" Score="7" Text="I think you mean sent by fax at some point in the **past** ;)" CreationDate="2010-07-21T12:42:57.863" UserId="46" />
  <row Id="272" PostId="165" Score="2" Text="Here is my favorite paper about the topic: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.7133&amp;rep=rep1&amp;type=pdf" CreationDate="2010-07-21T13:28:05.280" UserId="88" />
  <row Id="273" PostId="217" Score="1" Text="That's unfortunately one of Protovis's weak points (but it's an issue with any SVG library because IE doesn't support that).  Fortunately, Jamie Love has come up with a solution using SVGWeb.  See here: http://groups.google.com/group/protovis/browse_thread/thread/1a80f98a16736658?pli=1." CreationDate="2010-07-21T13:37:53.340" UserId="5" />
  <row Id="274" PostId="410" Text="I've updated the question with the information on kind of time series/mean I'm talking about." CreationDate="2010-07-21T13:41:09.180" UserId="219" />
  <row Id="275" PostId="399" Score="1" Text="Tufte's Visual Display of Quantitative Information (http://www.amazon.com/o/ASIN/0961392142/ref=nosim/gettgenedone-20) is better than Beautiful Evidence IMO. All four of his books are good though, and if you have an opportunity to attend one of his courses, do it." CreationDate="2010-07-21T13:57:31.187" UserId="36" />
  <row Id="276" PostId="415" Text="Thanks!  I was out of town last week and missed that MO post." CreationDate="2010-07-21T14:01:32.033" UserId="89" />
  <row Id="277" PostId="133" Text="I thought Chi-Sq was primarily for categorical data (contingency tables) vs. continuous?" CreationDate="2010-07-21T14:17:35.707" UserId="23" />
  <row Id="278" PostId="13" Score="1" Text="it's similar to using planet gravitational models to urban traffic. I find it absurd, but it works quiet accurately actually" CreationDate="2010-07-21T14:25:01.087" UserId="59" />
  <row Id="279" PostId="405" Text="We have done this in two different ways and it has proved very useful with the large movements this week lcearly being outside the histogram of typical results. " CreationDate="2010-07-21T14:47:33.937" UserId="210" />
  <row Id="280" PostId="402" Text="Heh - this was actually one of the plots I made that didn't make it into the post.  The problem I had is deciding how to calculate the bounds - my initial attempt was with Poisson bounds, with lambda set to the mean of my data, but variance is too high to be a proper Poisson (high enough to matter in practice?  I don't know).  " CreationDate="2010-07-21T15:01:32.140" UserId="71" />
  <row Id="281" PostId="402" Text="A further problem is that the center of the distribution can change over time - for example, it wouldn't make sense to set those bounds using data from the early 1900s, when Colorado was a haven for TB patients.  So what's an appropriate way to keep the lines up-to-date with long-term changes in the process, while still being able to identify deviations?" CreationDate="2010-07-21T15:02:21.543" UserId="71" />
  <row Id="282" PostId="217" Text="IE 9 will also support SVG- so as long as you don't need to work with IE 6..." CreationDate="2010-07-21T15:08:29.597" UserId="13" />
  <row Id="283" PostId="349" Text="The problem with the binning approach is that we do not have a good upper bound for the data, and so the midpoint for the largest bin would have to be huge. So, we'd need a huge number of bins (not enough memory for that), or have pretty wide bins (which would then lead to a fairly inaccurate answer.)&#xA;&#xA;And the data is not very sparse." CreationDate="2010-07-21T15:13:35.893" UserId="247" />
  <row Id="284" PostId="404" Text="I'll have to read it more thoroughly later on, but yes, this package is definitely addressing the kinds of problems I'm facing here.  Thanks!  And also, thanks for the kind words about the plots ;p" CreationDate="2010-07-21T15:16:07.733" UserId="71" />
  <row Id="285" PostId="352" Text="The data set is potentially too big to read in half of it...it is in a networking context where the device doing the processing can see tens of thousands of items per second, and probably has enough memory to store only a few hundred.&#xA;&#xA;Also the data is definitely not Gaussian. In fact it does not fit well to any of the common distributions." CreationDate="2010-07-21T15:17:56.660" UserId="247" />
  <row Id="286" PostId="424" Score="1" Text="Absolutely love this one." CreationDate="2010-07-21T15:23:11.967" UserId="13" />
  <row Id="287" PostId="346" Score="3" Text="@PeterR: Perhaps, asking on Stackoverflow may get better answers. " CreationDate="2010-07-21T15:33:20.303" />
  <row Id="288" PostId="113" Score="2" Text="While model selection typically involves the scoring of models within a family of distributions, based on their fit and penalizing the number of parameters used (a la AIC and BIC), whereas method selection is more general.  Method selection involves being faced with a problem (e.g. test, classify, predict) for which we have some background knowledge (variables are known to be (e.g. independence, data type), and for which auxiliary assumptions are made (e.g. normality, homoscedasticity), and we must select a method. " CreationDate="2010-07-21T15:50:24.250" UserId="39" />
  <row Id="289" PostId="113" Score="1" Text="Now there are mathematical prescriptions along the lines of measurement type, convergence results, optimality, and time/space complexity, but no framework for their systematic application, that I am aware of, thus the question." CreationDate="2010-07-21T15:53:08.243" UserId="39" />
  <row Id="290" PostId="260" Text="Aptamer active motif selection, forest ground humidity forecasting, digit OCR, multispectral satellite image analysis, musical information retrieval, chemometry... " CreationDate="2010-07-21T15:58:58.210" UserId="88" />
  <row Id="291" PostId="348" Text="The way I wrote it up, specifically with the bayesian not knowing much about cat reproduction, at the beginning only the frequentist would bet on there being kittens.  The relevant points of my *very crude example* were mostly that the frequentist made his prediction based on the data at the beginning, then sat back without incorporating new supplementary data, while the bayesian didn't have much data to begin with, but continued to incorporate relevant data as it become available.  " CreationDate="2010-07-21T16:09:12.023" UserId="24" />
  <row Id="292" PostId="135" Text="Thanks for the information, drknexus." CreationDate="2010-07-21T16:53:25.537" UserId="39" />
  <row Id="294" PostId="36" Score="4" Text="should be community wiki" CreationDate="2010-07-21T17:08:26.247" UserId="219" />
  <row Id="295" PostId="428" Text="This is interesting, and where some statistical advice could come in! Assume in total I've got (say) 500,000 i.i.d. points and I look at groups of (say) 1,000 of them, and calculate the median of each group. Now I've got 500 medians. Is there theory that could allow me to calculate a confidence interval for the overall median based on these 500 medians?" CreationDate="2010-07-21T17:10:16.133" UserId="247" />
  <row Id="298" PostId="223" Score="2" Text="Is he looking to perform statistical analyses, interpret the output or critique published papers that use statistical methods?" CreationDate="2010-07-21T17:23:52.493" UserId="215" />
  <row Id="300" PostId="437" Text="I guess I should take the first suggestion as a vote of confidence." CreationDate="2010-07-21T17:52:02.240" UserId="89" />
  <row Id="301" PostId="410" Text="@Silent: We would still need to guess what your problem really is. What is N? Number of sunflares in hundreds or the voltage you measure on some battery? It would suggest you plot your monthly means in a histogram. Fit this with a Gaussian an see if it fits and if you can justify a Gaussian fit from your model." CreationDate="2010-07-21T19:24:05.017" UserId="56" />
  <row Id="302" PostId="244" Text="+1 for Graphviz which is often missed and really useful" CreationDate="2010-07-21T19:33:01.453" UserId="212" />
  <row Id="303" PostId="457" Score="1" Text="Are you referring to a particular paper? I imagine I could find an answer to my question if I researched, studied, read a lot, but so could 95% of the questions other people ask here... Also, in some cases, particularly with macroeconomics data (which is my area), there is no more data to be collected. Data is scarce (the number of observations, I mean), and you just have to live with it. There is no &quot;get more data&quot; solution. I was hoping someone here would know the topic, but it doesn't seem like. Maybe once the website is opened to the general public?" CreationDate="2010-07-21T20:07:46.660" UserId="90" />
  <row Id="304" PostId="461" Text="Indeed some people are taking Tufte as gospel and not being particularly flexible..." CreationDate="2010-07-21T20:31:16.727" UserId="259" />
  <row Id="305" PostId="464" Text="No, nothing to do with Qnotifier, and that's not mine, it's a commercial product. The data I'm working with is very similar to that though.&#xA;&#xA;Completely agree with the &quot;color the part over the threshold&quot; but the plotting tools in use currently here don't allow that level of control. :-(" CreationDate="2010-07-21T20:42:35.483" UserId="259" />
  <row Id="306" PostId="450" Text="Can you give examples of alternatives? I'd like to look into that." CreationDate="2010-07-21T20:56:10.893" UserId="77" />
  <row Id="307" PostId="450" Score="3" Text="The mostly known and the simplest is the Median-Median regression, well known from smart calculators (Sigh!). Consult also Wikipedia  http://en.wikipedia.org/wiki/Robust_regression and maybe CRAN's Robust task view http://cran.r-project.org/web/views/Robust.html  " CreationDate="2010-07-21T21:20:20.287" UserId="88" />
  <row Id="308" PostId="464" Text="Too bad... Still you can make the same trick only with the color of the threshold line." CreationDate="2010-07-21T21:23:35.953" UserId="88" />
  <row Id="309" PostId="170" Score="1" Text="Should definitely be community wiki." CreationDate="2010-07-21T22:00:29.143" UserId="5" />
  <row Id="310" PostId="133" Score="1" Text="Hmmm I actually like the KS test answer better than mine !" CreationDate="2010-07-22T00:07:38.607" UserId="139" />
  <row Id="311" PostId="185" Text="I actually scanned that at work yesterday. It's an interesting read - I wish I had more time to absorb the material in it, but I had to get what I needed and move on." CreationDate="2010-07-22T00:39:46.840" UserId="110" />
  <row Id="312" PostId="457" Text="I suspect the answer to your question will be domain/model specific and hence I am not sure I can recommend a specific paper. " CreationDate="2010-07-22T01:19:43.857" />
  <row Id="313" PostId="247" Score="3" Text="More specifically, the asker may be interested in [JFreeChart](http://www.jfree.org/jfreechart/) which powers a lot of Incanter graphics." CreationDate="2010-07-22T03:23:01.707" UserId="13" />
  <row Id="314" PostId="262" Text="Actually, an even easier way is to use R+deducer with ggplot2 (there is a new release of this which is about to come out in the next few months.  A beta is currently available)" CreationDate="2010-07-22T03:57:44.747" UserId="253" />
  <row Id="315" PostId="423" Score="1" Text="I do have to ask though- how come cartoons are in and jokes are out?" CreationDate="2010-07-22T05:09:53.513" UserId="13" />
  <row Id="316" PostId="423" Text="@sharpie: are jokes out?  We obviously don't want the entire site to be humor, but everyone benefits from a little educational humor in small doses. " CreationDate="2010-07-22T05:15:40.057" UserId="5" />
  <row Id="317" PostId="479" Text="Totally agree. By making it a background it is clearly not data, but my making it coloured it shows a clear change in 'state'." CreationDate="2010-07-22T09:04:20.237" UserId="210" />
  <row Id="318" PostId="461" Text="Leaving the reader to figure things out has been a source of countless problems. You have to communicate your message properly and if your message is the data over this line is an issue you must shown the line which is concerning. " CreationDate="2010-07-22T09:47:28.597" UserId="210" />
  <row Id="319" PostId="151" Score="5" Text="&quot;Squaring always gives a positive value, so the sum will not be zero.&quot; and so does absolute values." CreationDate="2010-07-22T09:54:23.443" UserId="223" />
  <row Id="320" PostId="483" Text="&#xA;&#xA;Sorry, bu I don't really understand your answer. What do you mean by &quot;In detail, each tree is build on a sample of objects drawn with replacement from the original set&quot;&#xA;&#xA; Can you give more precision on where I find the details &quot;here&quot;? " CreationDate="2010-07-22T10:04:51.520" UserId="223" />
  <row Id="321" PostId="262" Score="2" Text="Nice example, but the plot is hardly publication quality. Or at least none of the journals I publish in would accept it. " CreationDate="2010-07-22T11:02:42.330" UserId="214" />
  <row Id="322" PostId="411" Score="1" Text="I like the question, there may be already most of the possible answer in the question... do you have an idea of the type of answer/developpement you want? " CreationDate="2010-07-22T11:18:44.207" UserId="223" />
  <row Id="323" PostId="485" Score="2" Text="Should be community wiki." CreationDate="2010-07-22T11:21:23.360" UserId="88" />
  <row Id="324" PostId="483" Text="This is how bagging works; check out http://en.wikipedia.org/wiki/Bootstrap_aggregating . Here is a link  (hardly visible in that theme I admit) to the detailed RF reference." CreationDate="2010-07-22T11:24:12.683" UserId="88" />
  <row Id="325" PostId="48" Text="can you give one of those or advise a particular one in the book? " CreationDate="2010-07-22T11:30:13.300" UserId="223" />
  <row Id="326" PostId="490" Text="Is it a question or a pool? If the latter, it should be community wiki. If the first, give more details about what you want to achieve? For instance, is it all-relevant or rather  minimal-optimal selection? How much is many? How   hard is the classification problem?" CreationDate="2010-07-22T11:38:20.697" UserId="88" />
  <row Id="327" PostId="492" Text="What about slow (standard) Fourier transform." CreationDate="2010-07-22T11:45:24.460" UserId="88" />
  <row Id="328" PostId="490" Text="pool... many means 1000 features or more and less than 100 observations. " CreationDate="2010-07-22T11:58:08.117" UserId="223" />
  <row Id="329" PostId="498" Score="1" Text="Also, I know this might get flagged off-topic, or not statistical, but maybe not. I wouldn't mind seeing any SAS/R/SPPS/Stata question be fair game here. Statisticians should be the most experienced in these packages. " CreationDate="2010-07-22T11:59:44.197" UserId="62" />
  <row Id="330" PostId="498" Text="imho offtopic. Even though it is an &quot;Analysis&quot; program, the question has definitely nothing to do with statistical analysis. Suggestion: contact the manufacturer of SAS, they can fix the problem." CreationDate="2010-07-22T12:05:58.567" UserId="190" />
  <row Id="331" PostId="498" Score="1" Text="See also http://meta.stats.stackexchange.com/questions/1/how-to-answer-r-questions . Option 3 seems to be favored here, which means that questions that have not anything to do with statistics are closed/move. I think this is the same case." CreationDate="2010-07-22T12:12:06.527" UserId="190" />
  <row Id="332" PostId="503" Text="Yes, certainly when I need to I output what I want into excel, or csv, or whatever, but at times, I just want a quick  C &amp; P. It behave very oddly. . .not even giving me the option of &quot;copy&quot; from the &quot;edit&quot; drop-down. " CreationDate="2010-07-22T12:33:19.247" UserId="62" />
  <row Id="333" PostId="168" Text="If I could vote up more than once, I would do it!" CreationDate="2010-07-22T12:38:01.420" UserId="223" />
  <row Id="334" PostId="460" Text="The procedure I already gave can be applied in large dimension, as I said, using a gaussian assumption.  If the dimension is really large with respect to the sample size (i.e. p&gt;&gt;n) then you can  make some sparcity assumption (assume that the parameters of your gaussian distribution lie in a low dimensional space for example)  and use a thresholding estimation procedure for the estimation of the parameters... " CreationDate="2010-07-22T13:16:30.577" UserId="223" />
  <row Id="335" PostId="151" Score="5" Text="@robin girard: That is correct, hence why I preceded that point with &quot;The benefits of squaring include&quot;. I wasn't implying that anything about absolute values in that statement. I take your point though, I'll consider removing/rephrasing it if others feel it is unclear." CreationDate="2010-07-22T13:19:20.007" UserId="81" />
  <row Id="336" PostId="498" Score="2" Text="I think questions about how to implement a statistical analysis in SAS/R/whatever are ok. But, purely interface questions are off-topic." CreationDate="2010-07-22T13:38:33.143" />
  <row Id="337" PostId="423" Text="@Sharpie, feel free to close or reopen according to your feelings! I agree with Shane, a bit is ok, but not too much. For example, this question already included a funny cartoon. The jokes question not really a funny joke...." CreationDate="2010-07-22T13:58:03.300" UserId="190" />
  <row Id="338" PostId="423" Score="8" Text="These cartoons are useful too; they can be included in a lecture on a particular topic where you are trying to explain a concept (e.g. correlation/causation above).  A little humor can help to keep an audience engaged.  " CreationDate="2010-07-22T14:22:11.213" UserId="5" />
  <row Id="339" PostId="411" Score="1" Text="Not very specifically.  I'm quite ignorant of statistics and one of my reasons for asking is to learn which criteria statisticians would use to pick between different metrics.  Since I did already describe one important practical advantage of 1 (you can actually compute it) I'm especially interested in theoretical motivations.  Say, is the information provided by estimates of Kolmogorov distance frequently of direct use in applications?" CreationDate="2010-07-22T14:50:34.407" UserId="89" />
  <row Id="340" PostId="328" Text="Given the current state of national economies due to the rescue of the various financial institutions, one may question the value of accepted knowledge in this field, save for greater fool theory." CreationDate="2010-07-22T15:06:12.763" UserId="229" />
  <row Id="341" PostId="36" Score="3" Text="That pirates / global warming chart is clearly cooked up by conspiracy theorists - anyone can see they have deliberately plotted even spacing for unequal time periods to avoid showing the recent sharp increase in temperature as pirates are almost entirely wiped out.&#xA;We all know that as temperatures rise it makes the rum evaporate and pirates cannot survive those conditions.&#xA;;-)" CreationDate="2010-07-22T16:08:37.597" UserId="270" />
  <row Id="342" PostId="461" Text="... right.  I wasn't suggesting he leave it to the reader *if it was a part of his message*.  I was saying that if the threshold weren't a part of the plot, he could either leave it out entirely or incorporate a subtle visual cue to make the threshold easier to find if the reader wanted to look for it." CreationDate="2010-07-22T16:09:04.540" UserId="71" />
  <row Id="343" PostId="155" Score="4" Text="Should be community wiki." CreationDate="2010-07-22T16:13:35.253" UserId="88" />
  <row Id="344" PostId="517" Score="3" Text="First, two lines from wiki: &quot;In computer science, semi-supervised learning is a class of machine learning techniques that make use of both labeled and unlabeled data for training - typically a small amount of labeled data with a large amount of unlabeled data. Semi-supervised learning falls between unsupervised learning (without any labeled training data) and supervised learning (with completely labeled training data).&quot; Does that help?" CreationDate="2010-07-22T16:25:36.087" />
  <row Id="345" PostId="11" Text="Can you be a little more specific?" CreationDate="2010-07-22T16:26:29.423" UserId="88" />
  <row Id="346" PostId="517" Text="@Ami What do you have in mind with &quot;Algorithmic approaches&quot;? I gave some examples of applications in my answer, is that what you are looking for?" CreationDate="2010-07-22T16:49:10.537" UserId="190" />
  <row Id="347" PostId="520" Text="Why the  downvote? Is it not a good answer, or do you think it's not a good tool?" CreationDate="2010-07-22T17:06:47.843" UserId="190" />
  <row Id="348" PostId="411" Text="I forgot to end my previous comment with the more or less obvious: and if so, how?" CreationDate="2010-07-22T17:11:20.193" UserId="89" />
  <row Id="349" PostId="479" Score="1" Text="If you don't want too much colour, just show a band of a neutral eg light grey - but do it in the *foreground* with some high level of transparency. This means that any part of the data line which shows above this threshold will automatically &quot;pop&quot; as it has greater contrast to the background than to your 'normal' band." CreationDate="2010-07-22T17:11:24.467" UserId="270" />
  <row Id="350" PostId="26" Score="10" Text="Is this really the kind of question we want on this site? A simple google search, opening any stats book, or checking out wikipedia would instantly provide an answer." CreationDate="2010-07-22T17:36:12.757" UserId="247" />
  <row Id="352" PostId="26" Score="2" Text="Totally agree with PeterR" CreationDate="2010-07-22T18:13:11.813" />
  <row Id="353" PostId="113" Text="Can you give an example of method selection with more details (a link to a page or a paper could be fine), this could help me to figure out more precisely. Thanks in advance" CreationDate="2010-07-22T18:14:49.250" UserId="223" />
  <row Id="354" PostId="452" Text="I am not sure of what you meant by &quot;Robust Standard Errors are reported as a matter of course&quot; standard errors of what? You said testing for &quot;it&quot; what is the test you are talking about?  " CreationDate="2010-07-22T18:21:09.143" UserId="223" />
  <row Id="355" PostId="498" Score="1" Text="Vote to keep. Stat software questions have to go somewhere. Stackoverflow is not the right place. Either we need to answer them here, or start statssoftware.stackexchange.com " CreationDate="2010-07-22T18:22:12.137" UserId="74" />
  <row Id="356" PostId="520" Text="I second the python recommendation here, especially if you have trouble fitting the data set into r. If you do go the python route,  have a look at &quot;Programming Collective Intelligence&quot; by Toby Segaran" CreationDate="2010-07-22T18:29:15.477" UserId="247" />
  <row Id="358" PostId="519" Text="Interesting example, because it looks, at first glance, like a probable cause-and-effect relationship, unlike many of the silliest examples. " CreationDate="2010-07-22T19:10:45.160" UserId="77" />
  <row Id="359" PostId="498" Text="Chief makes a good point. If not here, where? Is this purely &quot;statistical analysis&quot; just because of the name? Or is it a place that people working in the field can come for answers. Sometimes I just need a piece of SAS syntax that I've forgotten. I can dig through my archives, but I wouldn't mind coming here to get an answer/an alternate/a better way/ or just leave mental residue for future questioners. " CreationDate="2010-07-22T19:51:37.467" UserId="62" />
  <row Id="360" PostId="26" Text="The same concept may have different explanations at different levels... A smart sixth grader's understanding of the concept might be very different from that of a phd student, who has thought about it a lot more. It would be nice to see the basics revisited in light of more advanced concepts. That would help me understand how everything connects" CreationDate="2010-07-22T20:23:56.797" UserId="35" />
  <row Id="361" PostId="494" Text="Even if the constant is included, the AIC (AICc) can be negative." CreationDate="2010-07-22T23:15:41.563" UserId="159" />
  <row Id="362" PostId="155" Score="2" Text="Should definitely be community wiki. " CreationDate="2010-07-23T01:54:10.883" UserId="5" />
  <row Id="364" PostId="532" Text="Thanks for this answer (should be a comment ? ) &#xA;&#xA;As I already mentionned, as a comment to Rich answer's High dimension are not a problem (even 1000 could work) if you make parametric structural assumption. &#xA;&#xA;&#xA;&#xA;" CreationDate="2010-07-23T06:43:46.077" UserId="223" />
  <row Id="365" PostId="532" Text="@rob &quot;I'm not sure what would be a good threshold &quot; this would be the purpose of the multiple testing procedure I mentionned .... but I fully agree that things have to be filled in and I really like the outlier detection in the outlier detection !  who wants to writte a paper :) ? " CreationDate="2010-07-23T06:47:22.747" UserId="223" />
  <row Id="367" PostId="494" Text="That's what I've written." CreationDate="2010-07-23T08:17:51.990" UserId="88" />
  <row Id="370" PostId="501" Text="But the question says there are more variables than observations. So it is not possible to begin with the full set." CreationDate="2010-07-23T09:50:30.923" UserId="159" />
  <row Id="371" PostId="534" Score="7" Text="Correlation and a strong underlying reason for a link suggest causation until proven otherwise is probably the best you can get." CreationDate="2010-07-23T09:54:58.050" UserId="229" />
  <row Id="372" PostId="452" Text="Good point....I'm talking about the Standard Errors of regression coefficients in OLS regression and the problem of heteroscedasticity. The traditional approach would be to test for the presence of heteroscedasticity using, for example, White's test or Breusch Pagan test. If heteroscedasticity is found then one would report Robust Standard Errors, usually White Standard Errors. " CreationDate="2010-07-23T10:09:35.607" UserId="215" />
  <row Id="373" PostId="486" Text="Thanks for the helpful answers." CreationDate="2010-07-23T10:37:02.760" UserId="266" />
  <row Id="374" PostId="541" Text="&quot;[T]hen I am making very strong assumptions about the relative difference between consecutive values of the ordinal variable.&quot;&#xA;&#xA;I think this is the key point, really.  i.e. how strongly can you argue that the difference between groups 1 and 2 is comparable to that between 2 and 3?" CreationDate="2010-07-23T10:43:29.803" UserId="266" />
  <row Id="375" PostId="518" Text="Nice list, it will be very useful." CreationDate="2010-07-23T11:31:31.180" UserId="166" />
  <row Id="376" PostId="520" Text="What downvote? I'm not familiar with Python yet, but it seems promising, I'll take a look." CreationDate="2010-07-23T11:32:09.430" UserId="166" />
  <row Id="377" PostId="520" Text="@Victor, the first vote was a downvote (so it's now 2 upvotes/1 downvote). Maybe someone didn't like this tool. Happy I could help :)" CreationDate="2010-07-23T11:34:05.013" UserId="190" />
  <row Id="378" PostId="26" Score="5" Text="I don't think the purpose of this site is to answer 6th graders questions. And my kid, when faced with such a question, would google for the answer.&#xA;&#xA;If there is a specific part of the definition you don't understand, ask away. But such an unfocused question on such a basic topic indicates (to me anyway) that the poster didn't even try to find an answer. What is going to be next &quot;What is a number and how are they used?&quot;" CreationDate="2010-07-23T12:05:21.897" UserId="247" />
  <row Id="382" PostId="519" Score="1" Text="What I like is that you can provoke lots of discussion about whether the &quot;effect&quot; was to actually impact fertility (in a medical sense of ability to conceive) or was it social (&quot;I don't want to bring a child into this bad world&quot;). Then drop the bombshell about the Pill if no-one else has brought it up. And then point out that even this can only be one possible factor and discuss some of the others." CreationDate="2010-07-23T13:32:48.203" UserId="270" />
  <row Id="383" PostId="501" Text="What's the problem? " CreationDate="2010-07-23T13:41:29.023" UserId="88" />
  <row Id="384" PostId="486" Text="So consider accepting one." CreationDate="2010-07-23T13:48:15.700" UserId="88" />
  <row Id="385" PostId="541" Text="I think you should make some assumption about how the continuous variable should be distributed and then try to fit this &quot;psudohistogram&quot; of each categorical variable frequency (I mean find bin widths which will transform it into a fitted histogram). Still, I'm not an expert in this field, its a fast&amp;dirty idea." CreationDate="2010-07-23T13:56:36.227" UserId="88" />
  <row Id="387" PostId="555" Score="1" Text="If I am not mistaken, &#xA;&#xA;linear regression is the estimation of coefficients that define a good linear map from X to Y.&#xA;&#xA; ANOVA is a test to know if there is significant differences in X when Y take two different values.  &#xA;&#xA;Can you explain us why you think they are the same? " CreationDate="2010-07-23T15:29:16.043" UserId="223" />
  <row Id="388" PostId="541" Text="Recasting binary categories as {0,1} makes sense, but turning that into a continuous [0,1] interval seems like a bit of a leap. On the broader front, I'm totally with your reluctance to weight ordinals equally unless there are powerful arguments from the model." CreationDate="2010-07-23T15:29:27.463" UserId="174" />
  <row Id="389" PostId="544" Text="+1 Thanks, this is very interesting. Optimal scaling seems worth investigation, but first few Google hits feel a bit ad hoc. Do you have any good refs in this direction?" CreationDate="2010-07-23T15:37:20.113" UserId="174" />
  <row Id="390" PostId="455" Score="5" Text="Statistical voyeurism? And there we were wondering what to call the site..." CreationDate="2010-07-23T15:48:25.713" UserId="174" />
  <row Id="391" PostId="557" Text="Thanks for the Gelman reference. I will read his paper. But, can't we analyze multilevel models using classical maximum likelihood? I agree that OLS is inefficient/inappropriate for multi-level models." CreationDate="2010-07-23T15:50:47.240" />
  <row Id="392" PostId="557" Score="1" Text="+1 just for the lovely lucid even-handedness of this answer." CreationDate="2010-07-23T15:56:32.307" UserId="174" />
  <row Id="393" PostId="556" Score="3" Text="Data can be discrete without being restricted to integers. Or numbers, for that matter. It's always possible to *represent* discrete data with integers, but that doesn't mean the data can only take such values." CreationDate="2010-07-23T16:06:37.887" UserId="174" />
  <row Id="394" PostId="557" Score="2" Text="@Srikant - there any many ways to deal with multilevel data and Gelman is &quot;the king&quot; of this field. His point is that ANOVA is a simple/clear method of capturing the key features of complex and hierarchical data structures or study designs and ANOVA is a simple/clear way of presenting the key results. In this sense it's role is complementary or exploratory." CreationDate="2010-07-23T16:30:00.160" UserId="215" />
  <row Id="395" PostId="565" Text="That's a great example!" CreationDate="2010-07-23T17:03:23.050" UserId="5" />
  <row Id="396" PostId="566" Score="2" Text="Indeed, blending is one of the possible ensemble techniques. In particular, there are two when you combine the same sort of classifier, boosting (like Adaboost) and bagging (like Random Forest), and blending, where you combine different classifiers (what was Shane's question about)." CreationDate="2010-07-23T17:10:02.027" UserId="88" />
  <row Id="397" PostId="470" Score="2" Text="Why pay $70 for the book?  To support the authors and to have a physical copy.  That's why I did it, anyway!" CreationDate="2010-07-23T17:13:42.517" UserId="5" />
  <row Id="398" PostId="411" Text="I just reread my long comment above and realized that the last question I raised is as much a practical consideration as theoretical.  In any case, that's one of the kinds of issues I'd be interested to learn about." CreationDate="2010-07-23T17:23:48.337" UserId="89" />
  <row Id="399" PostId="26" Score="1" Text="you are right.. this question is way too basic.. (but there are others like http://stats.stackexchange.com/questions/118 which are things I has confused by at some point and am glad to see discussions on on a website such as this)" CreationDate="2010-07-23T17:32:09.993" UserId="35" />
  <row Id="400" PostId="566" Score="2" Text="For blending, this paper from the netflix competition is worth reading: http://www.the-ensemble.com/content/feature-weighted-linear-stacking." CreationDate="2010-07-23T17:42:07.760" UserId="5" />
  <row Id="401" PostId="563" Score="3" Text="Don't you think that the Wikipedia article about it is enough?" CreationDate="2010-07-23T17:59:06.980" UserId="88" />
  <row Id="402" PostId="534" Score="1" Text="@James said it best." CreationDate="2010-07-23T18:30:50.473" UserId="260" />
  <row Id="404" PostId="563" Text="Questions such as this require a wiki / blog post type of response. I do think questions should not require such long answers." CreationDate="2010-07-23T19:50:07.990" />
  <row Id="406" PostId="563" Text="I'm not sure the right thing to do is to simply ignore this question and refer the asker to the wiki - especially during beta where we are trying to build up the content of the site.  Perhaps the question asker should submit each of these questions individually so that they can be better addressed." CreationDate="2010-07-23T20:52:00.570" UserId="196" />
  <row Id="407" PostId="529" Text="Sorry, I was meaning an analytical measurement method.  I've re-worded the question." CreationDate="2010-07-23T20:58:13.660" UserId="114" />
  <row Id="408" PostId="529" Text="In that case, I think the two-sample test of equality for means/proportions is what you may want to do." CreationDate="2010-07-23T21:04:24.660" />
  <row Id="412" PostId="579" Text="Does that mean that for certain sample sizes BIC may be less stringent than AIC?" CreationDate="2010-07-23T21:36:49.540" UserId="196" />
  <row Id="413" PostId="121" Text="@rich said &quot;it's continuously differentiable (nice when you want to minimize it)&quot; do you mean that the absolute value is difficult to optimize ? " CreationDate="2010-07-23T21:40:12.833" UserId="223" />
  <row Id="414" PostId="118" Score="1" Text="Do you think the term standard means this is THE standard today ? Isn't it like asking why principal component are &quot;principal&quot; and not secondary ?" CreationDate="2010-07-23T21:44:37.093" UserId="223" />
  <row Id="415" PostId="311" Text="My opinion is that this book is THE book about extrem value theory " CreationDate="2010-07-23T21:56:20.037" UserId="223" />
  <row Id="416" PostId="532" Text="The principal component is a good powerfull idea. It will work in a lot of case. However, it is to avoid when the distribution has a trend or is multimodal... in those cases the direction of largest variation are not related to outliers. " CreationDate="2010-07-23T22:00:48.053" UserId="223" />
  <row Id="417" PostId="579" Score="1" Text="Stringent is not a best word here, rather more tolerant for parameters; still, yup, for the common definitions (with natural log) it happens for 7 and less objects." CreationDate="2010-07-23T22:13:56.467" UserId="88" />
  <row Id="419" PostId="121" Score="3" Text="@robin: while the absolute value function is continuous everywhere, its first derivative is not (at x=0). This makes analytical optimization more difficult. " CreationDate="2010-07-23T23:59:23.210" UserId="7" />
  <row Id="420" PostId="470" Text="I agree with Shane... and a hardcopy makes reading it much easier. " CreationDate="2010-07-24T00:07:57.327" UserId="7" />
  <row Id="421" PostId="581" Text="What do you mean by 'viterbi training' exactly? " CreationDate="2010-07-24T00:40:50.910" UserId="240" />
  <row Id="422" PostId="579" Text="AIC is asymptotically equivalent to cross-validation." CreationDate="2010-07-24T01:47:58.633" UserId="159" />
  <row Id="423" PostId="121" Text="Yeah, finding quantiles in general (which includes optimizing absolute values) tends to churn up linear programming type problems, which -- while they're certainly tractable numerically -- can get fiddly.  They typically don't have an analytical closed-form solution, and are a bit slower and a bit more difficult to implement than least-square-type solutions." CreationDate="2010-07-24T02:55:02.763" UserId="61" />
  <row Id="424" PostId="574" Text="There is a lot of training involved. Training a reasonable Finnish acoustic model can take 60-100 hours of computing time. Besides that, most corpora work with standardized training, development and evaluation sets." CreationDate="2010-07-24T04:50:42.187" UserId="190" />
  <row Id="425" PostId="588" Text="If I'm right you mix up Viterbi training and Viterbi decoding." CreationDate="2010-07-24T05:00:00.143" UserId="190" />
  <row Id="427" PostId="121" Text="I do not agree with this. First, theoretically, the problem may be of different nature (because of the discontinuity) but not necessarily harder (for example the median is easely shown to be arginf_m E[|Y-m|]).  Second, practically, using a L1 norm (absolute value) rather than a L2 norm makes it piecewise linear and hence at least not more difficult. Quantile regression and its multiple variante is an example of that. " CreationDate="2010-07-24T06:01:42.113" UserId="223" />
  <row Id="428" PostId="118" Text="My understanding of this question is that it could be shorter just be something like: what is the difference between the MAE and the RMSE ? otherwise it is difficult to deal with. " CreationDate="2010-07-24T06:08:14.627" UserId="223" />
  <row Id="429" PostId="527" Text="Can you give more details in your question? I don't understand what is &quot;the concentration of a particular molecule in a matrix&quot;. " CreationDate="2010-07-24T06:41:14.763" UserId="223" />
  <row Id="430" PostId="566" Score="1" Text="IT is fun that meteorologist also use the word &quot;ensemble&quot; but not for combination: they use it for an ensemble of prediction (like scenario) obtained by perturbation of the initial conditions of the numerical model.  " CreationDate="2010-07-24T06:46:49.650" UserId="223" />
  <row Id="431" PostId="529" Score="1" Text="Wouldn't a test of means/proportions only give you a point estimate of whether the two methods gave the same average response for a given set of responses?  Couldn't that approach yield a result of &quot;equal&quot; even if the two methods were actually negatively correlated with one another?" CreationDate="2010-07-24T07:44:03.637" UserId="196" />
  <row Id="432" PostId="579" Text="@Rob Can you give a reference? I doubt that it is general." CreationDate="2010-07-24T08:03:12.467" UserId="88" />
  <row Id="433" PostId="579" Text="@Rob For what I could found, this is true only for linear models. " CreationDate="2010-07-24T08:13:15.780" UserId="88" />
  <row Id="434" PostId="587" Text="AIC is equivalent to K-fold cross-validation, BIC is equivalent to leve-one-out cross-validation. Still, both theorems hold _only_ in case of linear regression." CreationDate="2010-07-24T08:23:58.813" UserId="88" />
  <row Id="436" PostId="574" Text="E, 100 cores and one may live with that. Seriously, this looks dubious from a ML point of view, still I can understand the reasons why you do it in such a way. So then stick to the Srikant solution." CreationDate="2010-07-24T08:44:37.530" UserId="88" />
  <row Id="437" PostId="585" Text="I meant significantly different results. I also think there is none, at least real-world example. Still, I think I'll wait some time more." CreationDate="2010-07-24T09:04:44.470" UserId="88" />
  <row Id="438" PostId="590" Text="This question is a repost of http://stats.stackexchange.com/questions/536/when-a-serious-statistician-calls-a-geometric-distribution-a-geometric-density-t (see comments there). @Hibernating is the original question-asker" CreationDate="2010-07-24T09:09:35.053" UserId="190" />
  <row Id="439" PostId="121" Score="3" Text="Yes, but finding the actual number you want, rather than just a descriptor of it, is easier under squared error loss.  Consider the 1 dimension case; you can express the minimizer of the squared error by the mean: O(n) operations and closed form.&#xA;&#xA;You can express the value of the absolute error minimizer by the median, but there's not a closed-form solution that tells you what the median value is; it requires a sort to find, which is something like O(n log n).&#xA;&#xA;Least squares solutions tend to be a simple plug-and-chug type operation, absolute value solutions usually require more work to find." CreationDate="2010-07-24T09:10:00.387" UserId="61" />
  <row Id="441" PostId="573" Text="The question is interesting. The typos make it hard to read." CreationDate="2010-07-24T09:24:27.443" UserId="200" />
  <row Id="442" PostId="573" Text="@Ivo I am sorry, there were a lot of mistakes indeed. I corrected a lot of them. " CreationDate="2010-07-24T09:40:23.567" UserId="190" />
  <row Id="444" PostId="587" Text="mbq, it's AIC/LOO (not LKO or K-fold) and I don't think the proof in Stone 1977 relied on linear models.  I don't know the details of the BIC result." CreationDate="2010-07-24T11:01:35.353" UserId="251" />
  <row Id="445" PostId="93" Score="4" Text="I think that if you ask a question that can be understood by people that don't know what the hazard function is and if you elaborate a bit more on what you do (how do you estimate the parameter of you'r gausian process, how do you use the gaussian process at the end) you will increase the chances to get an answer and this will be an added value for stats.stackexchange :) " CreationDate="2010-07-24T11:01:43.027" UserId="223" />
  <row Id="446" PostId="529" Text="That is a good point." CreationDate="2010-07-24T12:16:03.207" />
  <row Id="447" PostId="590" Text="I am guessing it is a typo or an oversight." CreationDate="2010-07-24T12:27:31.700" />
  <row Id="448" PostId="587" Text="ars is correct. It's AIC=LOO and BIC=K-fold where K is a complicated function of the sample size." CreationDate="2010-07-24T12:42:31.373" UserId="159" />
  <row Id="449" PostId="512" Text="We had a look at it. Worked okay but in this case the noise still seemed to be a bit too strong and if we changed the parameters to even out the distributions enough it appeared that the trend was damped down too much. Maybe in this case there just is no solution to the data and it is just a bit too noisy. " CreationDate="2010-07-24T17:10:43.037" UserId="210" />
  <row Id="450" PostId="588" Text="You're right.  I wasn't aware that there was a procedure that used only the Viterbi algorithm to compute the transition probabilities as well.  It looks -- on further reading -- like there's some overlap of nomenclature between discrete time/discrete state HMM analysis, and discrete time/continuous state analysis using Gaussian mixture distributions.  My answer speaks to the DTDS HMM setup, and not the mixture model setup." CreationDate="2010-07-24T19:08:38.050" UserId="61" />
  <row Id="451" PostId="587" Text="Congratulations, you've got me; I was in hurry writing that and so I made this error, obviously it's how Rob wrote it. Neverthelss it is from Shao 1995, where was an assumption that the model is linear. I'll analyse Stone, still I think you, ars, may be right since LOO in my field has equally bad reputation as various *ICs." CreationDate="2010-07-24T20:10:12.740" UserId="88" />
  <row Id="453" PostId="603" Text="No, no, no, it is about machine learning *not* model selection." CreationDate="2010-07-24T23:09:02.483" UserId="88" />
  <row Id="454" PostId="605" Text="To get the behavior you wanted, try using a simple CART." CreationDate="2010-07-24T23:29:48.473" UserId="88" />
  <row Id="455" PostId="604" Text="I am not sure why you feel that in regression you will get R2 = 1 if you try to predict the predicted variable. Can you clarify? " CreationDate="2010-07-25T01:37:03.440" />
  <row Id="457" PostId="603" Score="1" Text="Interesting distinction. I thought model selection was central to machine learning, in almost all meanings of the term." CreationDate="2010-07-25T02:40:53.223" UserId="30" />
  <row Id="458" PostId="13" Score="2" Text="I am interested in the last statement: &quot;the very first commercially available (and working) Bankruptcy model implemented by the credit bureaus was created through a plain old linear regression model targeting a 0-1 outcome&quot;. Which model was it? I believe that the first model was RiskCalc by Moody's, and even the first version was a logistic regression model. The developers of that model were not CS people with a background in ML, but rather in econometrics." CreationDate="2010-07-25T02:58:55.117" UserId="30" />
  <row Id="459" PostId="587" Text="The description on Wikipedia (http://en.wikipedia.org/wiki/Cross-validation_(statistics)#K-fold_cross-validation) makes it seem like K-fold cross-validation is sort of like a repeated simulation to estimate the stability of the parameters. I can see why AIC would be expected to be stable with LOO (since LOO can wasily be conducted exhaustively), but I don't understand why the BIC would be stable with K-fold unless K is also exhaustive. Does the complex formula underlying the value for K make it exhaustive?  Or is something else happening?" CreationDate="2010-07-25T03:18:18.817" UserId="196" />
  <row Id="460" PostId="593" Text="Thanks, I think the Friedman test is interesting, but I can't quite figure out how it is doing that adjustment for Type I error in the post-hoc.  The comments say it is a &quot;Wilcoxon-Nemenyi-McDonald-Thompson test&quot; but I've never heard of that before could you explain it?" CreationDate="2010-07-25T04:29:21.710" UserId="196" />
  <row Id="461" PostId="501" Text="You can't fit a model that has more variables than observations. There are not enough degrees of freedom for parameter estimation." CreationDate="2010-07-25T04:42:10.433" UserId="159" />
  <row Id="462" PostId="411" Text="I know you did not meant to be exhaustive but you could add Anderson darling statistic (see http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test). This made me remind of a paper fromo Jager and Wellner (see http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1194461721) which extands/generalises  Anderson darling statistic (and include in particular higher criticism of Tukey)..." CreationDate="2010-07-25T06:38:39.197" UserId="223" />
  <row Id="463" PostId="604" Score="1" Text="You could improve clarity in your question. Note that A (good) classification rule may make classification errors in the learning set, this is sometime a clue to avoide overfitting. " CreationDate="2010-07-25T07:37:44.797" UserId="223" />
  <row Id="464" PostId="596" Score="1" Text="Can you elaborate on what you really want to do (what is the model you want to fit) ? If you don't give a real question there won't be a real answer. " CreationDate="2010-07-25T07:40:01.560" UserId="223" />
  <row Id="465" PostId="594" Text="I am not sure but I think it is possible to interpret it as a stocastic gradient descent optimization procedure. I'll think about that... " CreationDate="2010-07-25T07:41:33.880" UserId="223" />
  <row Id="466" PostId="534" Score="3" Text="Isn't it Karl Popper that said man can't establish causality: scientific theories are abstract in nature. They can be falsifiable and the fact that we encouter difficulties in falsifying something make us think about causality... " CreationDate="2010-07-25T07:48:30.790" UserId="223" />
  <row Id="467" PostId="244" Text="+1 for matplotlib and the collection... which one is your favorite ? " CreationDate="2010-07-25T07:51:32.343" UserId="223" />
  <row Id="468" PostId="501" Text="This is machine learning, so you can." CreationDate="2010-07-25T08:08:08.653" UserId="88" />
  <row Id="469" PostId="603" Text="All those things work for trivial (mostly linear) models when you have few parameters and you just want to fit them to data to say something about it, like you have y and x and you want to check whether y=x^2 or y=x. Here I talk about estimating error of models like SVMs or RFs which can have thousands of parameters and are still not overfitting due to complex heuristics.  " CreationDate="2010-07-25T08:22:10.460" UserId="88" />
  <row Id="470" PostId="604" Text="I think it is a good question; if I haven't known what I know I would thought the same." CreationDate="2010-07-25T08:36:42.360" UserId="88" />
  <row Id="471" PostId="590" Text="Please fix spelling of Geometric. None of us have enough points yet to do editing." CreationDate="2010-07-25T09:04:57.253" UserId="159" />
  <row Id="473" PostId="512" Text="Exponentially weighted moving averages are a special case of a kernel smoother (assuming you used a 2-sided MA rather than 1-sided). Better estimates that are generalizations of this are loess or splines -- see my answer." CreationDate="2010-07-25T09:15:25.727" UserId="159" />
  <row Id="474" PostId="50" Score="2" Text="I think this question could be formulated with more precision. With your formulation, we don't know who is &quot;they&quot;, and we don't know if you want a general abstract mathematical definition, or an intuitive explanation for someone that don't know anything about statistic or mathematic...   " CreationDate="2010-07-25T11:10:57.363" UserId="223" />
  <row Id="475" PostId="47" Text="@radek  I don't see why you want to cluster your users without any subjective input (I refer to a comment you made @reed), you said you want &quot;the most appropriate number of clusters&quot;, but unfortunatly you don't have a clear objective, if you want to cluster your population to show something particular, you should tell us what you want to show ? If you want statistic (the data) to tell you what you want to show this is another problem :) " CreationDate="2010-07-25T11:24:42.817" UserId="223" />
  <row Id="476" PostId="194" Text="Maybe you are asking a bit to much to statistic: you said &quot;patterns that you extract out from the data are indeed true patterns, not statistical fluke&quot; and then you ask if there is a statistical procedure to answer the question... can I send that to xkcd :) ? more seriously, I think you should try to see if the pattern are meaningfull by trying to understand if they mean something " CreationDate="2010-07-25T11:29:52.403" UserId="223" />
  <row Id="477" PostId="242" Text="I really like this type of question, I think this type of precise real worl problem will increase the interest of the site. It would be even better if you had the possibility to add a link to the data, or to tell us (as a complement to the post) what you finally did, what was the conclusions .... however I understand that this can be confidential ... " CreationDate="2010-07-25T11:38:04.900" UserId="223" />
  <row Id="478" PostId="242" Text="I whish I could vote up again to make you pass over the question about the definition of a random variable ;) " CreationDate="2010-07-25T11:41:11.637" UserId="223" />
  <row Id="479" PostId="608" Text="+1 The code would be simple to write, still I'm very interested in seeing a clear, illustrative dataset." CreationDate="2010-07-25T12:31:16.200" UserId="88" />
  <row Id="480" PostId="544" Score="2" Text="This paper is a good start&#xA;http://www.jstatsoft.org/v31/i04/paper .&#xA;it also has several good references." CreationDate="2010-07-25T13:37:59.673" UserId="183" />
  <row Id="481" PostId="614" Score="3" Text="Community wiki!" CreationDate="2010-07-25T15:02:22.450" UserId="88" />
  <row Id="482" PostId="614" Text="On the other hand, can a book be open source? It rather applies to code, so probably the better word is &quot;open book&quot;." CreationDate="2010-07-25T15:08:59.453" UserId="88" />
  <row Id="483" PostId="614" Text="Yes, any list question like this should be community wiki." CreationDate="2010-07-25T15:25:40.080" UserId="5" />
  <row Id="484" PostId="185" Text="agreed, that is an excellent book. It pretty much explains how Google works :)" CreationDate="2010-07-25T18:41:18.913" UserId="74" />
  <row Id="485" PostId="615" Score="1" Text="I'm still struggling to understand equations (biologist ahoy), which is why I turned to the community here, hoping it will help me explain the difference in layman's terms." CreationDate="2010-07-25T19:18:23.953" UserId="144" />
  <row Id="486" PostId="613" Text="Thanks for your answer, it makes sense :)" CreationDate="2010-07-25T19:29:14.800" UserId="90" />
  <row Id="487" PostId="611" Text="I am afraid I do not follow much of what you wrote as I am not a trained mathematician. Are you saying that all 'discrete distributions' are in some sense densities under a very general definition? " CreationDate="2010-07-25T19:39:03.403" />
  <row Id="488" PostId="611" Score="2" Text="Without much mathematic you could say that a continuous variable has a density with respect to the Lebesgue measure, and a discrete random variable has a density with respect to the counting measure.  " CreationDate="2010-07-25T19:56:22.177" UserId="223" />
  <row Id="489" PostId="601" Text="Well, yes, that's the term for the general idea, but can you cite a specific unsupervised method, other than plain self-training, that works in the discriminative arena?" CreationDate="2010-07-25T20:02:30.353" UserId="240" />
  <row Id="490" PostId="6" Score="4" Text="subjective and argumentative? " CreationDate="2010-07-25T20:03:54.860" UserId="240" />
  <row Id="492" PostId="26" Score="3" Text="+1 to Peter... I think that we should directly close questions that find a direct answer on wikipedia http://en.wikipedia.org/wiki/Standard_deviation. I voted to close. " CreationDate="2010-07-25T20:07:35.910" UserId="223" />
  <row Id="493" PostId="50" Score="2" Text="In addition, did you try google before asking ? &#xA;ranked second (after wikipedia which is a bit theoretical on this question) on my google: http://www.stats.gla.ac.uk/steps/glossary/probability_distributions.html" CreationDate="2010-07-25T20:10:04.720" UserId="223" />
  <row Id="494" PostId="2" Score="1" Text="did you try google/ wikipedia first ? &#xA;http://en.wikipedia.org/wiki/Normal_distribution" CreationDate="2010-07-25T20:13:20.310" UserId="223" />
  <row Id="495" PostId="206" Text="Did you try google first ? for me, it gives http://infinity.cos.edu/faculty/woodbury/stats/tutorial/Data_Disc_Cont.htm" CreationDate="2010-07-25T20:15:33.723" UserId="223" />
  <row Id="496" PostId="109" Score="3" Text="Can you give a definition or a link to the definition of the different test. Can you state the hypothesis you want to test (otherwise it is difficult to discuss the power ...). " CreationDate="2010-07-25T20:20:36.190" UserId="223" />
  <row Id="497" PostId="244" Text="I tend to use some more than others- the tool I use most often for visualization is R and associated packages, but I left it off of this list because there is no easy way to compile R scripts to stand-alone &quot;executables&quot; that the OP wanted.  I can't really claim a single favorite- I would have to say it depends on 1) The task at hand and 2) The tools I am using" CreationDate="2010-07-25T20:26:16.973" UserId="13" />
  <row Id="498" PostId="603" Text="These results are valid for regression of general linear models with arbitrary number of independent variables. The variables can be arbitrary learners. The crucial assumption is that as the number of observations goes to infinity the number of learners describing the true model stays finite. All of this works for regression, so for a classification task like yours I am not sure it helps." CreationDate="2010-07-25T20:57:47.927" UserId="30" />
  <row Id="499" PostId="615" Text="I think the ideology is that FA assumes that the process is driven by some 'hidden factors', while the data we have consists of some combinations of them. Because of that, the problem of FA is to reconstruct the hidden factors somehow. And there goes PCA -- a method which iteratively builds a new variables (PCs) by mixing the old ones such to greedy absorb the variance of the data. One may say the PCs are equal to the FA's factors, and here they will be indistinguishable. But one may also make some changes to the PCA to make it a base of some other 'FA sort', and so the problem begins. " CreationDate="2010-07-25T22:56:50.603" UserId="88" />
  <row Id="500" PostId="615" Text="So basically, you should think of what you want to do (not which buzzword you want to use). I know it is hard, especially while having biologists around (to some extend use-buzzword works well in biology, so they just assume that this is common to other disciplines); still this is the way science should be done. Than use Google (or this site) to assess the good algorithm for it. Finally, use the docks to find a function/button that does it and type/click it. " CreationDate="2010-07-25T23:19:04.467" UserId="88" />
  <row Id="501" PostId="601" Text="Like clustering? It creates the decision that well suits data, so it definitely optimizes classification. " CreationDate="2010-07-25T23:37:24.903" UserId="88" />
  <row Id="502" PostId="596" Text="Ok, I admit that now I am also in deep confusion. Probably you could translate some of the document-retrieval jargon. " CreationDate="2010-07-25T23:45:52.910" UserId="88" />
  <row Id="503" PostId="566" Text="@robin It's not fun, its physics." CreationDate="2010-07-26T00:03:50.127" UserId="88" />
  <row Id="504" PostId="596" Text="Doc retrieval? None of that. However, I can elaborate." CreationDate="2010-07-26T00:25:00.770" UserId="240" />
  <row Id="505" PostId="608" Text="I'm not sure what all would need to be in a clear and illustrative dataset, but I've made an attempt to include a sample dataset." CreationDate="2010-07-26T05:50:33.367" UserId="196" />
  <row Id="506" PostId="566" Score="1" Text="@mbq in fact they call themselves forecaster and they use statistic quite a lot ...  " CreationDate="2010-07-26T06:46:28.900" UserId="223" />
  <row Id="507" PostId="457" Score="1" Text="Sorry for the late reply. I like your suggestion of simulation. That is not really easy, though. The truth is, what I see in practice is that researchers just do the test that is computationally easier or that give them the result they want." CreationDate="2010-07-26T07:41:27.907" UserId="90" />
  <row Id="508" PostId="566" Score="1" Text="@robin I know, this is just why it's called &quot;ensemble&quot; not a set or something like this. " CreationDate="2010-07-26T07:46:15.477" UserId="88" />
  <row Id="509" PostId="608" Text="So look: what you provided is an example of an useless set, because the BIC and AIC give the same results: 340 v. 342 for AIC and 349 v. 353 for BIC -- so good.model wins in both cases. The whole idea with that convergence is that certain cross-validation will select the same model as its corresponding IC." CreationDate="2010-07-26T07:53:50.170" UserId="88" />
  <row Id="510" PostId="570" Text="The term &quot;SEM&quot; is vague. It could also mean &quot;Search Engine Marketing&quot;, for instance, for someone looking for statistical analysis techniques for studying ad click data or evaluating advertising effectiveness. Consider making the title more verbose." CreationDate="2010-07-26T09:10:42.060" UserId="87" />
  <row Id="511" PostId="608" Text="I've made a simple scanning and for instance for seed 76 the ICs disagree." CreationDate="2010-07-26T09:45:12.620" UserId="88" />
  <row Id="512" PostId="625" Text="Thanks for the suggestions, I've actually came across your book a couple of years ago. " CreationDate="2010-07-26T12:30:56.427" UserId="59" />
  <row Id="513" PostId="600" Score="1" Text="Two quick notes on your notes. 1. The C-vM distance is precisely the L^2 cousin of the Kolmogorov (L^infinity) and (univariate) K-R (L^1) distances, and hence interpolates between them.  2. One advantage I didn't mention of the K-R and B-L distances is that they generalize more naturally to higher dimensional spaces." CreationDate="2010-07-26T13:31:30.177" UserId="89" />
  <row Id="514" PostId="609" Text="Yes, what I called the Kantorovitch-Rubinstein distance is also called the L^1 Wasserstein distance or W1.  It goes by many other names too." CreationDate="2010-07-26T13:37:04.810" UserId="89" />
  <row Id="515" PostId="411" Text="All the answers so far are very nice and add some nice perspective from different directions.  I won't accept any because I see no reasonable criterion for singling out just one." CreationDate="2010-07-26T13:39:31.297" UserId="89" />
  <row Id="517" PostId="25" Text="Should be community wiki." CreationDate="2010-07-26T14:54:10.180" UserId="5" />
  <row Id="518" PostId="614" Text="Done. Sorry about not doing it in the first place." CreationDate="2010-07-26T15:32:04.330" UserId="107" />
  <row Id="519" PostId="614" Text="Any policy on how to aggregate the four answers into one wiki answer?" CreationDate="2010-07-26T15:32:58.280" UserId="107" />
  <row Id="520" PostId="614" Text="They should be separate answers (so people can vote on them), but the moderator or the individuals should make their answers community wiki as well.  All new answers with be CW by default." CreationDate="2010-07-26T15:36:44.847" UserId="5" />
  <row Id="522" PostId="93" Text="Upvoted for doing a bounty..." CreationDate="2010-07-26T15:37:59.280" UserId="107" />
  <row Id="524" PostId="621" Text="Sure, sorry for that." CreationDate="2010-07-26T15:48:48.157" UserId="88" />
  <row Id="526" PostId="620" Text="Thanks Shane, Done." CreationDate="2010-07-26T15:52:28.077" UserId="253" />
  <row Id="527" PostId="581" Text="In my problem I have an array of real valued data which I am modeling as a HMM (speficially a mixture of multiple density functions each with unknown parameters).   For now I assume that I know the state transition probabilites.   What I mean by Viterbi Trainig is the following algorithm.&#xA;&#xA;1) Arbitrarily assign a state to each data point ( initialization)&#xA;2) Perform MLE of the density function parameters.&#xA;3) Re-estimate state for each point ( can be done with Viterbi Alg).&#xA;4) Goto step 2 and repeat unless stopping criteria is met. " CreationDate="2010-07-26T16:05:22.033" UserId="99" />
  <row Id="528" PostId="632" Text="Isn't that a function of estimator is still an estimator? I still don't know \sigma, only X_i." CreationDate="2010-07-26T16:45:07.923" UserId="88" />
  <row Id="529" PostId="632" Text="ok, then you will possibly estimate the square root of the variance of the estimation of the square root of the variance... right :) should be something like $\hat{\sigma}/n$ ? " CreationDate="2010-07-26T17:09:48.947" UserId="223" />
  <row Id="530" PostId="567" Text="+1 For this resource. As the name says, excellent for a practical approach to engineering problems." CreationDate="2010-07-26T17:42:34.383" UserId="77" />
  <row Id="531" PostId="608" Text="Thanks mbq - I didn't understand what you meant/needed in terms of an illustrative dataset.  Also, in general, in terms of empirical demonstration I'd imagine that a single example from a single seed won't really do the trick.  I was imagining something like a metric from the employment of a cross-validation method being shown to be correlated with the calculated corresponding information criterion.  In that way the ICs don't necessarily need to give different answers in order to demonstrate the relation between the IC and the cross validation method." CreationDate="2010-07-26T18:04:31.987" UserId="196" />
  <row Id="532" PostId="636" Score="1" Text="I didn't downvote, but I must say that some explanation or reasoning why to use random forest could make this answer very much more interesting. ;)" CreationDate="2010-07-26T18:14:28.533" UserId="190" />
  <row Id="533" PostId="636" Text="I didn't downvote either, but why random forest over SVM (for instance)?" CreationDate="2010-07-26T18:16:06.860" UserId="5" />
  <row Id="534" PostId="635" Text="Thanks a lot for your answer ! I have read this book from first to last page, but I think it was edition 1... I didn't know it was available online. " CreationDate="2010-07-26T18:24:35.673" UserId="223" />
  <row Id="535" PostId="636" Text="I didn't downvote either, it didn't let me ;-) . @Peter extended. @Shane and why binomial regression? This is just my favorite algorithm and it may work well in this case. " CreationDate="2010-07-26T18:25:43.130" UserId="88" />
  <row Id="536" PostId="636" Text="+1 @mbq because to some extent, that's the &quot;standard&quot; and/or &quot;obvious&quot; answer when it comes to regression. :) (hope that's vague enough...) When it comes to a machine-learning approach, it doesn't seem to me that there is an obvious answer; but your experience is more than enough." CreationDate="2010-07-26T18:26:52.067" UserId="5" />
  <row Id="537" PostId="625" Score="1" Text="Rob, your book is great! " CreationDate="2010-07-26T18:37:01.453" UserId="74" />
  <row Id="538" PostId="26" Score="4" Text="I think this question is ok. Actually, it was the most upvoted example on topic question on Area 51. Basics are ok here!" CreationDate="2010-07-26T18:52:44.213" UserId="190" />
  <row Id="539" PostId="50" Score="2" Text="@robin We don't have a policy that we can not ask question, before you have googled them. We want to be the resource that comes on top when you google this question :)" CreationDate="2010-07-26T18:55:12.197" UserId="190" />
  <row Id="540" PostId="609" Text="Just to clarify for anyone unfamiliar with Wasserstein distances who reads this and gappy's answer: the L^2 Wasserstein distance (W2) is *not* the same as the Cramer-von Mises distance." CreationDate="2010-07-26T18:55:17.140" UserId="89" />
  <row Id="541" PostId="130" Text="This should be community wiki since it is subjective." CreationDate="2010-07-26T18:55:47.077" UserId="5" />
  <row Id="542" PostId="641" Text="Should be community wiki?" CreationDate="2010-07-26T19:25:49.283" UserId="5" />
  <row Id="544" PostId="645" Text="This seems very vague to me.  What kind of data, and what kind of analysis?  Also, this should be community wiki if it is subjective." CreationDate="2010-07-26T19:36:21.660" UserId="5" />
  <row Id="545" PostId="650" Text="Peter, you beat me to the punch! I completely agree with storing data as text, though depending on the size (hundreds of millions of obs) it may be necessary to move into a map-reducible database (e.g., Pig, Cassandra, or one of the NoSQL options)." CreationDate="2010-07-26T19:37:03.767" UserId="302" />
  <row Id="547" PostId="652" Score="5" Text="Related: http://stats.stackexchange.com/questions/421/what-book-would-you-recommend-for-non-statistician" CreationDate="2010-07-26T19:40:56.657" UserId="5" />
  <row Id="549" PostId="652" Score="3" Text="Should be community wiki." CreationDate="2010-07-26T19:41:32.633" UserId="5" />
  <row Id="550" PostId="652" Text="You tagged this as bayesian and machine-learning.  What kind of data analysis are you interested in?" CreationDate="2010-07-26T19:46:47.423" UserId="5" />
  <row Id="551" PostId="650" Text="Oh ok interesting! So just take data for each variable and lay it out in row-column format and get to number crunching eh? Are there any tools I should be looking at or should I just be programming something?" CreationDate="2010-07-26T19:47:31.743" UserId="274" />
  <row Id="552" PostId="652" Text="Shane: Honestly, I don't really know yet. What kind of data analysis is there? Should I be posing this as yet another question for the site?" CreationDate="2010-07-26T19:49:24.203" UserId="274" />
  <row Id="553" PostId="647" Text="I can do Monte Carlo, I just wanted to do in a more 'sciency' way; still you're right that the distribution is not normal, so this sd will be useless for testing." CreationDate="2010-07-26T19:50:17.197" UserId="88" />
  <row Id="554" PostId="650" Text="R is a very extensive (and free) toolkit/programming language/library for statistics. My favorite for most things is however Python with SciPy / NumPy" CreationDate="2010-07-26T19:50:23.187" UserId="190" />
  <row Id="555" PostId="648" Text="I thought it shows the binomial distribution; I don't think that its asymptotics have a direct link with CLT." CreationDate="2010-07-26T19:52:46.723" UserId="88" />
  <row Id="556" PostId="655" Text="May be a bit too academic for myself being such a beginner..." CreationDate="2010-07-26T19:53:40.310" UserId="274" />
  <row Id="558" PostId="652" Text="There are many kinds.  :)  Those are just two specific areas, so it seems odd for tags on such a general question." CreationDate="2010-07-26T19:54:21.243" UserId="5" />
  <row Id="559" PostId="652" Text="@Justin If you don't know it yet, this question is too broad and vague. A nice thing to learn what a field contains, is to look on the tags of this sites and see what questions match with it. Also wikipedia and the books you bought can give you and idea of what you want (although the books maybe not name the field wherein they operate, more on the practical overall part)" CreationDate="2010-07-26T19:54:54.987" UserId="190" />
  <row Id="560" PostId="648" Score="2" Text="bean machine by the author of the package animation... http://yihui.name/en/wp-content/uploads/2010/07/animation-useR2010-Xie.pdf" CreationDate="2010-07-26T19:55:09.510" UserId="223" />
  <row Id="562" PostId="643" Text="This should be a community wiki." CreationDate="2010-07-26T19:57:08.537" UserId="88" />
  <row Id="565" PostId="648" Score="1" Text="@mbq take a look at http://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation" CreationDate="2010-07-26T20:01:14.113" UserId="223" />
  <row Id="566" PostId="658" Text="what is the relation between the Cauchy distribution and the CLT or the failling of the CLT ? " CreationDate="2010-07-26T20:02:15.373" UserId="223" />
  <row Id="567" PostId="652" Text="Why is this question not closed yet as duplication ? " CreationDate="2010-07-26T20:03:09.773" UserId="223" />
  <row Id="568" PostId="656" Text="That's great.  The standard way of dealing with outliers." CreationDate="2010-07-26T20:06:32.370" UserId="5" />
  <row Id="569" PostId="660" Text="What do you mean by *types* of &quot;textual&quot; content?" CreationDate="2010-07-26T20:08:17.403" UserId="68" />
  <row Id="570" PostId="648" Text="@robin I have wrote about it, what's the problem?" CreationDate="2010-07-26T20:09:41.097" UserId="88" />
  <row Id="571" PostId="652" Text="@Peter Smit: Thanks I'll give that a shot.&#xA;@Shane: Thanks as well... I was trying to give some idea of context with my tags.&#xA;&#xA;Beware the noobs!" CreationDate="2010-07-26T20:10:14.077" UserId="274" />
  <row Id="572" PostId="658" Score="1" Text="@robin Consult http://en.wikipedia.org/wiki/Cauchy_distribution#Properties" CreationDate="2010-07-26T20:12:30.357" UserId="88" />
  <row Id="573" PostId="652" Text="@robin There are only 7 users able to vote for closing... so it can take a small while" CreationDate="2010-07-26T20:17:54.540" UserId="190" />
  <row Id="574" PostId="600" Text="Regarding 1., that's correct. Regarding 2. In principle all of the above distances could carry over to R^n, however I don't know of popular non-parametric tests based on *any* distance. It would be interesting to know if there are any." CreationDate="2010-07-26T20:18:19.937" UserId="30" />
  <row Id="575" PostId="641" Text="I think a general wiki for where to get data is great, with a section for survey data." CreationDate="2010-07-26T20:21:15.657" UserId="302" />
  <row Id="576" PostId="666" Text="Flagged as spam. I don't want to have to click on that links without any information in the post related to the question. Also, I think your blog posts are not really layman explanations as the question asker had in mind." CreationDate="2010-07-26T20:22:32.533" UserId="190" />
  <row Id="577" PostId="165" Text="Should be community wiki?" CreationDate="2010-07-26T20:22:54.280" UserId="5" />
  <row Id="579" PostId="660" Text="Could you show some sample data? " CreationDate="2010-07-26T20:27:15.987" />
  <row Id="580" PostId="666" Text="Interesting... To be able to actually listen to music seemed like something anyone can appreciate. The second one was definitely not laymen you're right.  You should reconsider flagging this as spam." CreationDate="2010-07-26T20:30:47.320" UserId="274" />
  <row Id="581" PostId="672" Score="4" Text="Related: http://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english" CreationDate="2010-07-26T20:34:14.470" UserId="5" />
  <row Id="583" PostId="658" Text="great ! I did not know that ! " CreationDate="2010-07-26T20:47:47.713" UserId="223" />
  <row Id="589" PostId="262" Score="1" Text="&quot;Hardly publication quality&quot;???? I realise that it isn't perfect - the phrase &quot;...should you get you started..&quot; covers that bit. But with a little additional work, i.e. axis labels, I would say it's fine. BTW, what journals do you publish in?" CreationDate="2010-07-26T21:20:59.987" UserId="8" />
  <row Id="590" PostId="652" Text="@robin @peter: It isn't clear to me that it's an exact duplicate either.  Are you both referring to the one I linked above or another one?" CreationDate="2010-07-26T21:23:14.653" UserId="5" />
  <row Id="592" PostId="667" Text="So probability is pure mathematics and statistics is applied mathematics?" CreationDate="2010-07-26T21:24:35.293" UserId="327" />
  <row Id="593" PostId="673" Text="So statistics is synonymous with data analysis?" CreationDate="2010-07-26T21:25:04.843" UserId="327" />
  <row Id="594" PostId="652" Text="@shane I agree that it is not exact duplicate, but then it is too broad or not enough and the title is quite similar. In addition, my view is that someone asking for a book on stat.stack could at least say for what purpose and for what level. " CreationDate="2010-07-26T21:29:12.007" UserId="223" />
  <row Id="595" PostId="652" Text="@robin: I agree 100% with that." CreationDate="2010-07-26T21:33:06.110" UserId="5" />
  <row Id="597" PostId="675" Text="&#xA;	&#xA;So statistics observes what happens in the physical world, theorizes about the underlying process, and then having found the process, uses it in the sense of probability to predict what will happen next?" CreationDate="2010-07-26T21:34:19.947" UserId="327" />
  <row Id="598" PostId="651" Text="I work in medical/epidemiological research, and I see this book on colleagues' shelves all the time.  Still haven't read it myself, but I can attest to its popularity." CreationDate="2010-07-26T21:35:13.770" UserId="71" />
  <row Id="599" PostId="673" Text="I don't see any distinction. " CreationDate="2010-07-26T21:39:17.563" UserId="25" />
  <row Id="600" PostId="667" Score="1" Text="Statistics may be applied and may be not; still the concept of data is always present. " CreationDate="2010-07-26T21:42:34.723" UserId="88" />
  <row Id="601" PostId="685" Text="I feel that your question is not precise enough to get a reasonable answer." CreationDate="2010-07-26T21:44:31.073" />
  <row Id="602" PostId="685" Text="It could be rephrased as: In what ways are statistics misleadingly reported or cited?" CreationDate="2010-07-26T21:51:18.217" UserId="327" />
  <row Id="603" PostId="479" Text="Found a way to do this (high transparency overlay on the &quot;danger&quot; part of the graph) with our toolset, thanks!" CreationDate="2010-07-26T21:53:17.343" UserId="259" />
  <row Id="604" PostId="685" Score="1" Text="Even if it is not off-topic, it should be community wiki." CreationDate="2010-07-26T22:08:47.550" UserId="88" />
  <row Id="605" PostId="687" Text="About the glass, I think that just the boundary between phases lies in the half of its height. " CreationDate="2010-07-26T22:10:48.190" UserId="88" />
  <row Id="607" PostId="685" Text="@hslc: Your re-stated question is much better. I would either suggest asking another question along those lines or better still edit the current one along the lines of your comment." CreationDate="2010-07-26T22:20:56.700" />
  <row Id="610" PostId="13" Text="I bet they used discriminant analysis before logistic regression, as DA was invented well before LR" CreationDate="2010-07-26T22:56:40.980" UserId="74" />
  <row Id="612" PostId="194" Text="@robin, that would be the hard part. Sometimes when thousands of variables influencing something, it's hard to make sense of the patterns that emerged. " CreationDate="2010-07-27T00:09:31.367" UserId="175" />
  <row Id="613" PostId="675" Text="I'm not a statistician, but from my understanding I'd say, yes, that *part* of what statistics does." CreationDate="2010-07-27T00:10:53.173" UserId="89" />
  <row Id="615" PostId="349" Text="Since you are only interested in the median why couldn't you make the bins wider at higher values of your variable?" CreationDate="2010-07-27T00:23:15.770" UserId="196" />
  <row Id="616" PostId="570" Text="drpaulbrewer, thanks for the suggestion." CreationDate="2010-07-27T00:49:44.233" UserId="251" />
  <row Id="617" PostId="702" Score="1" Text="With a millon points and an 8 parameter model, a goodness of fit test like chi-squared tells me that there is essentially no chance that the model is correct. (Which is not surprising, as there are endless factors influencing reality that are not in the model)&#xA;&#xA;RMSE gives me a sense as to how good the model fits the data, but does not give me a sense of whether there is a better model" CreationDate="2010-07-27T01:07:20.330" UserId="72" />
  <row Id="618" PostId="702" Text="Well in order to find out if there is a better model, you could either experiment with different formulations or you could use various plots (e.g.,  exit times vs time) to see if the data is consistent with your model assumptions. You could also plot predicted exit times for a small sample selected at random vis-a-vis actual times to for model improvement ideas." CreationDate="2010-07-27T01:17:13.693" />
  <row Id="619" PostId="612" Text="Consider adding a tag that directly references the R package you are using." CreationDate="2010-07-27T04:00:46.060" UserId="196" />
  <row Id="620" PostId="712" Score="4" Text="What is your data, and what do you want to do with the anonymized data?" CreationDate="2010-07-27T04:42:39.970" UserId="190" />
  <row Id="621" PostId="495" Score="1" Text="I have watched all of those videos. It's a very good introduction to probability and counting. " CreationDate="2010-07-27T06:12:47.193" UserId="339" />
  <row Id="623" PostId="732" Score="8" Text="I don't mind the down vote, but I maintain that this is a deep statistical point, not to be taken lightly.  ;-)  " CreationDate="2010-07-27T07:18:30.913" UserId="251" />
  <row Id="624" PostId="675" Text="great answer! +1 for Persi and Mark" CreationDate="2010-07-27T07:21:49.237" UserId="223" />
  <row Id="625" PostId="242" Text="I will come back to tell you what the results were, but it will be a while as I am working my way through this alongside lots of other tasks. Wasn't sure what you meant about &quot;pass over the question about random variable&quot;? Is there a question you recommend I look at?" CreationDate="2010-07-27T07:46:32.190" UserId="199" />
  <row Id="626" PostId="744" Score="8" Text="I like this one, could be put as an advise when people  write questions on this site ? " CreationDate="2010-07-27T08:48:41.993" UserId="223" />
  <row Id="627" PostId="705" Text="Srikant, thanks for the response!  I'm not sure what you mean by contour plots of covariances (obs v est) -- could you elaborate?  Thanks." CreationDate="2010-07-27T08:52:31.750" UserId="251" />
  <row Id="628" PostId="612" Text="Perhaps Peter can edit it - my rep is not high enough yet." CreationDate="2010-07-27T08:54:56.253" UserId="144" />
  <row Id="629" PostId="675" Score="2" Text="Induction vs Deduction?" CreationDate="2010-07-27T09:14:39.757" UserId="434" />
  <row Id="630" PostId="686" Text="I hadn't seen these ones before. They look good." CreationDate="2010-07-27T09:40:01.247" UserId="183" />
  <row Id="631" PostId="749" Score="1" Text="+1 Wow, nice list." CreationDate="2010-07-27T09:51:43.303" UserId="88" />
  <row Id="632" PostId="734" Score="1" Text="i think only one of the classes in the Iris data set is linearly separable. (From the OP's Question, i think he's after data sets w/ *only* linearly separable classes)." CreationDate="2010-07-27T09:52:30.323" UserId="438" />
  <row Id="634" PostId="734" Text="@doug, good point, still I don't think that one can get any non-synthetic fully linearly separable problem. Nevertheless I'll wait for the OP's reaction." CreationDate="2010-07-27T10:13:30.293" UserId="88" />
  <row Id="635" PostId="740" Text="Is the question related to some specific output from a statistical package? If so, a mention of the software and a screenshot of the output may perhaps help." CreationDate="2010-07-27T10:48:23.613" />
  <row Id="636" PostId="705" Text="See this: http://en.wikipedia.org/wiki/Level_set. Let Sigma be a a 2 dimensional covariance matrix and Y ~ N(0, Sigma). An iso-contour line would plot the set of points Y for which f(Y|sigma) = c where c is a constant. Note that Y is a 2-dimensional vector. You would choose various values of c and hence obtain different iso-contour lines which would give you a sens of the spread of the distribution." CreationDate="2010-07-27T10:54:50.017" />
  <row Id="637" PostId="759" Text="thanks a lot for this answer! " CreationDate="2010-07-27T11:04:27.207" UserId="223" />
  <row Id="638" PostId="759" Text="You're welcome." CreationDate="2010-07-27T11:16:21.010" UserId="88" />
  <row Id="640" PostId="748" Score="8" Text="Still it looks promising." CreationDate="2010-07-27T11:20:02.893" UserId="88" />
  <row Id="642" PostId="741" Text="Thanks, this is just what I was looking for.&#xA;&#xA;This is essentially a million subjects each with an entry and exit time.&#xA;&#xA;Yes we are conditioning to account for the censoring.  " CreationDate="2010-07-27T12:03:40.630" UserId="72" />
  <row Id="643" PostId="693" Score="1" Text="+1 I really like this explanation and think it is very clear." CreationDate="2010-07-27T12:16:21.463" UserId="81" />
  <row Id="644" PostId="337" Text="I'm not interested into *truth* but in getting a function that works. I'm a bioinformatician and taught not to seek dogmatic *truth* but to seek statistics that work. I don't think that there work done with the kind of data that I want to work with that specifics what entropy works best. That's kind of the point why I want to work with the data." CreationDate="2010-07-27T12:17:57.387" UserId="237" />
  <row Id="645" PostId="337" Text="Right, but this is not a discussion about dogmatic truths but about words. You have asked about entropy, so I answered about entropy. Because now I see that you indeed need an answer about time series descriptors, write a question about time series descriptors, only then you'll get an useful answer. " CreationDate="2010-07-27T12:32:45.983" UserId="88" />
  <row Id="646" PostId="695" Text="Both of these are really excellent." CreationDate="2010-07-27T12:35:16.527" UserId="247" />
  <row Id="647" PostId="768" Text="So we can only find patterns that we were looking for in the first place?" CreationDate="2010-07-27T12:49:35.373" UserId="327" />
  <row Id="648" PostId="769" Score="3" Text="Statistics is not a subset of data analysis -- it is a theory that is used in data analysis." CreationDate="2010-07-27T12:56:46.077" UserId="88" />
  <row Id="649" PostId="618" Text="Nice, but I can't see anyone trying to draw a conclusion of causality there. Or are mexican lemon-truck drivers notoriously dangerous once they get over the border?" CreationDate="2010-07-27T12:57:59.863" UserId="270" />
  <row Id="650" PostId="674" Text="What if the random errors become greater than the observable factors over time?" CreationDate="2010-07-27T12:59:24.027" UserId="327" />
  <row Id="652" PostId="579" Text="@mbq. I was thinking of Shao 1995 which is, indeed, only for linear models. I don't know if the result has been extended to other models." CreationDate="2010-07-27T13:30:26.377" UserId="159" />
  <row Id="653" PostId="769" Score="4" Text="Whatever you do in Excel does not count.  Just kidding..." CreationDate="2010-07-27T13:36:13.320" UserId="334" />
  <row Id="654" PostId="769" Text="@Dirk: Wow...your hatred towards Excel knows no bounds.  :)" CreationDate="2010-07-27T13:37:37.183" UserId="5" />
  <row Id="655" PostId="769" Text="This should probably be community wiki since it is subjective/argumentative." CreationDate="2010-07-27T13:40:46.570" UserId="5" />
  <row Id="656" PostId="252" Text="+1 for omegahat - very good source" CreationDate="2010-07-27T13:41:47.660" UserId="22" />
  <row Id="657" PostId="780" Score="2" Text="+1 Nice pointer to the OR exchange site." CreationDate="2010-07-27T13:59:54.027" UserId="5" />
  <row Id="658" PostId="772" Text="Wow, Shane, thanks for the very fast response! I'll look into those references!" CreationDate="2010-07-27T14:02:33.473" UserId="445" />
  <row Id="660" PostId="744" Score="4" Text="Absolutely...asking the right question is one of the most important skills." CreationDate="2010-07-27T14:17:30.600" UserId="5" />
  <row Id="661" PostId="790" Score="1" Text="maybe a reference to the paper could help ? " CreationDate="2010-07-27T14:52:37.723" UserId="223" />
  <row Id="662" PostId="790" Text="Slightly related question: http://stats.stackexchange.com/questions/298/in-linear-regression-when-is-it-appropriate-to-use-the-log-of-an-independent-var/" CreationDate="2010-07-27T14:53:37.487" UserId="5" />
  <row Id="664" PostId="349" Text="drknexus - because we don't know what the largest bin should be." CreationDate="2010-07-27T14:58:50.750" UserId="247" />
  <row Id="665" PostId="47" Text="@robin: thanks for comment. i haven't seen much research in this area and the input from data provider was minimal [so far they were not interested/capable of investigating it further]. after initial exploration of data it was quite clear for me that there are couple of distinctive patterns [examples being 'heavy downloaders' in just a few locations or 'frequent hoppers' with lots of small sessions in large number of locations].&#xA;my goal at this stage was to try to use data itself to tell me how can i divide it best and minimize subjective input." CreationDate="2010-07-27T14:59:00.983" UserId="22" />
  <row Id="666" PostId="790" Text="You may find this paper of interest, which discusses the log mean and how it refines the 'arithmetic mean - geometric mean' inequality: http://www.ias.ac.in/resonance/June2008/p583-594.pdf" CreationDate="2010-07-27T15:03:53.810" UserId="81" />
  <row Id="667" PostId="704" Text="but then how to choose eta, and what doe sthis then mean statistically? i.e. how to form confidence intervals for the median from this result?" CreationDate="2010-07-27T15:08:20.650" UserId="247" />
  <row Id="668" PostId="428" Text="So, according to a long lost colleague, the best apropoach seems to be Chiranjeeb Buragohain and Subhash Suri. Quantiles on Streams.&#xA;http://www.cs.ucsb.edu/~suri/psdir/ency.pdf&#xA;&#xA;I also like Ian's approach, as these medians of smaller data sets will converge to a normal distribution, and so I can form conf intervals for the medians." CreationDate="2010-07-27T15:10:34.293" UserId="247" />
  <row Id="669" PostId="1" Score="1" Text="Although I've accepted an answer, I would recommend that interested people should look at all the answers." CreationDate="2010-07-27T15:22:18.650" UserId="8" />
  <row Id="670" PostId="801" Text="sorry, i should have mentioned that i need to do this in an automated way. the option of &quot;doing it multiple times until i find the one that best suits my purpose&quot; won't work for me. has to be done computationally..." CreationDate="2010-07-27T15:34:03.697" UserId="476" />
  <row Id="671" PostId="794" Text="oh the bayesian is soooo Good..." CreationDate="2010-07-27T15:39:02.963" UserId="223" />
  <row Id="672" PostId="799" Score="1" Text="... and in that vein also the lme4 package (which I find easier to use than lme or nlme) and related packages from Baayen's above referenced book, languageR." CreationDate="2010-07-27T15:40:33.200" UserId="196" />
  <row Id="674" PostId="804" Text="always these bayesian guys..." CreationDate="2010-07-27T15:45:32.327" UserId="442" />
  <row Id="675" PostId="720" Text="I found all the answers to this question helpful, but I think that this one is the most practical.  " CreationDate="2010-07-27T15:46:34.823" UserId="266" />
  <row Id="676" PostId="486" Text="Oddly enough, I was rather interested in hearing what other people had to say..." CreationDate="2010-07-27T15:52:13.307" UserId="266" />
  <row Id="677" PostId="779" Text="As I always understood it, the central limit theorem does not postulate something about averaging a large number of iid random variables. Rather, it states that when sampling means, the distribution of the means becomes normal (independent of the distribution underlying what is sampled from). So I question whether the antecedent for your question holds." CreationDate="2010-07-27T15:57:43.397" UserId="442" />
  <row Id="678" PostId="779" Text="But, if the sampling mean becomes normal irrespective of the distribution of the underlying distribution then is that not the same as saying 'averaging a large number of iid random variables' get us a normal distribution. To me they seem to be equivalent statements." CreationDate="2010-07-27T16:11:46.867" />
  <row Id="679" PostId="660" Text="@Srikant Vadali - sample data could be press releases, news stories, etc .. the textual data would be free-form, likely obtained from rss feeds or similar. Market data for a given company is what I'm looking to analyze/correlate. So maybe Blogger Bill writes a story about an upcoming VMware feature release, and VMW jumps 10%. (Oversimplified, I know)" CreationDate="2010-07-27T16:11:57.037" UserId="292" />
  <row Id="680" PostId="677" Text="that looks like a really promising start :)" CreationDate="2010-07-27T16:12:31.903" UserId="292" />
  <row Id="681" PostId="750" Score="2" Text="I hate this quote. It makes professions using statistics look like you could cheat. But, when someone profoundly uses statistics one knows that actually you cannot cheat. Because when provided with enough information about the statistical procedures used, one can draw a conclusion on the soundness of the procedures/results. If not enough information on the statistical (and other) procedures are provided, you should immediately question the results." CreationDate="2010-07-27T16:15:40.670" UserId="442" />
  <row Id="682" PostId="811" Text="is it ? then we are all intelligent persons here:) " CreationDate="2010-07-27T16:22:37.590" UserId="223" />
  <row Id="683" PostId="799" Text="thanks for the comment, I totally agree with you. lme4 is simply the best around. " CreationDate="2010-07-27T16:22:39.947" UserId="447" />
  <row Id="684" PostId="779" Text="Not in my eyes (but i would like to be convinced otherwise).&#xA;In the one case (the one i think of being meant by CLT) you draw samples from one distribution. Their means are normally distributed.&#xA;What i understand from the question and the quote &quot;average a large number of iid random variables&quot; is sth differnt: individual instantiations from different iid random variables determine (or make up) a trait. Hence, no averaging (i.e., computing a mean) from a single distribution and, hence, no application of the CLT.&#xA;I think mbq's answers points to the same issue." CreationDate="2010-07-27T16:27:57.773" UserId="442" />
  <row Id="685" PostId="815" Text="+1 for harmonic oscillator." CreationDate="2010-07-27T16:35:39.140" UserId="88" />
  <row Id="686" PostId="779" Text="Well the distribution need not be identical if some conditions hold. See: http://en.wikipedia.org/wiki/Central_limit_theorem#Lack_of_identical_distribution" CreationDate="2010-07-27T16:42:43.140" />
  <row Id="687" PostId="787" Text="Hillaire Belloc?  Nice work on digging that up." CreationDate="2010-07-27T16:43:12.517" UserId="5" />
  <row Id="688" PostId="252" Text="Well, RCurl and XML and both on CRAN too..." CreationDate="2010-07-27T16:47:41.510" UserId="334" />
  <row Id="689" PostId="307" Text="As can be read in my and dave's post, saturated models do not per definition lead to perfect fit. but if you use the n-1  polynominal as the model they will. see Sue Doe Nihm's seminal paper on this topic http://psych.fullerton.edu/mbirnbaum/papers/Nihm_18_1976.pdf" CreationDate="2010-07-27T16:52:10.383" UserId="442" />
  <row Id="690" PostId="783" Text="&#xA;	&#xA;I would love to take this one as my accepted answer ! too good to be true ! " CreationDate="2010-07-27T17:10:56.593" UserId="223" />
  <row Id="691" PostId="825" Score="1" Text="IMO, this question belongs on stackoverflow.  Voting to close as off-topic." CreationDate="2010-07-27T17:11:36.873" UserId="5" />
  <row Id="692" PostId="820" Text="Some comments would be appreciated for the down-vote." CreationDate="2010-07-27T17:12:21.957" UserId="488" />
  <row Id="693" PostId="827" Text="Hah. You tricked me. I saw your first (one-line, no links) answer and then filled mine in.  We need mutexes here :)" CreationDate="2010-07-27T17:12:51.957" UserId="334" />
  <row Id="694" PostId="825" Score="1" Text="Dunno. This is middle ground and suitable for either.  Xrefs, maybe?" CreationDate="2010-07-27T17:13:41.177" UserId="334" />
  <row Id="695" PostId="828" Text="What, are you copying my answers now?  :)  This is of course, needless to say, your domain." CreationDate="2010-07-27T17:14:16.587" UserId="5" />
  <row Id="696" PostId="828" Text="Well, mutexes needed. As I commented on your answer, I only saw the first (raw) version and figured well, I may expand on mc and Rmpi.  And then you did and I look like a copycat.  Such is life." CreationDate="2010-07-27T17:15:55.587" UserId="334" />
  <row Id="697" PostId="828" Text="On the other hand, my answer is derived from reading your paper/presentation in the past.  So I guess I'm copying you as well." CreationDate="2010-07-27T17:16:37.660" UserId="5" />
  <row Id="698" PostId="825" Text="@Shane Agree, this is for SO." CreationDate="2010-07-27T17:18:46.907" UserId="88" />
  <row Id="699" PostId="827" Text="Yep...with basic questions like this, I find that it's best to follow the ol' &quot;answer quickly and then improve&quot; routine...may not be a good practice for everyone else though." CreationDate="2010-07-27T17:18:53.007" UserId="5" />
  <row Id="700" PostId="825" Text="I just don't see how the fact that importing data and running SVM has any relevance to the question.  That's why I think it's more of an SO question.  But I could see Xrefs as being a good long-term solution since it is R..." CreationDate="2010-07-27T17:20:58.323" UserId="5" />
  <row Id="701" PostId="822" Text="&quot;market basket analysis&quot; seems like it's what I'm looking for, thanks for input." CreationDate="2010-07-27T17:22:33.487" UserId="488" />
  <row Id="702" PostId="827" Text="thanks for the prompt replies guys. multicore package looks like a good choice for my needs. " CreationDate="2010-07-27T17:27:49.640" UserId="480" />
  <row Id="703" PostId="830" Score="3" Text="I disagree somewhat. Revolution does a great sales job in getting mindshare (as evidenced by your post) but as of right now there is very little in the product you would not already get with the normal R (at least on Linux). Intel MKL, sure, but you can get Goto Blas.  On Windows, they offer doSMP which helps as multicore cannot be built there." CreationDate="2010-07-27T17:42:14.653" UserId="334" />
  <row Id="704" PostId="658" Text="The CLT requires that the MGF's exist in a neighborhood of 0. The Cauchy distribution does not have that property. CLT Win. &#xA;&#xA;Cauchy doesn't even satisfy the weaker requirements of a stronger version of CLT where all that is required is that mean and variance exist. The Cauchy distribution shows that the mean is required to exist for the CLT to hold. It doesn't make the CLT fail. " CreationDate="2010-07-27T17:46:38.767" UserId="62" />
  <row Id="705" PostId="744" Score="3" Text="I remember once where a private industry company commissioned a mathematician to solve a garbage collection routing problem. Long story short, the mathematician complained that the company was only interested in finding a &quot;close enough&quot; solution rather than an optimal solution. I think, ultimately he was fired, and an operations researcher was brought in instead. " CreationDate="2010-07-27T17:59:21.297" UserId="59" />
  <row Id="706" PostId="658" Text="@Baltimark You have misunderstood my post -- its obvious that Cachy is not covered by CLT because of CLT assumptions, otherwise it would be impossible to prove CLT. I have gave this example because people believe that CLT works for all distributions; probably &quot;fail&quot; is not a perfect word, but still I don't think it is a reason for downvote. Ok, I have even changed it to not applicable." CreationDate="2010-07-27T18:05:10.507" UserId="88" />
  <row Id="707" PostId="658" Text="I prefer your edit. The Cauchy distribution is definitely very cool. " CreationDate="2010-07-27T18:11:05.113" UserId="62" />
  <row Id="708" PostId="834" Score="3" Text="I would suggest reviewing the wiki link: http://en.wikipedia.org/wiki/Akaike_information_criterion and then edit the question so that you can highlight the aspect of AIC that you do not understand." CreationDate="2010-07-27T18:14:03.267" />
  <row Id="709" PostId="658" Text="Yup; the another nice &quot;paradox&quot; comes from basic physics -- if you have a particle that has flown between two detectors with an uniform speed and you've measured the distance and time-of-flight with a normally distributed error, the speed obtained from v=s/t is Cauchy distributed, so one may say that it is undefined ;-)" CreationDate="2010-07-27T18:19:06.127" UserId="88" />
  <row Id="710" PostId="834" Score="2" Text="Read also this question: http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other" CreationDate="2010-07-27T18:22:29.223" UserId="88" />
  <row Id="711" PostId="827" Text="I mainly use multicore, still I like snowfall more than snow and Rmpi for its fault tolerance and clean interface." CreationDate="2010-07-27T18:25:39.873" UserId="88" />
  <row Id="712" PostId="754" Text="I guess I should remove it... poor Karl Pearson, one of the inventor of hypothesis testing not understood by the 21st century ... I would vote up if I could, but it's me that put it here :) " CreationDate="2010-07-27T18:31:32.247" UserId="223" />
  <row Id="713" PostId="478" Score="1" Text="The Statistical sleuth is used a the textbook on that great introductory course (there are 64 lectures in total)  http://video.google.com/videoplay?docid=-3474013489970580510&amp;hl=en&amp;emb=1#" CreationDate="2010-07-27T18:36:31.503" UserId="339" />
  <row Id="714" PostId="842" Text="+1 Great point about how this doesn't deal with splitting up the cross-validation." CreationDate="2010-07-27T18:45:28.817" UserId="5" />
  <row Id="716" PostId="709" Text="There are also more advanced treatments on the course pages for PS 236 and PS 239 (graduate-level political science methods courses) at my website: http://cgibbons.berkeley.edu/teaching.html" CreationDate="2010-07-27T19:09:24.027" UserId="401" />
  <row Id="717" PostId="847" Score="3" Text="Have you tried asking Google itself?" CreationDate="2010-07-27T19:24:25.663" UserId="88" />
  <row Id="718" PostId="846" Text="Maybe Wikipedia will be sufficient here: http://en.wikipedia.org/wiki/F-test#Regression_problems" CreationDate="2010-07-27T19:34:49.773" UserId="88" />
  <row Id="719" PostId="806" Score="2" Text="What do you precisely mean by normalization?" CreationDate="2010-07-27T20:13:18.697" UserId="88" />
  <row Id="720" PostId="744" Score="1" Text="@dassouki I think the quote is more about the question .... something like science is not about finding good answer but about finding good questions !" CreationDate="2010-07-27T20:21:49.737" UserId="223" />
  <row Id="721" PostId="680" Text="This was awesome. Thank you so much for this response. You've given me a great jumping off point. Any books you recommend since you seem to &quot;get&quot; where I'm at." CreationDate="2010-07-27T20:24:17.453" UserId="274" />
  <row Id="722" PostId="731" Text="Thanks for the paper, I upvoted you for it." CreationDate="2010-07-27T20:24:58.433" UserId="274" />
  <row Id="723" PostId="680" Text="you're very welcome. books:  &#xA;&#xA;Statistics in Plain English to start. &#xA;Multivariate Data Analysis by Hair after that .   &#xA;  &#xA;These are good web resources:  &#xA;  &#xA;http://www.itl.nist.gov/div898/handbook/  , &#xA;http://www.statsoft.com/textbook/  " CreationDate="2010-07-27T20:54:37.240" UserId="74" />
  <row Id="724" PostId="845" Score="4" Text="gappy specifically stated that he has paired, i.e. non-independent samples, so assumption 2 is violated right away." CreationDate="2010-07-27T22:08:42.250" UserId="279" />
  <row Id="725" PostId="349" Text="Do you have **any** intuition as to what the range will be?  If you're fairly sure that over half of the answers will be below number N, then you can make your last bin as large as you want.  Maybe your last bin is all numbers greater than 1 trillion - would that be high enough?  With the amount of memory in modern systems you can store a LOT of bins and achieve fairly high resolution. In terms of data structures, we're not talking anything fancy and memory intensive here." CreationDate="2010-07-27T22:17:59.170" UserId="54" />
  <row Id="726" PostId="850" Text="It's a good idea, but&#xA;&#xA;    &quot;mtcars$dummy &lt;- 1; lrm(am ~ dummy, data=mtcars);&quot; &#xA;&#xA;gives back:&#xA;&#xA;    singular information matrix in lrm.fit (rank= 1 ) Offending variable(s):&#xA;    dummy &#xA;    Error in lrm(am ~ dummy, data = mtcars) : &#xA;    Unable to fit model using “lrm.fit”&#xA;" CreationDate="2010-07-27T23:03:52.330" UserId="501" />
  <row Id="727" PostId="858" Text="This will due for now.  Is the fact that this doesn't work in the Design package a bug?" CreationDate="2010-07-27T23:06:16.693" UserId="501" />
  <row Id="728" PostId="734" Text="yes - only one of the classes in the iris data set is linearly separable - so, I've now started experimenting with algorithmically generated datasets." CreationDate="2010-07-27T23:22:51.767" UserId="130" />
  <row Id="729" PostId="845" Text="Sorry I missed that. I'll try and think of something else. " CreationDate="2010-07-27T23:41:17.480" UserId="8" />
  <row Id="730" PostId="845" Text="Thanks anyway. I learned something from the wrong answer as well." CreationDate="2010-07-28T00:08:09.173" UserId="30" />
  <row Id="731" PostId="845" Score="1" Text="@gappy: I've updated my answer. Hopefully this will get you started." CreationDate="2010-07-28T00:09:12.573" UserId="8" />
  <row Id="732" PostId="841" Score="2" Text="I'm not sure why the information that the observations are paired is important to the hypothesis being tested; could you explain?" CreationDate="2010-07-28T00:11:01.913" UserId="196" />
  <row Id="733" PostId="834" Score="1" Text="Consider asking the general question about the AIC separately from the stata question." CreationDate="2010-07-28T00:16:29.750" UserId="196" />
  <row Id="734" PostId="666" Text="+1 from me, i actually think this is the best answer here. The OP asked for a layperson explanation of MCMC. If one of the marketing people i work with asked me to explain MCMC to them, and i then asked &quot;Would you prefer a summarized, informal recitation of the Textbook definition, or would you prefer an example?&quot;--i know which they would choose. Aside from that, being able to supply a concise, illustrative implementation of a concept is stronger evidence of actual knowledge of the concept--anyone can talk, but code either works or it doesn't. " CreationDate="2010-07-28T00:17:01.073" UserId="438" />
  <row Id="735" PostId="674" Text="In that case you re-work your model as it is no longer consistent with reality." CreationDate="2010-07-28T01:06:13.940" />
  <row Id="736" PostId="815" Text="the maximum entropy principle is also another reason why the Gaussian distribution is used.&#xA;For example, what are good reasons for using Gaussian errors in the linear model, except tractability ? " CreationDate="2010-07-28T02:10:27.877" UserId="368" />
  <row Id="737" PostId="868" Text="Also help with laying out tables on this site is much welcomed and appreciated." CreationDate="2010-07-28T03:15:57.280" UserId="274" />
  <row Id="738" PostId="869" Text="Interesting... I'm not exactly sure how to apply that to what I've done but I'll think on it a bit more so I can articulate a question. Thanks!" CreationDate="2010-07-28T04:22:26.177" UserId="274" />
  <row Id="739" PostId="705" Text="@Srikant, thanks for the suggestion.  I spent some time trying it out and it seems like a good start at getting a quick visual comparison, especially when the fit is bad." CreationDate="2010-07-28T04:35:31.360" UserId="251" />
  <row Id="740" PostId="870" Text="your reference about q-value should be http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1074290335" CreationDate="2010-07-28T05:54:26.307" UserId="223" />
  <row Id="741" PostId="866" Text="&quot;Say I want to estimate a large number of parameters&quot; this could be made more precise: What is the framework ? I guess it is linear regression? " CreationDate="2010-07-28T05:59:31.147" UserId="223" />
  <row Id="742" PostId="870" Text="The Benjamini-Hochberg procedure is not for calculating the FDR, it is for controlling the FDR (keeping it under a predefined threshold)" CreationDate="2010-07-28T06:11:26.677" UserId="223" />
  <row Id="743" PostId="870" Text="Your question, as it stands, is difficult to understand. What do you mean by &quot;referred to&quot; ? " CreationDate="2010-07-28T06:15:01.757" UserId="223" />
  <row Id="744" PostId="858" Text="Rather a non-implemented feature, still you can send a bug report." CreationDate="2010-07-28T09:23:10.217" UserId="88" />
  <row Id="745" PostId="497" Text="Change to community wiki." CreationDate="2010-07-28T09:27:17.553" UserId="88" />
  <row Id="747" PostId="835" Text="aligatou gozaimasu" CreationDate="2010-07-28T09:35:29.553" UserId="223" />
  <row Id="748" PostId="497" Text="@mbq Done......" CreationDate="2010-07-28T09:36:32.063" UserId="190" />
  <row Id="749" PostId="252" Text="@Dirk yes, that is correct, have edited my answer to include this information, cheers." CreationDate="2010-07-28T10:00:28.783" UserId="81" />
  <row Id="750" PostId="789" Text="This is Creative Commons but does not allow derivates... :( That way, I cannot use the bits in my own course material, just the bits I need them to know..." CreationDate="2010-07-28T11:00:41.763" UserId="107" />
  <row Id="751" PostId="880" Text="But still, what do you want to measure with CV and in what purpose? To get a cutoff of attribute number?" CreationDate="2010-07-28T11:15:17.963" UserId="88" />
  <row Id="752" PostId="880" Text="@mbq: thanks for the advice. I have edited the question accordingly, hope it is more clear now ! " CreationDate="2010-07-28T11:26:47.510" UserId="223" />
  <row Id="753" PostId="887" Text="What do you mean by: &quot;we would like to test whether it is an vis-a-vis the general population&quot;? and &quot;single sample S&quot;?" CreationDate="2010-07-28T12:06:44.597" />
  <row Id="754" PostId="888" Text="@PeterR The link does not work properly. I think the ending bracket at the end is not linked properly." CreationDate="2010-07-28T12:14:10.837" />
  <row Id="755" PostId="886" Score="2" Text="I don't see the interest of these questions about trying to bridge a fictive gap. what is the aim of all that ? in addition there are a lot of others idea that are fundamental in statistic... and loss function is at least 100 years old. can you reduce statistic like that ? maybe your question is about fondamental concept in datamining/statistic/machine learning however you call it ... Then the question already exists and is too wide http://stats.stackexchange.com/questions/372/what-are-the-key-statistical-concepts-that-relate-to-data-mining/381#381. " CreationDate="2010-07-28T12:19:52.493" UserId="223" />
  <row Id="756" PostId="890" Text="Maybe you can precise your problem?" CreationDate="2010-07-28T12:44:10.573" UserId="88" />
  <row Id="759" PostId="886" Text="Well, I do not know much about machine learning or its connections to statistics. In any case, look at this question: http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning which suggests that at the very least that the approaches to answer the same questions are different. Is it that 'unnatural' to wonder if there is some sort of link between them? Yes, I agree that there lot of ideas in statistics. That is why I have fundamental in quotes and restricted the scope to estimating parameters of interest." CreationDate="2010-07-28T13:13:32.700" />
  <row Id="760" PostId="886" Text="@Srikant link between what ? note that I really like to search link between to well defined objects, I find it really natural. " CreationDate="2010-07-28T13:20:05.620" UserId="223" />
  <row Id="761" PostId="893" Text="An excellent link. Thanks for sharing this one!" CreationDate="2010-07-28T13:43:41.130" UserId="1356" />
  <row Id="762" PostId="894" Score="2" Text="It is not quite correct to say &quot;you cannot change z&quot;. In fact, you have to change z to make the sum equal 10. But you have no choice (no freedom) about what it changes to. You can change any two values, but not the third. " CreationDate="2010-07-28T14:29:49.740" UserId="25" />
  <row Id="764" PostId="899" Text="Can you assume that the other two groups are from different Normal distributions?" CreationDate="2010-07-28T14:37:00.770" UserId="8" />
  <row Id="765" PostId="908" Text="@mbq said &quot;You miss one important issue -- there is almost never such thing as T[i]&quot;&#xA;&#xA; I wanted the answer to focus on the problem of selecting the number of variables. Construction (which I agree are not perfect) of T[i]  are discussed here http://stats.stackexchange.com/questions/490/variable-selection-procedure-for-binary-classification&#xA;&#xA;Sometime, it is also usefull to discuss problem separatly. " CreationDate="2010-07-28T15:21:55.357" UserId="223" />
  <row Id="766" PostId="902" Text="@PeterR: This is an interesting paper.  Could you add to your answer with some more references and include (for instance) the paper's title and author?  Are there any particular academic research labs working in this area that would be relevant?" CreationDate="2010-07-28T15:37:02.833" UserId="5" />
  <row Id="767" PostId="908" Score="1" Text="@robin But here you can't tear those apart. The most of algorithms mentions in that question were created to address this issue -- forward selection is to remove correlated features, backward elimination is to stabilize the importance measure, mcmc is to include correlated features...  " CreationDate="2010-07-28T15:37:10.917" UserId="88" />
  <row Id="768" PostId="908" Text="@robin the idea of making some exact importance measure was a base for so-called filter algorithms which are now mainly abandoned since they were just too weak. They have the advantage that they are computationally cheap, still this is not worth it." CreationDate="2010-07-28T15:42:44.737" UserId="88" />
  <row Id="769" PostId="910" Text="I'm not sure that this is entirely correct.  In what sense do machine learning methods work without parameter estimation (within a parametric or distribution-free set of models)? " CreationDate="2010-07-28T15:48:18.650" UserId="39" />
  <row Id="770" PostId="910" Score="1" Text="You are estimating/calculating something (the exact term may be different). For example, consider a neural network. Are you not calculating the weights for the net when you are trying to predict something? In addition, when you say that you train to match output to reality, you seem to be implicitly talking about some sort of loss function. " CreationDate="2010-07-28T15:59:39.143" />
  <row Id="771" PostId="779" Text="@Henrik Is there any meaningful difference between a single sample from each of N independent and *identically distributed* RVs and N independent measurements of a single RV?" CreationDate="2010-07-28T16:03:13.433" UserId="174" />
  <row Id="772" PostId="899" Text="@cgillespie: it is the same group, just with two modes, I guess, and therefore I probably cannot assume this." CreationDate="2010-07-28T16:03:55.587" UserId="219" />
  <row Id="773" PostId="910" Text="@John, @Srikant Learners have parameters, but those are not the parameters in a statistical sense. Consider linear regression y=a*x (without free term for simp.). a is a parameter that statistical methods will fit, feed by the assumption that y=a*x. Machine learning will just try produce a*x when asked for x within the range of train (this makes sense, since it is not assuming y=a*x); it may fit hundreds of parameters to do this." CreationDate="2010-07-28T16:07:24.413" UserId="88" />
  <row Id="774" PostId="909" Text="@Mike: `optimize` requires two distributions to be side-by-side as I understand. In my case one is inside the other, i.e., the values from the second population are on both side of the limits." CreationDate="2010-07-28T16:09:25.503" UserId="219" />
  <row Id="775" PostId="11" Score="1" Text="Interpolation involves 3 things: 1) a class of functions to interpolate, e.g. sound, pictures, terrain; 2) grids of known / unknown points, 4 cases regular &lt;-&gt; scattered; and 3) a model of noise added to 1). There are many many interpolation methods for various cases, most ad hoc, not &quot;in common use&quot;;&#xA;even IDW has variants. Can you describe what you're interpolating ?" CreationDate="2010-07-28T16:11:35.330" UserId="557" />
  <row Id="776" PostId="910" Text="@Srikant About the second issue; yes, it can be called a loss, but on unseen data, not the training one. When statistical models are used for prediction, it is assumed that the train is a perfect representation of reality and contains the full scope of the true process, which is not the case of machine learning." CreationDate="2010-07-28T16:18:22.453" UserId="88" />
  <row Id="778" PostId="349" Text="Any intuition? yes. And your approach could work in general. However, in this case we can not have a lot of memory/computation. It is in a networking application where the device could see tens of thousands of items per second, and have VERY little processing left over for this purpose. Not the ideal/typical scenario, I know, but that is what makes it interesting!" CreationDate="2010-07-28T16:43:50.167" UserId="247" />
  <row Id="781" PostId="894" Text="That's right, thanks for spotting an error!" CreationDate="2010-07-28T17:35:42.817" UserId="1356" />
  <row Id="782" PostId="924" Text="By the way, is this a duplicate of what you posted here? http://stats.stackexchange.com/questions/920/test-if-probabilities-are-statistically-different" CreationDate="2010-07-28T17:42:49.030" />
  <row Id="783" PostId="924" Text="I didn't think it was a duplicate. One question deals with probabilities and this one deals with a discrete variable. " CreationDate="2010-07-28T17:43:42.630" UserId="559" />
  <row Id="784" PostId="926" Text="When I saw your original answer, for a second, I lost faith in my intuition! :-)" CreationDate="2010-07-28T17:49:37.080" />
  <row Id="785" PostId="926" Text="It was surprising to me also! " CreationDate="2010-07-28T17:52:05.393" UserId="287" />
  <row Id="787" PostId="926" Text="#JoFrhwld, would you happen to know the Matlab function to calculate this? I don't have access to R :(" CreationDate="2010-07-28T18:01:10.910" UserId="559" />
  <row Id="788" PostId="930" Text="Indeed we have suggested something like this in this topic; I'll dig for a link." CreationDate="2010-07-28T18:05:47.203" UserId="88" />
  <row Id="789" PostId="930" Text="And I think your criterion for 'intervality' is valid only for a uniform distribution." CreationDate="2010-07-28T18:10:53.123" UserId="88" />
  <row Id="790" PostId="926" Text="Sorry, I don't have access to Matlab! I'd just google around for it. It's surprising you wouldn't have access to R, considering it's free and platform independent." CreationDate="2010-07-28T18:14:50.540" UserId="287" />
  <row Id="791" PostId="931" Text="And probably not easily put on an iPod, if that is essential to you." CreationDate="2010-07-28T18:18:25.183" UserId="561" />
  <row Id="792" PostId="929" Score="1" Text="I don't think it is a valid question here; just check out wiki and references there: http://en.wikipedia.org/wiki/Probability_interpretations" CreationDate="2010-07-28T18:20:32.713" UserId="88" />
  <row Id="793" PostId="929" Text="That wikipedia article is filled with &quot;dubious-discuss&quot; and other tags, so the question would become how comprehensive is that wiki article, what interpretations are missing from it." CreationDate="2010-07-28T18:26:20.673" UserId="560" />
  <row Id="795" PostId="935" Text="Not comprehensive at all then. Thanks." CreationDate="2010-07-28T18:41:44.730" UserId="560" />
  <row Id="796" PostId="928" Text="By the way, P70 - P50 represents the percentage of people who are between the 70th percentile and 50th percentile and that percentage is 20. Clearly that is the same as P50 -P30. When assessing if differences are equal I do not think you should look at the underlying scores. " CreationDate="2010-07-28T18:44:11.760" />
  <row Id="797" PostId="904" Text="&#xA;	&#xA;Fully agree ! the questions that are asked are different. Registration, landmarking, estimation of derivatives can arise from the functional view. This convince me ! so the big deal with functional data (as it stands in statistical literature) would not be that it is defined on a continuous set but more that it is indexed on an ordered set? " CreationDate="2010-07-28T18:44:58.467" UserId="223" />
  <row Id="798" PostId="929" Score="1" Text="Still references are quite informative. And nevertheless even if it is discussive and mentions something that is not present in the book it is some kind of a clue, isn't it?" CreationDate="2010-07-28T18:45:07.110" UserId="88" />
  <row Id="799" PostId="926" Text="@MJoFrhwld, I need to use Matlab so I can incorporate it into my simulation framework. " CreationDate="2010-07-28T18:53:16.423" UserId="559" />
  <row Id="801" PostId="439" Text="I've accepted this answer on the somewhat capricious basis that I now remember Wasserman's book being recommended to me by someone else several years ago.  The same person also recommended &quot;The Cartoon Guide to Statistics&quot; by Gonick and Smith." CreationDate="2010-07-28T19:13:50.953" UserId="89" />
  <row Id="802" PostId="290" Text="I am tempted to say &quot;R for Stata users&quot;, but I would get voted down for this :)" CreationDate="2010-07-28T19:19:36.570" UserId="253" />
  <row Id="803" PostId="917" Score="1" Text="I'd argue from a standpoint of trying to create good guess as to what video game a person likes coding 1s and 0s for like and didn't like isn't a good approach.  A scale of 1-5 or 1-7 is quite easy to elicit and will require fewer datapoints to generate a good model (because each data point provides more information).  With caveats about treating ordinal data as interval data of course applying, but probably not really that important in this context. " CreationDate="2010-07-28T19:23:52.880" UserId="196" />
  <row Id="804" PostId="936" Score="1" Text="Econtalk is one of the most intelligent podcasts out there." CreationDate="2010-07-28T19:36:06.157" UserId="319" />
  <row Id="805" PostId="917" Text="Agreed. I just answered in terms of the question, which was about logistic regression. I've fit proportional odds logistic regressions for ordered data like this in R using `MASS:polr`. " CreationDate="2010-07-28T19:42:06.937" UserId="287" />
  <row Id="806" PostId="918" Text="I was actually going to discuss this same thing. I had a job evaluating CEP/ESP tools, such as Aurora, STREAM, and Esper once..." CreationDate="2010-07-28T20:40:34.987" UserId="110" />
  <row Id="808" PostId="939" Text="Read carefully: http://en.wikipedia.org/wiki/Yates'_correction_for_continuity" CreationDate="2010-07-28T20:55:03.517" UserId="88" />
  <row Id="810" PostId="898" Text="Bayesian methods are theoretically well suited for real time analyses because the essential Bayesian method is to take a prior belief, and using a bit of data, compute a posterior belief -- which can become the prior for a new cycle.  While that part sounds good, there are practical difficulties and significant hurdles in specifying interesting models on real, changing data that are tractable and easily computable." CreationDate="2010-07-28T21:35:46.503" UserId="87" />
  <row Id="811" PostId="944" Score="6" Text="IMO total off-topic. Voted to close." CreationDate="2010-07-28T21:58:10.180" UserId="88" />
  <row Id="812" PostId="948" Text="Will t.test() compare the means across all levels of a factor? " CreationDate="2010-07-28T22:59:28.497" UserId="569" />
  <row Id="813" PostId="949" Text="Till we have latex support, could you please avoid using latex. It is hard to make out what you are saying. See this meta thread: http://meta.stats.stackexchange.com/questions/5/what-typographic-support-is-available-to-support-display-of-statistical-formula" CreationDate="2010-07-28T23:03:58.350" />
  <row Id="814" PostId="917" Text="Thanks guys! TONS O' INFO. :D So then what I'm doing now isn't exactly logistic regression and is instead proportional odds logistic regression?" CreationDate="2010-07-28T23:06:39.797" UserId="274" />
  <row Id="815" PostId="682" Text="I ordered Statistics in Plain English... I'll accept your answer when I get it on Thursday if I dig it as much as I think I will." CreationDate="2010-07-28T23:08:37.843" UserId="274" />
  <row Id="817" PostId="918" Text="VFDT looks very interesting; thanks for the references!" CreationDate="2010-07-29T00:21:10.253" UserId="5" />
  <row Id="818" PostId="928" Text="And the same stands for calculating correlation coefficients, I guess?" CreationDate="2010-07-29T00:26:04.627" UserId="1356" />
  <row Id="819" PostId="928" Text="Yes, that would be correct. In fact correlation would be ratio as 0 means no correlation and such a conclusion is scale invariant. By the way, I suspect that percentiles would also be classified as ratio as the 0 point is scale invariant but it does not really matter. Most statistic applications require interval level measurements not necessarily ratio." CreationDate="2010-07-29T00:53:56.147" />
  <row Id="820" PostId="939" Text="@mbq I did read the Wikipedia article but I wanted to confirm and verify it " CreationDate="2010-07-29T00:55:08.780" UserId="559" />
  <row Id="821" PostId="944" Score="3" Text="I agree. I think R programming question would be a better fit on StackOverlfow.com than here. http://stackoverflow.com/questions/tagged/r" CreationDate="2010-07-29T02:45:20.880" UserId="319" />
  <row Id="822" PostId="808" Text="Thanks. Had the opportunity to work with a great statistician for a while and was always amazed by how much information he could get out of the least amount of data by asking very pointed questions. This quotation so reminds me of hm" CreationDate="2010-07-29T03:30:12.343" UserId="482" />
  <row Id="823" PostId="917" Text="What you _want_ to do is a proportional odds logistic regression. I'm not exactly sure what to call what you _have_ done," CreationDate="2010-07-29T03:47:12.360" UserId="287" />
  <row Id="824" PostId="952" Text="How are you defining redundancy?" CreationDate="2010-07-29T04:15:57.833" />
  <row Id="825" PostId="957" Text="I am not sure I understand what you are saying. I am not sure there is any relationship between a new draw x and the current sample mean mean(S) as x ~ N(mu,sigma^2). Clearly, the draw of x can be anywhere in the support of the distribution. It is more likely to be around mu and less likely to be in the tails but it does not have anything to do with mean(S)." CreationDate="2010-07-29T04:20:42.963" />
  <row Id="826" PostId="955" Score="1" Text="Imo this question is offtopic on this site as it not goes about statistics itself. " CreationDate="2010-07-29T04:36:54.120" UserId="190" />
  <row Id="827" PostId="917" Text="Hahaha ok thanks man" CreationDate="2010-07-29T05:35:37.930" UserId="274" />
  <row Id="828" PostId="603" Text="It does not; GLM is not machine learning. True machine learning methods are wise enough to hold their level of complexity independent of growing number of objects (if it is sufficient of course); even for linear models this whole theory works quite bad since the convergence is poor. " CreationDate="2010-07-29T07:43:01.653" UserId="88" />
  <row Id="829" PostId="707" Text="Have you even read this paper? Nevertheless it is works only for linear models (even the title shows it!) it is about asymptotic behavior for infinite number of objects. 100 is way not enough." CreationDate="2010-07-29T07:49:16.083" UserId="88" />
  <row Id="830" PostId="707" Score="1" Text="And I wish you luck making 10-fold cross validation on set with 9 objects." CreationDate="2010-07-29T07:50:50.653" UserId="88" />
  <row Id="831" PostId="644" Score="2" Text="CV using full set for model selection, huh? It's a common error (still even Wikipedia mentions it), because it is a hidden overfit. You need to make a higher level CV or leave some test to do this right." CreationDate="2010-07-29T08:00:22.393" UserId="88" />
  <row Id="832" PostId="934" Score="1" Text="I'd love to see this loss minimizing in clustering, kNN or random ferns..." CreationDate="2010-07-29T08:09:31.527" UserId="88" />
  <row Id="833" PostId="922" Text="Statistical crowd is clicking randomly in SPSS until desired p-value appears..." CreationDate="2010-07-29T08:17:05.427" UserId="88" />
  <row Id="834" PostId="962" Text="+1 Very nice paper." CreationDate="2010-07-29T08:59:46.400" UserId="88" />
  <row Id="835" PostId="955" Text="Have you voted for close? It is strange that since without Shane we couldn't close any question..." CreationDate="2010-07-29T10:33:19.223" UserId="88" />
  <row Id="836" PostId="750" Score="1" Text="That would be true if everyone were knowledgeable enough in statistics to drive the correct conclusions. Alas, that quote is very applicable to many of those amusing human beings called politicians... " CreationDate="2010-07-29T11:22:35.377" UserId="582" />
  <row Id="837" PostId="955" Text="I do think this could be rephrased in such a way that it would be on-topic (eg about the kind of work that statistical consultants do), but this is more like a job request. " CreationDate="2010-07-29T11:30:08.797" UserId="5" />
  <row Id="838" PostId="607" Score="1" Text="Great post! Note that Vapnick had a PhD in statistics. I'm not sure there are a lot of computer scientist that know the name Talagrand and I'm sure 0.01% of them can state by memory one result of talagrand :) can you ? I don't know the work of Valiant :) " CreationDate="2010-07-29T11:30:57.080" UserId="223" />
  <row Id="840" PostId="955" Text="@mbq Sorry, I don't understand what you mean. Yes I voted to close. All 500+ rep users can (which means 8 users)." CreationDate="2010-07-29T11:45:13.843" UserId="190" />
  <row Id="841" PostId="955" Text="@Shane So you're here... I've got an impression that you should be holidating.&#xA;@Shane, @Peter Still, the problem exists, because not all of this eight is active in closing." CreationDate="2010-07-29T11:53:06.400" UserId="88" />
  <row Id="843" PostId="955" Text="@mbq: I am vacationing, but occasionally my iPhone gets service.  And let's face it: I'm addicted to this." CreationDate="2010-07-29T13:18:13.197" UserId="5" />
  <row Id="844" PostId="929" Text="See the meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed where this question is proposed to be closed." CreationDate="2010-07-29T13:25:38.920" />
  <row Id="845" PostId="966" Text="Your last line should have been with the log(P)'s, right? And isn't this also a simple consequence of Markov processes and the related exponential distribution?" CreationDate="2010-07-29T13:47:51.097" UserId="56" />
  <row Id="846" PostId="980" Text="I forgot this: The points along the x-axis come with varying spacing." CreationDate="2010-07-29T14:06:54.140" />
  <row Id="847" PostId="980" Text="I am not sure I understand. Don't you have a y-axis?" CreationDate="2010-07-29T14:07:59.053" />
  <row Id="848" PostId="966" Text="In the notation I borrowed:  x_t := log(X_t) to the p_t are the logs of the P_t." CreationDate="2010-07-29T14:08:04.870" UserId="334" />
  <row Id="849" PostId="980" Text="Ah, sorry. I misstyped. I have now changed it above." CreationDate="2010-07-29T14:11:05.440" />
  <row Id="850" PostId="947" Text="Thanks, this worked." CreationDate="2010-07-29T14:12:49.710" />
  <row Id="851" PostId="905" Score="3" Text="Yes, and a slight clarification is that online learning algorithms, at least as studied in Machine Learning, mostly make the assumption that your ability to store examples is very limited compared to the size of the data set. In the most limiting case, you only get to see one example at a time, and then you have to forget it after you've used it to update your classifier." CreationDate="2010-07-29T14:14:16.260" UserId="6" />
  <row Id="852" PostId="944" Score="1" Text="@mbq: @JohnD.Cook: This is not a programming question. It is also not a question about a common tool of programmers. It is a question about a common tool of statisticians. That is why I asked it here. I don't mind that it was closed, though, since I received and accepted the correct answer." CreationDate="2010-07-29T14:16:46.300" />
  <row Id="853" PostId="981" Text="Ah, interesting, but I need it to be predictable, i.e. to have the same result each time I view the data." CreationDate="2010-07-29T14:18:32.553" />
  <row Id="854" PostId="980" Text="I also think you need to provide a bit more information. For example, I still cannot visualize the graph. What is your goal?" CreationDate="2010-07-29T14:19:04.330" />
  <row Id="855" PostId="981" Text="In that case, generate the *n* indexes of the points you choose, and store those indexes." CreationDate="2010-07-29T14:21:07.587" UserId="8" />
  <row Id="856" PostId="981" Text="Or store the seed to the RNG before sampling." CreationDate="2010-07-29T14:25:18.947" UserId="334" />
  <row Id="857" PostId="981" Text="Dirk's solution regarding the seed is probably the better option." CreationDate="2010-07-29T14:31:10.027" UserId="8" />
  <row Id="858" PostId="983" Text="Is there any way to do this without knowing in advance what the relative varience will be or is it just a case of having to take a guess at the relative varience to do anything at all?  " CreationDate="2010-07-29T14:38:55.030" UserId="210" />
  <row Id="860" PostId="980" Text="Ok, sorry. I have added some more details above." CreationDate="2010-07-29T15:11:17.823" />
  <row Id="861" PostId="982" Text="Yep, visualization is what I want. I have added some more info in the question." CreationDate="2010-07-29T15:11:55.310" />
  <row Id="862" PostId="984" Text="Ok, sorry. I have added some more details above." CreationDate="2010-07-29T15:12:12.270" />
  <row Id="863" PostId="980" Text="Ok, just so I understand better- your x-axis is time with the 0 point being the time of the first sample. Your y-axis is beats per minute. Is that right? I still do not know your goal: Why do you want to reduce the number of data points? - Reduce clutter? or See patterns better? or the existing points are too close to each other for you to figure out what is happening?" CreationDate="2010-07-29T15:17:06.110" />
  <row Id="864" PostId="911" Text="Thanks! But I hoped for a statistic that is faster and more stable to calculate..." CreationDate="2010-07-29T15:25:23.073" UserId="506" />
  <row Id="866" PostId="963" Score="3" Text="You should be very careful in interpreting p-values from these t-tests with this looping approach. If you have 20 columns, and there is no real difference between any of them, at least one comparison is likely to appear significantly different at p &lt; 0.05. You should at least multiply the p-values by the number of tests, to produce the Bonferronoi correction. Or, you could multiply them by their inverse rank for the Holm correction." CreationDate="2010-07-29T15:36:54.343" UserId="287" />
  <row Id="867" PostId="982" Text="seconding plotting raw data with a smoothing line." CreationDate="2010-07-29T15:43:10.833" UserId="287" />
  <row Id="869" PostId="685" Text="	&#xA;See the meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed where this question is proposed to be closed." CreationDate="2010-07-29T15:48:02.003" />
  <row Id="870" PostId="971" Text="Thank you, that's exactly what I needed. I'm not completely sure of what you mean when you talk about the overdispersion (sorry, I'm not a statistician, maybe it's something very basic)... You say that the residual deviance should be equal to the residual degrees of freedom... how would I check that?" CreationDate="2010-07-29T16:10:48.980" UserId="582" />
  <row Id="871" PostId="944" Text="It is a question about using a computer program use; so I think you should try SuperUser. Dirk is there, so R questions will be answered in a blink of eye ;-)" CreationDate="2010-07-29T16:13:58.040" UserId="88" />
  <row Id="872" PostId="981" Text="Calculating averages per each second is ok, but what I do when there is no data for a specific second. I guess I could do some interpolation from the seconds before and after it, but it would be great with some specific (named) method for this, so I don't try to invent something already invented." CreationDate="2010-07-29T16:24:46.173" />
  <row Id="873" PostId="948" Text="I'm not sure what you mean. If you have some continuous variable column A (say people's heights) and some grouping factor column B (say Country of origin), you really want to do a `pairwise.t.test()`. You would do this like `pairwise.t.test(df$A , df$B)`. This function will automatically correct for multiple comparisons." CreationDate="2010-07-29T16:28:03.317" UserId="287" />
  <row Id="874" PostId="997" Text="excellent logic, but wouldn't that be binom.test(3,13,0.5) (which is the same as binom.test(10,13,0.5)" CreationDate="2010-07-29T17:31:51.323" UserId="605" />
  <row Id="875" PostId="982" Text="thirding plotting raw data with a smoothing line --- You might want to also plot the change in BPM over time as a separate visualization." CreationDate="2010-07-29T17:52:51.363" UserId="601" />
  <row Id="876" PostId="981" Text="The interpolation you are talking about is the moving average or smoothing bit." CreationDate="2010-07-29T18:21:06.423" UserId="8" />
  <row Id="877" PostId="993" Text="You should really should state would language you sample code is for." CreationDate="2010-07-29T18:33:14.403" UserId="8" />
  <row Id="878" PostId="730" Text="I use this quote a lot to explain the difficulties in mathematicians transitioning to statistics" CreationDate="2010-07-29T18:48:03.407" UserId="549" />
  <row Id="879" PostId="971" Text="If you give `summary(model1)` you'll see something like `Residual deviance: -2.7768e-28  on 0  degrees of freedom`" CreationDate="2010-07-29T18:55:47.400" UserId="339" />
  <row Id="882" PostId="138" Score="1" Text="Refer to SO. http://stackoverflow.com/questions/192369/books-for-learning-the-r-language/2270793" CreationDate="2010-07-29T19:17:06.187" UserId="1356" />
  <row Id="884" PostId="981" Text="But moving average will not reduce the number of data points, will it? As far as I know it will _only_ smoothen. Or are there optional moving averages that does both?" CreationDate="2010-07-29T19:29:56.217" />
  <row Id="885" PostId="763" Text="@doug: this is great information.  Do you know of any papers that cover this?  " CreationDate="2010-07-29T19:39:40.220" UserId="5" />
  <row Id="886" PostId="981" Text="I rewrote the answer. Hopefully it's clear and useful now." CreationDate="2010-07-29T20:07:35.143" UserId="8" />
  <row Id="887" PostId="993" Text="@csgillespie: sorry; it's Matlab code, as Elpezmuerto requested in his comment to the other answer." CreationDate="2010-07-29T20:44:37.253" UserId="506" />
  <row Id="888" PostId="899" Text="Do you know that members of the second group aren't included in the first group or are you just willing to mistakenly label those members as belonging to the first group?" CreationDate="2010-07-29T20:54:53.360" UserId="237" />
  <row Id="890" PostId="926" Text="There is a chi-square test function in Matlab Central File Exchange: http://www.mathworks.com/matlabcentral/fileexchange/4779" CreationDate="2010-07-29T21:19:19.083" UserId="128" />
  <row Id="891" PostId="1004" Text="KS test always turns out to be non-normal with very large sets (I have dataset larger than 1 million data points). Also, I would like to have a quantifying measure that tells me about goodness of fit rather than just a test.&#xA;&#xA;Am I asking for too much?&#xA;&#xA;Thanks in advance,&#xA;A" CreationDate="2010-07-29T21:21:41.907" UserId="608" />
  <row Id="892" PostId="1004" Text="I agree with the question answerer.  What you are looking for is exactly a KS test.  The heightened ability of a KS test to detect violations of normality in large datasets is not a reason to toss it aside.  But, you may want to set different thresholds in terms of the D statistic depending on your sample size or get into the guts of the equation and see if you can remove the increased likelihood of rejecting the null as a function of sample size." CreationDate="2010-07-29T21:50:42.850" UserId="196" />
  <row Id="893" PostId="1001" Text="Spearman's Correlation Coefficient is used to compare relative rank orders.  It strikes me that when comparing normal distributions in this way you are particularly unlikely to detect differences in kurtosis." CreationDate="2010-07-29T21:53:37.250" UserId="196" />
  <row Id="894" PostId="981" Text="Thanks, very clear! Random sampling sounds like a great tool. But what can I do if the original data points are very unevenly spread out - I still want, for example, one final data point per 5 second interval, and that might not be present (maybe not even before the random sampling). That means I need to do some kind of interpolation. " CreationDate="2010-07-29T22:09:51.903" />
  <row Id="895" PostId="1001" Text="Not adding that this code is a total junk; you must normalize histograms somehow, cutting the end is not a good idea (still I have no idea how to do it). And make code a code (indent with 4 spaces)." CreationDate="2010-07-29T22:13:58.017" UserId="88" />
  <row Id="896" PostId="1004" Text="And as drknexus wrote, it is easy to extract the statistic and use it for comparison. Even the p-values will do." CreationDate="2010-07-29T22:15:59.590" UserId="88" />
  <row Id="898" PostId="1004" Text="Is it actually valid to use p-values as a qualitative measure if they are not statistically significant (i.e. below .05)?  Wouldn't it be much better to have some kind of effect size?" CreationDate="2010-07-29T23:25:52.993" UserId="608" />
  <row Id="899" PostId="1012" Score="1" Text="What is &quot;not meaningful&quot; about the standard deviation of a uniform distribution? It is a measure of spread for the uniform, just as it is for almost every other distribution. It may not be the best measure of spread, but it is certainly meaningful." CreationDate="2010-07-29T23:32:38.380" UserId="159" />
  <row Id="900" PostId="965" Text="+1 from me--domain-independent answers are particularly useful here. It seems to me that in interdisciplinary domains, the &quot;why&quot; is often ignored because of the (understandable) emphasis on practical application.  " CreationDate="2010-07-29T23:32:42.013" UserId="438" />
  <row Id="901" PostId="1023" Score="3" Text="community wiki?" CreationDate="2010-07-29T23:55:06.170" />
  <row Id="902" PostId="1004" Score="4" Text="Why below 0.05, not 0.04973? Statistical significance does not have any in-depth meaning, it is just an accepted probability of analysis failure.  The operation of transforming statistic into a p-value is monotonic, so there is no problem with comparison. Still obtaining a significance level of this comparison is problematic (I have no better idea than bootstrap)." CreationDate="2010-07-30T00:20:42.420" UserId="88" />
  <row Id="903" PostId="1004" Text="Hi drknexus, you said &quot;set different thresholds in terms of the D statistic depending on your sample size&quot;. How can this be achieved with kstest. I did not find a way to manipulate thresholds. I assume that with &quot;getting into the guts of the equation&quot; you mean transformations before testing. My distribution looks multi-modal and I have already an idea what these multi-modal tendencies might be i.e. how I can single them out..." CreationDate="2010-07-30T00:42:25.733" UserId="608" />
  <row Id="904" PostId="429" Score="1" Text="Besides... Shapiro-Wilk's test is often used when estimating departures from normality in small samples. Great answer, John! Thanks." CreationDate="2010-07-30T01:24:13.157" UserId="1356" />
  <row Id="905" PostId="871" Text="See the meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed where this question is proposed to be closed." CreationDate="2010-07-30T02:00:21.963" />
  <row Id="906" PostId="997" Text="Thanks Andreas. I fixed it. You are right of course, and that is what I entered into R. I am not sure how I copy and pasted something totally different into this answer, but it is fixed now." CreationDate="2010-07-30T03:47:01.817" UserId="25" />
  <row Id="907" PostId="957" Text="But you don't and can't know mu or sigma. What you know are the mean and SD of the sample, mean(S) and stdev(S). " CreationDate="2010-07-30T03:51:27.510" UserId="25" />
  <row Id="908" PostId="1027" Text="Thanks for your thoughts. Maybe I wasn't clear though. I do have paired sets. They are the standardized quadratic scores of the test-retest results, ie some distributions were elicited twice. I will edit the question." CreationDate="2010-07-30T06:42:57.067" UserId="108" />
  <row Id="909" PostId="971" Text="@gd047: thanks!" CreationDate="2010-07-30T06:49:40.687" UserId="582" />
  <row Id="910" PostId="1012" Text="All I meant was that the standard deviation is not really a parameter that defines or describes the uniform distribution well.  In my mind, the standard deviation refers to the spread of a normal, or near normal distribution.  &#xA;&#xA;Simply because a value is calculable does not mean that it is interesting or meaningful.  For example, I might be able to calculate what rate parameter from an exponential distribution best matches a normal distribution, but to me such a value would not be particularly meaningful because the distribution being described is not actually exponential." CreationDate="2010-07-30T06:51:57.670" UserId="196" />
  <row Id="911" PostId="949" Score="1" Text="@Yaroslav Could you put $ signs around the LaTex? It now will be rendered." CreationDate="2010-07-30T07:04:38.770" UserId="190" />
  <row Id="912" PostId="916" Text="It also seems as if $y$ is always larger than some linear function of $x$, $y&gt;C x$." CreationDate="2010-07-30T10:11:17.393" UserId="56" />
  <row Id="914" PostId="981" Text="@Pete - see answer" CreationDate="2010-07-30T10:45:12.760" UserId="8" />
  <row Id="915" PostId="916" Text="Possibly; still it is hard to tell without a zoom on this dense area (points are not transparent, so one cannot judge the density, and so this plot may be deceiving)." CreationDate="2010-07-30T10:48:57.713" UserId="88" />
  <row Id="917" PostId="445" Text="In referring to other questions, don't say &quot;above&quot; or &quot;below&quot; as the order can change depending on votes or whether the answer is accepted, for example." CreationDate="2010-07-30T11:33:28.503" UserId="159" />
  <row Id="918" PostId="1012" Score="3" Text="I can define a uniform distribution with mean 0.5 and standard deviation 1/12. It is perfectly well defined, but it is not the most natural parameterization. There is nothing about standard deviations that implies normality." CreationDate="2010-07-30T11:36:04.613" UserId="159" />
  <row Id="921" PostId="1031" Text="Please add $ around your LaTeX. It should now be rendered ok. See http://meta.math.stackexchange.com/questions/2/tex-math-markup-is-sorely-needed" CreationDate="2010-07-30T13:10:35.327" UserId="159" />
  <row Id="922" PostId="21" Text="What kind of demographic data are you trying to forecast? Every data could (and should) be treated differently." CreationDate="2010-07-30T13:43:17.743" UserId="614" />
  <row Id="923" PostId="1040" Text="You can latex on this site. Please enclose the tex with $ $. See this meta thread: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats" CreationDate="2010-07-30T14:29:26.027" />
  <row Id="924" PostId="1027" Text="Could you clarify how you are computing standardized quadratic scores? And, what exactly your are correlating?" CreationDate="2010-07-30T14:44:55.257" />
  <row Id="927" PostId="1027" Text="@Srikant: I updated the question with some background information " CreationDate="2010-07-30T17:16:03.600" UserId="108" />
  <row Id="928" PostId="1045" Text="Not an answer (because I don't know) but I just saw this new book on the Springer web site: &quot;A Comparison of the Bayesian and Frequentist Approaches to Estimation&quot; http://www.springer.com/statistics/statistical+theory+and+methods/book/978-1-4419-5940-9 that (one would assume) may have some answers." CreationDate="2010-07-30T17:20:19.320" UserId="247" />
  <row Id="929" PostId="1019" Text="I don't understand... one set per person?" CreationDate="2010-07-30T17:40:15.650" UserId="88" />
  <row Id="930" PostId="1019" Text="@robin &amp; @mbq I would suggest keeping it one dataset per post. This so people can indicate with votes which of the suggested ones there also suggest/support" CreationDate="2010-07-30T17:59:24.413" UserId="190" />
  <row Id="931" PostId="1049" Text="Welcome -- Nice to see you here, Pat!" CreationDate="2010-07-30T18:35:19.240" UserId="334" />
  <row Id="932" PostId="949" Text="cool, mostly works, although \{ is not recognized and \ldots looks strange" CreationDate="2010-07-30T18:43:31.250" UserId="511" />
  <row Id="933" PostId="949" Text="mostly worked here on firefox/Linux.  the subscript 1 on x1,...,xd didnt appear as a subscript but the others did.  The issues with formatting are minor, mostly the latex appears higher than other characters in the line.  " CreationDate="2010-07-30T19:52:01.763" UserId="87" />
  <row Id="934" PostId="1051" Text="I understand what you wrote but I am not sure that answers my question. The method you describe and the methods in the linked question in my ps avoid estimating the nuisance parameter on the grounds that the nuisance parameter is not of interest. Such an approach is fine but then without estimating the nuisance parameter we cannot construct a confidence interval for this parameter. Thus, the sentence in the wiki makes sense only if we estimate the nuisance parameter using standard maximum likelihood. " CreationDate="2010-07-30T22:45:06.690" />
  <row Id="935" PostId="1051" Text="So my question is: if we estimate the nuisance parameter using maximum likelihood how is its treatment any different than estimating any other parameter. By the way there seems to be a typo in your eqns as the second term f(z|-) should not depend on theta, right?" CreationDate="2010-07-30T22:47:15.360" />
  <row Id="937" PostId="1051" Text="The equation looks right to me -- I hope I'm not having a blind moment staring at the screen here.  In each case, we have one component that depends on theta and the other (nuisance) component may or may not.  The crucial point is that we must find a transformation that isolates theta to *some* extent to obtain either a marginal or conditional.  When we can't, we must work with profile and other estimated likelihoods.  We could argue that ML is losing information, but that's another question entirely.  Still, perhaps this sheds light on the differences?" CreationDate="2010-07-30T23:33:50.077" UserId="251" />
  <row Id="939" PostId="934" Text="Well, for a loss function characterization of k-means nearest neighbor, see the relevant subsection (2.5) of this paper: http://www.hpl.hp.com/conferences/icml2003/papers/21.pdf" CreationDate="2010-07-30T23:41:56.227" UserId="39" />
  <row Id="940" PostId="628" Text="Note that the last definition here is an *Integrated* (or Bayesian) Likelihood, not a Marginal Likelihood." CreationDate="2010-07-31T00:09:58.993" UserId="251" />
  <row Id="941" PostId="628" Text="Is this correct in the RHS for partial likelihood: &quot;L2(θ|theta)&quot;?" CreationDate="2010-07-31T00:13:14.947" UserId="461" />
  <row Id="942" PostId="628" Text="oops. Thanks for spotting the typo. Fixed it now." CreationDate="2010-07-31T00:34:57.243" />
  <row Id="943" PostId="728" Text="It has actually led to very interesting articles... :)" CreationDate="2010-07-31T01:07:04.477" UserId="253" />
  <row Id="944" PostId="1053" Score="2" Text="community wiki please. Your question does not have an 'objective best' answer. " CreationDate="2010-07-31T02:04:28.317" />
  <row Id="946" PostId="1055" Text="Correlation is not necessarily linear - Spearman's rho relies on the monotonic function, and yet, we refer to it as a &quot;correlation coefficient&quot;, not &quot;mutual information coefficient&quot;. And for a good reason: it provides an information about association between two variables. Mutual information, redundant information, mutual variance, correlation - these terms are so similar, and this question refers to _network reconstruction_, so I guess that we ended up in the wrong area with right terminology. This is quite specific question..." CreationDate="2010-07-31T02:26:27.953" UserId="1356" />
  <row Id="947" PostId="1055" Text="Good point. I've edited my answer to include monotonic relationships. I don't know anything about network reconstruction." CreationDate="2010-07-31T03:04:58.373" UserId="159" />
  <row Id="948" PostId="1051" Text="I am not sure how far we should take this discussion as SE is not a good outlet for discussions. I agree with what you said but I am not sure you are addressing the issue I raised. If you use a nuisance free likelihood how can you construct a confidence interval for the nuisance parameter? In any case, I should probably stop here and I will let you have the last comment." CreationDate="2010-07-31T04:27:08.830" />
  <row Id="949" PostId="1015" Text="Is your your probability score identical to what is called the Brier score? (See: http://en.wikipedia.org/wiki/Brier_score)" CreationDate="2010-07-31T04:34:21.133" />
  <row Id="951" PostId="1051" Text="Sorry, I wrote we don't estimate the nuisance, let me try again with some examples that address it.  Partial likelihood just discards information about the nuisance.  Profile replaces it with the MLE at fixed theta, but this doesn't account for uncertainty in lambda.  Contrast with the Bayesian and specification of a prior.  I think I get what you're saying that it's just estimation like any other parameter.  But the treatment matters because it's how you account for the uncertainty due to the nuisance which affects the intervals, whether through the posterior or the likelihood." CreationDate="2010-07-31T04:54:33.057" UserId="251" />
  <row Id="952" PostId="894" Text="+1 I think this is the nicest/simplest example here." CreationDate="2010-07-31T05:18:59.897" UserId="251" />
  <row Id="953" PostId="1019" Text="@Peter, OK, I follow your idea, I have changed the question accordingly." CreationDate="2010-07-31T06:14:49.843" UserId="223" />
  <row Id="954" PostId="728" Text="@Tal: Fully agree! I think the whole area on optimal separation in minimax testing is starting from this idea ... and it is still so confused for a lot of statistician. For those interested see the paper of donoho http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1085408492 (and the references in the paper ! since things are much older than donoho's paper)" CreationDate="2010-07-31T06:19:15.793" UserId="223" />
  <row Id="955" PostId="1015" Text="@Srikant: Yes it is. It is known by both names." CreationDate="2010-07-31T06:49:42.253" UserId="108" />
  <row Id="956" PostId="659" Text="+1 from me--nice explanation." CreationDate="2010-07-31T10:37:32.073" UserId="438" />
  <row Id="957" PostId="1061" Text="Aww, you've beat me. +1" CreationDate="2010-07-31T10:44:50.723" UserId="88" />
  <row Id="958" PostId="1061" Text="Thanks! Is it the same as the df=6 that LogLik returns? Could I have used -2*logLik(fit)+2*(attr(logLik(fit),&quot;df&quot;))  ??" CreationDate="2010-07-31T10:58:19.690" UserId="339" />
  <row Id="959" PostId="255" Score="3" Text="Check out the outlier function in the randomForest package http://lib.stat.cmu.edu/R/CRAN/web/packages/randomForest/randomForest.pdf" CreationDate="2010-07-31T11:01:22.993" UserId="339" />
  <row Id="960" PostId="213" Score="2" Text="5th D in color hue, 6th D in color intensity, 7th D in point size and we can go ;-)" CreationDate="2010-07-31T12:36:14.223" UserId="88" />
  <row Id="961" PostId="963" Text="You are right. I was feeling that something was wrong. They constitute a family of confidence intervals. Perhaps its even better to adjust using Tukey Honest Significant Differences (TukeyHSD in R)" CreationDate="2010-07-31T15:45:10.863" UserId="339" />
  <row Id="962" PostId="1062" Text="Orthogonal parameterization has a broad range of uses.  Please be more specific.  What are you applying it to?" CreationDate="2010-07-31T15:48:18.880" UserId="601" />
  <row Id="963" PostId="1061" Text="@user603 Also, how did you &quot;draw&quot; that indexed sigma?" CreationDate="2010-07-31T15:48:24.497" UserId="339" />
  <row Id="964" PostId="1065" Text="On 2) I believe the confusion is what p(X=.01) means when X is a continuous random variable.  Intuitively, the probability seem to be zero everywhere because there is no chance X is exactly .01. The questioner should review the definition of a density function in the continuous case, which is defined as the derivative of the cumulative density function." CreationDate="2010-07-31T17:15:21.020" UserId="493" />
  <row Id="965" PostId="827" Text="@mbq +1 for snowfall- abstracts snow even further and makes parallel computing with R pretty simple." CreationDate="2010-07-31T18:14:41.427" UserId="13" />
  <row Id="966" PostId="823" Text="It's unfortunate that consensus has so much control over what kind of science gets funded." CreationDate="2010-07-31T18:26:15.927" UserId="13" />
  <row Id="967" PostId="827" Text="You could give it to my answer not Shane's... still thanks ;-)" CreationDate="2010-07-31T18:28:48.433" UserId="88" />
  <row Id="968" PostId="1066" Text="You mean that you are looking at the distribution of subsample **lengths** of subsegments covering each position, right? On the other hand (probably I'm wrong) I'm guessing it has something to do with the sequencing?" CreationDate="2010-07-31T18:40:22.233" UserId="88" />
  <row Id="969" PostId="1066" Text="No, I'm looking at the distribution of the **number of sub-segments** covering a specific position. For example, if we focus on position 1 in the large segment, in the first simulation we have 4 sub-segments covering it; in the second simulation we have 1 sub-segment covering it, etc.&#xA;&#xA;I don't care what are the lengths of the sub-segments covering each position, just how many sub-segments there are.&#xA;&#xA;You are not completely wrong, this has started as a part of an exercise I've been working on in a biology course, but has gone to another domain :)" CreationDate="2010-07-31T18:45:20.927" UserId="634" />
  <row Id="971" PostId="1066" Text="Still, one more doubt -- what is a distribution of the lengths of subsamples? I think it is crucial to the answer. (I'm suspecting it is a binomial distribution)" CreationDate="2010-07-31T18:58:00.717" UserId="88" />
  <row Id="972" PostId="1066" Text="the lengths of the sub-segments? A list of lengths is given. I believe the lengths distribute quite normally, but I did not check it." CreationDate="2010-07-31T19:01:45.513" UserId="634" />
  <row Id="973" PostId="1066" Text="sounds like a poisson distribution... " CreationDate="2010-07-31T19:08:22.567" UserId="601" />
  <row Id="974" PostId="1068" Text="I think I got the idea, although I will have to think about it some more. One other thing I intentionally did not mention at the beginning, is the fact we might have predefined &quot;hot&quot; subranges along our long segment. This means that every subrange that is drawn and completely includes one of the &quot;hot&quot; subranges will not be counted (it will be totally elimnated, not drawn again). " CreationDate="2010-07-31T19:12:59.453" UserId="634" />
  <row Id="975" PostId="1068" Text="For example, if we have a predefined &quot;hot&quot; subrange 20..25 and we are now drawing a subrange of length  40, and we happen to draw 11..50, than we don't count anything (we throw this subrange). However, if we happen to draw 21..60 we count as normal (since our subrange does not include the complete hot subrange, only part of it)." CreationDate="2010-07-31T19:13:25.787" UserId="634" />
  <row Id="976" PostId="1068" Text="If you want the pmf, then it seems the only sensible thing to do is to actually construct it.  Some sort of counting approach probably works best, similar to what I suggested where you don't need to actually count everything.  It's not clear what your goal is though.  If you're interested in the pmf, then an exact pmf is obviously better than some ad hoc approximation.  If you're interested in some underlying parameter, the fact that you can easily simulate from your distribution opens other possibilities." CreationDate="2010-07-31T19:25:14.010" UserId="493" />
  <row Id="977" PostId="1068" Text="OK, so what I really want is the following: I do not know if where the hotspots are (how many are there or how long is each hotspit), but I do have a few hypotheses (each hypothesis is a set of hotspots; each hotspot is just a subrange). These are hidden variables.&#xA;&#xA;I also have one &quot;true&quot; mapping - where all the lengths were drawn and mapped to the long range. This is my data.&#xA;&#xA;What I aim to do is to check which of my hypotheses is most likely given the data (the &quot;true&quot; mapping)." CreationDate="2010-07-31T19:36:44.343" UserId="634" />
  <row Id="978" PostId="1068" Text="(and sorry I can't vote-up yet, I'm still new here :))" CreationDate="2010-07-31T19:40:30.527" UserId="634" />
  <row Id="979" PostId="1068" Text="I don't exactly understand your data.  It sounds like you'll end up wanting to do something like EM or set up a full Bayesian model. If the probability of exactly matching your data using simulation is reasonable (above one in a million or something), you could just do rejection sampling to get the posterior probabilities of your hypotheses.  Unless your application is simple though rejection sampling may not be feasible since the match probability is too small.  In that case, having the pmf will help." CreationDate="2010-07-31T19:57:37.427" UserId="493" />
  <row Id="980" PostId="893" Score="1" Text="It would be nice to have an explanation for *why* degrees of freedom is important, rather than just what it is.  For instance, showing that the estimate of variance with 1/n is biased but using 1/(n-1) yields an unbiased estimator." CreationDate="2010-07-31T20:12:29.810" UserId="493" />
  <row Id="981" PostId="1068" Text="What I thought of doing is: for each scenario, get the pmf for each position, then calculate the likelihood of the data to be produced given the scenario (multiple probability of each position, assuming positions are independent, although they actually are not). Then we have a likelihood for the data given each of the scenarios and we can choose the most likely scenario (from the set of given scenarios).&#xA;&#xA;There is no chance I will fully match my data using simulations. There is too much noise and the segments are quite long." CreationDate="2010-07-31T20:20:15.360" UserId="634" />
  <row Id="982" PostId="1070" Text="I like your explanation (+1)-- it is probably more to the point for Alekk.  I'll leave mine up in the hope that it casts a different, perhaps useful, light on the role of orthogonality." CreationDate="2010-07-31T20:41:58.913" UserId="39" />
  <row Id="983" PostId="1068" Text="I've done some thinking. If I understand your suggestion correctly, I need to keep for each position in my long range m bernouli variables (one for each length, saying what is the probabilty the i-th subrange will cover this position).&#xA;Since I usually work with n=200k and m=20k, this means 4*10^9 variables, which is way too much. This could perhaps be &quot;compressed&quot; since many subsequent positions will have the exact same set of variables, but I can not think immediately of an easy way to do that. On the other hand, perhaps I can do all the calculations on the fly and not save those variables..." CreationDate="2010-07-31T20:44:17.923" UserId="634" />
  <row Id="985" PostId="1026" Text="Actually, to the best of my memory, Harrell strongly discourages the use of AIC.  I guess cross-validation would probably be the safest method around." CreationDate="2010-08-01T01:54:35.720" UserId="253" />
  <row Id="986" PostId="1072" Score="1" Text="Is there some way to quantify what is &quot; increasingly prefered &quot; these days ?" CreationDate="2010-08-01T01:55:20.693" UserId="253" />
  <row Id="987" PostId="1070" Text="Can you comment on how your definition of marginal likelihood relates to the one given in the wiki link? (See: http://en.wikipedia.org/wiki/Marginal_likelihood)" CreationDate="2010-08-01T01:57:42.390" />
  <row Id="988" PostId="1026" Score="1" Text="AIC is asymptotically equivalent to CV. See answers to http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other. I checked Harrell before I wrote that answer, and I didn't see any discouragement of the AIC. He does warn about significance testing after variable selection, with the AIC or any other method." CreationDate="2010-08-01T02:54:39.570" UserId="159" />
  <row Id="989" PostId="282" Score="2" Text="Effort is commendable -- BUT -- neither PC1 nor PC2 tells you who did best in all subjects.  To do so the PC subject coeffcients would all have to be positive.  PC1 has positive weights for Math and Music but negative for Science and English.  PC2 has positive weights for Math and English but negative for Science and Music. What the PCs tell you is where the largest variance in the dataset lies.  So by weighting the subjects by the coefficients in PC1, and using that to score the students, you get the biggest variance or spread in student behaviors. It can classify types but not performance." CreationDate="2010-08-01T02:58:57.017" UserId="87" />
  <row Id="991" PostId="1073" Text="Nice explanation.  Doesn't explain people's cognitive failings, but +1 anyway." CreationDate="2010-08-01T03:17:33.713" UserId="87" />
  <row Id="997" PostId="1051" Text="[moving this comment from the orthogonal question] You might find this paper by Berger et al. interesting: http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.ss/1009211804 -- marginal likelihood is defined on p.5 before they try to sell the integrated form to &quot;likelihoodists&quot;. Also check out the historical references on that page, Cox 1975 and Basu 1977. The former is the origin of partial likelihood and the latter is a critique of Fisher likelihood w.r.t. nuisance parameters. " CreationDate="2010-08-01T04:34:25.180" UserId="251" />
  <row Id="998" PostId="1070" Text="All the cites on that page (Bos; MacKay/p.29) are to Bayesian concepts of likelihood (post ~ lik x prior). There's a connection, but it's not the marginal likelihood in the sense of Fisher and Likelihood theory. [see comment on Berger paper here: http://stats.stackexchange.com/questions/1045/is-there-a-radical-difference-in-how-bayesian-and-frequentist-approaches-treat-nu/1051#1051 ]" CreationDate="2010-08-01T04:37:58.330" UserId="251" />
  <row Id="999" PostId="1026" Text="@Tal: Perhaps from one of his papers rather than the RMS book, I remember Harrell objecting to the use of AIC for simply choosing among a pool of *many* models.  I think his point was that you must add a variable at a time and compare two models methodically or use some similar strategy.  (To be clear, this is in line with Rob's answer.)" CreationDate="2010-08-01T05:14:48.807" UserId="251" />
  <row Id="1001" PostId="282" Text="+1 good comment, cheers. You are of course correct, I should have have written that better and have now edited the offending line to make it clear I hope." CreationDate="2010-08-01T09:29:48.767" UserId="81" />
  <row Id="1002" PostId="1081" Text="One remark -- expand nMDS. Is it just non-metric or something more exotic?" CreationDate="2010-08-01T09:35:29.957" UserId="88" />
  <row Id="1003" PostId="825" Score="1" Text="I have no problem with this sort of Q&amp;A here.  R isn't such a mainstream language (like Python or Java) that a quant would naturally say, &quot;Oh this is a general programming question so I should go to StackOverflow or similar and ask this or look there for solutions&quot;.  Actually it is more a question for an R mailing list or group site.  To serve those budding analysts who want to learn R we should be glad to have an answer here as well.  " CreationDate="2010-08-01T10:32:26.773" UserId="87" />
  <row Id="1006" PostId="226" Text="That's right - if the variance of PC is, say 3.5, then that PC &quot;explains&quot; variability of 3.5 variables from the initial set. Since PCs are additive, `PC1 &gt; PC2 &gt; ... &gt; PCn`, and the sum of their variances is equal to the sum of the variances of the initial variable set, since PCA is computed upon covariance matrix, i.e. variables are standardised (SD = 1, VAR = 1)." CreationDate="2010-08-01T11:51:45.170" UserId="1356" />
  <row Id="1007" PostId="282" Text="You could standardise the vars, hence calculate the sum, in order to see who's the best, or if you prefer, in R: `apply(dtf, 1, function(x) sum(scale(x)))`" CreationDate="2010-08-01T12:08:34.560" UserId="1356" />
  <row Id="1008" PostId="1072" Text="I think that it is recognized to be scientifically more correct in many field in the sense that the shrinkage approach is used more in recent applied stat papers than the *.IC approach. That shows a certain -at least tacit- theoretical consensus." CreationDate="2010-08-01T12:12:41.710" UserId="603" />
  <row Id="1009" PostId="1085" Text="Sweet now I have a name! I read the mentioned example, but it seems that the actual computation is being taken care of by the &quot;R&quot; library, so is of no help to me in understanding the equation." CreationDate="2010-08-01T12:32:17.117" UserId="642" />
  <row Id="1010" PostId="1085" Text="Ok: i misunderstood the question. I thought you were looking for an implementation. In this case, i guess you want simplicity more than statistical strength. I would go for an moving average (beside JAVA is not the right platform to estimate complicated models)" CreationDate="2010-08-01T13:35:07.040" UserId="603" />
  <row Id="1011" PostId="1086" Text="moving average is exactly what I needed, thanks!" CreationDate="2010-08-01T14:40:31.220" UserId="642" />
  <row Id="1012" PostId="981" Text="Great method for dealing with uneaven data. I will try that. A huge thanks for all your help!" CreationDate="2010-08-01T14:50:13.037" />
  <row Id="1013" PostId="1071" Text="It may help if you can clarify what you are trying to accomplish. I also have a few questions: Why is n=4 when you have a 5 rating scale? I am not sure how your weighting scheme works. What are numbers are you referring to when you say &quot; If I proportionally weight the numbers..&quot;? Also your weighting scheme seems a bit odd. Do you mean that you want to compute the weighted avg of the no of occurrences of a word with the weights being the number of 1, 2,.. star reviews?" CreationDate="2010-08-01T15:30:05.807" />
  <row Id="1014" PostId="632" Text="What Srikant found (and what seems confirmed at PhysicsForums) there should be $\sqrt{2}$, so rather $\hat{\sigma}\frac{\sqrt{2}}{2n}$." CreationDate="2010-08-01T15:34:01.873" UserId="88" />
  <row Id="1015" PostId="632" Text="Aww, those comments locks; $\frac{\hat{\sigma}}{\sqrt{2n}}$. At least this one gives the result in agreement with bootstrap." CreationDate="2010-08-01T15:58:24.160" UserId="88" />
  <row Id="1016" PostId="1069" Text="[Ack, deleted a previous comment when I thought it would edit, back to try again.]  This is interesting though I don't understand information geometry +1.  I'm intrigued by this question at quora, which is unfortunately unanswered: http://www.quora.com/What-does-it-mean-to-say-In-information-geometry-the-E-step-and-the-M-step-are-interpreted-as-projections-under-dual-affine-connections " CreationDate="2010-08-01T17:59:05.207" UserId="251" />
  <row Id="1017" PostId="1069" Text="I had wondered where that went!  If I get some time, I think I'll try to answer that question over at quora, though I should admit that I an information geometry novice.  Most of what I know of it is incidental to an an effort to pin down how the features of mathematical spaces (topological, algebraic, and otherwise) enable the application of learning methods.  This, in turn, is part of a quite difficult pet project of mine." CreationDate="2010-08-01T18:51:44.180" UserId="39" />
  <row Id="1018" PostId="1069" Text="If you would like to correspond about any of these subjects feel free to email me at johnnylogic at gmail." CreationDate="2010-08-01T19:09:46.810" UserId="39" />
  <row Id="1019" PostId="113" Text="The aforementioned paper addresses method selection, generally. As for specific examples and more details they may be found scattered among specific meta-methodological disciplines (measurement theory, algorithmic learning theory, statistical learning theory, complexity theory), but I have not found a systematic treatment, thus the question. If you wish to discuss these issues generally, you may email me at johnnylogic at gmail." CreationDate="2010-08-01T19:11:47.790" UserId="39" />
  <row Id="1020" PostId="1040" Text="Thanks for the pointer on Latex" CreationDate="2010-08-01T20:12:49.313" UserId="168" />
  <row Id="1021" PostId="1095" Text="Reg Latex: See this meta thread: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats" CreationDate="2010-08-01T22:58:47.517" />
  <row Id="1022" PostId="1094" Text="I am already modeling the probability of each cell as a Dirichlet, $p \sim Dirichlet(\alpha)$.  Normalizing by $n$ wouldn't quite work since a cell with large $p_i$ would have small likelihood whenever $n$ is fairly large, even when we indeed observed $x_i=1$ (unnormalized).  Does that make any sense?" CreationDate="2010-08-01T23:08:31.900" UserId="647" />
  <row Id="1023" PostId="1094" Text="I am afraid you lost me by the mention of a 'cell' when the question does not mention this word at all. Could you provide some more context so that I can understand better? " CreationDate="2010-08-01T23:13:24.467" />
  <row Id="1024" PostId="1094" Text="Sorry: By cell I mean each $x_i$.  " CreationDate="2010-08-01T23:14:44.807" UserId="647" />
  <row Id="1025" PostId="1093" Text="By the way, will the binomial not work for you? You have k trials and n successes. right?" CreationDate="2010-08-01T23:16:12.010" />
  <row Id="1026" PostId="1094" Text="In other words, I want to sample $n$ integers between 1 and $k$ without replacement using the probability vector $p$." CreationDate="2010-08-01T23:17:14.770" UserId="647" />
  <row Id="1027" PostId="1094" Text="This getting confusing to me. Your $X_i$ is either 0 or 1 but now you want to sample $n$ integers between 1 and $k$?" CreationDate="2010-08-01T23:23:32.467" />
  <row Id="1028" PostId="1094" Text="Sorry: I'm saying that it's an equivalent formulation to sample the indices of $X$ without replacement.  Then $x_i=1$ for those integers that are sampled and $x_i=0$ for those integers that are not.  " CreationDate="2010-08-01T23:39:34.080" UserId="647" />
  <row Id="1029" PostId="1094" Text="In that case, you have basically the following scenario: You flip a coin $k$ times and your constraint is that you need $n$ successes as that will ensure that $X_i$ sum to $n$. This is a binomial distribution. If this satisfies your requirements I will change my answer. If it does not can you explain why this will not work?" CreationDate="2010-08-02T00:04:19.553" />
  <row Id="1030" PostId="219" Text="Would this be good for implementing on GPUs?  Link to details, references?" CreationDate="2010-08-02T00:27:47.807" UserId="646" />
  <row Id="1031" PostId="146" Text="Why did you put `r` tag and what do you mean by &quot;why this is so&quot;? PC's are not correlated, i.e. they're orthogonal, additive, you cannot predict one PC with the another. Are you looking for a formula?" CreationDate="2010-08-02T00:38:33.680" UserId="1356" />
  <row Id="1032" PostId="167" Text="In the set with n variables, n PCs can be extracted, but you can decide how many you'd like to keep, e.g. Guttman-Keiser criterion says: keep all PCs that have eigenvalue (variance) larger than 1. So there..." CreationDate="2010-08-02T00:43:29.833" UserId="1356" />
  <row Id="1033" PostId="1094" Text="Slightly different: I have $k$ different coins, each with a different probability (ie. $p_i$), and I have $n$ successes.  Sorry for the confusion!" CreationDate="2010-08-02T01:14:41.317" UserId="647" />
  <row Id="1035" PostId="1093" Text="Slightly different: I have $k$ different coins, each with a different probability (ie. $p_i$), and I have $n$ successes. Sorry for the confusion!" CreationDate="2010-08-02T01:43:26.147" UserId="647" />
  <row Id="1036" PostId="1094" Text="For the second proposal, what is the form of this posterior?  Another Dirichlet?  It makes sense that the Dirichlet would be conjugate here, just like if we were using a multinomial.  But we're not using a multinomial.  So what is the pdf of the distribution you propose?  (It's a multivariate bernoulli that has been conditioned on observing n successes, right?)" CreationDate="2010-08-02T02:07:56.790" UserId="647" />
  <row Id="1037" PostId="1094" Text="Unfortunately, it is not a dirichlet. The bernoulli can be written as: $f(X_i;p_i) = p_i^{X_i} (1-p_i)^{1-X_i}$ which is not conjugate to the dirichlet. You will have to use Metropolis-Hastings to estimate the parameters." CreationDate="2010-08-02T02:14:54.870" />
  <row Id="1038" PostId="1094" Text="Thanks for your help. " CreationDate="2010-08-02T02:19:15.513" UserId="647" />
  <row Id="1039" PostId="910" Score="2" Text="[citation needed]. In other words, intriguing answer, although it doesn't jive (at least) with a lot of ML literature." CreationDate="2010-08-02T05:03:16.727" UserId="30" />
  <row Id="1040" PostId="1081" Text="Indeed, its non-metric multidimensional scaling." CreationDate="2010-08-02T06:09:19.373" UserId="144" />
  <row Id="1041" PostId="146" Text="I was wondering about the principles behind the logic (in my quest to understand PCA). I used R tag because R people might read this and maybe show R examples. :) " CreationDate="2010-08-02T06:13:21.407" UserId="144" />
  <row Id="1042" PostId="910" Score="1" Text="Classical one is Breiman's &quot;Statistical Modeling: The Two Cultures&quot;. " CreationDate="2010-08-02T06:42:56.153" UserId="88" />
  <row Id="1043" PostId="1025" Text="Interesting posting. I will look into that too. You are probably right." CreationDate="2010-08-02T07:53:58.937" />
  <row Id="1045" PostId="146" Text="Oh, why didn't you say so? Have you seen http://www.statmethods.net/advstats/factor.html" CreationDate="2010-08-02T11:01:14.140" UserId="1356" />
  <row Id="1046" PostId="1113" Text="Thanks for your answer. That solves the problem of bounding.&#xA;&#xA;For my data it goes to 1 very quickly for my data so I guess the next thing I need to do is to scale this information to concentrate on the interesting range which I could do based on the history of it without fear of leaving the bound, just hitting the limit." CreationDate="2010-08-02T15:19:49.210" UserId="652" />
  <row Id="1047" PostId="1114" Text="erf is not a very handy function, provided you don't want to rather use it for its derivative. " CreationDate="2010-08-02T15:26:48.747" UserId="88" />
  <row Id="1048" PostId="1026" Text="Doing a quick search, I found Harrell writing the following &quot;Beware of doing model selection on the basis of P-values, R-square, partial R-square, AIC, BIC, regression coefficients, or Mallows' Cp.&quot;  He wrote that on 12/14/08, on a mailing list titled [R] Obtaining p-values for coefficients from LRM function (package Design) - plaintext.  I guess I misunderstood his meaning." CreationDate="2010-08-02T16:20:19.767" UserId="253" />
  <row Id="1049" PostId="1115" Score="3" Text="Could you please make your question a little more understandable? " CreationDate="2010-08-02T16:38:41.897" UserId="88" />
  <row Id="1050" PostId="1118" Text="Thank you- mbq and stoplan - I computed mean in time1 and time2, and percent change. There are several obs for each idno, but not an equal  no. of obs for each idno in each time period.  There are 2 or 3 vars that cause change in dep. var, but the means are diff from time1 to time2. I am not clear on repeated measures - in SAS I used proc mixed w/repeated option but I am not clear on results! " CreationDate="2010-08-02T17:32:45.887" UserId="474" />
  <row Id="1051" PostId="1118" Text="There is a specific change in 1 indep. var, X1 in time2. What I want to do is show statistically what is causing change in dep var from time1 to time2.  How much is due to change in X1? Am I still unclear?" CreationDate="2010-08-02T17:37:02.900" UserId="474" />
  <row Id="1052" PostId="1118" Text="@Tailltu I think, ideally, editing the question itself to lend clarity is preferable. That way your question would show up in searches by other people who may have an issue similar to yours." CreationDate="2010-08-02T17:58:52.207" />
  <row Id="1053" PostId="1126" Score="2" Text="Community wiki? I do not think there is a 'correct' answer for your questions." CreationDate="2010-08-02T18:28:30.477" />
  <row Id="1054" PostId="763" Text="(didn't get a notifier from SA of your comment) Well, i wasn't referring to any papers when i wrote that, rather just informally summing pieces of my experience relevant to your Question. I'll look through my files and see what i have that is relevant though." CreationDate="2010-08-02T18:32:24.133" UserId="438" />
  <row Id="1055" PostId="1126" Score="2" Text="Community wiki..." CreationDate="2010-08-02T19:18:14.877" UserId="5" />
  <row Id="1056" PostId="1128" Text="+1 Very good point; my number 1." CreationDate="2010-08-02T19:29:21.060" UserId="88" />
  <row Id="1057" PostId="1126" Text="Fascinating question - thank you for it!" CreationDate="2010-08-02T20:27:48.907" UserId="253" />
  <row Id="1058" PostId="1115" Text="Or for example mention what stat package you are using ? (R, SAS, SPSS, etx ?)" CreationDate="2010-08-02T20:30:33.360" UserId="253" />
  <row Id="1059" PostId="1133" Text="I once asked the same question on the R mailing list, and didn't get a response.  I'd suggest you to change your title since your question is regarding &quot;post hoc analysis of chi square - to detect the cause of the significance&quot; (a shorter titles then the one I proposed would be better :) )" CreationDate="2010-08-02T20:33:00.797" UserId="253" />
  <row Id="1060" PostId="1144" Text="Thank you for your response, but what if the signal exhibits a high seasonality (i.e. a lot of network measurements are characterized by a daily and weekly pattern at the same time, for example night vs day or weekend vs working days)? An approach based on standard deviation will not work in that case. " CreationDate="2010-08-02T20:57:49.240" UserId="667" />
  <row Id="1061" PostId="1144" Text="For example, if I get a new sample every 10 minutes, and I'm doing an outlier detection of the network bandwidth usage of a company, basically at 6pm this measure will fall down (this is an expected an totaly normal pattern), and a standard deviation computed over a sliding window will fail (because it will trigger an alert for sure). At the same time, if the measure falls down at 4pm (deviating from the usual baseline), this is a real outlier." CreationDate="2010-08-02T20:58:18.850" UserId="667" />
  <row Id="1062" PostId="1132" Score="1" Text="I reckon loads of computer games are modelling problems dressed in disguise. SimCity for example - the goal of the game is to build as good a model as possible of the hidden game mechanics, then use that model to build a functioning city! (This is all probably a gross over-justification for wasting my youth playing SimCity)" CreationDate="2010-08-02T21:03:44.017" UserId="668" />
  <row Id="1065" PostId="1147" Score="1" Text="Yeah, this is exactly what I am doing: until now I manually split the signal into periods, so that for each of them I can define a confidence interval within which the signal is supposed to be stationary, and therefore I can use standard methods such as standard deviation, ...&#xA;The real problem is that I can not decide the expected pattern for all the signals I have to analyze, and that's why I'm looking for something more intelligent." CreationDate="2010-08-02T21:37:03.073" UserId="667" />
  <row Id="1066" PostId="1142" Score="1" Text="Just for clarity, here's the original question on SO: http://stackoverflow.com/questions/3390458/simple-algorithm-for-online-outlier-detection-of-a-generic-time-series" CreationDate="2010-08-02T21:42:06.913" UserId="71" />
  <row Id="1067" PostId="1142" Text="I think we should encourage posters to post links as part of the question if they have posted the same question at another SE site." CreationDate="2010-08-02T21:47:31.343" />
  <row Id="1068" PostId="886" Text="As, arguably, a machine learner, I'm here to tell you we maximise the heck out of likelihoods. All the time. Loads of machine learning papers start with &quot;hey look at my likelihood, look how it factorises, watch me maximise&quot;. I'd suggest that it's dangerous to claim a fundamental basis of either discipline in terms of inference techniques. It's more about which conference you go to! " CreationDate="2010-08-02T21:47:52.763" UserId="668" />
  <row Id="1069" PostId="1142" Text="yes, you're completely right. Next time I'll mention that the message is crossposted." CreationDate="2010-08-02T21:53:14.870" UserId="667" />
  <row Id="1070" PostId="1148" Text="Again, this works pretty well if the signal is supposed to have a seasonality like that, but if I use a completely different time series (i.e. the average TCP round trip time over time), this method will not work (since it would be better to handle that one with a simple global mean and standard deviation using a sliding window containing historical data)." CreationDate="2010-08-02T22:02:16.023" UserId="667" />
  <row Id="1071" PostId="1148" Score="1" Text="Unless you are willing to implement a general time series model (which brings in its cons in terms of latency etc) I am pessimistic that you will find a general implementation which at the same time is simple enough to work for all sorts of time series." CreationDate="2010-08-02T22:06:07.470" />
  <row Id="1072" PostId="1148" Text="Another comment: I know a good answer might be &quot;so you might estimate the periodicity of the signal, and decide the algorithm to use according to it&quot;, but I didn't find a real good solution to this other problem (I played a bit with spectral analysis using DFT and time analysis using the autocorrelation function, but my time series contain a lot of noise and such methods give some crazy results mosts of the time)" CreationDate="2010-08-02T22:06:34.413" UserId="667" />
  <row Id="1073" PostId="1148" Text="A comment to your last comment: that's why I'm looking for a more generic approach, but I need a kind of &quot;black box&quot; because I can't make any assumption about the analyzed signal, and therefore I can't create the &quot;best parameter set for the learning algorithm&quot;." CreationDate="2010-08-02T22:09:22.837" UserId="667" />
  <row Id="1074" PostId="1147" Text="Here is a one idea: Step 1: Implement and estimate a generic time series model on a one time basis based on historical data. This can be done offline. Step 2: Use the resulting model to detect outliers. Step 3: At some frequency (perhaps every month?), re-calibrate the time series model (this can be done offline) so that your step 2 detection of outliers does not go too much out of step with current traffic patterns. Would that work for your context?" CreationDate="2010-08-02T22:24:42.510" />
  <row Id="1075" PostId="108" Text="I love that blog!" CreationDate="2010-08-02T22:31:03.767" UserId="582" />
  <row Id="1076" PostId="1147" Text="Yes, this might work. I was thinking about a similar approach (recomputing the baseline every week, which can be CPU intensive if you have hundreds of univariate time series to analyze).&#xA;BTW the real difficult question is &quot;what is the best blackbox-style algorithm for modeling a completely generic signal, considering noise, trend estimation and seasonality?&quot;.&#xA;AFAIK, every approach in literature requires a really hard &quot;parameter tuning&quot; phase, and the only one automatic method I found is an ARIMA model by Hyndman (http://robjhyndman.com/software/forecast/). Am I missing something?&#xA;" CreationDate="2010-08-02T22:38:45.803" UserId="667" />
  <row Id="1077" PostId="1147" Text="Please keep in mind I'm not too lazy for investigating these parameters, the point is that these values need to be set according to the expected pattern of the signal, and in my scenario I can't make any assumption." CreationDate="2010-08-02T22:40:16.397" UserId="667" />
  <row Id="1078" PostId="1147" Text="ARIMA models are classic time series models that can be used to fit time series data. I would encourage you to explore the application of ARIMA models. You could wait for Rob to be online and perhaps he will chime in with some ideas. " CreationDate="2010-08-02T22:44:41.197" />
  <row Id="1079" PostId="1026" Text="@Tal. Thanks for finding that. While it sounds like a general statement, I *think* he probably meant &quot;Beware of looking at p-values after doing model selection on the basis of ....&quot; I don't think it makes sense otherwise. Certainly the discussion in that thread was all about p-values" CreationDate="2010-08-02T23:12:30.950" UserId="159" />
  <row Id="1080" PostId="1128" Score="1" Text="+1 Ha. Speaking of overlap, this may be the only answer with none over Epstein's reasons." CreationDate="2010-08-02T23:49:07.253" UserId="251" />
  <row Id="1081" PostId="1069" Text="Cool, thanks. :)" CreationDate="2010-08-02T23:49:50.873" UserId="251" />
  <row Id="1082" PostId="1098" Text="Interesting, thanks.  I wonder though if it's not the Fisher that's more appropriate here -- since in the urn formulation of the problem, N is observed after the experiment, and you condition on the sum.  Quite likely, I'm one of the &quot;confused&quot; the wikipedia article refers to. :-)" CreationDate="2010-08-02T23:52:10.697" UserId="251" />
  <row Id="1083" PostId="795" Text="Do you mean normalize to (0,1)? Please explain clearly what you are asking." CreationDate="2010-08-03T01:08:43.380" UserId="159" />
  <row Id="1084" PostId="1126" Text="I think I like the overall classification given by the answers here more than Epstein's." CreationDate="2010-08-03T01:44:08.383" UserId="251" />
  <row Id="1086" PostId="1149" Score="1" Text="Really great question.  The best way to understand something is from multiple direction of explanation." CreationDate="2010-08-03T02:29:31.550" UserId="253" />
  <row Id="1087" PostId="1026" Score="1" Text="@Tal, @Rob: In that thread, he does say &quot;Be sure to use the hierarchy principle&quot;.  Perhaps of interest, this discussion from medstats (scroll down for Harrell's response): http://groups.google.com/group/medstats/browse_thread/thread/86c44163b849572" CreationDate="2010-08-03T02:38:20.563" UserId="251" />
  <row Id="1088" PostId="1153" Text="+1 from me, excellent. So &gt; 1.5 X inter-quartile range is the consensus definition of an outlier for time-dependent series? That would be nice to have a scale-independent reference." CreationDate="2010-08-03T03:06:39.653" UserId="438" />
  <row Id="1089" PostId="1153" Text="The outlier test is on the residuals, so hopefully the time-dependence is small. I don't know about a consensus, but boxplots are often used for outlier detection and seem to work reasonably well. There are better methods if someone wanted to make the function a little fancier." CreationDate="2010-08-03T03:45:06.910" UserId="159" />
  <row Id="1090" PostId="538" Text="I have some trouble understanding why arrow directions in a corresponding Bayesian network have any relation to causation. For instance, A-&gt;B and B-&gt;A represent different directions for causality, but Bayesian networks for those two structures are equivalent" CreationDate="2010-08-03T06:41:21.290" UserId="511" />
  <row Id="1091" PostId="1160" Text="Probably better suited to stackoverflow since it has no particular data analysis relevance. " CreationDate="2010-08-03T08:28:02.897" UserId="5" />
  <row Id="1092" PostId="795" Text="This question is proposed to be closed. See this meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed" CreationDate="2010-08-03T08:58:14.633" />
  <row Id="1093" PostId="1160" Text="This question is proposed to be closed. See this meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed" CreationDate="2010-08-03T08:58:48.143" />
  <row Id="1094" PostId="1167" Score="1" Text="May be better suited to stackoverflow. " CreationDate="2010-08-03T09:25:06.817" UserId="5" />
  <row Id="1095" PostId="1167" Text="Yes, I was a bit unsure about that. From one perspective R is a scriptable tool for doing statistics, but it is also a programming language." CreationDate="2010-08-03T09:38:46.020" UserId="671" />
  <row Id="1096" PostId="1168" Text="Thank you. I'm not sure i follow you completely. u is supposed to be shorter because it's weekdays, since this is for a call center i have more people calling on mondays than fridays. On some events, like a letter being sendt I have an effekt i think might be additive (i.e. 1% of people recieving letters make a call). So if letter A is recieved on a monday 1 i get different total number of calls than if it's recieved on a friday." CreationDate="2010-08-03T09:44:51.917" UserId="671" />
  <row Id="1097" PostId="1167" Text="This question at least involves a data problem, but this will be an ongoing cause for confusion wig potentially no right answer. http://meta.stats.stackexchange.com/questions/1/how-to-answer-r-questions" CreationDate="2010-08-03T09:46:56.497" UserId="5" />
  <row Id="1098" PostId="1167" Text="Feel free to bounce me to stackoverflow :-) Is that something that can be done automaticly?&#xA;" CreationDate="2010-08-03T09:49:56.897" UserId="671" />
  <row Id="1099" PostId="1168" Text="How can you subtract a vector of length 10 from a vector of length 15 (for instance)?  R will simply repeat the first 5 elements of the shorter vector, which would not be what you want. So you need to merge them and carry forward the aggregated values. " CreationDate="2010-08-03T09:54:48.413" UserId="5" />
  <row Id="1100" PostId="1167" Text="No, let's just leave it here (IMO). " CreationDate="2010-08-03T09:55:41.283" UserId="5" />
  <row Id="1101" PostId="1168" Text="Thank you, I think I am missing some basic insight. To me my revised solution above looks almost the same as my initial solution, but an A-B test showed calendar misalignment, which is consistent with what you made me aware of." CreationDate="2010-08-03T10:19:35.343" UserId="671" />
  <row Id="1102" PostId="1171" Text="Ah, yes, I knew that overlapping CI's didn't necessitate zero difference, but hadn't connected that to this example, where clearly the above procedure would always compute difference &quot;CI&quot; that includes zero whenever the individual CIs overlap. Thanks!" CreationDate="2010-08-03T12:39:56.020" UserId="364" />
  <row Id="1103" PostId="1061" Score="1" Text="@gd047. LaTeX code: `$\sigma_{\epsilon}$`" CreationDate="2010-08-03T13:23:24.293" UserId="159" />
  <row Id="1104" PostId="353" Text="I think that you're right, Rich.  I can't think of any other reasons.  Thanks!" CreationDate="2010-08-03T13:36:15.590" UserId="187" />
  <row Id="1105" PostId="1170" Text="&quot;sometimes&quot; is an understatement... the authors logic isn't often this direct but the stimulus / reward scenario is such that people will do this as a matter of conditioning" CreationDate="2010-08-03T13:52:52.740" UserId="601" />
  <row Id="1106" PostId="1175" Text="The data isn't necessary rates. It could be anything." CreationDate="2010-08-03T14:07:10.257" UserId="8" />
  <row Id="1107" PostId="1171" Score="2" Text="A useful fact given in these references is that non-overlapping 84% confidence intervals *do* approximate a 95% level test." CreationDate="2010-08-03T14:08:29.083" UserId="279" />
  <row Id="1108" PostId="1176" Text="Is it available as a video? It sounds great." CreationDate="2010-08-03T14:12:15.673" UserId="442" />
  <row Id="1109" PostId="1176" Text="I think the word is &quot;will be eventually&quot; -- keynotes got recorded.  " CreationDate="2010-08-03T14:18:07.673" UserId="334" />
  <row Id="1110" PostId="1173" Score="1" Text="Helping your field come to a consensus on just the se v. sd question would be a huge advance.  They mean completely different things.&#xA;" CreationDate="2010-08-03T14:41:26.260" UserId="601" />
  <row Id="1111" PostId="1170" Text="I don't researchers are being dishonest so much as acting out of ignorance.  They don't understand what statistics mean or what assumptions they require, but as you said they clearly understand the stimulus/reward: p &gt; 0.05 =&gt; no publication.  " CreationDate="2010-08-03T14:44:25.520" UserId="319" />
  <row Id="1112" PostId="1175" Text="Subscription link, unfortunately." CreationDate="2010-08-03T14:57:26.783" UserId="71" />
  <row Id="1113" PostId="1176" Text="this is easy in ggplot I think, i.e. http://had.co.nz/ggplot2/geom_jitter.html" CreationDate="2010-08-03T14:58:29.120" UserId="668" />
  <row Id="1115" PostId="1175" Text="... but here's the Wikipedia link on funnel plots: http://en.wikipedia.org/wiki/Funnel_plot" CreationDate="2010-08-03T14:59:02.240" UserId="71" />
  <row Id="1116" PostId="1180" Score="1" Text="And so if the mean is the same as the variance, could you conclude that the data was Poisson? Hardly! " CreationDate="2010-08-03T14:59:46.230" UserId="247" />
  <row Id="1118" PostId="1182" Text="I should add that my &quot;plot only the data and uncertainty&quot; recommendation should be qualified: when presenting data to an audience that has experience/expertise with the variable being plotted, plot only the data and uncertainty. When presenting data to a naieve audience and when zero is a meaningful data point, I'd first show the data extending to zero so that the audience can get oriented to the scale, then zoom in to show just the data and uncertainty." CreationDate="2010-08-03T15:10:49.760" UserId="364" />
  <row Id="1119" PostId="1176" Text="`jitter` is also in plain R." CreationDate="2010-08-03T15:14:10.347" UserId="88" />
  <row Id="1120" PostId="1176" Text="Well, I didn't mention a need to jitter. Frank's example had relatively few points.  I simply mentions alpha blending because it may make mixing the dots and the bars easier." CreationDate="2010-08-03T15:18:15.560" UserId="334" />
  <row Id="1121" PostId="1173" Text="I agree - se is usually chosen because it gives a smaller region!" CreationDate="2010-08-03T15:20:39.920" UserId="8" />
  <row Id="1122" PostId="1182" Text="@Mike: since you've went to trouble of writing R code, could you include a jpeg image of the final plot. I find just uploading the image to http://img84.imageshack.us/ and linking to it is fairly easy. Oh thanks for the answer :)" CreationDate="2010-08-03T15:26:43.643" UserId="8" />
  <row Id="1123" PostId="1179" Text="@Chris: I've added a small section on why I dislike these plots." CreationDate="2010-08-03T15:34:58.710" UserId="8" />
  <row Id="1124" PostId="1121" Text="This can all be implemented in Max/MSP/Jitter with the [peak] and [trough] objects for the first example and with [jit.3m] for the second example. " CreationDate="2010-08-03T15:40:01.393" UserId="162" />
  <row Id="1126" PostId="1182" Text="@csgillespie: done." CreationDate="2010-08-03T15:46:09.180" UserId="364" />
  <row Id="1127" PostId="1153" Text="Really thank you for your help, I really appreciate. I'm quite busy at work now, but I'm going to test an approach like yours as soon as possible, and I will come back with my final considerations about this issue. One only thought: in your function, from what I see, I have to manually specify the frequency of the time series (when constructing it), and the seasonality component is considered only when the frequency is greater than 1. Is there a robust way to deal with this automatically?" CreationDate="2010-08-03T15:59:18.127" UserId="667" />
  <row Id="1128" PostId="1180" Text="True. Necessary but not sufficient." CreationDate="2010-08-03T16:44:18.040" UserId="319" />
  <row Id="1129" PostId="1173" Text="Maybe some more informative title?" CreationDate="2010-08-03T18:36:41.790" UserId="88" />
  <row Id="1130" PostId="1182" Text="I've found that it's easier to read a plot like this with `geom_ribbon()` indicating the error. If you don't like producing apparent estimates for regions between 1 and 2, at least reduce the width of the error bar." CreationDate="2010-08-03T19:20:49.397" UserId="287" />
  <row Id="1131" PostId="752" Score="1" Text="I don't get what is so deep in that one, is it only playing with words ? " CreationDate="2010-08-03T19:29:45.997" UserId="223" />
  <row Id="1132" PostId="1187" Text="Thank you, I will check these out." CreationDate="2010-08-03T19:40:35.717" UserId="573" />
  <row Id="1133" PostId="595" Text="Agreed; wavelets are excellent for picking out non-stationary behavior in high amounts of noise.  You do have to be careful with the DWT, though.  It's not rotation-invariant (although there are modifications of the DWT that are, see e.g. Percival and Walden 2000), so you can lose sharp transients depending on the starting point for your data.  Also, most implementations of the DWT do implicit circularization of the data, so you'd still need to control for that." CreationDate="2010-08-03T19:57:01.273" UserId="61" />
  <row Id="1134" PostId="1193" Text="Edited to disambiguate 'above.' " CreationDate="2010-08-03T20:10:46.107" UserId="455" />
  <row Id="1135" PostId="1194" Score="2" Text="This question is proposed to be closed. See: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed I see that it has 2 votes. Could the up-voters or the OP comment on why they would like to see the question stay open at the meta thread?" CreationDate="2010-08-03T20:33:45.763" />
  <row Id="1136" PostId="1194" Score="2" Text="Rather than saying &quot;this should be closed. Someone should defend it&quot; how about starting with explaining why you want it closed. Too vague? Then ask for clarification. This seems a reasonable question to me. The asker presents a paper and asks about the difference is between predictive and explanatory statistics. The only change I would make to the question is to clarify exactly the question thus making it easier to vote. " CreationDate="2010-08-03T20:39:24.053" UserId="29" />
  <row Id="1137" PostId="1194" Score="2" Text="I have already offered a reason on the meta thread. I feel that 'meta discussions' about the question would clutter up this particular page." CreationDate="2010-08-03T20:41:09.837" />
  <row Id="1138" PostId="1160" Score="1" Text="I wish we could move a question to Stack Overflow the way you can move a question from Super User..." CreationDate="2010-08-03T20:42:08.510" UserId="29" />
  <row Id="1139" PostId="1194" Score="1" Text="@Srikant @JD I'll beef up the question. Thanks for the feedback. I do think that this is a topic that merits discussion." CreationDate="2010-08-03T20:44:41.797" UserId="11" />
  <row Id="1140" PostId="1194" Score="1" Text="Your question would good for the community if, instead of telling us your life,  you could define what is (according to you) predictive model and explanatory model. I think nice debates start with clear definitions..." CreationDate="2010-08-03T20:49:23.517" UserId="223" />
  <row Id="1141" PostId="1194" Score="1" Text="Could you pose a question here?  Is the question whether the paper is right?  " CreationDate="2010-08-03T20:52:48.987" UserId="5" />
  <row Id="1142" PostId="595" Text="If my memory is good, package wavethresh contains translation invariant denoising (my reference was Coifman 1995) (Note that you talked about rotation, arn't we talking about temporal signals?). " CreationDate="2010-08-03T20:53:23.853" UserId="223" />
  <row Id="1143" PostId="1194" Text="@srikant you seem to not understand the nature of comments under the questions. They are, by definition, meta. They are not answers. They are not questions. They are meta. Having a convention where comments become a pointer to meta conversations in some other location is wasteful and silly. " CreationDate="2010-08-03T20:59:14.013" UserId="29" />
  <row Id="1144" PostId="1194" Text="I've made the question more specific" CreationDate="2010-08-03T21:18:07.387" UserId="11" />
  <row Id="1145" PostId="1194" Score="1" Text="@JD Perhaps. But, in order to get some control on the process of closing questions there is a meta thread devoted to this issue. If I do not make a mention of the fact that this question is proposed to be closed then the community will not get a chance to have a say whether it should stay open or not. One other issue is consider a person who stumbles on this question in the distant future. All this discussion about whether and why we should keep the question open is a bit irrelevant. I feel that comments should be used to clarify the question not to debate its merits." CreationDate="2010-08-03T21:19:05.843" />
  <row Id="1146" PostId="1194" Text="@srikant. That's articulately stated and clear. This discussion probably belongs in the Meta area, however, as it is not specific to the question above. :) yeah, ok, that was a bit of a jackass comment... I could not resist it!   You make a good point. I think we can agree that @wahalulu needed to clarify his question. I think he's moving in the right direction. " CreationDate="2010-08-03T21:46:16.827" UserId="29" />
  <row Id="1147" PostId="1194" Score="3" Text="Nevertheless, it should be community wiki." CreationDate="2010-08-03T22:17:23.380" UserId="88" />
  <row Id="1148" PostId="1031" Text="Since I am not a mathematician nor a statistician, I would like to restate what you were saying to make sure I did not mis-understand. &#xA;&#xA;So, you are saying that taking ds^2 (twice the KL) would have a similar meaning as R^2 (in a regression model) for a general distribution. And that this could actually be used to quantify  distances geometrically? Does ds^2 have a name so I can do more reading about this. Is there a paper that directly describes this metric and shows applications and examples? " CreationDate="2010-08-03T22:22:05.357" UserId="608" />
  <row Id="1149" PostId="608" Score="1" Text="Wow, this is something that will be even harder do obtain I'm afraid; my general point in the whole discussion is that the convergence of those theorems is too weak so the difference may emerge from random fluctuations. (And that it is not working for machine learning, but I hope this is obvious.)" CreationDate="2010-08-03T23:19:47.353" UserId="88" />
  <row Id="1150" PostId="1153" Text="Yes, I have assumed the frequency is known and specified. There are methods to estimate the frequency automatically, but that would complicate the function considerably. If you need to estimate the frequency, try asking a separate question about it -- and I'll probably provide an answer! But it needs more space than I have available in a comment." CreationDate="2010-08-03T23:40:47.553" UserId="159" />
  <row Id="1151" PostId="608" Text="That is interesting.  If the convergence is so weak I wonder why people even brought it up in the other question about AIC/BIC, or how they figured out that the cross validation and IC methods were actually convergent; I'm not really involved in machine learning, so the last bit that you hope is obvious isn't obvious to me, could you expand on that point?  Besides, the entire thing may be moot, as I say below - a correlational approach to the problem doesn't look like it will work (at least not while sample sizes are constant)." CreationDate="2010-08-04T00:05:11.603" UserId="196" />
  <row Id="1152" PostId="1153" Text="Thank you, I'll post a different question." CreationDate="2010-08-04T00:21:03.897" UserId="667" />
  <row Id="1153" PostId="109" Text="I've been watching this question for a while waiting for clarification from the question asker, but none has been forthcoming.  I've nominated this question to be closed: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed" CreationDate="2010-08-04T00:38:26.360" UserId="196" />
  <row Id="1154" PostId="1185" Text="But this raises an interesting pedagogical problem, at least in Psychology, because as far as I know most introductory statistics books being used in my field do not really discuss robust measures except as an aside." CreationDate="2010-08-04T00:43:19.297" UserId="196" />
  <row Id="1155" PostId="1205" Text="What is N? What is the nature of the weighting scheme? Are the weights known?" CreationDate="2010-08-04T01:25:01.977" />
  <row Id="1156" PostId="1182" Text="@JoFrwld: I like ribbons too, though I tend to reserve them for cases where the x-axis variable it truly numeric; my version of the &quot;don't draw lines unless the x-axis variable is numeric&quot; rule that I profess violating in my answer above :Op" CreationDate="2010-08-04T01:40:54.393" UserId="364" />
  <row Id="1157" PostId="1206" Text="@Rob Why/how would an important explanatory variable reduce predictive accuracy?" CreationDate="2010-08-04T02:34:10.030" />
  <row Id="1158" PostId="1210" Text="Interesting.  Did you ever come across an R implementation of this ?" CreationDate="2010-08-04T03:16:52.187" UserId="253" />
  <row Id="1159" PostId="1206" Score="1" Text="@Srikant. This can happen when the explanatory variable has a weak but significant relationship with the response variable. Then the coefficient can be statistically significant but hard to estimate. Consequently, the MSE of predictions can increase when the variable is included compared to when it is omitted. (The bias is reduced with its inclusion but the variance is increased.)" CreationDate="2010-08-04T05:15:22.890" UserId="159" />
  <row Id="1160" PostId="1181" Text="+1 thanks for the thorough example!" CreationDate="2010-08-04T06:49:26.163" UserId="634" />
  <row Id="1161" PostId="1178" Text="+1 thank you. often I get some &quot;weired&quot; results, for example, a normal distribution gets a higher p-value then a poisson one, where lambda is relatively small (so by looks only the normal and poisson are not similiar at all)" CreationDate="2010-08-04T06:51:10.857" UserId="634" />
  <row Id="1162" PostId="1212" Text="+1 thanks for the suggestions!" CreationDate="2010-08-04T07:04:35.153" UserId="634" />
  <row Id="1163" PostId="562" Text="i added what i think is a pretty good source (unfortunately a Textbook) w/ annotation, in light of your comment/question below my answer. I edited my original answer, so it appears at the end." CreationDate="2010-08-04T07:07:48.703" UserId="438" />
  <row Id="1164" PostId="1216" Text="ok, let's size I need to calculate for size 45. Every delivery is about once every 3 months.&#xA;I'm using excel here. =POISSON(4;98;FALSE). Ressult: 0... Could you give me some more clues?" CreationDate="2010-08-04T07:16:39.543" UserId="698" />
  <row Id="1165" PostId="1216" Text="If the deliveries are every 3 months, and the data in the question are annual, then the average sales between deliveries is 95/4. So you want y such that 1-POISSON(y,95/4,TRUE) is smaller than 0.05. Choosing y=32 will give the out-of-stock probability of 4.1%." CreationDate="2010-08-04T08:12:41.370" UserId="159" />
  <row Id="1166" PostId="752" Text="I like to think of it as the statisticians equivalent of &quot;guns don't kill people, people kill people&quot; not very deep, but important to realise from time to time" CreationDate="2010-08-04T09:17:53.497" UserId="127" />
  <row Id="1167" PostId="1222" Text="Good point, still the main differences are somewhere else;  first, statistics is about fitting a model to the data one has, ML is about fitting a model to data one will have; second, statistics ASSUME that a process one observes is fully driven by some embarassingly trivial &quot;hidden&quot; model that they want to excavate, while ML TRIES to make some complex enough to be problem-independent model behave like reality.   &#xA;" CreationDate="2010-08-04T09:28:38.720" UserId="88" />
  <row Id="1168" PostId="934" Text="@John Still, this is mixing aims with reasons. To great extent you can explain each algorithm in terms of minimizing something and call this something &quot;loss&quot;. kNN wasn't invented in such a way: Guys, I have thought of loss like this, let's optimize it and see what will happen!; rather Guys, let's say that decision is more less continuous over the feature space, then if we would have a good similarity measure... and so on." CreationDate="2010-08-04T09:38:57.930" UserId="88" />
  <row Id="1169" PostId="1176" Text="@Dirk I was commenting Mike's comment about ggplot. As a general comment I also used to like these seas of dots for the sake that nothing will better represent the distribution -- until we've made figures that were crushing Acrobat Reader ;-) . " CreationDate="2010-08-04T09:54:30.090" UserId="88" />
  <row Id="1170" PostId="1222" Text="@mbq. That's a rather harsh caricature of statistics. I've worked in five university statistics departments and I don't think I've met anybody who would think of statistics like that." CreationDate="2010-08-04T09:59:17.247" UserId="159" />
  <row Id="1172" PostId="1223" Score="1" Text="I don't think this is a duplicate because of the nature of the data. The problem discussed on the other question concerned regularly observed time series with occasional outliers (at least that's how I interpreted it). The nature of tick-by-tick data would lead to different solutions due to the exchange opening effect." CreationDate="2010-08-04T10:09:10.590" UserId="159" />
  <row Id="1173" PostId="1223" Text="possible duplicate of [Simple algorithm for online outlier detection of a generic time series](http://stats.stackexchange.com/questions/1142/simple-algorithm-for-online-outlier-detection-of-a-generic-time-series) This question is proposed to be closed as a duplicate. Could you please let us know at the meta thread if and how your context is different from the question I linked?" CreationDate="2010-08-04T10:10:59.450" />
  <row Id="1174" PostId="1223" Text="@Rob But the exchange opening effect only determines when you have to run the algorithm. The fundamental issue remains the same. Even in network data you have the 'office opening effect' where traffic peaks as soon as an office opens. At the very least, the OP should link to that question, scan the answers there and explain why the solutions there do not work so that a suitable answer can be posted for this question." CreationDate="2010-08-04T10:12:49.820" />
  <row Id="1176" PostId="1224" Score="1" Text="I think we need a bit more information. What are you comparing? What sort of data do you have? Is it a single outcome you a measuring per centre?" CreationDate="2010-08-04T10:41:18.757" UserId="8" />
  <row Id="1177" PostId="1223" Score="1" Text="I agree with @Rob. This kind of data can pose unique challanges, so this is not a duplicate. " CreationDate="2010-08-04T10:46:15.037" UserId="5" />
  <row Id="1178" PostId="1222" Text="@Rob Caricature? I think this is what makes statistics beautiful! You assume all those gaussians and linearities and it just works -- and there is a reason for it which is called Taylor expansion. World is hell of a complex, but in linear approx. (which is often ninety-something% of complexity) embarrassingly trivial. ML (and nonparametric statistics) comes in in these few per cent of situations where some more subtle approach is needed. This is just no free lunch -- if you want theorems, you need assumptions; if you don't want assumptions, you need approximate methods." CreationDate="2010-08-04T10:54:35.570" UserId="88" />
  <row Id="1179" PostId="1223" Text="This question might eventually get served better here due to its domain-specificity: http://area51.stackexchange.com/proposals/117/quantitative-finance" CreationDate="2010-08-04T11:09:19.307" UserId="5" />
  <row Id="1180" PostId="1222" Text="@mbq. Fair enough. I must have misinterpreted your comment." CreationDate="2010-08-04T11:21:07.587" UserId="159" />
  <row Id="1181" PostId="1205" Text="see also this related question: http://stats.stackexchange.com/questions/856/fishers-exact-test-with-weights" CreationDate="2010-08-04T11:22:20.907" UserId="442" />
  <row Id="1182" PostId="1126" Text="I don't want to pick just one right answer, so I've made this a community wiki page, based on two suggestions above. Thanks for all the ideas!" CreationDate="2010-08-04T11:41:05.200" UserId="660" />
  <row Id="1183" PostId="718" Text="+1 for David Howell, because his Intro to Statistics for Psychologists is the best you can get as an undergrad!!" CreationDate="2010-08-04T12:01:14.297" UserId="442" />
  <row Id="1185" PostId="1223" Score="1" Text="I think it belongs here. The question is about analyzing irregularly spaced, very noisy time series.&#xA;&#xA;Have you had a look at &quot;An Introduction to High-Frequency Finance&quot; by Dacorogna, Olsen and a bunch of others? Or the papers by the same authors?" CreationDate="2010-08-04T12:30:40.983" UserId="247" />
  <row Id="1186" PostId="1228" Score="1" Text="I took the liberty of trying to fix the grammar. I hope I understood the question correctly." CreationDate="2010-08-04T12:35:14.983" UserId="159" />
  <row Id="1187" PostId="1183" Score="1" Text="I'm not sure boxplots or vioplots would be suitable with such a small sample size (n = 6)" CreationDate="2010-08-04T12:39:43.527" UserId="8" />
  <row Id="1188" PostId="1099" Text="I think asking another question separately is much better than updating your previous question. " CreationDate="2010-08-04T12:42:04.597" />
  <row Id="1189" PostId="1223" Text="I saw the other answer and followed Rob's reasoning. I amended my question to address differences I see." CreationDate="2010-08-04T13:18:51.890" UserId="127" />
  <row Id="1190" PostId="1170" Score="4" Text="You must also present something that those &quot;in power&quot; (decision makers, supervisors, reviewers) understand. Therefore it has to be in the common language which evolves quite slowly, as those people tend to be older and more resistant to change, largely as it may invalidate their careers hitherto!" CreationDate="2010-08-04T13:29:34.113" UserId="229" />
  <row Id="1191" PostId="1164" Score="3" Text="The Black Swann by Nassim Nicholas Taleb explains why simple models have been used in the financial world and the dangers this has led to. A particular fault is equating very-low probabilities with zero and blindly applying the normal distribution in risk management!" CreationDate="2010-08-04T13:38:31.913" UserId="229" />
  <row Id="1192" PostId="1223" Text="@jilles I do not see any edits to your question. Did you save your edits? It may help if you post a link to that question as well and indicate the changes to your question by something like 'Edit'." CreationDate="2010-08-04T13:39:08.930" />
  <row Id="1193" PostId="1223" Text="I have the Olsen book and it doesn't address the exchange open/close question. " CreationDate="2010-08-04T13:39:26.560" UserId="5" />
  <row Id="1194" PostId="1223" Text="@Srikant: done, @PeterR: do you know of any specific paper of those authors that addresses this question?" CreationDate="2010-08-04T13:46:53.597" UserId="127" />
  <row Id="1195" PostId="1223" Text="I wish there is a way to undo my close vote! I think it is clear now that it is not a duplicate. " CreationDate="2010-08-04T13:49:44.400" />
  <row Id="1196" PostId="1234" Text="I've tried something like this, but this method is not very good at dealing with abrupt changes in the volatility. This leads to underfiltering in quiet periods and overfiltering during more busy times." CreationDate="2010-08-04T13:53:00.447" UserId="127" />
  <row Id="1197" PostId="1183" Text="Right, I admit I haven't read the question carefully enough, so it was rather a general idea; nevertheless I think that 6 points is minimal but enough for a boxplot. I have made some experiments and they were meaningful. On the other hand, obviously boxplot does not indicate the number of observations (which is an important bit of information here), so I would rather use a combination of it and points. " CreationDate="2010-08-04T13:55:26.417" UserId="88" />
  <row Id="1199" PostId="1235" Text="By using only the returns you become very vulnerable to ladders (i.e. a sequence of prices that climbs or drops away from the norm, where each individual return is acceptable, but as a group they represent an outlier). Ideally you'd use both the return and the absolute level." CreationDate="2010-08-04T14:01:02.803" UserId="127" />
  <row Id="1200" PostId="1224" Text="Agree; what is the null hypothesis? " CreationDate="2010-08-04T14:01:59.993" UserId="88" />
  <row Id="1201" PostId="1206" Text="First paragraph is a very, very good point. Still sometimes is even worse; here PMID: 18052912 is a great example that sometimes a better model can be made on the noise part of the set than on a true one -- it is obvious that one can do a good model on random data, but this is a bit shocking." CreationDate="2010-08-04T14:20:30.940" UserId="88" />
  <row Id="1202" PostId="825" Score="1" Text="Vote to keep open; very relevant to statisticians because the ways in which our problems can or can not be broken down into parallel streams is of relevance to the question being asked." CreationDate="2010-08-04T14:40:38.203" UserId="196" />
  <row Id="1203" PostId="830" Score="1" Text="But, of course, doSMP is exactly what the OP would be looking for if they were working in a Windows environment." CreationDate="2010-08-04T14:44:16.370" UserId="196" />
  <row Id="1204" PostId="1238" Text="Spam filter prevented my posting the TestU01 link:&#xA;&#xA;http://www.iro.umontreal.ca/~simardr/testu01/tu01.html" CreationDate="2010-08-04T14:48:00.137" UserId="729" />
  <row Id="1205" PostId="1238" Text="Fixed the link to TestU01" CreationDate="2010-08-04T14:49:37.450" />
  <row Id="1206" PostId="1214" Text="Thank you. Again, I will try this approach as soon as possible and will write here the final results." CreationDate="2010-08-04T14:49:55.500" UserId="667" />
  <row Id="1207" PostId="1239" Score="1" Text="Belongs on http://stackoverflow.com" CreationDate="2010-08-04T14:57:39.883" />
  <row Id="1208" PostId="1224" Text="Comparing a biomarker using hazard ratio as a measure of effect (will be adjusing for other covariates also)" CreationDate="2010-08-04T15:18:46.687" UserId="712" />
  <row Id="1209" PostId="1241" Text="What is a ROC curve? Could you please provide a link?" CreationDate="2010-08-04T15:19:32.367" />
  <row Id="1210" PostId="1243" Text="Your last point about regression would only be true if X is random. If X is fixed then Y would also be a normal. no?" CreationDate="2010-08-04T15:28:24.657" />
  <row Id="1211" PostId="1210" Text="No, not directly.  However, R will give you everything you need to do this--such as: the observed counts, the expected values, and the residuals for each cell.&#xA;&#xA;x &lt;- matrix(c(12, 5, 7, 7), ncol = 2)&#xA;chisq.test(x)$expected&#xA;chisq.test(x)$observed&#xA;chisq.test(x)$residuals&#xA;" CreationDate="2010-08-04T15:55:25.703" UserId="485" />
  <row Id="1212" PostId="1245" Text="+1 yes, nothing is perfect. Tickdata.com (whose paper is mentioned) also includes outliers and they also strip out too much good data (when compared with another source). Olsen's data is close to being terrible, and I generally just indicative. There's a reason that banks pay big operations teams to work on this. " CreationDate="2010-08-04T16:10:44.320" UserId="5" />
  <row Id="1213" PostId="1241" Text="http://en.wikipedia.org/wiki/Receiver_operating_characteristic" CreationDate="2010-08-04T16:54:06.347" UserId="88" />
  <row Id="1214" PostId="1210" Text="I'll give you the tick, since this should be useful for my research life. However, this approach is applicable to an i x j matrix. However, my question involves an i x j x k matrix," CreationDate="2010-08-04T17:02:58.283" UserId="287" />
  <row Id="1215" PostId="1183" Text="With 6 points - scatter plot is probably best (maybe with adding a red dot for the mean)" CreationDate="2010-08-04T17:20:09.427" UserId="253" />
  <row Id="1216" PostId="1031" Text="I think you are far from understanding the point, and I am not sure you should try to go further now. If you are motivated, you can read the paper from Bradley Efron I mentionned or that paper from Amari http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1176345779. " CreationDate="2010-08-04T17:25:16.690" UserId="223" />
  <row Id="1217" PostId="1183" Score="1" Text="I generally use boxplots with superimposed points, I find it very &quot;visual&quot;. Violin plots, instead, are a bit hard to understand in my opinion." CreationDate="2010-08-04T17:30:20.317" UserId="582" />
  <row Id="1219" PostId="1254" Score="1" Text="Something similar to that but includes more information rather than just the name of the test. We have some of these charts in urban and transportation modelling. They'll show a large table where they specify tests per type of problem. They also list caveats, expected time/duration, input and outputs, etc" CreationDate="2010-08-04T17:35:31.277" UserId="59" />
  <row Id="1220" PostId="1255" Text="To compute similarity, I need to count the number of times two items were rated together by the same person.  That count becomes my numerator.  The formula I'm using to calculate similarity between items A &amp; B is &#xA;&#xA;    AB / A + B - AB&#xA;&#xA;Where AB is the number of times items A and B were rated by the same person, and A and B are the number of times each item was rated overall.&#xA;&#xA;In order to perform that count, I need to generate all of the itemA,itemB pairs in my mapper, and then sum the counts in my reducer.&#xA;&#xA;The items are rated in a binary fashion-if a (userid,itemid) exists, then 1." CreationDate="2010-08-04T17:43:38.923" UserId="738" />
  <row Id="1221" PostId="1249" Score="2" Text="This question is for Mathoverflow: http://mathoverflow.net/. Seems to be a good question anyway. I remember I saw a generalization of the results in Baraud's paper done by Rozenholc an other collaborators but I can't find the paper anymore... " CreationDate="2010-08-04T17:44:19.610" UserId="223" />
  <row Id="1222" PostId="1210" Score="1" Text="Chi-square partitioning is extensible to multi-way contingency tables.  Here's the article that Agresti cites in his book, in fact...&#xA;&#xA;H. O. Lancaster (1951) &quot;Complex Contingency Tables Treated by the Partition of χ2&quot; Journal of the Royal Statistical Society. Series B (Methodological), Vol. 13, No. 2 &#xA;&#xA;" CreationDate="2010-08-04T18:13:20.323" UserId="485" />
  <row Id="1223" PostId="1214" Text="Your idea is quite good, but in my case, it fails to detect the periodicity of a really simple (and not so noisy) time series like http://dl.dropbox.com/u/540394/chart.png. With my &quot;empirical&quot; approach (based on the autocorrelation), the simple algorithm I wrote returns an exact period of 1008 (having a sample every 10 minute, this means 1008/24/6 = 7, so a weekly periodicity). My main problems are:&#xA;1) It's too slow to converge (it requires a lot of historical data) and I need a reactive, online approach;&#xA;2) It's inefficient as hell from a memory usage point of view;&#xA;3) It's not robust at all;" CreationDate="2010-08-04T18:14:17.607" UserId="667" />
  <row Id="1224" PostId="1257" Text="PS: This was posted on mathoverflow.net sometime back. See: http://mathoverflow.net/questions/18302/likelihood-function-for-sequential-random-variables" CreationDate="2010-08-04T18:29:49.823" />
  <row Id="1226" PostId="1102" Text="Didn't see this for a while. Users don't get notifications about new answers to questions they've contributed answers to themselves. If you leave a followup as a comment instead, you might get an additional response sooner." CreationDate="2010-08-04T19:18:37.620" UserId="287" />
  <row Id="1227" PostId="1102" Text="Actually, `P(y | grassland)` would be `exp(-2.8) / (1 + exp(-2.8))` A point in the forest would be `exp(-2.8 - 3) / (1 + exp(-2.8  - 3))`&#xA;&#xA;However, you could estimate those probabilities without doing a logistic regression. What would be more interesting would be to look at `exp(-3)`, or `1/exp(-3)`. This says that an animal is 20x more likely on grassland than forest, and if the p-value is &lt;0.05, then that difference is significant.&#xA;&#xA;" CreationDate="2010-08-04T19:23:49.840" UserId="287" />
  <row Id="1228" PostId="1249" Text="Do you want a lower or a upper bound? The first sentence says upper but the last sentence mentions lower." CreationDate="2010-08-04T19:24:01.340" />
  <row Id="1229" PostId="1256" Text="I assume that 'without multicore' means without parallelism at all?" CreationDate="2010-08-04T19:37:59.470" UserId="88" />
  <row Id="1230" PostId="1249" Text="@Srikant: you should read the mentionned paper by Barraud which is great to understand the whole thing... heuristically lower bounds are larger (testing is more difficult) when distributions (from the null and alternative) are close. Hence the upper bound asked by mkolar is related to the chi square distance between two distributions (the least favorable in the alternative and the null) if this distance is small the minimum error will be large..." CreationDate="2010-08-04T19:40:09.047" UserId="223" />
  <row Id="1231" PostId="1263" Score="1" Text="And actually the sapply, could be coded into a multi core framework..." CreationDate="2010-08-04T19:49:36.047" UserId="253" />
  <row Id="1232" PostId="1160" Text="I moved it manually to stackoverflow. JD - I agree with you." CreationDate="2010-08-04T20:14:47.510" UserId="253" />
  <row Id="1233" PostId="1263" Text="Agree; still OP wrote that this is not an option." CreationDate="2010-08-04T20:33:25.307" UserId="88" />
  <row Id="1235" PostId="1268" Text="Good question! I think you can improve the title, there could be something like &quot;missing data&quot; somewhere or &quot;times series of different length&quot;.  " CreationDate="2010-08-04T21:19:03.420" UserId="223" />
  <row Id="1236" PostId="1269" Text="+1 replication/resampling are definitely better than zero-padding.. still I'll wait and see if there are any other ideas out there :)" CreationDate="2010-08-04T21:43:04.953" UserId="170" />
  <row Id="1237" PostId="1268" Score="1" Text="I wouldn't call it &quot;missing data&quot;, perhaps &quot;SVD dimensionality reduction for time series of different length&quot;?" CreationDate="2010-08-04T21:45:44.267" UserId="170" />
  <row Id="1238" PostId="1261" Text="Do i, j, k have to be integers? In general you have an unidentifiable model (that is the solution is not unique) because if you let p2=p^i, then p^j=p2^j2, where j2=j/i." CreationDate="2010-08-04T21:45:48.200" UserId="279" />
  <row Id="1239" PostId="1263" Text="#Actually sapply is about 3 times slower than my baseline method&#xA;library(rbenchmark);&#xA;baseline &lt;- function()&#xA;{;&#xA;	ncol &lt;- 100;&#xA;	nrow &lt;- 100;&#xA;	x &lt;- matrix(rnorm(ncol*nrow),nrow,ncol);&#xA;	y &lt;- matrix(rnorm(ncol*nrow),nrow,ncol);&#xA;	return(diag(cor(t(x),t(y))));&#xA;};&#xA;sapply.method &lt;- function()&#xA;{;&#xA;	ncol &lt;- 100;&#xA;	nrow &lt;- 100;&#xA;	x &lt;- matrix(rnorm(ncol*nrow),nrow,ncol);&#xA;	y &lt;- matrix(rnorm(ncol*nrow),nrow,ncol);&#xA;	return(sapply(1:100,function(i) cor(x[i,],y[i,])));&#xA;};&#xA;benchmark(baseline=baseline(),sapply.method(),replications=10);" CreationDate="2010-08-04T21:49:38.477" UserId="196" />
  <row Id="1240" PostId="1263" Text="(your mileage may vary on by what factor it is slower)" CreationDate="2010-08-04T22:10:15.153" UserId="196" />
  <row Id="1241" PostId="1265" Text="Simulation supports your answer, and even with a transpose operation added to handle cases with more items than subjects, sapply beats diag(cor()), but only when there are notable asymmetries between the N of items and the N of subjects." CreationDate="2010-08-04T22:21:18.920" UserId="196" />
  <row Id="1242" PostId="1243" Text="Yes, this is true, but for general regression problems (as opposed to anova or designed problems), X really isn't fixed but are observations from the underlying process. However, for the Poisson case, the point still holds, since mixtures of Poissons aren't necessarily Poisson. " CreationDate="2010-08-04T22:23:54.907" UserId="732" />
  <row Id="1243" PostId="1274" Score="4" Text="Correlated with what? Each obs to the one before it?" CreationDate="2010-08-04T22:26:35.010" UserId="287" />
  <row Id="1244" PostId="1263" Score="1" Text="For 100x100 matrix it is obvious; I believe that the problem requires a lot larger matrices and there my method will be faster. Still, I have written &quot;your code is equivalent to:&quot; instead of &quot;use:&quot; having that in mind -- this is how possible C/Fortran chunk should work. " CreationDate="2010-08-04T22:29:35.553" UserId="88" />
  <row Id="1245" PostId="1256" Text="That is correct." CreationDate="2010-08-04T22:32:32.473" UserId="196" />
  <row Id="1246" PostId="1263" Text="Also, from no-multicore requirement one can conclude that this is indeed memory-bounded, so reducing memory complexity from $N^2$ to $N$ is also something worth playing." CreationDate="2010-08-04T22:35:12.953" UserId="88" />
  <row Id="1247" PostId="1194" Text="I've got this 3 votes as an agreement for making it CV." CreationDate="2010-08-04T22:37:49.207" UserId="88" />
  <row Id="1248" PostId="1261" Text="I think to demonstrate non-identifiability you have to show that the likelihood is invariant to a transformation of parameters. I am not sure if what you did is sufficient to demonstrate lack of  identification." CreationDate="2010-08-04T22:50:19.640" />
  <row Id="1249" PostId="1239" Text="Question is proposed to be closed. See: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed" CreationDate="2010-08-04T22:59:28.953" />
  <row Id="1250" PostId="1170" Score="2" Text="Good point. &quot;I understand p-values.  Just give me a p-value.&quot;  Ironically, they probably do *not* understand p-values, but that's another matter." CreationDate="2010-08-05T00:22:37.490" UserId="319" />
  <row Id="1251" PostId="1277" Text="Will the usual dw test be appropriate?" CreationDate="2010-08-05T00:33:10.403" UserId="273" />
  <row Id="1252" PostId="752" Text="So it's an inane platitude used for quibbling over semantics? &quot;Cigarettes don't cause cancer; people cause cancer.&quot; &quot;Landmines don't maim people; people maim people.&quot;" CreationDate="2010-08-05T00:36:02.957" UserId="753" />
  <row Id="1253" PostId="823" Score="2" Text="That's because consensus is a practical means of establishing the validity of a scientific position. Sound science cannot be conducted by a single person. It requires peer review. Having achieved a popular consensus implies that the science in question has passed peer review. Whereas a theory or position held by a lone individual and opposed by the rest of the scientific community is likely to have failed the process of peer review.&#xA;&#xA;If you're not an expert in field X, then statistically you're better off following popular consensus within field X." CreationDate="2010-08-05T00:50:10.697" UserId="753" />
  <row Id="1254" PostId="1277" Score="1" Text="The dw test is designed for checking autocorrelation in the residuals from a regression. There is no regression here, so it is not appropriate." CreationDate="2010-08-05T00:57:48.733" UserId="159" />
  <row Id="1255" PostId="823" Text="To elaborate: if you're a politician who has no background in field X, then it is far better that you simply consult the consensus of experts in that field of research than to misinterpret the data first-hand.&#xA;&#xA;What is problematic is when lay persons disregard overwhelming scientific consensus for data misrepresented to them by a fringe minority--especially when these lay persons are in charge of policy decisions. It's much easier to mislead a handful of politicians than thousands of expert researchers..." CreationDate="2010-08-05T01:02:53.433" UserId="753" />
  <row Id="1256" PostId="1273" Text="this seems interesting, but sadly I can't find even a portion of the book from google books, will have a look on it, thanks!" CreationDate="2010-08-05T01:14:49.277" UserId="588" />
  <row Id="1257" PostId="1228" Text="thanks! i submitted this question in a rush and so maybe i missed the grammar...." CreationDate="2010-08-05T01:15:13.847" UserId="588" />
  <row Id="1258" PostId="1099" Text="sure, thanks for the comments!" CreationDate="2010-08-05T01:16:54.873" UserId="588" />
  <row Id="1259" PostId="1099" Score="1" Text="You should consider up voting the accepted answer. After all you found that answer useful." CreationDate="2010-08-05T01:33:43.950" />
  <row Id="1260" PostId="1271" Text="Thank you for the lead Brett!  I wonder if someone got to implement it by now in R (I'd guess not). " CreationDate="2010-08-05T01:43:35.037" UserId="253" />
  <row Id="1261" PostId="1261" Text="@Srikant Vadali I just did. Replace p^i with p, j with j/i and k with k/i and you get the exact same probabilities as an output with a different set of parameters (unless i=1 - with that constraint the model is identifiable)." CreationDate="2010-08-05T03:08:28.630" UserId="279" />
  <row Id="1262" PostId="1278" Text="Can you have a control group (ward)?" CreationDate="2010-08-05T03:10:53.713" UserId="279" />
  <row Id="1263" PostId="1271" Text="Right. R has lots of routines for assisting with bootstrap and other randomization methods, but I don't know that you'll find anything specific to this problem." CreationDate="2010-08-05T03:15:46.050" UserId="485" />
  <row Id="1264" PostId="1274" Text="I'd hope not - the correlation at lag one is very close to 0." CreationDate="2010-08-05T03:37:28.007" UserId="196" />
  <row Id="1265" PostId="1214" Text="Try v2........." CreationDate="2010-08-05T04:04:43.560" UserId="159" />
  <row Id="1266" PostId="1278" Text="no, the ward itself will be its control" CreationDate="2010-08-05T04:34:52.127" UserId="588" />
  <row Id="1267" PostId="1277" Text="I agree. Books mislead again. http://sas-and-r.blogspot.com/2010/07/example-81-digits-of-pi.html" CreationDate="2010-08-05T04:47:04.203" UserId="273" />
  <row Id="1268" PostId="1274" Text="Or in Prof Wecker's words: &quot;The result was a = .01, which implies little or no serial dependence in the data.&quot;" CreationDate="2010-08-05T04:55:59.030" UserId="273" />
  <row Id="1269" PostId="1268" Score="1" Text="@Amro: I like the title you propose !  " CreationDate="2010-08-05T05:27:53.393" UserId="223" />
  <row Id="1270" PostId="1286" Text="Has anyone noticed that the number of questions that should be in StackOverflow R rocketed up here recently?" CreationDate="2010-08-05T05:32:45.083" UserId="601" />
  <row Id="1271" PostId="1256" Text="the question in addition to the answers make me think this should be stackexchange... there should be a discussion on meta about questions with &quot;how to speed up&quot; ...." CreationDate="2010-08-05T05:33:21.887" UserId="223" />
  <row Id="1272" PostId="1267" Text="Huber's cost function (and M-estimation in general) ????? it should be something like 1970 !!!  " CreationDate="2010-08-05T05:42:04.583" UserId="223" />
  <row Id="1273" PostId="1286" Score="1" Text="Why not making random sampling?" CreationDate="2010-08-05T07:21:32.873" UserId="88" />
  <row Id="1274" PostId="752" Score="1" Text="well... there is a fine line between inane platitude and profound wisdom. I like the quote for it's poetic quality. Any insight is of secondary importance to me." CreationDate="2010-08-05T07:25:08.297" UserId="127" />
  <row Id="1275" PostId="1245" Text="I like your idea about using known arbitrage relations. have you tried this at all in your previous job?" CreationDate="2010-08-05T07:27:53.860" UserId="127" />
  <row Id="1276" PostId="1256" Text="The discussion Robin called for is now on meta: http://meta.stats.stackexchange.com/questions/248/how-to-speed-up-questions" CreationDate="2010-08-05T07:51:58.757" UserId="196" />
  <row Id="1277" PostId="1263" Text="The no-multicore was more a practical restriction, my colleague does not have a multicore machine, so parallelism wasn't going to help him.  I'm sure there is an entire other question to be asked regarding how to know a priori whether a parallel approach (with associated overhead) will be faster than a single threaded approach." CreationDate="2010-08-05T07:58:09.417" UserId="196" />
  <row Id="1278" PostId="1286" Score="1" Text="@John: If you feel that way discuss the issue at http://meta.stats.stackexchange.com/questions/248/how-to-speed-up-questions, no need to be snarky." CreationDate="2010-08-05T07:59:42.723" UserId="196" />
  <row Id="1279" PostId="1286" Text="@mbq:  Random sampling will quickly provide a reasonable approximate, especially with well behaved data.  However, I did specify that my goal was an exact test." CreationDate="2010-08-05T08:04:02.603" UserId="196" />
  <row Id="1280" PostId="1286" Text="@drknexus That's why it was a comment not an answer." CreationDate="2010-08-05T09:19:40.690" UserId="88" />
  <row Id="1281" PostId="1289" Score="2" Text="I think you mean bar charts rather than histograms" CreationDate="2010-08-05T10:09:06.270" UserId="159" />
  <row Id="1283" PostId="1261" Text="@Aniko Perhaps, I am missing something here. But, when I substituted the two sets of parameters into the likelihood function that I suggested they are not identical (assuming that I did not make any errors which is always possible). " CreationDate="2010-08-05T10:26:27.343" />
  <row Id="1284" PostId="1289" Text="@Rob: Isn't histogram a special type of bar chart that represents a frequency distribution? I am trying to visualize category frequencies for many bookstores." CreationDate="2010-08-05T10:28:07.700" UserId="760" />
  <row Id="1285" PostId="1290" Text="But will this cope with very large combinations?" CreationDate="2010-08-05T10:40:03.770" UserId="8" />
  <row Id="1286" PostId="1289" Text="@nimcap No, because histogram is over a continuous variable, and book category is a categorical variable.  " CreationDate="2010-08-05T10:46:38.593" UserId="88" />
  <row Id="1287" PostId="1291" Text="Thank you for your valuable answer. Situation is hard to describe even in my native language :) Let me  try. I am not interested if bookstores are favoring particular categories but I want to see if they favor categories. Actually this is what I am expecting. Let's say I have 3 bookstores (B1, B2, B3) and 4 categories (C1, C2, C3, C4). These are their sales data: B1(1, 1, 20, 20) B2(90, 1, 1, 1), B3(1, 1, 1, 30). Looking at this data I can tell they favor some categories to others. But if data was like B1(20, 30, 20, 20) B2(90, 100, 100, 100), B3(30, 30, 40, 40) I cant say that." CreationDate="2010-08-05T10:53:24.123" UserId="760" />
  <row Id="1288" PostId="1289" Text="@mbq Let's say a book store has 3 books, and their categories are: B1:[c1, c2, c3] B2:[c1, c3] B3:[c1, c4]. When we aggregate the category counts we get [c1 x 3, c2 x 1, c3 x 2, c4 x 1]. Isn't this enough to generate a histogram?" CreationDate="2010-08-05T10:56:34.337" UserId="760" />
  <row Id="1289" PostId="1290" Text="@csgillespie Well, I believe so -- it works *in situ*,  so only one combination is stored in memory at a time, and the results of simulation can be also aggregated to eliminate the need of storing them. This will of course work terribly long, but exhaustive searches usually do. For speed it could be written in C, but then along with the simulation part, which probably is way slower than a generator step." CreationDate="2010-08-05T11:11:21.973" UserId="88" />
  <row Id="1290" PostId="1289" Score="1" Text="@nimcap No, it is enough to generate a bar chart. Histogram can be done for instance for a price of a book." CreationDate="2010-08-05T11:13:42.140" UserId="88" />
  <row Id="1291" PostId="1289" Text="@mbq I really don't get it, wherever I look it says histogram is about frequencies, can you point to any sources mentioning histogram is for continuous variables." CreationDate="2010-08-05T11:39:43.660" UserId="760" />
  <row Id="1292" PostId="1287" Text="I assume you have the book, what's in chapter 17 (it's referenced in the document you provided)" CreationDate="2010-08-05T11:41:58.197" UserId="59" />
  <row Id="1293" PostId="1245" Text="No, we never fully formalized that.  But I think we used some simple ones (ie ETF vs underlying index etc).  It's been a few years though." CreationDate="2010-08-05T11:43:48.130" UserId="334" />
  <row Id="1294" PostId="1294" Score="1" Text="Parallel plots depend heavily on the &quot;right&quot; ordering of variables, so for too many categories this will become tedious. And the correct source seems to be A.Inselberg, 1981." CreationDate="2010-08-05T11:44:55.993" UserId="56" />
  <row Id="1295" PostId="1294" Score="3" Text="They're called parallel coordinate plots: http://en.wikipedia.org/wiki/Parallel_coordinates" CreationDate="2010-08-05T12:12:39.053" UserId="495" />
  <row Id="1296" PostId="1298" Text="Well I already know p. I also know the amount of events detected: k. So the total events is somewhere around k/p. I would like to find out an interval around k/p so I can be say 95% sure that the total number of events is inside it. Does that make more sense?" CreationDate="2010-08-05T12:24:21.903" UserId="762" />
  <row Id="1297" PostId="1295" Text="i'm wonder if this is a practical limitation though--for a variable that only weakly influences classification, my intuition is that Tree won't likely split on that variable (i.e., it's not going to be a node) which in turn means it's invisible as far as Decision Tree classification goes." CreationDate="2010-08-05T12:52:48.133" UserId="438" />
  <row Id="1298" PostId="1291" Text="In my example, shops O-Y favour romance books. This is why these shops are in a distinct group in the PC plot. " CreationDate="2010-08-05T13:00:10.447" UserId="8" />
  <row Id="1299" PostId="1295" Text="I am talking of weak interactions, not weak effects on classification. An interaction is a relationship between two of the predictor variables." CreationDate="2010-08-05T13:03:44.480" UserId="159" />
  <row Id="1300" PostId="1295" Text="This may be inefficient, but tree structure can handle it." CreationDate="2010-08-05T13:12:07.147" UserId="88" />
  <row Id="1301" PostId="1295" Text="That's why I said inefficient rather than biased or incorrect. If you have loads of data, it doesn't matter much. But if you fit a tree to a few hundred observations than the assumed interactions can greatly reduce the predictive accuracy." CreationDate="2010-08-05T13:15:49.477" UserId="159" />
  <row Id="1302" PostId="1273" Text="Actually, you don't need the book for this (you definitely should read it, but not for your current problem). All you have to do is plot the time intervals between infections and build your control chart based on this variable. Try it, you'll see that it's quite simple." CreationDate="2010-08-05T13:18:39.300" UserId="666" />
  <row Id="1303" PostId="1289" Text="@nimcap It is about frequencies that some continuous variable fits in a particular interval (bin). After making the cut of space into bins, it boils down to a bar chart, still with richer interpretation. " CreationDate="2010-08-05T13:21:43.727" UserId="88" />
  <row Id="1304" PostId="1294" Text="@Simon thanks; @honk I agree, this is one reason why I don't use them." CreationDate="2010-08-05T13:24:00.787" UserId="88" />
  <row Id="1309" PostId="1308" Score="3" Text="@John Is there any reason for your negative tone?" CreationDate="2010-08-05T14:31:51.997" />
  <row Id="1310" PostId="1309" Text="@Shane I am not sure if the multiple types solution will work here." CreationDate="2010-08-05T14:32:32.040" />
  <row Id="1311" PostId="1293" Text="Converted to community wiki." CreationDate="2010-08-05T14:33:26.487" UserId="88" />
  <row Id="1312" PostId="1308" Text="I presume that it's bioinformatics problem" CreationDate="2010-08-05T14:41:27.627" UserId="8" />
  <row Id="1313" PostId="1309" Text="@Srikant you're right, thanks. " CreationDate="2010-08-05T14:42:33.193" UserId="5" />
  <row Id="1314" PostId="1309" Text="I think that generalisation still only works for 2 or more people sharing a birthday - just that you can have different sub-classes of people." CreationDate="2010-08-05T14:45:02.190" UserId="765" />
  <row Id="1315" PostId="1308" Text="It is actually a bioinformatics problem, but since it boils down to the same concept as the birthday paradox I thought I'd save the irrelevant specifics!" CreationDate="2010-08-05T14:50:57.560" UserId="765" />
  <row Id="1316" PostId="1308" Score="2" Text="Normally I would agree with you, but in this case the specifics might matter since there could already be a bioconductor package that does what you ask." CreationDate="2010-08-05T14:53:30.957" UserId="8" />
  <row Id="1317" PostId="1291" Score="2" Text="I voted this up as a good general answer but as a practical answer, dealing with that many data points is going to be brutal." CreationDate="2010-08-05T15:16:13.710" UserId="601" />
  <row Id="1318" PostId="1308" Text="If you really want to know, it's a pattern finding problem where I'm trying to accurately estimate the probability of a given level of enrichment of a subsequence within a set of larger sequences.  I therefore have a set of subsequences with associated counts and I know how many subsequences I observed and how many theoretically observable sequences are available.  If I saw a particular sequence 10 times out of 10,000 observations I need to know how likely that was to have occurred by chance." CreationDate="2010-08-05T15:21:10.573" UserId="765" />
  <row Id="1319" PostId="1290" Score="1" Text="That looks almost identical to how R's combn function is already doing things.  I wrote up a version of combn that does take combinations off the stack one at a time, and as mbq says because it is only storing one combination in memory at a time it can handle very large combinations.  The problem with doing it in R is that doing a step-by-step approach in a function typically involves reading the state variables into the function, manipulating them, then storing them back out to global - which seems to just slow everything /way/ down." CreationDate="2010-08-05T15:34:43.297" UserId="196" />
  <row Id="1320" PostId="1311" Text="Will this solution suffer from the curse of dimensionality? If instead of n=365, n=10^6 is this solution still feasible?" CreationDate="2010-08-05T15:38:04.510" UserId="8" />
  <row Id="1322" PostId="1311" Text="Some approximations may have to be used to deal with high dimensions. Perhaps, use Stirling's approximation for factorials in the binomial coefficient. To deal with the product terms you could take logs and compute the sums instead of the products and then take the anti-log of the sum. " CreationDate="2010-08-05T15:45:53.913" />
  <row Id="1324" PostId="1312" Text="There may be notational ambiguities in what I wrote but $X_3$ is a random variable as it is a function of two random variables $Y_1$ and $Y_2$. In fact, you can compute $P(X_3=X_{31}) = P(Y_1 \  Y_2 &gt; 0)$." CreationDate="2010-08-05T16:11:28.367" />
  <row Id="1325" PostId="1311" Text="There are also several other forms of approximations possible using for example the Taylor series expansion for the exponential function. See the wiki page for these approximations: http://en.wikipedia.org/wiki/Birthday_problem#Approximations" CreationDate="2010-08-05T16:15:05.790" />
  <row Id="1326" PostId="1214" Text="Thank you. Unfortunately, this still doesn't work as I would expect. For the same time series of the previous comment it returns 166, which is only partially right (from my point of view, the evident weekly period is more interesting). And using a very noisy time series, like this one http://dl.dropbox.com/u/540394/chart2.png (a TCP receiver window analysis), the function returns 10, while I would expect 1 (I can't see any obvious periodicity). BTW I know that it will be really difficult to find what I'm looking for, since I'm dealing with too different signals." CreationDate="2010-08-05T16:17:41.543" UserId="667" />
  <row Id="1327" PostId="1312" Text="$X_3$ is random, but a deterministic function of other random quantities in your model. It therefore should not occur explicitly in the likelihood. In the context of graphical modelling, you would refer to $X_3$ as a deterministic node." CreationDate="2010-08-05T16:30:06.397" UserId="643" />
  <row Id="1328" PostId="1291" Score="1" Text="+1 This is certainly not what OP wants, still it is certainly what she/he should want.  " CreationDate="2010-08-05T16:39:07.180" UserId="88" />
  <row Id="1329" PostId="1297" Score="1" Text="From a ML point of view trees can be tested in a same way as any other classifier (CV for instance). Still it rather shows that heavy overfit happened ;-) Also RF escapes multicollinearity not because it is ensemble, but because its trees are suboptimal. " CreationDate="2010-08-05T16:48:36.623" UserId="88" />
  <row Id="1330" PostId="1295" Score="1" Text="Agree; I just wanted to highlight it. Still I think that reduction of predictive accuracy can be removed by using proper training; in phylogenetics the similar problem (greedyness) is reduced by Monte Carlo scanning of the possible tree space to find maximum likelihood ones -- I don't know is there a similar approach in stats, probably no-one was bothered by this problem to such extent." CreationDate="2010-08-05T16:55:49.580" UserId="88" />
  <row Id="1331" PostId="1284" Text="Thank you for the link. I'll try to contact the author sometime in the future." CreationDate="2010-08-05T17:52:15.700" UserId="253" />
  <row Id="1333" PostId="1153" Text="I finally had the time to play a bit with this method, and this seems  a good compromise (even if I still have to evaluate how complex is the stl() routine from a memory/cpu point of view), which works pretty well if we known the frequency of the time series. BTW, I noticed that the results of the detection heavily depend on the kind of signal I'm analyzing, and in order to obtain the optimum results I have to manually tune the probabilities for the quantiles or the multiplier for the IQR.&#xA;Since you're an expert of this topic, is there a &quot;smart&quot; and &quot;de facto&quot; solution to deal with this issue?" CreationDate="2010-08-05T18:33:56.440" UserId="667" />
  <row Id="1334" PostId="1314" Score="1" Text="Should be marked as Community Wiki." CreationDate="2010-08-05T19:01:38.897" />
  <row Id="1335" PostId="1314" Text="Definitely community wiki. " CreationDate="2010-08-05T19:11:26.587" UserId="5" />
  <row Id="1336" PostId="1314" Text="Done, CW. Still I think it is off-topic. " CreationDate="2010-08-05T19:33:19.117" UserId="88" />
  <row Id="1337" PostId="1316" Text="i was actually going to efforts to *avoid* mentioning the Gamma distribution. i saw it on Wikipedia, i cannot actually find the formula for the distribution, or the formulas to estimate the parameters in that formula. And then i got really nervous when i saw *&quot;There is no closed-form solution for k.&quot;*  And i tried it anyway with some formulas - but when you get a packet that comes back in 0ms, the ln(0) blows up." CreationDate="2010-08-05T19:40:33.160" UserId="775" />
  <row Id="1338" PostId="1316" Text="Because while i have good understanding of the normal distribution, from my university days, i am over my head when we get to things like *&quot;Kullback–Leibler divergence&quot;*." CreationDate="2010-08-05T19:42:41.593" UserId="775" />
  <row Id="1339" PostId="1315" Score="1" Text="Just looking at it, it looks like a skewed normal distribution. Are you sure the outliers are necessary for your analysis?" CreationDate="2010-08-05T19:43:35.447" UserId="776" />
  <row Id="1340" PostId="1316" Text="If you have a packet that comes back in 0ms it is not 'really' zero, right? Perhaps, you can set it to a small value and in any case you need not worry about ln(0) as the parameter $k$ does not refer to the data you have. I will update my answer with some details about the estimation process." CreationDate="2010-08-05T19:45:12.510" />
  <row Id="1341" PostId="1315" Score="1" Text="My analysis will consist solely of drawing a pretty graph over-top the bars :)   But it would be cheating to pretend there was no top tail..." CreationDate="2010-08-05T19:53:13.040" UserId="775" />
  <row Id="1342" PostId="1316" Text="Yes, technically it should be referred to as `&lt;1ms`. And this plot doesn't include zero, because it's going over a higher latency link  (modem). But i can run the program just as well over a faster link (i.e. ping another machine on the LAN), and routinely get `&lt;1ms` and `1ms`, with much less occurrences of `2ms`. Unfortunately Windows only provides resolution of `1ms`. i could manually time it using a high-performance counter, getting µs; but i was still hoping to be able to put them into buckets (to save memory). Perhaps i should add 1ms to everything... `1ms ==&gt; (0..1]`" CreationDate="2010-08-05T19:55:28.323" UserId="775" />
  <row Id="1343" PostId="1318" Text="Really i want to draw the mathematical curve that follows the distribution. Granted it might not be a known distribution; but i can't imagine that this hasn't been investigated before." CreationDate="2010-08-05T19:57:17.073" UserId="775" />
  <row Id="1344" PostId="1315" Text="@Ian: Since your histogram looks like you have 33 bins I would guess a 33rd order polynomial would fit nicely ;)" CreationDate="2010-08-05T20:12:16.263" UserId="56" />
  <row Id="1345" PostId="138" Score="1" Text="http://stackoverflow.com/questions/3375808/learning-r-where-does-one-start on SO" CreationDate="2010-08-05T21:05:06.750" UserId="776" />
  <row Id="1346" PostId="1321" Score="1" Text="You mean like any one of the numerous ggplot2 example charts such as&#xA;[this](http://learnr.wordpress.com/2009/05/18/ggplot2-three-variable-time-series-panel-chart/) or did you have something more specific in mind?  " CreationDate="2010-08-05T21:46:47.150" UserId="334" />
  <row Id="1347" PostId="1321" Score="1" Text="For what it is worth, I like plotting with `xts` and `zoo` objects where different series can be merged easily." CreationDate="2010-08-05T21:52:29.420" UserId="334" />
  <row Id="1348" PostId="1315" Text="@honk Please no! Even imaging it hurts." CreationDate="2010-08-05T22:04:39.287" UserId="88" />
  <row Id="1349" PostId="1315" Text="@Brandon Please no! Even imag(in)ing it hurts." CreationDate="2010-08-05T22:07:20.063" UserId="174" />
  <row Id="1350" PostId="1267" Text="Nein: Huber's *not* robust (a robust loss function has to be re-descending). The theory of re-descending dates from the 80's (MCD,S). The actual (usable, i.e. non combinatorial) estimators from early 2000 (check FAST-MCD, FAST-LTS, FAST-S)." CreationDate="2010-08-05T22:23:55.147" UserId="603" />
  <row Id="1351" PostId="1314" Text="I've closed this as it is off-topic. The site is not intended to be a place for career advice for statisticians." CreationDate="2010-08-05T23:37:12.533" UserId="159" />
  <row Id="1352" PostId="1326" Text="I doubt that anyone is going to top this one. Thanks Matt, great answer!" CreationDate="2010-08-05T23:48:22.267" UserId="1356" />
  <row Id="1354" PostId="1322" Score="3" Text="Here's an example of your suggestion from the NYTimes&#xA;&#xA;http://www.nytimes.com/imagepages/2010/05/02/business/02metrics.html" CreationDate="2010-08-06T00:57:28.210" UserId="287" />
  <row Id="1355" PostId="1297" Score="2" Text="For a probabilistic framework of decision trees, see DTREE (url: http://www.datamining.monash.edu.au/software/dtree/index.shtml) which is based on the paper &quot;Wallace C.S. &amp; Patrick J.D., `Coding Decision Trees', Machine Learning, 11, 1993, pp7-22&quot;." CreationDate="2010-08-06T01:17:00.280" UserId="530" />
  <row Id="1357" PostId="1322" Text="That's an interesting visualization. JoFrhwld, thanks for the example." CreationDate="2010-08-06T02:14:34.127" UserId="776" />
  <row Id="1358" PostId="1314" Text="A possible place for this type of question might be  http://groups.google.com/group/sci.stat.consult." CreationDate="2010-08-06T02:27:52.730" UserId="159" />
  <row Id="1359" PostId="1337" Text="I made this community wiki as there is no correct answer." CreationDate="2010-08-06T02:44:34.817" UserId="159" />
  <row Id="1360" PostId="1214" Text="166 is not a bad estimate of 168. If you know the data are observed hourly with a weekly pattern, then why estimate the frequency at all?" CreationDate="2010-08-06T03:01:01.493" UserId="159" />
  <row Id="1361" PostId="1340" Text="http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1043351251&#xA;&#xA;Freely available document written by Peter Huber on John Tukey's contributions to robust statistics. Reasonably easy read, light on the formulae." CreationDate="2010-08-06T03:08:39.293" UserId="781" />
  <row Id="1362" PostId="1267" Score="4" Text="The median is pretty robust and it has been around for a while." CreationDate="2010-08-06T03:41:17.933" UserId="159" />
  <row Id="1363" PostId="1340" Text="@Wesley. It is best to avoid referring to other answers &quot;above&quot; and &quot;below&quot; as they can move depending on votes and the ordering tab at the top of the answers section." CreationDate="2010-08-06T03:42:37.603" UserId="159" />
  <row Id="1364" PostId="1214" Text="Because I have to analyze a lot of time series (suppose 100 network metrics), and only some of them have a weekly periodicity. In any case, I guess in my implementation I will use an algorithm similar to your function, and I'll manually distinguish the weekly periodicity. Really thanks for your support, I really appreciate (and keep up the good work with the forecast library :-))" CreationDate="2010-08-06T03:55:59.060" UserId="667" />
  <row Id="1365" PostId="1339" Text="Thank you for your precious informations, I'll look at that book for sure." CreationDate="2010-08-06T03:57:38.187" UserId="667" />
  <row Id="1367" PostId="1304" Text="Thanks, that looks great. I think this is the answer I was looking for." CreationDate="2010-08-06T06:18:16.917" UserId="762" />
  <row Id="1368" PostId="1327" Text="+1 i thought binomial as well, when i first saw the histogram. (Not sure why this got downvoted)." CreationDate="2010-08-06T06:35:35.057" UserId="438" />
  <row Id="1369" PostId="1346" Score="5" Text="You should also quote the title attribute of the image: &quot;Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'.&quot;" CreationDate="2010-08-06T07:15:51.640" UserId="582" />
  <row Id="1370" PostId="1349" Score="1" Text="A simple help through the rseek.org or r-help mailing list will get you on your way with dealing with most (all?) of the methods in R (which is the program package I would suggest to anyone). Good link." CreationDate="2010-08-06T07:47:25.943" UserId="144" />
  <row Id="1372" PostId="1355" Text="I wanted to add the tag ga,es but couldn't.  I will later when I pass the rep threshold." CreationDate="2010-08-06T09:48:44.230" UserId="114" />
  <row Id="1373" PostId="1356" Text="I think originaly taken from Yihui XIE's Statistics Jokes Slides (http://www.yihui.name/en/attachment.php?f=attachment/jokes_yihui.pdf)" CreationDate="2010-08-06T09:57:54.707" UserId="114" />
  <row Id="1374" PostId="1164" Text="Excellent recommendation, thanks." CreationDate="2010-08-06T10:36:18.407" UserId="438" />
  <row Id="1375" PostId="1355" Score="2" Text="It'd be fun to see your code once you write it :)  Sorry, I don't remember coming across such a thing..." CreationDate="2010-08-06T11:04:58.043" UserId="253" />
  <row Id="1376" PostId="1318" Score="1" Text="Look up 'density estimation'." CreationDate="2010-08-06T11:21:01.607" UserId="247" />
  <row Id="1377" PostId="1355" Text="I did something similar a while back to analyze the most-frequently-landed-on spaces in Monopoly (wrote a small simulator)... not sure if there's another way of doing it." CreationDate="2010-08-06T11:47:54.013" UserId="292" />
  <row Id="1378" PostId="1355" Text="Seems off-topic. I am not sure I see the link to statistical analysis in the question. This seems appropriate for http://stackoverflow.com." CreationDate="2010-08-06T12:21:39.367" />
  <row Id="1379" PostId="1355" Text="Strikant - since it can be solved with simulation, I think keeping the question will be o.k." CreationDate="2010-08-06T12:45:52.503" UserId="253" />
  <row Id="1380" PostId="1365" Text="Thank you, this is really helpful. (Shame on me for not recognizing a Fourier series even when hurtling towards it...)" CreationDate="2010-08-06T12:55:07.460" UserId="650" />
  <row Id="1381" PostId="1355" Text="I agree with it Tal. It's a statistical computing problem." CreationDate="2010-08-06T12:58:32.320" UserId="8" />
  <row Id="1382" PostId="752" Text="@Jilles: thanks a lot, I also like the poetic quality. " CreationDate="2010-08-06T14:19:16.390" UserId="223" />
  <row Id="1383" PostId="1275" Text="extrapolation would include smoothness in the filled part that does not exists in the existing part. You have to add randomness... hence resampling (and resmapling on the extrapolation seems to be a good idea) " CreationDate="2010-08-06T14:28:13.300" UserId="223" />
  <row Id="1384" PostId="1369" Score="3" Text="I don't really understand what you are asking. Could you try and reword your question?" CreationDate="2010-08-06T14:28:50.943" UserId="8" />
  <row Id="1385" PostId="1275" Text="Extrapolating the model would require sampling the error term which would induce the desired randomness." CreationDate="2010-08-06T14:53:04.030" />
  <row Id="1386" PostId="1264" Text="thanks for this explanation" CreationDate="2010-08-06T15:02:18.257" UserId="684" />
  <row Id="1387" PostId="1350" Score="2" Text="A 'correct' answer is dependent on the model and the code/commands you are using. So, a brief description of both would be useful." CreationDate="2010-08-06T15:08:53.033" />
  <row Id="1388" PostId="1355" Text="Definitely in topic" CreationDate="2010-08-06T15:10:40.767" UserId="582" />
  <row Id="1389" PostId="1359" Score="1" Text="One of the problems that I have found with several of the DOE books that I am familiar with is that they heavily emphasize analysis.  Is there are good book that really does the design elements well--e.g. blocking, replication, randomization, choosing factor levels, using repeated measurements, split-plots, etc.  What about the (even more often neglected) design of non-experimental analytic studies, like case-control designs or surveys for analytical purposes (a very different prospect than simply estimating a population parameter in terms of how you might structuring that research)." CreationDate="2010-08-06T15:21:35.050" UserId="485" />
  <row Id="1390" PostId="1359" Text="I would look for books in epidemiology and economics for what you are describing.  But sadly, I don't have a recommendation to give." CreationDate="2010-08-06T15:46:26.883" UserId="253" />
  <row Id="1391" PostId="1340" Text="I realized that after I saw my statement float around. I'll edit it to refer to the specific comment." CreationDate="2010-08-06T15:49:03.607" UserId="781" />
  <row Id="1392" PostId="1282" Text="I'm giving you the tick, because I'll probably implement this." CreationDate="2010-08-06T15:59:51.023" UserId="287" />
  <row Id="1393" PostId="1340" Text="@Wesley Burr thanks for the princeton study: i did not know of it." CreationDate="2010-08-06T16:16:17.073" UserId="603" />
  <row Id="1394" PostId="1267" Text="Sure Rob, but the median is really a small part of robust statistics. By and large, usable robust estimation Technics only became widely available after 2000. " CreationDate="2010-08-06T16:18:48.960" UserId="603" />
  <row Id="1395" PostId="1234" Text="I do not understand this &#xA;&quot;This leads to underfiltering in quiet periods and overfiltering during more busy times&quot;&#xA;care to explain ?" CreationDate="2010-08-06T16:25:05.877" UserId="603" />
  <row Id="1396" PostId="1374" Text="minus five minus seven? so minus twelve? that doesn't make sense" CreationDate="2010-08-06T17:24:55.827" UserId="74" />
  <row Id="1397" PostId="1365" Text="Does this mean that the moments of a circular distribution should be compared to the characteristic function of a linear distribution rather than to its moments?" CreationDate="2010-08-06T17:53:04.410" UserId="650" />
  <row Id="1398" PostId="1374" Text="Five to seven -- I'll try to convert it to en-dash." CreationDate="2010-08-06T17:53:36.657" UserId="88" />
  <row Id="1399" PostId="1338" Text="That is certainly true!" CreationDate="2010-08-06T18:30:34.200" UserId="253" />
  <row Id="1400" PostId="1374" Text="That made me laugh, my god!" CreationDate="2010-08-06T18:31:09.193" UserId="253" />
  <row Id="1401" PostId="1338" Score="1" Text="My mom in law said to me that this is based on the line &quot;being in love means you never need to say you are sorry&quot;.  From a book called &quot;love story&quot; by arik sigall (I think).  " CreationDate="2010-08-06T18:33:19.983" UserId="253" />
  <row Id="1402" PostId="1372" Text="Thanks that got me on the right way and i was able to solve it!" CreationDate="2010-08-06T18:46:05.887" UserId="791" />
  <row Id="1403" PostId="1365" Text="@Rasmus: I guess that depends on exactly what you want to do with the information, but in general I'd say yes." CreationDate="2010-08-06T19:13:44.303" UserId="89" />
  <row Id="1404" PostId="1368" Score="5" Text="This joke falls flat for me because I can't imagine a statistic professor saying this - a student who failed the course, sure." CreationDate="2010-08-06T19:25:07.807" UserId="196" />
  <row Id="1405" PostId="1316" Text="simply fitting gammas with R: http://docs.google.com/viewer?a=v&amp;q=cache:bl4TbieigsEJ:cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf+fitting+gamma+distribution+R&amp;hl=en&amp;gl=us&amp;pid=bl&amp;srcid=ADGEESjkOC5S7YMbhWRHBDF9cjFaKv4F33c19EPChCnV9rgtdRyO28LCncXNgq9NYJHYe1DeqHPoW3Pl5uM9m0uLeofqruGfCfH6vRhZ6ioinju5ukyWdUYm6D4CVRYOF5xVymC7dTHs&amp;sig=AHIEtbSdTddbrI7ADPWiM_N7gLaLeara0w" CreationDate="2010-08-06T20:15:00.690" UserId="291" />
  <row Id="1406" PostId="1376" Text="thanks for editing to have proper math. I couldn't find info on how to render math in 'markdown' language. looks just like raw \LaTeX does it?" CreationDate="2010-08-06T21:33:34.720" UserId="795" />
  <row Id="1407" PostId="1376" Text="Yes, it is Latex. See this meta thread: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats" CreationDate="2010-08-06T21:46:02.110" />
  <row Id="1408" PostId="1275" Text="IMO both suggestions boil down to predicting future values from existing ones (AR/ARMA models perhaps?). I guess I'm still hoping for a solution that doesn't involve sampling values (thus the possibility of introducing error).. Besides estimating such models is in itself a form of dimensionality reduction :)" CreationDate="2010-08-06T22:19:00.363" UserId="170" />
  <row Id="1409" PostId="1335" Text="Here is a post with the code and some illustrations:&#xA;&#xA;http://www.markmfredrickson.com/thoughts/2010-08-06-combinadics-in-r.html" CreationDate="2010-08-06T23:50:07.863" UserId="729" />
  <row Id="1410" PostId="786" Score="3" Text="According to Wikiquote it is misattributed to Joseph Stalin; the origin is Kurt Tucholsky: http://en.wikiquote.org/wiki/Joseph_Stalin#Misattributed" CreationDate="2010-08-07T00:36:13.317" UserId="509" />
  <row Id="1411" PostId="485" Text="Just noticed this related question: http://stackoverflow.com/questions/570029/learning-applied-statistics-with-a-focus-on-r" CreationDate="2010-08-07T08:36:28.870" UserId="183" />
  <row Id="1412" PostId="1371" Score="2" Text="I remember seeing George Burns on TV being interviewed on his 100th birthday. He was puffing on a cigar. The interviewer made some comment about the incongruity of longevity and smoking.&#xA;&#xA;&#xA;George Burns: &quot;Twenty years ago my doctor told me that these cigars were going to kill me&quot;&#xA; &#xA;Interviewer: &quot;What does he say now?&quot;&#xA; &#xA;George Burns: &quot;I don't know. He's dead&quot;&#xA;" CreationDate="2010-08-07T09:17:31.607" UserId="521" />
  <row Id="1413" PostId="1383" Text="you should include a better description of what data you want to compress... otherwise we will have 10 different answers trying to summarize different part of the huge theory of compression.  &#xA;&#xA;In addition, the two point you give to be more specific are not detailed enough to be understood. " CreationDate="2010-08-07T09:32:00.023" UserId="223" />
  <row Id="1414" PostId="1378" Text="+1 for interesting real case ! can you help us to understand what is blood test ? (how do you calculate it)" CreationDate="2010-08-07T09:35:06.900" UserId="223" />
  <row Id="1415" PostId="165" Text="The link doesn't work:&#xA;http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.7133&amp;rep=rep1&amp;type=pdf" CreationDate="2010-08-07T09:47:53.430" UserId="521" />
  <row Id="1416" PostId="1391" Score="1" Text="+1 nice and clear answer. Jeromy, can I ask a question about point 3? I do understand the reasoning behind transforming the data, but something always bothered me about doing that. What is the validity of reporting the results of the t-test on the transformed data to the untransformed data (where you're not &quot;allowed&quot; to do a t-test)? In other words, if two groups are different when data is, for instance, log transformed, on what bases can you say the raw data is different too? Bare in mind, I'm not a statistician, so maybe I just said something absolutely stupid :)" CreationDate="2010-08-07T09:50:59.823" UserId="582" />
  <row Id="1417" PostId="618" Text="Obviously an unforseen side-effect of the profusion of lemon laws in the US. For example see:&#xA;http://en.wikipedia.org/wiki/Lemon_law" CreationDate="2010-08-07T10:03:20.123" UserId="521" />
  <row Id="1418" PostId="823" Score="11" Text="-1 This is an intellectually dishonest quote from a bad novelist, playing up a popular romantic myth of scientists as Rand-esque revolutionary loners in order to pander to anti-science crankery. Moreover, it has nothing whatsoever to do with statistics. What is it even doing in this list?" CreationDate="2010-08-07T14:34:40.253" UserId="174" />
  <row Id="1419" PostId="1391" Score="2" Text="@nico I'm not sure about how to report or think about the results, but if all you want to show is that for some X and Y, mu_X != mu_Y, it should be true that for all X_i &lt; X_j, log(X_i) &lt; log(X_j) and for all all X_i &gt; X_j, log(X_i) &gt; log(X_j). That's why for non-parametric tests which operate with ranks, transformations of the data don't affect the result. I think from this, you can assume that if some test shows that mu_log(X) != mu_log(Y), then  mu_X != mu_Y." CreationDate="2010-08-07T15:57:56.737" UserId="287" />
  <row Id="1420" PostId="1391" Text="thanks for the answer(s). indeed, the t-test appears to maintain nominal type I rate under mildly skewed/kurtotic input. however, I was hoping for something with more power. re: 2, I have implemented Wilcox' `trimpb` and `trimcibt`, but they are a bit too slow to do my power tests, at least for my taste. re: 3, I had thought of this method, but I am interested in the mean of the un-transformed data (i.e., I am not comparing 2 R.V.s with a t-test, in which case, a monotonic transform would be fine for a rank-based comparison, as noted by @JoFrhwld.)" CreationDate="2010-08-07T16:11:08.237" UserId="795" />
  <row Id="1421" PostId="1390" Text="the data is most definitely not normal. the excess kurtosis is on the order of 10-20, the skew is on the order of -0.2 to 0.2. I am doing a 1-sample t-test, so I'm not sure I follow you regarding 'unequal variances', or the U-test." CreationDate="2010-08-07T16:14:18.203" UserId="795" />
  <row Id="1422" PostId="1378" Text="Sure - briefly, it's a test for latent tuberculosis infection.  Blood is drawn from the patient into three tubes: one that has no antigens, one that has selected TB antigens, and one that has mitogen.  We then compare amount of immunologic response in the tube with no antigens to the one with TB - if they have a strong boost of response in the TB tube, they may have latent TB.  (The mitogen tube serves as a check to make sure that the person is capable of producing an immunologic response at all - most people have very strong reactions to it)" CreationDate="2010-08-07T17:25:37.110" UserId="71" />
  <row Id="1423" PostId="1396" Text="It won't help if just the numerical algorithm is diverging for certain parameters." CreationDate="2010-08-07T18:43:22.720" UserId="88" />
  <row Id="1424" PostId="1393" Text="interesting idea, thanks for sharing" CreationDate="2010-08-07T22:20:11.457" UserId="170" />
  <row Id="1425" PostId="1405" Score="1" Text="what's the raw information you have?... i.e. how did you come to two DORs?" CreationDate="2010-08-08T01:37:27.917" UserId="601" />
  <row Id="1426" PostId="1391" Score="2" Text="@nico If the population distribution of residuals is the same in two groups, then I imagine any time there is a difference in raw population group means there would also be differences in group means of an order-preserving transformation. That said, p-values and confidence intervals will tend to change slightly based on whether you are using raw data or transformed data. In general I prefer to use transformations when they seem like a meaningful metric for understanding the variable (e.g., Richter scale, decibels, logs of counts, etc.)." CreationDate="2010-08-08T03:49:01.213" UserId="183" />
  <row Id="1427" PostId="1337" Text="It probably makes sense to leave cartoons in this question: http://stats.stackexchange.com/questions/423/what-is-your-favorite-data-analysis-cartoon" CreationDate="2010-08-08T11:49:27.500" UserId="183" />
  <row Id="1428" PostId="1408" Text="The first point you mention is false in general I suggest that you read the first sentence of the chapter http://books.google.fr/books?id=Nxnh48rS9jQC&amp;pg=PA167&amp;lpg=PA167&amp;dq=Orthonormal+bases+of+compactly+supported+wavelets,&amp;source=bl&amp;ots=tbUc1teQFB&amp;sig=gmzBJyET8S5SXd-oPmzyh8BWDKg&amp;hl=fr&amp;ei=DPNeTOeBC8nT4gbsuazaBw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=4&amp;ved=0CDAQ6AEwAw#v=onepage&amp;q=Orthonormal%20bases%20of%20compactly%20supported%20wavelets%2C&amp;f=false in the book of Daubechie. In addition, If you had read my answer I already mentionned the nice property of the DWT in the 2nd part of my answer..." CreationDate="2010-08-08T18:13:13.153" UserId="223" />
  <row Id="1429" PostId="1405" Text="If you have the raw information available that you used to calculate the odds ratios a statistical test will be possible.  Alternatively there might be some reasonable simulation approaches if you know how many observations went into each odds ratio.&#xA;&#xA;P.S. For those of us who only have a passing familiarity with DORs would you please provide the formula used for calculating one?" CreationDate="2010-08-08T18:17:59.023" UserId="196" />
  <row Id="1430" PostId="1414" Text="I'm not sure I understand your answer.  I think you are saying I do not need to specify a link function because the intercept of my model will adjust based on the data and effectively account for the fact that at high difficulties people will not be able to perform the task.  Is that correct?  If so, how does this prevent a flattening of the slope estimate due to the same values at difficulties 4 and 5?  I may need a less technical explanation (if possible)." CreationDate="2010-08-08T18:40:26.170" UserId="196" />
  <row Id="1431" PostId="1408" Text="To the first point, you're right.  I should have said &quot;Most commonly used/implemented discrete wavelet basis functions&quot;; I'll edit to reflect that.&#xA;&#xA;To the second point, you gave a good answer for how the some CWTs (most often a DOG wavelet or the related Ricker wavelet; something like e.g. the Gabor wavelet would not provide the behavior your describe) can detect anomalies of the singularity kind.  I was trying to give an analogous description of how DWT can be used for detecting other kinds of anomalies." CreationDate="2010-08-08T19:21:26.600" UserId="61" />
  <row Id="1432" PostId="242" Text="sorry if I wasn't clear, I mean't that I prefer (personal subjective opinion) questions like yours than the question that asks &quot;what is a random variable&quot;... but I guess my pleasure is not that of everyone :) " CreationDate="2010-08-08T19:27:36.960" UserId="223" />
  <row Id="1433" PostId="1414" Text="*I'm not sure I understand your answer...Is that correct?*&#xA;&#xA;yes&#xA;&#xA;The limitation with the logit is one of symmetry; going back to your example, you force the slope between 4 and 5 to be the same as the slope between 1 and 2.&#xA;&#xA;If symmetry is a problem, you might want to try a complementary log-log link (not implemented in LMER)." CreationDate="2010-08-08T20:09:07.203" UserId="603" />
  <row Id="1434" PostId="1364" Text="That's perfect ! I was hoping for a pointer but that's a really good answer.  Thanks." CreationDate="2010-08-08T20:11:57.883" UserId="114" />
  <row Id="1435" PostId="1414" Text="I think your example is somewhat boggus in that the variable &quot;difficulty of the discrimination&quot; is really a ordinal scale and should be introduced in your regression as such. The problem you would have in this case would be due to you wrongly introducing &quot;difficulty of the discrimination&quot; as a continuous predictor when  it is not, not to the logit link per see." CreationDate="2010-08-08T20:17:42.503" UserId="603" />
  <row Id="1436" PostId="1414" Text="@kwak: I introduced it as an ordinal because it is easier to describe that way.  Difficulty of discrimination in many of these experiments is a quantifiable ratio scale variable, e.g. contrast or noise introduced to the signal." CreationDate="2010-08-08T21:29:05.707" UserId="196" />
  <row Id="1437" PostId="1414" Text="@kwak: Does your suggestion to use a complementary log-log link still apply given that the IV is an interval scale variable?" CreationDate="2010-08-08T21:36:55.993" UserId="196" />
  <row Id="1438" PostId="1414" Text="@drknexus#1:&gt; understood, but the mishandled ordinal scale was what was causing the expected behavior of the logit seem inappropriate, so i pointed it out.&#xA;@drknexus#2:&gt; what you mean by &quot;IV&quot;? " CreationDate="2010-08-08T21:46:14.610" UserId="603" />
  <row Id="1439" PostId="1421" Text="The example is a case where a single predictor linear equation, though rationally sound given the dataset, will provide a suboptimal solution to the problem.  If in the example case you fit a quadratic as well you'd likely get an improved model fit would you not?  The improved model fit wouldn't be a consequence of some underlying real quadratic effect of the IV, just that you'd hit a measurement floor." CreationDate="2010-08-08T22:06:43.480" UserId="196" />
  <row Id="1440" PostId="1421" Text="I'll take a look at Dixon - thanks for the reference - it looks right up my alley.  The alternative link I was imagining was an explicitly 2AFC link, e.g. mafc.logit(2) in the psyphy package of R." CreationDate="2010-08-08T22:14:26.050" UserId="196" />
  <row Id="1441" PostId="1414" Text="I'm not sure what you mean by &quot;mishandled ordinal scale&quot;.  Is the problem that the data itself is not spread out in a continuous fashion?  I'd like to simulate this with continuous data, but I can't think of a way of generating values that don't beg the question.  Any ideas?&#xA;&#xA;By IV I meant &quot;independent variable&quot;/&quot;predictor&quot;, sorry for using shorthand." CreationDate="2010-08-08T23:19:29.193" UserId="196" />
  <row Id="1443" PostId="608" Text="mbq; Is the question in its current form unanswerable or just difficult?" CreationDate="2010-08-08T23:40:54.377" UserId="196" />
  <row Id="1444" PostId="1424" Text="By 'such event' do you mean exactly 3 aces, or at least 3 aces?" CreationDate="2010-08-09T00:20:36.233" UserId="455" />
  <row Id="1445" PostId="219" Score="2" Text="Thomas, Howes, Luk - 2009 - A comparison of CPUs, GPUs, FPGAs, and massively parallel processor arrays for random number generation. http://doi.acm.org/10.1145/1508128.1508139.&#xA;&#xA;Discussion + benchmarks of a set of PNGs executing on CPU, GPU, FPGA and Massively Parallel Processor Arrays." CreationDate="2010-08-09T01:12:09.227" UserId="154" />
  <row Id="1446" PostId="1424" Text="That's right, exactly 3 aces, because you roll exactly 3 dices." CreationDate="2010-08-09T01:39:46.790" UserId="1356" />
  <row Id="1447" PostId="1425" Text="Could you, please, point us to a correct calculation? ... in &quot;as if&quot; case?" CreationDate="2010-08-09T01:59:02.097" UserId="1356" />
  <row Id="1448" PostId="1425" Text="It coccured to me to do it a few minutes ago, so it's already there. Did you want more detail?" CreationDate="2010-08-09T02:00:12.077" UserId="805" />
  <row Id="1450" PostId="1419" Text="Yes, I now remember looking at one or two of these videos a few years back. They are quite mathematical." CreationDate="2010-08-09T06:33:34.333" UserId="183" />
  <row Id="1451" PostId="1429" Text="I am not trying to compute a reduced-rank approximation of X, rather a transformed X. You see my goal is not to filter noisy sequences, but to find a representation with a reduced dimensionality (to be used for classification/clustering of time series)... Could you elaborate a bit on the EM-PCA approach?" CreationDate="2010-08-09T06:38:02.670" UserId="170" />
  <row Id="1452" PostId="1430" Text="I would like to add a stochastic processes tag, but don't have the reputation, perhaps somebody could edit this for me?" CreationDate="2010-08-09T06:51:17.527" UserId="352" />
  <row Id="1453" PostId="1430" Text="@Robby retag done ! " CreationDate="2010-08-09T06:59:16.823" UserId="223" />
  <row Id="1454" PostId="1421" Text="Doesn't the AIC adequately assess model fit for these purposes? I only provided the curves to demonstrate/visualize what is going wrong.  The problem isn't simply because the floor has been hit.  To convince yourself of this, try difficulty 4 at 60% and 5 at 55%.  The logistic curve still is dropping down into &quot;impossible&quot; territory.  Yes, it might be nice to not assess difficulty levels where the task is &quot;impossible&quot; but you don't always know where those levels are going to be in advance of preliminary data collection." CreationDate="2010-08-09T07:09:45.270" UserId="196" />
  <row Id="1455" PostId="1421" Text="My take on it is this: the problem we are observing is analogous to why using percentages as linear predictors is fine until you start reaching the extremes of the percentages - the logistic function and a linear function are reasonably matched until you get upwards of 80%. Likewise, in the low end here the basic logistic function is a reasonable match until the probability of correct answers starts dipping below 60% or so. Then you find yourself in part of the curve where the logistic function is predicting a continued drop off in accuracy whereas the 2AFC function predicts a leveling." CreationDate="2010-08-09T07:15:08.003" UserId="196" />
  <row Id="1456" PostId="430" Text="This is the way I explain things, fully agree ! " CreationDate="2010-08-09T07:24:50.600" UserId="223" />
  <row Id="1457" PostId="1408" Text="The second point you mention is also likely to be false: wavelet support (if it is compact) is giving information about the temporal localization of the wavelet not the frequency localization. " CreationDate="2010-08-09T07:30:30.167" UserId="223" />
  <row Id="1458" PostId="1431" Text="Perhaps I am a little confused, or we are using different definitions?  Am I correct in thinking that a process is 2nd order stationary if the joint marginal cdfs&#xA;$F_{X(t),X(t+τ)}$ are equal for all $\tau$?&#xA;&#xA;Similarly for a process to be 1st-order stationary the marginal cdfs $F_{X(t)}$ need to be the same for every $t$. So all the moments of the $X(t)$ must be equal. &#xA;&#xA;2nd-order stationary implies 1st-order stationary, correct?" CreationDate="2010-08-09T07:44:33.563" UserId="352" />
  <row Id="1459" PostId="1431" Text="Extending this, a process is $N$th order stationary if for every $t_1, t_2, \dots, t_N$ the marginal cdfs $F_{X(t_1 + \tau), X(t_2 + \tau)\dots,X(t_N+\tau)}$ are the same for all $\tau$.&#xA;&#xA;Strictly stationary is Nth order stationary for all N." CreationDate="2010-08-09T07:45:28.883" UserId="352" />
  <row Id="1460" PostId="1431" Text="see http://www.statistik.tuwien.ac.at/public/dutt/vorles/geost_03/node49.html for order 2 stationnarity. Anyway, I have tryed to clarify my answer, hope it is better now..." CreationDate="2010-08-09T08:05:01.520" UserId="223" />
  <row Id="1461" PostId="833" Text="thanks for the update simon ! " CreationDate="2010-08-09T08:06:15.527" UserId="223" />
  <row Id="1462" PostId="76" Text="why is this answer not community wiki ?" CreationDate="2010-08-09T08:18:24.810" UserId="223" />
  <row Id="1463" PostId="1431" Text="I believe that article describes stationary in the weak sense and this is not what I meant. I should clarify this in the question. See http://en.wikipedia.org/wiki/Stationary_process for a description of the various types of stationarity." CreationDate="2010-08-09T08:24:15.200" UserId="352" />
  <row Id="1464" PostId="1431" Text="@robby OK... I didn't know this &quot;second order stationnarity&quot; I think you should not say second order stationnarity in the question and give the definintion instead. For more clarity you should ask another question. do you have a paper that refers to this second order stationnarity ? " CreationDate="2010-08-09T08:32:42.733" UserId="223" />
  <row Id="1465" PostId="1425" Text="No, this should suffice. Thanks!" CreationDate="2010-08-09T08:41:22.930" UserId="1356" />
  <row Id="1466" PostId="501" Text="Can you define how you &quot;check the error increase&quot; please?" CreationDate="2010-08-09T09:03:43.360" UserId="521" />
  <row Id="1467" PostId="497" Text="How do you &quot;train&quot; your classifier? Presumably this is done on the training set. If it is a Support vector Machine (SVM) there are several parameters to try during training. Is each tested against the validation (test) set? Or are you using k-fold cross validation? How many times are you using the validation (test) set to check your performance - presumably this is accuracy. Sorry to be pedantic, but this is a poorly defined answer and risks over-fitting." CreationDate="2010-08-09T09:08:36.263" UserId="521" />
  <row Id="1468" PostId="1431" Text="I'll put a second question together that better describes what I mean.  In the mean time, your answer is correct for the question I stated!" CreationDate="2010-08-09T09:11:45.773" UserId="352" />
  <row Id="1469" PostId="1291" Score="1" Text="+1 Nice example of a &quot;down-to-Earth&quot; application of PCA." CreationDate="2010-08-09T10:49:45.883" UserId="582" />
  <row Id="1470" PostId="1438" Text="not long at all, I am comparing different methods, although I was not sure whether correlating with main components does make sense in terms of time series data. On the other hand the major trends are indeed orthogonal hence PCA should give me solid results" CreationDate="2010-08-09T10:56:26.593" />
  <row Id="1471" PostId="1203" Text="note that this will be your accepted answer for the bounty (in case you do not choose an answer the bounty automatically select the answer with the most points)  " CreationDate="2010-08-09T11:21:10.687" UserId="223" />
  <row Id="1472" PostId="6" Score="1" Text="It is maybe good to restate what is written in the faq: &quot;Avoid asking questions that are subjective, argumentative, or require extended discussion&quot;" CreationDate="2010-08-09T11:34:49.367" UserId="223" />
  <row Id="1473" PostId="1421" Text="As to your first comment, I see that you have a good answer about how to look at the residuals from another posted question but you still don't have why.  AIC, log-likelihood, etc. all tell you how good you fit is but they don't tell you the nature of the fit.. it's like comparing SD and histograms for looking at variability.  They both tell you about variability.  " CreationDate="2010-08-09T12:22:15.760" UserId="601" />
  <row Id="1474" PostId="1421" Text="As to your second problem, no.  Of course the chance end of the scale can be variable but pretty much anyone who's extensively studied SAT curves can attest (Ratcliff, Lappin, Pachella) that there isn't much useful variability in the chance end of the scale. It just ditches catastrophically.  All of the above names work with models other than just logistic to come to this.  It's also a simple measurement issue.  At the low end of the scale you have a good measure of error.  At the high end you don't.  So, you can trust your low accuracy measures better." CreationDate="2010-08-09T12:22:37.540" UserId="601" />
  <row Id="1475" PostId="1421" Text="And your comment about imagining that accuracy started going up... typically in one curve you want to model a single psychological process.  I doubt accuracy increasing after difficulty exceeded some point does that.  It would be tapping a different process... or just be some chance variation." CreationDate="2010-08-09T12:24:55.630" UserId="601" />
  <row Id="1476" PostId="1421" Text="BTW, to really understand modelling accuracy over a change in difficulty you might want to look at the SAT literature since it does exactly that." CreationDate="2010-08-09T12:25:27.300" UserId="601" />
  <row Id="1477" PostId="1406" Text="I think it is really popular.... but not it statistic (we don't want to loose our job because physisics are improving their experiment :) ). Anyway I think Rutherford belongs to this class of spiritual scientist... +1" CreationDate="2010-08-09T12:32:10.467" UserId="223" />
  <row Id="1478" PostId="1393" Text="+1 for the web site VideoLecture, I did not know, did you mention it in the question about video lectures ? " CreationDate="2010-08-09T12:34:34.143" UserId="223" />
  <row Id="1479" PostId="1393" Text="I've only being reading about this stuff recently. I really like Candes and Tao's recent paper on the topic http://arxiv.org/abs/0903.1476" CreationDate="2010-08-09T12:55:23.790" UserId="352" />
  <row Id="1480" PostId="1386" Text="Is there a reason that prohibit the use of the Welch t-test? Have a look at my answer to this question (http://stats.stackexchange.com/questions/305/when-conducting-a-t-test-why-would-one-prefer-to-assume-or-test-for-equal-varia) where i refer to a paper advocating the use of Welch in case of non-normality and heteroscedasticity." CreationDate="2010-08-09T13:09:08.537" UserId="442" />
  <row Id="1481" PostId="338" Score="2" Text="Also the paper &quot;Information Theoretic Inequalities&quot; by Dembo Cover and Thomas reveals a lot of deep aspects " CreationDate="2010-08-09T13:36:23.177" UserId="223" />
  <row Id="1482" PostId="6" Score="2" Text="Thanks @robin; made CW.  Although I don't entirely see this as &quot;argumentative&quot;; there are two fields which have informed each other (this is a fact), and the question is how much they have evolved together over the last decade." CreationDate="2010-08-09T14:17:51.890" UserId="5" />
  <row Id="1483" PostId="76" Text="@robin: must be a bug; since my answer (and several of the others) were given before the question was made CW." CreationDate="2010-08-09T14:37:42.607" UserId="5" />
  <row Id="1484" PostId="1441" Text="I see round 2 is more difficult :) Mathoverflow may give you a faster answer than stat.overflow... " CreationDate="2010-08-09T14:41:14.627" UserId="223" />
  <row Id="1485" PostId="1447" Text="your question is not about R vs R-square (you understand that $0.8^2=0.64$) it is about interpretation of $r^2$. Please reformulate the title. " CreationDate="2010-08-09T15:03:53.263" UserId="223" />
  <row Id="1486" PostId="1408" Text="Discrete wavelets -- or at least the vast majority of ones that are implemented and commonly used -- are typically designed to have useful frequency-based properties under the compact support constraint.  The vanishing moment condition of Daubechies, for instance, is more or less equivalent to flatness in the pass-band.  The frequency localization properties of wavelets are usually what lead the coefficients to be sparse representations and allow estimation of the noise variance under the &quot;signal + additive zero-mean noise&quot; assumption." CreationDate="2010-08-09T15:05:47.657" UserId="61" />
  <row Id="1488" PostId="1450" Text="@Brett: Upload your image to http://img84.imageshack.us/ and link to it. " CreationDate="2010-08-09T15:59:24.180" UserId="8" />
  <row Id="1489" PostId="1410" Text="Thanks Jeromy.  Just read the Wikipedia entry for MDS -- seems like it could lead somewhere." CreationDate="2010-08-09T16:24:53.383" UserId="251" />
  <row Id="1490" PostId="1452" Text="hmm, can't get the latex &quot;begin cases&quot; to accept new lines.  :-/" CreationDate="2010-08-09T16:44:36.263" UserId="251" />
  <row Id="1491" PostId="1450" Text="Great.  Done and thanks." CreationDate="2010-08-09T16:45:32.797" UserId="485" />
  <row Id="1492" PostId="1452" Text="@ars I fixed the eqns to use begin cases. I hope I did not mangle the eqns in the process." CreationDate="2010-08-09T16:55:57.247" />
  <row Id="1493" PostId="1386" Text="well, the problem is that I want a 1-sample test, not a 2-sample test! I am testing the null $E[X] = \mu$, and not $E[X_1] = E[X_2]$. I will look up the Kubinger et. al., paper (Ich kann schlecht Deutsche)." CreationDate="2010-08-09T17:02:43.710" UserId="795" />
  <row Id="1494" PostId="1452" Text="@Srikant Looks good, thanks!" CreationDate="2010-08-09T17:13:21.540" UserId="251" />
  <row Id="1495" PostId="1421" Text="I understand now why looking at the residuals directly might be useful.  In this case it would indicate where the model was beginning to fail.  I don't think I said that accuracy should go up at increased levels of difficulty (although that is what the quadradic model would imply).  I just said that accuracy in 2AFC will hit a floor after which it will not continue to go down as a function of increased difficulty.  This is a result of a fairly simple psychological process interacting with the 2AFC design." CreationDate="2010-08-09T17:16:52.797" UserId="196" />
  <row Id="1496" PostId="1421" Text="I'll grant that in the current model, it may make sense simply to drop the high difficulty levels that are giving the logistic model problems. Though if, as you say, you can trust your low accuracy measures better, then aren't you are advocating I throw away the very observations that should be most trusted?  I'll shelve the question for now and may come back to it in a couple months after I've had time to look further at the literature you've cited." CreationDate="2010-08-09T17:25:19.193" UserId="196" />
  <row Id="1497" PostId="1428" Text="Hi Rob,&#xA;&#xA;Thanks for the tip on BFAST - it looks very promising for my purpose! As I briefly mentioned, in my case the rolling aggregation doesn't actually remove the seasonality (there are slacks and surges in effort on a more-or-less predictable schedule, coupled with turnover in the measure denominator), and I have noticed that even with a lag(12) in my breakpoints() model specification, strucchange is a little more sensitive to these seasonal spikes than I'd prefer." CreationDate="2010-08-09T17:46:42.490" UserId="394" />
  <row Id="1498" PostId="1428" Text="Follow up q: Given this, is it fair to say that the pre-aggregation of the data may somewhat dilute (but not completely eliminate) the sensitivity of the BFAST approach to detect breaks in the seasonal component?" CreationDate="2010-08-09T17:47:52.847" UserId="394" />
  <row Id="1499" PostId="1457" Text="Have no idea why latex looks so bad. Can someone please help and fix this?" CreationDate="2010-08-09T18:05:55.353" />
  <row Id="1500" PostId="847" Text="[The question has been asked here](http://www.google.com/support/forum/p/websiteoptimizer/thread?tid=449ac5085f82912e&amp;hl=en), where Google employees patrol, and there has been no answer. Also I asked someone that was able to talk to the engineers to ask this question for me, and he said that they would not say; so I wonder.." CreationDate="2010-08-09T18:08:39.250" UserId="500" />
  <row Id="1501" PostId="1456" Text="I had the feeling it would be something of this sort - many thanks!" CreationDate="2010-08-09T18:13:10.487" UserId="253" />
  <row Id="1502" PostId="497" Text="@Thylacoleo This is a very crude basic and greedy method. Often you keep your validation set the same over runs, but whatever you like is ok. " CreationDate="2010-08-09T18:20:12.313" UserId="190" />
  <row Id="1503" PostId="1463" Text="Thanks, it's been very helpful.  I'll vote it up when I have enough rep." CreationDate="2010-08-09T19:23:41.040" UserId="148" />
  <row Id="1504" PostId="1458" Score="4" Text="not really an answer to your question, but have you encountered False Discovery Rates (FDR)? &quot;Beyond Bonferroni&quot; by Narum: http://www.springerlink.com/content/c5047h0084528056/" CreationDate="2010-08-09T20:21:27.547" UserId="291" />
  <row Id="1505" PostId="1203" Text="well - awarding the bounty to myself seems silly - but nobody else has submitted an answer." CreationDate="2010-08-09T21:01:31.007" UserId="196" />
  <row Id="1506" PostId="1468" Text="I think you have a typo in your hypotheses -- they both seem to be the same..." CreationDate="2010-08-09T21:50:44.347" UserId="174" />
  <row Id="1507" PostId="1428" Text="Pre-aggregation will remove or greatly reduce the seasonality. But I don't think that would reduce the ability of BFAST to detect breaks in the seasonal component. It is rare to have breaks in seasonality (at least in all the data I've looked at). Slow changes in seasonality are more common." CreationDate="2010-08-09T22:12:37.153" UserId="159" />
  <row Id="1509" PostId="1474" Text="See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions the current consensus is that this is an appropriate question, if you disagree chime in at meta, don't torment the new question asker." CreationDate="2010-08-09T22:53:26.200" UserId="196" />
  <row Id="1510" PostId="1474" Text="ok, but you'll probably have a bigger audience to get responses for a pure R question (at least for now) -- hence, the &quot;probably&quot;. &quot;torment&quot; is a strong word eh? :)" CreationDate="2010-08-09T23:51:42.210" UserId="291" />
  <row Id="1511" PostId="1474" Text="Sure, torment is a bit strong; I'm sorry about that.  I was/am just frustrated with repeat offender users (of which you are not one) who persist in redirecting question askers to StackOverflow without having made any comment on meta (where the consensus seems pretty clear)." CreationDate="2010-08-10T00:15:39.493" UserId="196" />
  <row Id="1512" PostId="1397" Text="yes, that is the paper I am looking for, but I cannot find a version of it online. The R-code might be useful, but I am working in Matlab, and don't have time to do the translation (the R code looks nontrivial). I would even be willing to settle for the bibliography of this paper at the moment. " CreationDate="2010-08-10T00:18:02.327" UserId="795" />
  <row Id="1513" PostId="1390" Text="I am accepting the 'use a parametric test' advice. it doesn't exactly solve my question, but my question was probably too open-ended." CreationDate="2010-08-10T00:35:20.870" UserId="795" />
  <row Id="1514" PostId="1466" Text="Good point.  If it's not a normal distribution, how do I go about finding  a different &quot;fit&quot; ?   (What terminology should I research?)" CreationDate="2010-08-10T00:41:22.523" UserId="832" />
  <row Id="1515" PostId="1465" Text="Thank you so much. This is very helpful!" CreationDate="2010-08-10T00:54:57.547" UserId="834" />
  <row Id="1516" PostId="1461" Text="Thank you so much. It seems that the log transformation is widely practiced in economics. It is not part of economic theory but is widely accepted. I appreciate your very useful response. " CreationDate="2010-08-10T00:56:28.027" UserId="834" />
  <row Id="1518" PostId="1435" Text="Reg images: Use one of the free image hosting sites (search google) , upload the plot to that site and link it here." CreationDate="2010-08-10T01:05:50.377" />
  <row Id="1519" PostId="1435" Text="I've corrected an error in my original answer. I first wrote p=logit(X*beta). In fact the predicted probability is the inverse logit of the linear combination, p=inv-logit(X*beta). In R this is calculated as p&lt;-plogit(X*beta), which is p=exp(X*beta)/(1+exp(X*beta))." CreationDate="2010-08-10T01:07:25.020" UserId="521" />
  <row Id="1520" PostId="1471" Score="2" Text="We're happy to take R questions here. See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions" CreationDate="2010-08-10T01:22:32.630" UserId="159" />
  <row Id="1521" PostId="1385" Score="2" Text="There's also &quot;reduced-model&quot; approach where you train a classifier for every pattern of missing values encountered during testing. IE, to make prediction for x where i'th attribute is missing, remove i'th attribute from all instances of training data and train on that. http://jmlr.csail.mit.edu/papers/v8/saar-tsechansky07a.html" CreationDate="2010-08-10T01:27:11.713" UserId="511" />
  <row Id="1522" PostId="1452" Text="@ars. I think the `boxcox` function in MASS only estimates $\lambda_1$ and assumes $\lambda_2=0$." CreationDate="2010-08-10T01:27:33.897" UserId="159" />
  <row Id="1523" PostId="1448" Text="I do appreciate your attempt at helping, but unfortunately, this just made things 10x worse.  Are you really introducing trigonometry to explain r^2?  You're way too smart to be a good teacher!" CreationDate="2010-08-10T01:49:49.367" UserId="832" />
  <row Id="1524" PostId="1450" Text="&gt; The ratio between the variation explained and the original variation is your R^2&#xA;&#xA;Let's see if I got this.  &#xA;&#xA;&#xA;&#xA;If the original variation from mean totals 100, and the regression variation totals 20, then the ratio = 20/100 = .2   You're saying R^2 = .2 b/c 20% of the mean variation (red) is accounted for by the explained variation (green)  &#xA;&#xA;&#xA;&#xA;(In the case of r=1) If the original variation totals 50, and the regression variation totals 0, then the ratio = 0/50 = 0 = 0% of the variation from the mean (red) is accounted for by the explained variation (green)  I'd expect R^2 to be 1, not 0.&#xA;" CreationDate="2010-08-10T01:51:53.620" UserId="832" />
  <row Id="1525" PostId="1452" Score="1" Text="@Rob: Oh, sorry.  Diggle's geoR is the way to go -- but specify `lambda2=TRUE` in the arguments to `boxcox.fit`.  (Also updated the answer.)" CreationDate="2010-08-10T02:01:31.553" UserId="251" />
  <row Id="1530" PostId="1450" Text="R^2 = 1-(SSR/SST) or (SST-SSR)/SST. So, in your examples, R^2=.80 and 1.00. The difference between the regression line and each point is that left UNexplained by the fit. The rest is the proportion explained. Otherwise, that's exactly right. " CreationDate="2010-08-10T04:17:49.067" UserId="485" />
  <row Id="1531" PostId="1450" Text="I edited that last paragraph to try to make it a bit clearer. Conceptually (and computationally) all you need is there. It might be clearer to actually add the formula and refer to the SST SSE and SSR, but then I was trying to get at it conceptually" CreationDate="2010-08-10T04:18:19.300" UserId="485" />
  <row Id="1532" PostId="1468" Text="@walkytalky: whooops, thanks ! " CreationDate="2010-08-10T05:11:21.430" UserId="223" />
  <row Id="1533" PostId="498" Text="SAS questions (and questions related to other statistical software) are now allowed. See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions for discussion." CreationDate="2010-08-10T05:36:45.163" UserId="159" />
  <row Id="1534" PostId="1469" Score="1" Text="You need to compute the Fisher matrix ? what is your model ? I am sure you can make your question more precise.  " CreationDate="2010-08-10T05:50:21.163" UserId="223" />
  <row Id="1535" PostId="1466" Text="@stats: See my edit." CreationDate="2010-08-10T06:53:52.817" UserId="56" />
  <row Id="1536" PostId="1488" Text="Weka's URL is http://www.cs.waikato.ac.nz/ml/weka/ (due to stack exchange paranoia, I cannot post more than one link in an answer)" CreationDate="2010-08-10T07:00:00.887" UserId="213" />
  <row Id="1537" PostId="1489" Text="Would you please give a link or two to explanation of data standardization?" CreationDate="2010-08-10T07:05:58.687" UserId="213" />
  <row Id="1538" PostId="1448" Text="I thought that you wanted to know why correlation^2 = R^2. In any case, different ways of understanding the same concept helps or at least that is my perspective." CreationDate="2010-08-10T08:47:06.010" />
  <row Id="1539" PostId="1493" Score="3" Text="your gaussian variables are iid ? " CreationDate="2010-08-10T08:56:47.737" UserId="223" />
  <row Id="1540" PostId="1478" Text="Could you give a couple of examples of what your code does?" CreationDate="2010-08-10T09:42:11.907" UserId="8" />
  <row Id="1541" PostId="1497" Text="Does this book provide the code to reproduce the graphics that it shows? (I think it's S that is used)" CreationDate="2010-08-10T11:08:22.910" UserId="339" />
  <row Id="1542" PostId="1483" Text="Raw data is from a 2 x 2 diagnostic contingency table. DOR equation = (TP*TN)/(FP*FN)" CreationDate="2010-08-10T11:10:51.487" />
  <row Id="1543" PostId="1386" Text="Thanks for clarifying. In this case the Kubinger paper will not be very helpful to you. I am Sorry." CreationDate="2010-08-10T11:18:25.560" UserId="442" />
  <row Id="1544" PostId="1472" Text="Are you working on the automated procedure?  :)" CreationDate="2010-08-10T13:31:06.803" UserId="5" />
  <row Id="1545" PostId="1501" Score="3" Text="possible duplicate of [Locating freely available data samples](http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples)" CreationDate="2010-08-10T13:55:30.473" />
  <row Id="1546" PostId="1450" Text="ie: R^2 is the proportion of the total variation from mean (SST) that is the difference b/w the expected regression value and mean value (SSE).  In my example of hours vs. score, the regression value would be the expected test score based on correlation with hours studied.  Any additional variation from that is attributed to SSR.  For a given point, hours studied variable/regression explained x% of the total variation from the mean (SST).  With a high r-value, &quot;explained&quot; is big percentage of SST compared to SSR.  With a low r-value, &quot;explained&quot; is a lower percentage of SST compared to SSR." CreationDate="2010-08-10T13:56:30.207" UserId="832" />
  <row Id="1547" PostId="1467" Text="I don't know R syntax.  I almost need pseudo-code to get the gist.   I will google some of the concepts you used.  Thanks!" CreationDate="2010-08-10T13:58:05.820" UserId="832" />
  <row Id="1548" PostId="1501" Text="Maybe that going into a more fused and detailed catalogue such as Free data set for very high dimensional classification: http://stats.stackexchange.com/questions/973/free-data-set-for-very-high-dimensional-classification could be an idea but otherwise I also feel this is a duplicate. " CreationDate="2010-08-10T14:17:26.180" UserId="223" />
  <row Id="1549" PostId="1498" Text="Thank you, that's exactly what I was trying to avoid (having a lot of samples as baseline), because I would like a really reactive approach (e.g. online detection, maybe &quot;dirty&quot;, after 1-2 weeks of baseline)" CreationDate="2010-08-10T14:49:29.207" UserId="667" />
  <row Id="1550" PostId="1470" Text="This is also definitely worth citing: http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg" CreationDate="2010-08-10T14:59:40.647" UserId="582" />
  <row Id="1551" PostId="1482" Text="Thanks for the suggestion, but the ability to choose my own measure of distance is critical, so this won't work to me. Someone else may find it useful, though." CreationDate="2010-08-10T15:06:57.837" />
  <row Id="1552" PostId="1484" Text="Thanks, but is there any benefit to using clusterfly rather than calling ggobi directly? The website only mentions clustering methods, which are interesting, but not my primary goal just yet. gcexplorer has less informative website, but looks like it is for visualizing data after it has already been split to clusters. I will give them a try once I get to that point, but not what I need right now." CreationDate="2010-08-10T15:12:13.177" />
  <row Id="1553" PostId="1491" Text="Thanks for the suggestion, @Shane. ggobi looks promising, I am installing it right now and will give it a try :)" CreationDate="2010-08-10T15:12:59.887" />
  <row Id="1554" PostId="1502" Text="@Shelby, I thought of this but there doesn't seem to be a reasonable cut point, and domain experts can't justify one either." CreationDate="2010-08-10T15:15:48.673" />
  <row Id="1555" PostId="1500" Text="Looks like a useful program, but their webpage does not do a good job of convincing me it will solve this exact problem. It looks like it may be too broad, too many features I don't care about, making it hard to do the simple things. I will give it another look if the other choices don't work out." CreationDate="2010-08-10T15:19:55.883" />
  <row Id="1556" PostId="1499" Text="Here's some criticism of that paper in 4 from the guy who runs eager eyes:&#xA;http://junkcharts.typepad.com/junk_charts/2010/05/8-red-flags-about-the-useful-chartjunk-paper.html&#xA;&#xA;http://junkcharts.typepad.com/junk_charts/2010/05/8-red-flags-about-the-useful-chartjunk-paper.html" CreationDate="2010-08-10T15:21:27.100" UserId="287" />
  <row Id="1557" PostId="1488" Text="Orange's features page says 'under construction' and they don't list screenshots like what I am doing. weka has no features list at all. They may be able to do what I want, but if they don't promote the feature, how can I tell. I am more convinced by the other choices." CreationDate="2010-08-10T15:24:40.237" />
  <row Id="1558" PostId="1501" Score="1" Text="I voted to close as duplicate." CreationDate="2010-08-10T15:36:09.417" UserId="5" />
  <row Id="1559" PostId="1501" Text="@Shane if I could I would do the same" CreationDate="2010-08-10T16:01:36.483" UserId="223" />
  <row Id="1560" PostId="1501" Text="I already voted to close. Perhaps, one of the of the moderators could close this question." CreationDate="2010-08-10T16:16:12.043" />
  <row Id="1561" PostId="1507" Text="Why would you calculate the probability of exactly 39 in case 1, but 39 or higher in case 2?" CreationDate="2010-08-10T16:20:35.893" UserId="196" />
  <row Id="1562" PostId="1507" Text="@drknexus Case 2 is a continuous approximation of case 1. Thus, it does not make sense to talk about P(Z=10.3) in case 2." CreationDate="2010-08-10T16:24:29.650" />
  <row Id="1563" PostId="1478" Text="briefly `pv = AD_test_uni(xs)` takes a sample vector `xs`, which are restricted to the range $[0,1]$ and returns a p-value, `pv` under the null that `xs` are drawn from the Uniform distribution. I can then use this elsewhere to test other tests which spit out a p-value." CreationDate="2010-08-10T16:28:49.007" UserId="795" />
  <row Id="1564" PostId="1479" Text="There is indeed an infinite regress problem here. However, it is certainly not the case that all external libraries are error-free (I have found such errors in the past). Because all of my tests ultimately rest on this A-D test, I thought it might be worthy of extra scrutiny." CreationDate="2010-08-10T16:44:51.017" UserId="795" />
  <row Id="1565" PostId="1502" Text="I would think this could be fairly arbitrary for your stated purpose - honestly, you might not even need to actually cut into binary, just recode a tie value label on a scale of 1 to some manageable number, then progressively hide/show the ties at various levels (optionally also hiding/eliminating any pendants &amp; orphans along the way).&#xA;&#xA;Not directly responding to your request as written, but why not take a more typical approach and use a hybrid clustering method that doesn't use initial centroids to identify preliminary clusters, then feed the centroids from that result into your new analysis?" CreationDate="2010-08-10T16:45:55.197" UserId="394" />
  <row Id="1566" PostId="1479" Text="Perhaps, you can use multiple external libraries whose code base does depend on each other. It is unlikely that independent libraries have the same errors which should eliminate some of your concerns." CreationDate="2010-08-10T16:49:52.710" />
  <row Id="1567" PostId="1491" Score="1" Text="Works fine on other platforms, but gtk does not play nice with OSX. " CreationDate="2010-08-10T17:02:53.457" />
  <row Id="1568" PostId="1502" Text="@Shelby, I am guessing you mean to try for many different cutoffs until I see some nice results? I wish to avoid that for standard multiple comparisons reasons. re: your second suggestion I guess I just trust myself better than those algorithms. I use the computer to process large amounts of data too tedious to do by hand, not to replace my thinking." CreationDate="2010-08-10T17:06:47.270" />
  <row Id="1569" PostId="1493" Text="What kind of answer are you looking for?  A name?  A simple formula for the density?" CreationDate="2010-08-10T17:23:56.213" UserId="89" />
  <row Id="1570" PostId="1499" Text="very interesting. thx for pointing this one out!" CreationDate="2010-08-10T17:34:56.570" UserId="22" />
  <row Id="1571" PostId="1501" Text="It's just my two cents, but I think if we limit stats.se to 1 datasets question it makes it harder to share that knowledge.  I'm sorry if I didn't word my question/explanation better to differentiate this question, but I foresee there being lots of 'what open dataset exists to do x?' where x is what stops them from all being dupes." CreationDate="2010-08-10T17:55:45.410" UserId="114" />
  <row Id="1572" PostId="1503" Text="I have seen that question, and your answer was good (and I do rely on a lot of the datasets that ship with R) but not many of them include defined (or should I say well characterized) answers to particular statistical questions. (although I believe that if the dataset has been used in an example to a function in the helpfile, then the package will include the results of running that example, so you could use that)" CreationDate="2010-08-10T17:58:26.543" UserId="114" />
  <row Id="1573" PostId="1507" Text="@srikant-vadali By the common definition of p-value it should include more extreme cases in addition to the one observed. So the first calculation should be P(39 or 40). Still very small though." CreationDate="2010-08-10T18:05:53.237" UserId="279" />
  <row Id="1574" PostId="1507" Text="@Aniko I think we need to calculate p-values if the question involves inference about a parameter. But, the OP's qn is about the probability of observing a particular event. So, I think case 1 calculation is fine. If we want to compute the probability that we would see at least 39 such events then you would compute P(39 or 40) but perhaps I misinterpreted the OP's goals here. " CreationDate="2010-08-10T18:19:19.450" />
  <row Id="1575" PostId="1501" Text="I think it makes a lot of sense for this question to remain open if you could edit the question to indicate how your request is different from the one that exists and why the answers to the existing question do not satisfy your requirements. See Robin's comment as an example of what I am suggesting." CreationDate="2010-08-10T18:24:15.303" />
  <row Id="1576" PostId="1469" Text="I could give more details if that's of interest...as an overview, I'm trying to compare efficiency/bias of n-iteration Contrastive Divergence estimator for fitting Boltzmann Machines with an estimator that uses n iterations of Belief Propagation" CreationDate="2010-08-10T18:31:36.163" UserId="511" />
  <row Id="1577" PostId="1312" Text="I thought a bit more about what you said and it makes sense. I suppose one would write $f(Y_3|Y_1,Y_2)$ as $f(Y_3|X_{31},-) I(Y_1 Y_2&gt;0)$" CreationDate="2010-08-10T18:55:01.157" />
  <row Id="1578" PostId="1511" Text="I do not understand your answer. Suppose, you do not know P(H) and you have to estimate it. You have to write Prob(4H, 1T 39 times out of 40) in terms of P(H) and estimate P(H). Your answer suggests that the probability of observing the event in question is always 1 and hence independent of P(H). I am not sure if that makes sense. Did I miss something?" CreationDate="2010-08-10T19:13:43.613" />
  <row Id="1579" PostId="1507" Text="@srikant-vadali Then per drknexus the second calculation should also do the same, something like P(38.5&lt;X&lt;39.5). " CreationDate="2010-08-10T19:27:03.677" UserId="279" />
  <row Id="1580" PostId="1511" Score="2" Text="The student observed a result that appears odd.  But once it's observed it's no longer odd, it simply is random chance.  The probability of observed data occurring after you've already observed it is always 1.0.  I put the smiley in because I recognize that you want to know if it's plausible.  But that would involve not only working out the probability of it occurring once but the probability of it occurring given all the students who've ever attempted it and the number of attempts they made.  (also, consider that 4 heads and 1 tail in 39 of 40 attempts has the same probability)" CreationDate="2010-08-10T19:36:42.093" UserId="601" />
  <row Id="1581" PostId="1507" Text="@Aniko Perhaps, that would be a better choice." CreationDate="2010-08-10T19:40:51.413" />
  <row Id="1582" PostId="1511" Text="How does what you say reconcile with the estimation problem I posed in my comment? " CreationDate="2010-08-10T19:42:03.110" />
  <row Id="1583" PostId="1511" Score="1" Text="First, I was making a stats joke. But there is a serious point. If you believe the student then the probability is 1.0 because the data have already been observed." CreationDate="2010-08-10T19:52:38.340" UserId="601" />
  <row Id="1584" PostId="1507" Text="Is this a statistics assignment?  If so, then if you are testing the student's plausibility you need to calculate the number of students you've ever assigned this task to and multiply the result by that." CreationDate="2010-08-10T19:54:25.717" UserId="601" />
  <row Id="1585" PostId="1511" Text="But, that is exactly the point isn't it? The OP does not trust the student and wants to make an assessment of how likely the event is under reasonable assumptions (P(H)=0.5, independence etc). " CreationDate="2010-08-10T20:00:30.320" />
  <row Id="1586" PostId="1507" Text="Or do I?  I'd think I can just base the calc on theoretical probability of the fair coin, and ignore Experimental Probability of what students got?  " CreationDate="2010-08-10T20:34:15.700" UserId="832" />
  <row Id="1587" PostId="1507" Text="Yes, in case 1, I could have done P(30 or 40), but I would have just added another 0%, so I skipped it.  I used the inequality in case 2 b/c I was interested in the odds of 39 or anything more extreme." CreationDate="2010-08-10T20:35:37.507" UserId="832" />
  <row Id="1588" PostId="1511" Text="John, I got the joke.  LOL" CreationDate="2010-08-10T20:36:13.667" UserId="832" />
  <row Id="1589" PostId="740" Text="I'm confused by the question -- &quot;this variable also has&quot;...which variable?" CreationDate="2010-08-10T21:20:40.343" UserId="511" />
  <row Id="1590" PostId="538" Text="They're not equivalent in the face of interventions." CreationDate="2010-08-10T22:35:51.347" UserId="858" />
  <row Id="1591" PostId="571" Score="1" Text="Might as well use Pearl's word for &quot;directly setting [a variable's] value&quot;: an intervention." CreationDate="2010-08-10T22:36:53.263" UserId="858" />
  <row Id="1592" PostId="1470" Text="Awesome poster - thanks for linking to it!" CreationDate="2010-08-10T22:48:58.863" UserId="561" />
  <row Id="1593" PostId="1517" Text="You could learn lot of things but for useful suggestions you should ideally let us know what kinds of questions you like to ask and what kind of answers do you want to those questions. Some exemplars would really help as otherwise your question is too broad." CreationDate="2010-08-10T22:57:41.880" />
  <row Id="1594" PostId="1517" Text="My thought was that the tools I'm seeking are generic.  Standard Deviation is a generic tool.  What's another one like it?  The basic goal is to remove noise from data.  A bit later I will provide more about why Inventory Turn Rate is difficult for us and why I think more stats knowledge would help.  " CreationDate="2010-08-10T23:15:43.580" UserId="857" />
  <row Id="1595" PostId="1507" Text="No, you can't ignore how many students do the task.  If you have a 1000 students doing the same assignment and one of them reports a 1:1000 event then that's not terribly unusual.  If you have 200 students a year for the last 10 years and one reports a 1:2000 event it's not unusual... even a higher on like 1:20000, which is now 1:10 is not unusual.  This is similar to the Bulgarian lottery problem   &#xA;(of course this is just a theoretical argument... the probability here is far too low for it to matter)" CreationDate="2010-08-10T23:47:55.430" UserId="601" />
  <row Id="1596" PostId="1518" Text="Linear Regression is precisely what I'm looking for to deal with one problem (forecasting sales from current customer inquiries).  It looks difficult, but I will read more.  I can't tell about Comparing Means.  Thank you." CreationDate="2010-08-10T23:52:10.180" UserId="857" />
  <row Id="1597" PostId="1518" Score="1" Text="&quot;Forecasting sales from customer inquiries&quot; is the type of example that you should have mentioned in the question as that enables a better response. Another tool that helps in this context is correlation coefficient (See: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient). This concept is closely related to linear regression." CreationDate="2010-08-11T00:37:42.773" />
  <row Id="1598" PostId="1518" Score="1" Text="Likewise, you should indicate whether you are interested in individual sales--will this customer buy--or aggregate sales--how much will we sell this quarter--as those will require different models." CreationDate="2010-08-11T03:22:30.037" UserId="485" />
  <row Id="1599" PostId="1501" Text="I like your question (presumably after being edited).&#xA;It sounds like your question is not especially about datasets. Rather its about the resources and infrastructure for comparing reference results to a new implementation. &#xA;" CreationDate="2010-08-11T04:53:40.700" UserId="183" />
  <row Id="1600" PostId="847" Text="How did I not answer your question? Read the JMP book. It's free. " CreationDate="2010-08-11T06:33:44.667" UserId="74" />
  <row Id="1601" PostId="1523" Score="1" Text="What's the duplicate? I can't see anything obvious." CreationDate="2010-08-11T06:43:03.150" UserId="159" />
  <row Id="1602" PostId="1523" Score="1" Text="Pretty similar to this one I thought: http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning" CreationDate="2010-08-11T07:25:58.733" UserId="74" />
  <row Id="1603" PostId="1521" Score="1" Text="duplicate? http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning" CreationDate="2010-08-11T07:26:33.567" UserId="74" />
  <row Id="1604" PostId="1523" Score="1" Text="ok. I was searching for data mining, not machine learning. Please vote to close if you think it is a duplicate." CreationDate="2010-08-11T07:36:37.613" UserId="159" />
  <row Id="1605" PostId="1525" Text="I'm just wondering whether the edited version of this question should retain the aspect of the original question regarding how the distinction between probabilities and proportions could be described in lay terms." CreationDate="2010-08-11T08:26:37.557" UserId="183" />
  <row Id="1606" PostId="1527" Score="8" Text="+1 for proportion is empirical, and it is often a good estimate of a probability which is theoretical ! " CreationDate="2010-08-11T08:38:14.050" UserId="223" />
  <row Id="1607" PostId="944" Score="1" Text="Following on from http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions I've reopened this question." CreationDate="2010-08-11T10:42:09.300" UserId="8" />
  <row Id="1608" PostId="1160" Text="Following on from http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions I've reopened this question." CreationDate="2010-08-11T10:43:21.717" UserId="8" />
  <row Id="1609" PostId="1523" Text="Hmm, so Data Mining == Machine Learning?  " CreationDate="2010-08-11T10:47:42.087" UserId="251" />
  <row Id="1610" PostId="1531" Score="7" Text="You need to change your question as many people won't (and shouldn't) run a random excel file on their computer. Pull out the formula from the file and ask a question about that." CreationDate="2010-08-11T11:06:49.453" UserId="8" />
  <row Id="1611" PostId="1502" Score="1" Text="You're using hypothesis testing language but yet talking about a very exploratory, know-it-when-you-see-it approach @ the same time - so it's not clear what your goal really is for this part of your analysis. If you have hypotheses you're testing later (e.g. predicting cluster membership or using clust membership as predictor) then you can choose not to do things that will tempt bias there. But &quot;multiple comparison&quot; issues don't really figure into the exploratory problem you're describing. The viz cutoffs are just to help you see what's there - but your trust may still be misplaced." CreationDate="2010-08-11T11:48:50.397" UserId="394" />
  <row Id="1612" PostId="1517" Text="After learning the basics, it would be quite helpful if you use some tool i.e. Matlab, R, Excel to view your results in a more human understandable format." CreationDate="2010-08-11T12:01:36.487" UserId="734" />
  <row Id="1613" PostId="303" Text="I guess you can put $$ now ? " CreationDate="2010-08-11T12:51:26.480" UserId="223" />
  <row Id="1614" PostId="1535" Text="@Dirk: Thanks. I completely agree with your point and changed my question." CreationDate="2010-08-11T13:07:08.347" UserId="8" />
  <row Id="1616" PostId="1521" Text="If it's duplicate I can guess that data mining and machine learning are the same thing! " CreationDate="2010-08-11T14:16:22.953" UserId="339" />
  <row Id="1617" PostId="1532" Text="This is a great paper and is consistent with my perspective on what data mining is and how it differs from statistics.  The catch is, it's from 1997!  Note an indictment of the paper or your recommendation, but the degree to which I have kept up with data mining.  It sounds like I need to grab a current book on data mining to catch up." CreationDate="2010-08-11T14:19:58.720" UserId="485" />
  <row Id="1619" PostId="1522" Text="I think points 2 and 3 are useful comments and consistent with what I see as the distinction between the two SA and DM.  I'm not so sure about your first point.  I've done statistical work where I was interested in improving understanding about causal relationships.  However, I've also done statistical work where the task was to take known relationships and develop models with the sole purpose of prediction but which shared non of the other features of &quot;data mining&quot;. " CreationDate="2010-08-11T14:39:04.117" UserId="485" />
  <row Id="1620" PostId="1526" Text="I can buy that.  Data mining is more exploratory application of statistical techniques.  Though, I don't think that distinction is enough.  When I'm doing EDA on my set of 100 observations from a designed experiment, I don't think anyone would call that data mining, would they?" CreationDate="2010-08-11T14:42:14.797" UserId="485" />
  <row Id="1621" PostId="1523" Text="1) I'm not seeing the comp stat distinction. There's not much that statisticians do that doesn't require a computer. I suppose you mean computationally intensive procedures such as iterative solutions, etc? But then, these are also common in modern statistical work that is not data mining. 2) In my own (stats) work, I've been interested in model building for explanation and for prediction, depending on the problem-I wouldn't have considered that data mining. 3) I'm left with the conclusion that modern DM is a particular application of statistics, which I think is a fine conclusion." CreationDate="2010-08-11T14:44:15.727" UserId="485" />
  <row Id="1623" PostId="1509" Text="A really helpful link, thank you." CreationDate="2010-08-11T14:59:37.967" UserId="667" />
  <row Id="1625" PostId="1539" Text="@robin girard: Shouldn't we apply the CLT to all finite-dimensional marginal distributions?" CreationDate="2010-08-11T15:26:01.897" UserId="650" />
  <row Id="1626" PostId="1470" Text="@propofol: I'm sure they had lots of fun asking a dead salmon about its emotions!!!" CreationDate="2010-08-11T15:48:04.470" UserId="582" />
  <row Id="1627" PostId="1551" Text="About N.B.2 - I think you're exactly right regarding the connotation of data mining and I had not made the connection to machine learning.  My training always emphasized the problems of over-fitting, spuriousness, and capitalizing on chance and as such I have been skeptical on DM--and still am, perhaps until someone actually tells me WHAT they are doing and HOW.  Thanks." CreationDate="2010-08-11T16:05:00.513" UserId="485" />
  <row Id="1628" PostId="1539" Text="you asked for an intuitive answer :) also I choose not to bother you with the tricky mathematical part which is to show that the convergence for all t implies the convergence (in law) of the supremum... do you want me to complete the answer ? " CreationDate="2010-08-11T16:11:46.967" UserId="223" />
  <row Id="1629" PostId="1551" Score="1" Text="My only quibble on the ML/DM distinction would be that I think DM is broader.  For example, OLAP and related tools include mining technologies.  But these come from the database side of computer science rather than machine learning.  The role of commerce in shaping the &quot;meaning&quot; of data mining is hard to ignore -- it brings in elements of management sciences, operations research, machine learning and statistics as required.  It also gives the impression of something flimsy, but that's usually a problem for purists not practitioners." CreationDate="2010-08-11T16:47:59.927" UserId="251" />
  <row Id="1630" PostId="1539" Text="Dear robin girard, I think your answer is fine as it stands. Thank you!" CreationDate="2010-08-11T17:16:09.460" UserId="650" />
  <row Id="1631" PostId="1532" Text="Heh, I kept the date out on purpose because I thought it would be amusing to notice the time span.  :)  The books by Michael Berry and Gordon Linoff are pretty good and will appeal to statisticians (for the broader exposure rather than learning statistical techniques).  If you want a sense of the fuzzy, &quot;enterprise&quot; side of this field, skimming through one of the books on a vendor product, like SAS's Enterprise Miner or SPSS's Clementine, may help.  I wouldn't recommend buying them unless you're going to work with the product itself." CreationDate="2010-08-11T17:50:21.430" UserId="251" />
  <row Id="1633" PostId="1338" Score="1" Text="@Tal Galili: http://en.wikipedia.org/wiki/Love_means_never_having_to_say_you%27re_sorry. (My favorite is the line from *What's Up, Doc?*)" CreationDate="2010-08-11T19:23:58.057" UserId="877" />
  <row Id="1634" PostId="1551" Text="@ars: I agree.  I was trying to say that a little by saying &quot;machine learning techniques are used in data mining&quot; (i.e. data mining is a super-set).  Your point about the commercial applications is also spot on.  Although someone in a commercial application now-a-days might refer to their work as something else (e.g. &quot;data science&quot;).  " CreationDate="2010-08-11T19:36:59.117" UserId="5" />
  <row Id="1635" PostId="1551" Text="@Shane: Right, I should have said I was trying to flesh out the differences, rather than actually quibble with what you wrote.  Apologies for the misdirection.  Good point on changing times and terms like the adoption of &quot;data science&quot;.  Doesn't one of Gelman's books start with something like &quot;statistics is the science of data&quot;?  So &quot;they're&quot; stealing from statisticians.  Again. :)" CreationDate="2010-08-11T20:39:16.520" UserId="251" />
  <row Id="1636" PostId="1555" Score="2" Text="You need to explain the data structure more, otherwise it is not clear how would a chi-square test apply. What do the values represent? Are the rows meaningful?" CreationDate="2010-08-11T21:45:06.457" UserId="279" />
  <row Id="1639" PostId="1282" Text="@Aniko Just implemented this. Worked like a dream. Thanks so much!" CreationDate="2010-08-11T23:31:13.757" UserId="287" />
  <row Id="1640" PostId="1031" Text="This seems to be a characterization of directional derivative of KL rather than of KL itself, and it doesn't seem possible to get KL divergence out of it because unlike the derivative, KL-divergence doesn't depend on the geometry of the manifold" CreationDate="2010-08-11T23:37:49.470" UserId="511" />
  <row Id="1641" PostId="1304" Text="+1 thanks for the link to the Feldman/Cousins paper." CreationDate="2010-08-12T00:08:51.970" UserId="251" />
  <row Id="1642" PostId="1572" Text="the priors are distributed normal N(b0, B0^-1). note above, that I use a precision (B0) of .00001, which is pretty diffuse." CreationDate="2010-08-12T00:59:36.530" UserId="291" />
  <row Id="1643" PostId="1572" Score="1" Text="Actually, I think one other factor that matters is sample size. I am not sure if you can expect identical values with a sample size of just 15 odd points. With only 15 points there is not enough information in the likelihood to outweigh the prior. In other words, the likelihood is not 'concentrated enough' which suggests that when you integrate out the parameters you may get a different mcmc marginal likelihood. Does that make any sense? You can do a small simulation to check the above." CreationDate="2010-08-12T01:13:30.637" />
  <row Id="1644" PostId="1573" Text="your paper will take time to digest. it looks like the -logLik changes fairly significantly if I changed the number of digits in the prior precision. MCMCpack doesn't let me specify other priors (aside from uniform), so I'm going to try another package. btw, what is a &quot;disjoint support?&quot;" CreationDate="2010-08-12T01:55:20.850" UserId="291" />
  <row Id="1645" PostId="1573" Text="On second though, my paper may not be so important here. But the point about the priors is important.  Look at the differences in the priors at the empirical mean of the data.  If that difference is large, that could explain your problem." CreationDate="2010-08-12T02:10:24.543" UserId="319" />
  <row Id="1646" PostId="1573" Text="As for the paper, disjoint support means there is no region to which both hypotheses assign positive probability.  i.e. the alternative really is an alternative." CreationDate="2010-08-12T02:12:21.450" UserId="319" />
  <row Id="1647" PostId="1573" Text="as I changed the precision of the prior from .01, .001, .0001, .00001, .000001, the -logLik were: -0.85, -2.00, -3.15, -4.31, -5.46 respectively. All the estimates (mean of mean, sd; sd of mean, sd) were the same at least to the 8th digit. (i used mean of 0.38 ~ the sample avg)" CreationDate="2010-08-12T02:23:19.783" UserId="291" />
  <row Id="1648" PostId="1412" Text="BTW Brian D. Ripley thinks it is a problem too: https://stat.ethz.ch/pipermail/r-help/2006-December/122353.html" CreationDate="2010-08-12T02:27:04.893" UserId="196" />
  <row Id="1649" PostId="1573" Text="@John Why would the priors matter as sample size goes to infinity? Asymptotically, the likelihood function 'collapses' (technically, a dirac delta function) at the true parameter vector. Thus, it seems to me that the effect of the prior on marginal likelihood should weaken as sample size increases and should vanish asymptotically. Is the above intuition not correct?  " CreationDate="2010-08-12T03:12:07.187" />
  <row Id="1650" PostId="1525" Score="6" Text="If you eat Hamburgers _every_ Tuesday, the probability of you eating a hamburger in any given week is 1." CreationDate="2010-08-12T03:28:16.310" UserId="776" />
  <row Id="1651" PostId="1573" Score="1" Text="it looks this problem has been widely known since 1939, called Jeffrey-Lindley paradox. Gelman's book (p185 &amp; 250) also says uninformative priors w/ Bayes Factors are a no-no. Basically, they say that formulating point null hypotheses are stupid, and tests should be constructed from the posteriors. http://www.artsci.uc.edu/collegedepts/economics/research/docs/Wppdf/2009-01.pdf" CreationDate="2010-08-12T04:50:06.480" UserId="291" />
  <row Id="1652" PostId="1581" Text="Thanks for the answer. I had these thoughts.&#xA;(a) Multicollinearity: I agree. Wihtout it, the coefficients should not change. (b) Is it interesting? I actually think that the sign flipping can have interesting theoretical interpretations in some instances; but perhaps not from a pure prediction perspective. (c) Residualisation: I'd be keen to hear what other people think of this approach." CreationDate="2010-08-12T06:05:59.343" UserId="183" />
  <row Id="1653" PostId="1581" Text="I'm not sure if multicollinearity could be interesting. Say you had some outcome `O`, and your predictors are `Income` and `Father's Income`. The fact that `Income` is correlated with `Father's Income` is intrinsically interesting, but that fact would be true no matter the value of `O`. That is, you could establish that `O`'s predictors are all collinear without ever collecting your outcome data, or even knowing what the outcome is! Those facts shouldn't get especially more interesting once you know that `O` is really `Education`." CreationDate="2010-08-12T06:25:19.883" UserId="287" />
  <row Id="1654" PostId="1581" Text="I'm suggesting that the suppressor effect can be theoretically interesting, of which presumably multicollinearity provides a starting point for an explanation." CreationDate="2010-08-12T06:35:33.963" UserId="183" />
  <row Id="1655" PostId="1559" Score="1" Text="Hmmmm... maybe you should have a look at http://en.wikipedia.org/wiki/Percentage 1 and 100% ARE the same, as are 0.35 and 35% or 2.24 and 224%." CreationDate="2010-08-12T06:42:33.890" UserId="582" />
  <row Id="1657" PostId="1413" Text="It is unfortunate that nobody could chime in with a package where this is easy to do.  However, it turned out that adding the appropriate functions to lme4 wasn't as difficult as I had feared.  Now the only challenge that remains is to find the &quot;correct&quot; link function (see: http://stats.stackexchange.com/questions/1583/appropriate-link-function-for-2afc-data).  So I'll accept ars' answer, but someone may want to ask this again in the future since next time the answer may be different." CreationDate="2010-08-12T07:09:59.960" UserId="196" />
  <row Id="1658" PostId="1582" Text="Thanks for the suggestion to explore ridge or PCA regression.&#xA;Just a side point regarding your comment &quot;if your variables are positively correlated, then the coefficients will be negatively correlated leading to sign reversal.&quot;: positively correlated predictors do not typically lead to sign-reversal." CreationDate="2010-08-12T07:11:15.230" UserId="183" />
  <row Id="1659" PostId="1432" Text="There are elements to this question that remain unanswered, e.g. the nature of the &quot;pearson&quot;, &quot;working&quot;,&quot;response&quot;, and &quot;partial&quot; residuals, but for now I will accept Thylacoleo's answer." CreationDate="2010-08-12T07:12:16.117" UserId="196" />
  <row Id="1660" PostId="1335" Text="I'm accepting this answer because it solves (what I think) is the harder of the problems I was looking for a solution to - picking a particular combination out without calculating the preceding values.  Unfortunately, it is still very slow.  Perhaps as mentioned here and elsewhere a binary search would help speed things up.  Perhaps the best approach is to have one thread generating the combinations stepwise as in mbq's answer and another thread reading them off and testing them." CreationDate="2010-08-12T07:15:40.647" UserId="196" />
  <row Id="1661" PostId="1582" Text="Sorry, that's a botched one line explanation written in haste.  Fixed now, thanks.  " CreationDate="2010-08-12T07:19:53.383" UserId="251" />
  <row Id="1662" PostId="1586" Text="Good point. All the examples of Simpson's Paradox apply to categorical variables. Is the concept of a supressor variable the numeric equivalent?" CreationDate="2010-08-12T07:55:46.557" UserId="183" />
  <row Id="1663" PostId="967" Text="I guess Val Harian is not a statistician if he is not kidding... what is a sexy job ? for me it is like the sitation with the sword of the century... fun but a bit trivial :)  " CreationDate="2010-08-12T07:58:10.670" UserId="223" />
  <row Id="1664" PostId="1585" Text="+1 for this paper ! my reference for the link between thresholding and penalized estimation was http://www.google.fr/url?sa=t&amp;source=web&amp;cd=1&amp;ved=0CB0QFjAA&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.8.6694%26rep%3Drep1%26type%3Dpdf&amp;ei=X6xjTILlHsiU4gashYThCg&amp;usg=AFQjCNE2kNFinNcuvG4qkBBPQAi2omY-9A&amp;sig2=D0NgkuN_KfgV8ZdzADKcxg" CreationDate="2010-08-12T08:11:28.517" UserId="223" />
  <row Id="1665" PostId="435" Score="1" Text="This one reminds me of the recent bailout in the States, where they just made up 700 billion number - they said they just wanted a really large number. :)" CreationDate="2010-08-12T08:53:40.443" UserId="144" />
  <row Id="1668" PostId="1493" Text="yes , they are iid" CreationDate="2010-08-12T09:49:25.610" UserId="852" />
  <row Id="1672" PostId="1595" Text="@csgillespie: Thanks for adding the python tag. :) I wanted to, but as a new user I couldn't." CreationDate="2010-08-12T10:48:58.367" UserId="890" />
  <row Id="1673" PostId="1595" Text="@Fabian: You're welcome!" CreationDate="2010-08-12T10:50:18.523" UserId="8" />
  <row Id="1674" PostId="1569" Text="+1 I like this interpretation! could you clarify &quot; p below e &quot;? why do you take small e ? you say &quot;the probability of making the opposite mistake is&quot; it is an upper bound  or exact probability? If I remember, this type of approach is due to Chernoff, do you have the references (I find your first reference is not clarifying the point :) ) ?   " CreationDate="2010-08-12T11:08:21.647" UserId="223" />
  <row Id="1675" PostId="1383" Text="Thank you for updating! you said: &quot;I'm talking about universal compression obviously&quot; even if things are obvious within your community you maximise the chance of having original instructive answer by recalling things we few words or a link.   " CreationDate="2010-08-12T11:31:11.297" UserId="223" />
  <row Id="1676" PostId="1383" Text="My personal view is that we should not mix comments and question. Comment are made to give feedback or to ask questions about the question, most often it is a dialogue between two people. The question should be something clear, easy to read and understandable by as much people as possible... it's you explaining your problem to get an answer. " CreationDate="2010-08-12T11:36:18.137" UserId="223" />
  <row Id="1678" PostId="1249" Text="@srikant an upper bound is needed here. The last sentence just states that an upper bound on the desired quantity will be used for proving the minimum error in the testing problem, as @robin explained. " CreationDate="2010-08-12T13:13:54.340" UserId="168" />
  <row Id="1679" PostId="1573" Score="1" Text="Srikant: The problem is that you have *two* posteriors becoming more concentrated and you're looking at their relative fit.  Either prior alone would become irrelevant in the limit, but the priors continue to impact the ratio of posterior model probabilities." CreationDate="2010-08-12T13:56:30.090" UserId="319" />
  <row Id="1680" PostId="1555" Text="@Aniko, the data structure is a frequency structure where each column represent a specific treatment and the rows represents result broken up into bins. " CreationDate="2010-08-12T14:31:34.373" UserId="559" />
  <row Id="1681" PostId="1598" Score="3" Text="Inserting shameless plug for Rcpp :)" CreationDate="2010-08-12T14:37:32.143" UserId="334" />
  <row Id="1682" PostId="1546" Text="i am trying to measure whether or not i am getting more visitors after i turn on SEO, and if possible to quantify that &quot;lift&quot;.  it's important to compare before and after on a per-account basis, and then say something about the comparison in aggregate.&#xA;&#xA;it is possible that there is a &quot;ramp up&quot; period after turning on SEO, so i'm not sure that just looking at the days where SEO is on and the days where it is off is sufficient (maybe an average within a window before and after turning it on?)" CreationDate="2010-08-12T14:44:45.760" UserId="125" />
  <row Id="1683" PostId="1589" Text="It is an interesting idea to fit the previous dataset here, it would tell me the extent to which the &quot;problem&quot; I'm worried about is solved by using these functions.  Also, I may fit existing psychometric data individually by subject using glm and produce the output curves on the probability scale.  Perhaps one will pass the eyeball test and the other won't.  I'll update the question when I know more." CreationDate="2010-08-12T14:49:59.117" UserId="196" />
  <row Id="1684" PostId="1589" Text="ASIDE: I'll grant that the model should match the process.  In the case of modeling successes and failures of the robot arm you propose, values past its maximum movement are non-nonsensical, consequently they are past the point of observed values; so if I did that I'd be committing another error, attempting to extrapolate beyond the range of observed values." CreationDate="2010-08-12T14:57:40.567" UserId="196" />
  <row Id="1685" PostId="1589" Text="ASIDE continued: However, in psychophysics we have a system that starts off not working at all, gradually starts being able to process the information provided to it, and then asymptotes near 100% when the information is clearly available.  Here I may have presented stimuli that are too hard to discriminate (beyond its range on the lower bound) or presented stimuli that are too easy to discriminate (beyond its range on the upper bound).  Here it is reasonable ask what the system will do at those extreme values." CreationDate="2010-08-12T15:03:25.373" UserId="196" />
  <row Id="1686" PostId="956" Text="We sort the test results in increasing order by their un-corrected original p-value, then, iterating over the list, calculate the FDR expected if we were to reject the null hypothesis for this and all tests prior in the list, using the B-H correction using an alpha equal to the observed, un-corrected p-value. We then take, as what we've been calling our &quot;q-value&quot;, the maximum of the previously corrected value (FDR at iteration `i - 1`) or the current value (at `i`), to preserve monotonicity. Does this sound like the procedure you described in your second paragraph?" CreationDate="2010-08-12T15:46:43.107" UserId="520" />
  <row Id="1687" PostId="1603" Text="Given that $z^2$ can be quite large, ignoring higher order terms in the Taylor expansion is pretty fishy." CreationDate="2010-08-12T15:48:20.983" UserId="89" />
  <row Id="1688" PostId="1603" Text="hmm on second thoughts I may have my inequality reversed. The bound I computed is a lower bound and not a upper bound. Please vote it down to oblivion. :-)" CreationDate="2010-08-12T15:53:35.007" />
  <row Id="1689" PostId="1605" Score="1" Text="I believe that the sums need to be normalized (e.g., use the mean score) for the distribution to tend to normality." CreationDate="2010-08-12T16:05:45.593" />
  <row Id="1690" PostId="870" Text="@robin Many thanks for your comments. I apologize for my confusion of terminology. I have updated the question to include a more complete description of our correction procedure, in the hopes that it provides clarification. I have also updated the q-value link; thanks for pointing me to that." CreationDate="2010-08-12T16:09:26.790" UserId="520" />
  <row Id="1691" PostId="1588" Text="I like his 2008 paper on FDR. I don't use R, though :(" CreationDate="2010-08-12T16:21:50.837" UserId="795" />
  <row Id="1692" PostId="1560" Text="I guess I'm wondering *why* the James-Stein estimator isn't widely used. Is it subsumed by these other techniques, or are the conditions of the theorem not met in practice? " CreationDate="2010-08-12T16:23:45.047" UserId="795" />
  <row Id="1695" PostId="1160" Text="Now you tell me :D  (thanks anyway)" CreationDate="2010-08-12T16:56:01.157" UserId="253" />
  <row Id="1696" PostId="1587" Text="I think you have the interpretation of the R code correct (to be sure, hadcru.mcmc.zero has no slope parameter, it is just a mean model). I don't understand why the prior should be constrained just to the positive values though (I also wouldn't know how to code or write it down mathematically either; I want a diffuse prior but ignore negative values?). Shouldn't the H1 model include all values of the slope? Maybe I need to test whether H0: \theta &lt;= 0 and H1: \theta &gt; 0, in that case, I wouldn't know how to code it either :)" CreationDate="2010-08-12T17:07:31.950" UserId="291" />
  <row Id="1698" PostId="1605" Score="1" Text="Yes, that is correct. In my example I assumed the classes would have the same number of students, which is not realistic. Thank you." CreationDate="2010-08-12T17:22:06.107" UserId="666" />
  <row Id="1699" PostId="1249" Text="Link to the paper http://math.unice.fr/~baraud/publications/min-rev1.pdf " CreationDate="2010-08-12T17:30:09.167" UserId="223" />
  <row Id="1700" PostId="1285" Text="Thanks, that is what I needed!" CreationDate="2010-08-12T18:12:39.290" />
  <row Id="1702" PostId="1610" Text="Terminology is a bit vague. I changed error to typeI-errors and typeII-errors. Hope that is fine. Also, your question should be community wiki as there is no correct answer to your question." CreationDate="2010-08-12T20:00:22.107" />
  <row Id="1703" PostId="1610" Text="@Srikant: in that case, we should make questions like this cw as well: http://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english." CreationDate="2010-08-12T20:01:51.153" UserId="5" />
  <row Id="1704" PostId="1610" Text="Older literature calls H2 the null hypothesis, H1 the alternate hypothesis, then it's natural to call type i error as the error of mistakenly accepting Hi hypothesis" CreationDate="2010-08-12T20:03:10.947" UserId="511" />
  <row Id="1705" PostId="1610" Text="@Shane: I will abstain from commenting on your point as the present top voted answer is mine (conflict of interest). I will go with what the community feels is appropriate." CreationDate="2010-08-12T20:04:53.140" />
  <row Id="1706" PostId="1610" Text="@Shane you're totally right I think the question you mentionned should be community too. " CreationDate="2010-08-12T20:06:40.377" UserId="223" />
  <row Id="1707" PostId="1610" Text="The tags in this case should be type-i-errors and type-ii-errors as the dashes replace spaces. A terminology tag would help people to find questions that ask about definitions of words, like this one. Finally, I'm not making this community wiki as there is a correct answer that I will be accepting - whatever technique I feel best helps me to remember the difference." CreationDate="2010-08-12T20:07:02.480" UserId="110" />
  <row Id="1708" PostId="1569" Text="Why do I take small e...hmm...that's what Balasubramanian's paper did, but now, going back to Kullback, it seems his bound holds for any e, and he also gives bound for finite n, let me update the answer" CreationDate="2010-08-12T20:07:21.047" UserId="511" />
  <row Id="1709" PostId="1249" Text="Can you give us the hypothesis you want to test ? " CreationDate="2010-08-12T20:08:37.610" UserId="223" />
  <row Id="1710" PostId="1610" Text="ok I will edit as per your request. Reg CW- fine with me as long as that makes sense to everyone else." CreationDate="2010-08-12T20:10:04.003" />
  <row Id="1711" PostId="1610" Score="2" Text="Honestly, perhaps the community wikiness of this question should be discussed on meta. I personally feel that there is a singular right answer to this question - the answer that helps me. However, that singular right answer won't apply to everyone (some people might find an alternative answer to be better). Personally, I want to give reputation to the person or people who help me with my problem, but if the community wants this to be community wiki, I can make it happen (but not without a discussion on meta first)." CreationDate="2010-08-12T20:11:46.837" UserId="110" />
  <row Id="1712" PostId="1610" Text="I started a thread here: http://meta.stats.stackexchange.com/questions/290/what-is-community-wiki.  I think that I agree with @Thomas; we should reward good answers on questions like these." CreationDate="2010-08-12T20:16:14.597" UserId="5" />
  <row Id="1715" PostId="1612" Text="I've never even thought of it pictorially before. Normally, thinking in pictures doesn't work for me, but I'll read that article and maybe this is a special case where it will help me." CreationDate="2010-08-12T20:27:30.903" UserId="110" />
  <row Id="1716" PostId="1614" Text="Isn't that agism?  :)" CreationDate="2010-08-12T20:33:46.193" UserId="5" />
  <row Id="1717" PostId="1608" Text="@Aniko, The binning is unavoidable. Each bin represents a geometric region. I also use the Kruskal-Wallis test for continuous variables for the entire region, which was worked out fine. &#xA;&#xA;The rows are not ordered." CreationDate="2010-08-12T20:42:55.063" UserId="559" />
  <row Id="1718" PostId="1614" Score="2" Text="How about &quot;once bitten, twice shy&quot;? No funnier, but commonplace enough to remember. And no ageism required!" CreationDate="2010-08-12T20:54:07.470" UserId="174" />
  <row Id="1720" PostId="1616" Text="I kind of like that. I'm thinking this might work for me." CreationDate="2010-08-12T21:42:13.480" UserId="110" />
  <row Id="1721" PostId="1616" Text="it's sort of like how in elementary school kids would ask &quot;are you not not cool?&quot; " CreationDate="2010-08-12T22:10:11.923" UserId="900" />
  <row Id="1722" PostId="1569" Text="ok, we don't need small e (now called b, Type II error) to be small for bound to hold, but b=0 is the value for which the simplified (exp(-n KL(p,q)) bound matches the more complicated bound above. Curiously enough, lower bound for Type I error given 0 Type II error is &lt;1, I wonder if &lt;1 Type II error rate is actually achievable" CreationDate="2010-08-12T22:54:33.343" UserId="511" />
  <row Id="1723" PostId="1619" Text="But you still have to associate type I with an innocent man going to jail and type II with a guilty man walking free. So in the end, it really doesn't get me anywhere." CreationDate="2010-08-12T23:07:53.913" UserId="110" />
  <row Id="1724" PostId="1616" Text="yes, now you just need to remember *which* hypothesis (null or alternate) :P" CreationDate="2010-08-12T23:17:20.717" UserId="511" />
  <row Id="1725" PostId="1616" Text="It's easier to remember that both types of errors deal with the null hypothesis. I'm seriously considering accepting this answer - I think this one helps me the most." CreationDate="2010-08-12T23:33:33.170" UserId="110" />
  <row Id="1726" PostId="1609" Text="Good references, but I don't understand your last comment about a parametric distribution. Splines are usually used for modelling the conditional mean of Y|X and are not directly to do with the distribution of either Y or X. I know you can fit spline-based density estimates, but I doubt that is what Henry was asking about." CreationDate="2010-08-12T23:45:04.853" UserId="159" />
  <row Id="1727" PostId="1616" Score="1" Text="Actually it's the alternate hypothesis in both cases" CreationDate="2010-08-13T00:10:23.680" UserId="511" />
  <row Id="1728" PostId="134" Score="1" Text="Re-opened following discussion at http://meta.stats.stackexchange.com/questions/276/should-we-unclose-computing-questions" CreationDate="2010-08-13T00:31:38.923" UserId="159" />
  <row Id="1729" PostId="1603" Text="by pigeonhole principle, $\bar{P} &gt;= 1/(n+1)$, thus I think your proposed upper bound based on the mode is going to be, asymptotically, the same as the trivial upper bound." CreationDate="2010-08-13T01:52:33.443" UserId="795" />
  <row Id="1730" PostId="495" Text="I gave the answer to this one because (a) it has the most votes, and (b) it is a good course. But some of the courses at http://www.uccs.edu/~math/vidarchive.html represent more mathematical treatments of statistics." CreationDate="2010-08-13T01:55:11.407" UserId="183" />
  <row Id="1731" PostId="1582" Text="Great point about the importance of causal mechanisms." CreationDate="2010-08-13T02:24:18.780" UserId="183" />
  <row Id="1732" PostId="1446" Score="1" Text="Every answer to my question has provided useful information and I've up-voted them all. But I can only select one answer and Srikant's provides the best overview IMO." CreationDate="2010-08-13T02:25:01.587" UserId="159" />
  <row Id="1733" PostId="1267" Text="the first edition of Wilcox' Robust Estimation and Hypothesis Testing was printed in 1997. the book was nonempty. (granted, the second edition is a bit bigger...)" CreationDate="2010-08-13T02:34:59.707" UserId="795" />
  <row Id="1734" PostId="1609" Text="Yes, the latter, but due to meandering and unrelated thoughts, not meant for &quot;publication&quot;.  I was thinking about density estimation because of some other question and, in light of this question, was working through explaining spline methods to *myself*.  Strangely, I remember thinking what I typed was cogent -- because of what was in my *head*.  Ridiculous.  :-/  Thanks for the catch; impressed that it occurred to you, though I shouldn't be based on your quality answers." CreationDate="2010-08-13T02:44:42.490" UserId="251" />
  <row Id="1735" PostId="1616" Text="Yaroslav - I thought the same.  But well, the researcher doesn't usually want the H0 (H NULL) to be true..." CreationDate="2010-08-13T03:33:10.347" UserId="253" />
  <row Id="1736" PostId="1610" Text="I just wanted to say I loved your question - and I love the answers even more!" CreationDate="2010-08-13T03:35:25.357" UserId="253" />
  <row Id="1738" PostId="1579" Text="The rule of thumb there is highly useful. Thanks for that." CreationDate="2010-08-13T03:38:38.353" UserId="776" />
  <row Id="1739" PostId="1619" Score="2" Text="+1, I like.  @Thomas: Given an &quot;innocent until proven guilty&quot; system, you could think of type I as a primary error to avoid (imprisoning the innocent) and type II as a secondary error (guilty go free)." CreationDate="2010-08-13T03:42:03.563" UserId="251" />
  <row Id="1740" PostId="1444" Score="1" Text="I've summarized some of the answers plus some other material at http://robjhyndman.com/researchtips/transformations/" CreationDate="2010-08-13T04:28:23.740" UserId="159" />
  <row Id="1741" PostId="151" Text="Much of the field of robust statistics is an attempt to deal with the excessive sensitivity to outliers that that is a consequence of choosing the variance as a measure of data spread (technically scale or dispersion).&#xA;http://en.wikipedia.org/wiki/Robust_statistics" CreationDate="2010-08-13T05:15:29.947" UserId="521" />
  <row Id="1742" PostId="1619" Text="That's actually quite deep ars, thanks! My way of remembering was admittedly more pedestrian: &quot;innocent&quot; starts with &quot;I&quot;." CreationDate="2010-08-13T05:32:10.217" UserId="830" />
  <row Id="1743" PostId="1622" Text="When you say that the Sharpe ratio is the mean divided by the sample standard deviation... &quot;Up to a constant factor (sqrt(n), where n is the number of observations)&quot;... what does this last part mean?  Is it really the mean divided by the standard error of the mean?" CreationDate="2010-08-13T05:44:29.807" UserId="196" />
  <row Id="1744" PostId="1590" Score="1" Text="Consider selecting a more informative question title; someone asking a similar question in the future is unlikely to check the answer under a vague title than under a specific one." CreationDate="2010-08-13T05:58:54.037" UserId="196" />
  <row Id="1746" PostId="1570" Text="You don't need assumption about 2x2 table but rather kxm table.&#xA;" CreationDate="2010-08-13T06:06:22.257" UserId="419" />
  <row Id="1748" PostId="1570" Text="Of course. Now fixed." CreationDate="2010-08-13T06:32:31.767" UserId="159" />
  <row Id="1749" PostId="1456" Text="Some correction should be made to formula above. It is var(log(OR)) not var(OR)." CreationDate="2010-08-13T06:34:43.027" UserId="419" />
  <row Id="1750" PostId="1456" Text="@Wojtek: fixed, thanks." CreationDate="2010-08-13T06:44:41.213" UserId="251" />
  <row Id="1751" PostId="1587" Score="2" Text="The Bayes factor is the ratio of the marginal likelihood assuming H1 to the marginal likelihood assuming H0.  In this case, the alternative hypothesis H1 is that the slope is positive, so your prior fo H1 must exclude negative values for the slope parameter as H1 says they are implausible a-priori.  One way of achieving this might be to use an exponential prior, which is strictly positive, for the slope and a Gaussian prior for the intercept.  You could then implement a &quot;negative slope&quot; null hypothesis by flipping the data upside-down and using the same model.&#xA;&#xA;HTH" CreationDate="2010-08-13T07:23:13.433" UserId="887" />
  <row Id="1752" PostId="1598" Text="@csgillespie, curious if you've tried PyMC and how the performance compares (relative to python/C) for your models." CreationDate="2010-08-13T07:31:00.843" UserId="251" />
  <row Id="1753" PostId="1607" Text="@ars If we apply t-tools to Box-Cox transformed data we will get inferences about the difference in means of the transformed data. How can we interpret those on the original scale of measurement? (The mean of the transformed values is not the transformed mean). In other words (if I'm correct), taking the inverse transform of the estimate of the mean, on the transformed scale, does not give an estimate of the mean on the original scale. " CreationDate="2010-08-13T07:39:21.070" UserId="339" />
  <row Id="1754" PostId="1452" Text="Is there a page where I can find all those latex symbols (including begin{cases}) ?" CreationDate="2010-08-13T08:45:33.440" UserId="339" />
  <row Id="1755" PostId="1621" Text="Fisher's information gives lower bound in parameter estimation. It is a natural metric because it roughly says something like &quot;in this direction the difficulty of my problem cannot decrease more than that&quot;.  What you call generalization bounds are upper bounds ? do you want to know the performence of the method that use Fisher metric (the large body you mention is a good list)? sorry but I don't really get the question :) can you reformulate that point ? " CreationDate="2010-08-13T08:45:45.920" UserId="223" />
  <row Id="1756" PostId="1637" Score="7" Text="One point: t-tests use the t distribution not the Z distribution" CreationDate="2010-08-13T10:12:21.990" UserId="183" />
  <row Id="1758" PostId="1444" Text="@rob excellent way to transform and promote stat.stackoverflow !" CreationDate="2010-08-13T11:49:16.957" UserId="223" />
  <row Id="1759" PostId="1598" Text="@ars: In the case above, each iteration (of the 10^8 iterations) involved solving 5 ODEs. This really had to be done in C. The rest of the code was fairly simple and so the C code was straightforward. My application was non-standard and so PyMC wasn't applicable - also it was ~2 years ago." CreationDate="2010-08-13T12:33:10.720" UserId="8" />
  <row Id="1760" PostId="452" Text="Would you put a link to Angrist and Pischke." CreationDate="2010-08-13T13:37:18.213" UserId="8" />
  <row Id="1761" PostId="1608" Text="@Elpezmuerto If each row represents a region and they are not ordered, then how are you doing a Kruskall-Wallis test? That test only applies to ordered outcomes. Please, please, please add a compelete description of your problem by editing your original question." CreationDate="2010-08-13T13:50:46.057" UserId="279" />
  <row Id="1762" PostId="1651" Score="1" Text="I hope I did not mangle the eqns while converting them to latex. You can use latex on this site. See: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats Could you also please clean up the question a bit. For example, the first line is a bit odd, what do you mean by &quot; I know the value of $y_{ij}$, but it could be more than that value.&quot; etc." CreationDate="2010-08-13T14:32:35.100" />
  <row Id="1763" PostId="1651" Text="ooh nice! Thank you very much! :)" CreationDate="2010-08-13T14:34:58.650" />
  <row Id="1764" PostId="1639" Score="2" Text="I can look up a citation, but this is easy enough to to test empirically.  F from an ANOVA with two groups is exactly equaly to t^2 and the p-values will be exactly the same.  The only reason it wouldn't be equivalent in the case of unequal variances is if you apply a correction.  Otherwise, they are the same." CreationDate="2010-08-13T14:54:42.113" UserId="485" />
  <row Id="1765" PostId="1621" Text="Let's say that Fisher's Information Matrix gives our Riemannian metric tensor. It lets us find arclength of any curve by integrating. Then you define the distance between p and q as the smallest arclength over all curves connecting p and q. This is the distance measure I'm asking about. Same with volume." CreationDate="2010-08-13T15:19:41.580" UserId="511" />
  <row Id="1766" PostId="1653" Text="Would be interested to know how to fit repeated measures ANOVA models using lm." CreationDate="2010-08-13T15:51:49.717" UserId="702" />
  <row Id="1767" PostId="1653" Text="The issues of coding categorical variables, the equivalence of regression and ANOVA models, and regression coding for repeated measures are described in this article. http://dionysus.psych.wisc.edu/Lit/Topics/Statistics/Contrasts/Wendorf2004a.pdf Here's the citation... Wendorf, C. A. (2004). Primer on multiple regression coding: Common forms and the additional case of repeated contrasts. Understanding Statistics 3, 47-57. " CreationDate="2010-08-13T16:11:17.003" UserId="485" />
  <row Id="1768" PostId="1648" Text="Thanks Shane, great answer! Well, &quot;the others&quot; from my field often use SPSS, so they use Kolmogorov-Smirnov (if they check normality at all), though IMHO the Lilliefors' test is a better choice when the data is gathered from a sample (when parameters are unknown). I was taught that Shapiro-Wilk's is appropriate for small samples, and just wanted to get more info about &quot;small samples normality tests&quot;...  BTW, I use nortest in R! =)" CreationDate="2010-08-13T16:11:51.620" UserId="1356" />
  <row Id="1769" PostId="1646" Text="Not to start a sub-question, but I've never understood the utility of intra-class correlation; it strikes me that the scenario you describe can be adequately captured by talking about (1) the variance of the team means and (2) the mean within-team variance." CreationDate="2010-08-13T16:15:00.807" UserId="364" />
  <row Id="1770" PostId="1658" Text="Do we vote for the first one or the second one :) ? " CreationDate="2010-08-13T16:35:07.670" UserId="223" />
  <row Id="1771" PostId="1635" Text="yes, my question was poorly worded and ambiguous. to your last point: there are well known techniques for comparing Sharpe given paired observations. My Q is for independent observations, applicable for:&#xA;1. non-overlapping times, or&#xA;2. the case where summary statistics are provided by a third party (e.g. in a fund prospectus)." CreationDate="2010-08-13T16:46:03.257" UserId="795" />
  <row Id="1772" PostId="1647" Text="I agree. I performed a quick test of the A-D test, Jarque-Bera, and Spiegelhalter's test (1983), under the null, with sample size 8, repeating 10,000 times. The A-D test maintains nominal rejection rate, and gives uniform pvals, while J-B test is terrible, Spiegelhalter is middling." CreationDate="2010-08-13T17:18:29.993" UserId="795" />
  <row Id="1773" PostId="1607" Text="@gd047, some tests assume normality of the distribution of the mean, not the data.  t-test tends to be pretty robust w.r.t to underlying data.  You're right though -- with post-transformation tests, results are reported after inverse-transforming, and interpretation can be very problematic.  It comes down to how &quot;un-normal&quot; your data is, can you get away without transforming or applying, say, a log transform which is easier to interpret.  Otherwise, it's contextual on the actual transformation and domain and I don't really have a good answer.  Might be worth asking to see what others say?" CreationDate="2010-08-13T17:42:32.747" UserId="251" />
  <row Id="1774" PostId="1525" Text="good discussion on here. thanks guys!" CreationDate="2010-08-13T17:44:05.367" UserId="74" />
  <row Id="1775" PostId="1452" Score="2" Text="@gd047: here's a nice reference: http://elevatorlady.ca/doc/refcard/expressions.html" CreationDate="2010-08-13T17:45:12.853" UserId="251" />
  <row Id="1776" PostId="1598" Text="@cgillespie: curiosity sated.  sounds interesting, thanks. :)" CreationDate="2010-08-13T17:46:37.937" UserId="251" />
  <row Id="1777" PostId="1661" Text="What is the question you want to answer? Do you want to compare males to females, or wing length to tail length, or something else? And why are you restricted to a univariate test?" CreationDate="2010-08-13T17:50:24.760" UserId="279" />
  <row Id="1778" PostId="1662" Text="I'm testing individual component means" CreationDate="2010-08-13T18:04:05.077" />
  <row Id="1779" PostId="1662" Text="I'm restricted to unvariate t-tests." CreationDate="2010-08-13T18:05:27.273" />
  <row Id="1780" PostId="1573" Text="i'm going to make this the best answer, since it provided a good starting point of study. Thanks for the discussion guys!" CreationDate="2010-08-13T18:06:57.707" UserId="291" />
  <row Id="1781" PostId="328" Text="@James: Alternatively, this might argue for more education in the field, not less..." CreationDate="2010-08-13T18:11:14.860" UserId="5" />
  <row Id="1782" PostId="1662" Text="What do you mean by individual components? What are these 'components'? Ideally, you should edit the question to clarify so that others can also post appropriate answers. See also Aniko's comments to your question." CreationDate="2010-08-13T18:13:08.507" />
  <row Id="1783" PostId="1651" Text="I don't know of an R procedure that would do it, however there is a Stata module CENSORNB that does exactly this. Perhaps looking at the code might give you ideas." CreationDate="2010-08-13T19:23:40.617" UserId="279" />
  <row Id="1784" PostId="1642" Text="Interesting idea and it makes sense. However, the exam that I'm studying for uses Type I and Type II errors." CreationDate="2010-08-13T20:07:47.593" UserId="110" />
  <row Id="1785" PostId="1664" Text="i need to compare tail length male vs tail length female and in a separate test compare wing length male vs wing length female.&#xA;&#xA;Ho: X1-male = X1-female&#xA;H1: not equal&#xA;&#xA;and&#xA;&#xA;Ho: X2-male = X2-female&#xA;H1: not equal&#xA;&#xA;I thinking 2 separate univariate t-tests?" CreationDate="2010-08-13T20:45:05.053" />
  <row Id="1786" PostId="1664" Text="That's what my answer describes. " CreationDate="2010-08-13T20:58:11.683" UserId="8" />
  <row Id="1787" PostId="1569" Text="Actually a much easier to understand reference for this is Cover's &quot;Elements of Information Theory&quot;, page 309, 12.8 &quot;Stein's Lemma&quot;" CreationDate="2010-08-13T21:25:20.333" UserId="511" />
  <row Id="1788" PostId="1664" Text="In R:&#xA;&#xA;##X1 TEST&#xA;&#xA;t.test(T6.11.dat[1],T5.12.dat[1])&#xA;&#xA;RIGHT?" CreationDate="2010-08-13T21:28:04.970" />
  <row Id="1789" PostId="1669" Text="Bookmarked. I wish they offered PDFs, but you can't go wrong with NIST's documents." CreationDate="2010-08-14T00:47:50.010" UserId="110" />
  <row Id="1790" PostId="1676" Text="Debating making this Community Wiki - I really want to see what the first couple of answers are, first, though. If someone can provide a list of &quot;must have&quot; R packages, I'll probably accept that. If it begins to turn into a list of suggestions (or someone has an otherwise good reason), I'll CW this thing up." CreationDate="2010-08-14T01:40:13.547" UserId="110" />
  <row Id="1791" PostId="1677" Text="Good call. I use SO and I was unaware that question even existed. It's good to have a cross-linking for times when questions might appear on multiple exchanges." CreationDate="2010-08-14T01:46:17.650" UserId="110" />
  <row Id="1792" PostId="1676" Score="2" Text="I would say that this should be community wiki. " CreationDate="2010-08-14T02:13:08.023" UserId="5" />
  <row Id="1793" PostId="1676" Score="7" Text="Possible duplicate: http://stats.stackexchange.com/questions/73/what-r-packages-do-you-find-most-useful-in-your-daily-work/" CreationDate="2010-08-14T02:14:21.220" UserId="5" />
  <row Id="1794" PostId="452" Text="Angrist, Joshua D. and Jorn-Steffen Pischke. 2009. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press: Princeton, NJ." CreationDate="2010-08-14T02:40:04.060" UserId="401" />
  <row Id="1795" PostId="1676" Text="I agree it should be CW as the answer depends entirely on what you want to do. I load a lot of forecasting and time series packages, but others may not need them at all." CreationDate="2010-08-14T05:13:41.537" UserId="159" />
  <row Id="1796" PostId="1594" Text="Thanks, your post looks really nice, and its what I'd expect from&#xA;a statistician too. I'd appreciate the explanation of the choice&#xA;of logistic regression though :)&#xA;But unfortunately it doesn't help me at all. Like, your function&#xA;fitting is LMS, right? Even with logit, does it really minimize&#xA;the (abs) entropy? Also all the common models in data compression are discrete -&#xA;eg. a bit run model.&#xA;Anyway, I guess I'd really have to ask a few more specific questions&#xA;instead.&#xA;" CreationDate="2010-08-14T09:15:01.753" UserId="799" />
  <row Id="1797" PostId="1676" Text="Making community wiki, and this is a really close duplicate of the other. Is the searching acting up? I searched for &quot;r packages&quot; with and without quotes and didn't see that...I wonder if it had something to do with the downtime last night." CreationDate="2010-08-14T09:41:38.147" UserId="110" />
  <row Id="1798" PostId="1594" Text="It's usually fitted using maximum likelihood estimation.  You really need to say some more about what might have generated the binary data to progress any further.  Also is there a specific data set?  I see you linked to one, but was that just randomly chosen or do you have another specific one in mind?" CreationDate="2010-08-14T10:51:58.903" UserId="702" />
  <row Id="1799" PostId="154" Text="Looks very interesting indeed!" CreationDate="2010-08-14T11:29:43.413" UserId="702" />
  <row Id="1800" PostId="1594" Text="Its good then if its maximum likelihood, but it turns into bruteforce in real cases which is not very practical. As to &quot;specific data&quot;, I can post some (eg from a text compressor), but the whole point is that we don't know a good model for it. So instead we have to &quot;mix&quot; predictions of a few not-so-good models." CreationDate="2010-08-14T11:46:12.530" UserId="799" />
  <row Id="1801" PostId="1676" Text="This one's _definitely_ for SO!" CreationDate="2010-08-14T11:58:19.323" UserId="1356" />
  <row Id="1802" PostId="1594" Text="posted an example" CreationDate="2010-08-14T12:11:31.277" UserId="799" />
  <row Id="1803" PostId="1676" Text="I disagree that it's for SO, especially considering the discussions on meta that supporting tools for statistical analysis (including software) is on-topic...." CreationDate="2010-08-14T12:25:54.850" UserId="110" />
  <row Id="1804" PostId="1684" Score="1" Text="No disrespect, but that strikes me as entirely subjective. I have used latex for since the late 80s and R since the 90s yet I see no good reason to move away from embedding eps files. Everybody's mileage will differ here, and Thomas should maybe look at Task Views for particular problem domains." CreationDate="2010-08-14T13:04:26.380" UserId="334" />
  <row Id="1805" PostId="63" Score="1" Text="Judgement is a good word, since all we can ever observe is correlation. All that experiments and/or clever statistics can do is allow us to exclude some alternative explanations for what could have caused an effect." CreationDate="2010-08-14T18:18:08.860" UserId="198" />
  <row Id="1806" PostId="1600" Score="1" Text="I have done a lot of number crunching, one website and few administrative scripts in R and they are working quite nice." CreationDate="2010-08-14T19:40:50.960" UserId="88" />
  <row Id="1807" PostId="501" Text="@Thylacoleo By comparing the error of a full set and the set with some feature knock-outed, for instance by shuffling its values across objects. Read about RF importance to learn more about the concept." CreationDate="2010-08-14T19:49:45.370" UserId="88" />
  <row Id="1808" PostId="1676" Text="While not being moderator, I would vote to close as a duplicate; we don't need more than one R package feed." CreationDate="2010-08-14T19:56:33.717" UserId="88" />
  <row Id="1809" PostId="1664" Text="@Matt: that's correct" CreationDate="2010-08-14T21:42:37.077" UserId="8" />
  <row Id="1810" PostId="1691" Text="On second thoughts my answer may not make much sense. I will leave it up in case it gives you some ideas." CreationDate="2010-08-14T22:15:33.003" />
  <row Id="1811" PostId="1586" Text="At the page most are categorical but there is an figure at the top of the page one could imagine a variety of derivatives of... Specifically, if you have a categorical and continuous predictor then the continuous predictor could easily flip sign if the categorical one is added and within each category the sign is different than for the overall score." CreationDate="2010-08-14T23:07:11.833" UserId="601" />
  <row Id="1812" PostId="538" Text="Those Bayesian networks are equivalent in a sense that given data sampled from one of them, you can't tell which one it was" CreationDate="2010-08-14T23:52:21.040" UserId="511" />
  <row Id="1813" PostId="1678" Text="+1 for lattice." CreationDate="2010-08-15T00:30:18.983" UserId="389" />
  <row Id="1814" PostId="596" Text="EM is simply one way of maximizing likelihood when your log-likelihood function involves logs of sums. This can happen when you have latent variables either in unsupervised or unsupervised models. So for instance, if you fit a CRF and assume some of the outputs are unobserved, it would be suitable to use EM. But you could also maximize that (conditional) likelihood by gradient descent." CreationDate="2010-08-15T06:48:34.390" UserId="511" />
  <row Id="1815" PostId="1699" Score="1" Text="The link you gave points to your own machine - 127.0.0.1" CreationDate="2010-08-15T07:00:41.767" UserId="8" />
  <row Id="1816" PostId="1699" Text="@csgillespie Fixed." CreationDate="2010-08-15T08:41:40.557" UserId="88" />
  <row Id="1820" PostId="1709" Text="It is not clear what you want. Do you want a visualization of the 5 points you collected or a plot of the theoretical distribution that you feel generated the data? Please clarify the question so that an appropriate answer can be provided." CreationDate="2010-08-15T15:08:06.470" />
  <row Id="1822" PostId="1709" Text="I'm not sure I understand. If you know they're distributed normally then... well they'll be distributed normally! If you don't know how they are distributed then plot your data and think of which distribution seems to be the more likely to fit them. Also, do you have any knowledge at all about the process you are analysing? Could you infer the distribution from that?" CreationDate="2010-08-15T15:24:36.453" UserId="582" />
  <row Id="1823" PostId="1709" Text="@Srikant: I tried to clarify the question. Better?" CreationDate="2010-08-15T15:27:11.980" UserId="198" />
  <row Id="1824" PostId="1709" Text="@nico: I don't know how the data are distributed. I mentioned the normal distribution only to help you understand the problem. Since we are trying to find out what process generates the distribution, we unfortunately don't know anything a priori about the distribution. I would like to visualize the data this way so that we can better identify patterns in the distribution and how the distribution changes with experimental perturbations." CreationDate="2010-08-15T15:31:45.553" UserId="198" />
  <row Id="1826" PostId="1708" Score="2" Text="Can you write some crash-introduction of what you want to do (including Eigenstrat step)? People are rather lazy, and are much more prone to answer questions that don't require too much googling just to be understood." CreationDate="2010-08-15T16:12:09.910" UserId="88" />
  <row Id="1827" PostId="1706" Text="That is not finding the tangency portfolio. It is just discretizing the returns space and plotting points of min variance on the graph." CreationDate="2010-08-15T18:47:03.793" UserId="862" />
  <row Id="1828" PostId="1708" Text="For example, are you working with SNP arrays?" CreationDate="2010-08-15T18:57:02.150" UserId="8" />
  <row Id="1829" PostId="1716" Text="Thank you for the suggestions. These are what I'd normally do with this kind of data. However, as I hope to explain in my edit, I really want to try something different." CreationDate="2010-08-15T20:07:55.577" UserId="198" />
  <row Id="1830" PostId="1717" Text="Thank you very much. This looks very promising! What if the data is not normally distributed? How bad is it to still use $\kappa=3-L$ ? Also, I guess if I wanted to have points with equal weight, I'd just disregard the first one, right?" CreationDate="2010-08-15T20:09:39.627" UserId="198" />
  <row Id="1832" PostId="1621" Text="So, just as an example, Rodriguez gets a significant improvement by using &quot;information volume&quot; as measure of model complexity, but surprisingly I can't see anyone else trying this out" CreationDate="2010-08-16T00:54:39.823" UserId="511" />
  <row Id="1834" PostId="1720" Text="Inference is conditional on the BC parameter $\lambda$ -- does this have an easy interpretation on the original scale?  I think the usual course is simply to report it that way and leave it at that (usually resting on some result about asymptotic equivalence which may not apply as generally)." CreationDate="2010-08-16T01:06:16.333" UserId="251" />
  <row Id="1836" PostId="1702" Text="+1 the Marsland book is quite good and filled a big gap in the existing selection of ML books." CreationDate="2010-08-16T03:11:15.627" UserId="251" />
  <row Id="1837" PostId="1714" Text="I can only second that... from what I've seen can you select data with the mouse in one projection while looking how the selected subset looks like in another projection." CreationDate="2010-08-16T06:22:04.397" UserId="961" />
  <row Id="1838" PostId="1681" Text="deviation from the mean of what? what are the assumption on your random variable or what type of bound do you need (exponential?). I think your question would be easier to answer if you could formalize it a bit more and if you add some details. take a look at http://en.wikipedia.org/wiki/Large_deviations_theory" CreationDate="2010-08-16T06:37:35.847" UserId="223" />
  <row Id="1839" PostId="1717" Text="The choice of $\kappa$ has to do with the properties of the *transformed* points (if you want to non-lineraly transform your random variables). A proper choice of $\kappa$ can also scale the points towards or away from the mean. You can safely select $\kappa$ so that $0\lt L+\kappa \lt k $ where $k$ the kurtosis of your distribution. Set $\kappa=0$ to disregard the point on the mean, without even changing the formulas." CreationDate="2010-08-16T06:43:50.793" UserId="339" />
  <row Id="1840" PostId="1720" Text="@Rob Hyndman Thank you. Maybe because the sample (from a population that I think it should follow an approximately symmetric distribution) might just happened to be skewed by chance." CreationDate="2010-08-16T07:10:32.687" UserId="339" />
  <row Id="1841" PostId="1729" Text="Thanks for the tags Rob, I was unable to create the latex tag myself." CreationDate="2010-08-16T08:42:16.583" UserId="913" />
  <row Id="1842" PostId="1731" Text="Thank you for the tips on writing a custom export and the useful functions." CreationDate="2010-08-16T08:47:58.710" UserId="913" />
  <row Id="1843" PostId="1730" Text="This works perfectly, an easy method of achieving exactly what I wanted." CreationDate="2010-08-16T08:48:27.287" UserId="913" />
  <row Id="1844" PostId="1719" Text="Should be a community wiki" CreationDate="2010-08-16T09:32:58.870" UserId="8" />
  <row Id="1846" PostId="63" Text="Very good comment about the symmetric/asymmetric relations. One also might claim that global warming causes piracy to increase. " CreationDate="2010-08-16T11:02:28.133" UserId="961" />
  <row Id="1847" PostId="1725" Text="I've tried that series before, and that guy although he's awesome; he umms and umms and umms and umms a lot. It is really distracting" CreationDate="2010-08-16T11:21:58.523" UserId="59" />
  <row Id="1848" PostId="1735" Score="3" Text="A few suggestions:&#xA;1. Replace acronyms with complete words,&#xA;2. Elaborate a little more on what you are trying to achieve statistically.&#xA;Are you simply trying to determine which regression coefficients are larger or smaller?&#xA;" CreationDate="2010-08-16T11:45:12.003" UserId="183" />
  <row Id="1849" PostId="1710" Text="It's also a good example of the ecological fallacy (i.e., making inferences about the individual-level from group-level data)." CreationDate="2010-08-16T11:50:18.967" UserId="183" />
  <row Id="1850" PostId="1736" Score="4" Text="Use Rpy to call R from Python ;-)" CreationDate="2010-08-16T12:35:36.380" UserId="88" />
  <row Id="1852" PostId="1709" Text="Picking one of your actual experimental results randomly is a good way to illustrate a typical result and can be used for simulation as well (a'la bootstrap)." CreationDate="2010-08-16T13:44:05.787" UserId="279" />
  <row Id="1853" PostId="1739" Text="Thank You! currently looking into this." CreationDate="2010-08-16T14:00:42.533" />
  <row Id="1854" PostId="1740" Text="Thank You! This worked, although I had to drop a comma in the call to data.frame.&#xA;&#xA;    stats = ddply(&#xA;    .data = ords&#xA;    , .variables = .(Symbol,SysID,Hour) &#xA;    , .fun = function(x){ &#xA;        to_return = data.frame(&#xA;             s = sum(x$Profit)&#xA;            , m = mean(x$Profit)&#xA;        )&#xA;        return(to_return)&#xA;    }&#xA;    , .progress = 'text'&#xA;)&#xA;    " CreationDate="2010-08-16T16:06:37.417" />
  <row Id="1855" PostId="1739" Text="fastest running correct answer" CreationDate="2010-08-16T16:37:33.770" UserId="601" />
  <row Id="1856" PostId="1181" Text="This line: freq.ex&lt;-(dpois(0:max(x.poi),lambda=lambda.est)*200) produces errors with some real world data, because the length of freq.ex won't match freq.obs on this line acc &lt;- mean(abs(freq.os-trunc(freq.ex))).  I adapted this line to freq.ex&lt;-(dpois(seq(0,max(x.poi))[seq(0:max(x.poi)) %in% x.poi],lambda=lambda.est)*200) but something still isn't quite right because goodfit produces warnings." CreationDate="2010-08-16T17:15:34.603" UserId="196" />
  <row Id="1857" PostId="1747" Text="PS -- the real reason I included the code is to prompt responses from anyone in the programming realm.  This function is probably found (and more elegantly) in some 'VBA stats snippets' resource ... I should search on that. " CreationDate="2010-08-16T17:18:20.713" UserId="857" />
  <row Id="1858" PostId="1527" Text="You change the viewpoint here. You could just as easily say, &quot;the proportion of heads on any one flips is .50&quot;. &#xA;&#xA;I contend that probabilities and proportions are essentially the same. " CreationDate="2010-08-16T18:13:04.040" UserId="74" />
  <row Id="1859" PostId="1751" Text="Using the information that what I was looking for was the negative binomial distribution I was able to find the correct function in R: qnbinom(c(.025,.975),1,1/104) - Thanks a bunch!" CreationDate="2010-08-16T18:27:32.137" UserId="196" />
  <row Id="1860" PostId="1753" Text="What was on the x, what was on the y, and what was the point of the graph?" CreationDate="2010-08-16T18:46:40.523" UserId="702" />
  <row Id="1861" PostId="1753" Text="@Andy: It doesn't matter. This is a software package which customers will use to their own needs with figures that can represent literally anything. In my test case I have a set of six points of data which are all positive and sit in the upper, right cartesian quadrant." CreationDate="2010-08-16T18:55:11.880" UserId="968" />
  <row Id="1862" PostId="1753" Text="Perhaps, the line is drawn such that it is consistent with your points 1, 2 and 3. In any case, your question is so vague that I am not sure a reasonable answer can be given. " CreationDate="2010-08-16T19:11:44.513" />
  <row Id="1863" PostId="1644" Score="2" Text="To give an example: Often one wants to know the P(model | data) ). Frequentist analysis gives you P(data | model) however (which then people often read as P (model |data). By assuming a prior probability P(model) you can get P(model | data) in Bayesian statsitics. But then you can debate what P(model) should be." CreationDate="2010-08-16T19:20:24.543" UserId="961" />
  <row Id="1864" PostId="1753" Text="Added a rephrase of the question to hopefully clarify via an example." CreationDate="2010-08-16T19:37:30.187" UserId="968" />
  <row Id="1865" PostId="1267" Text="Again, most of the technical content of the first edition is outdated by now." CreationDate="2010-08-16T19:40:35.293" UserId="603" />
  <row Id="1866" PostId="1753" Score="3" Text="No, what was on the axes definitely does matter.  Based on what you've written, I can imagine two different plots: one that is illustrating change in the variability of Y as X increases (in which case a simple line might be valid), and one that's illustrating the actual relationship of X and Y, which might include the mean and and SD lines... or might not.  Why 1 SD?  Why not 2?  We can only give plotting advice with some kind of context.  If people will be plotting whatever they want, all you can do is make it as flexible as possible." CreationDate="2010-08-16T19:50:46.800" UserId="71" />
  <row Id="1867" PostId="1754" Text="Thanks. That's exactly what i'm wrestling with. I have a deprecated app from a defunct company that i'm supposed to resurrect while at the same time troubleshooting their original intent. For example, the only clue (and the basis of my question) is a check box on a graph titled &quot;show standard dev.&quot; Now, i'm supposed to recreate the functionality. Thanks for your patience btw. Any further insight is greatly appreciated." CreationDate="2010-08-16T20:04:55.097" UserId="968" />
  <row Id="1869" PostId="1681" Text="Hm...I thought it was unambiguous given that I'm talking about Chernoff bound for absolute error...added a link in question (it seems to break in comment)...in that notation, I'm looking for epsilon as a function of m" CreationDate="2010-08-16T21:55:47.023" UserId="511" />
  <row Id="1870" PostId="1754" Text="Wow.  I don't even you that task.  In that context, I think what I'd do is have use the mean line + 2 SD for continuous X, and then use bars like csgillespie's for categorical data.  Not sure how you'd be able to differentiate reliably between cont. and cat., but I'm sure many of attempted that before." CreationDate="2010-08-16T22:29:48.727" UserId="71" />
  <row Id="1871" PostId="1525" Score="1" Text="Personally I liked the first title: &quot;Your blonde girlfriend asks, &quot;Hey, how's a probability different than a plain old proportion?&quot; Answer her in plain English.&quot;" CreationDate="2010-08-17T01:58:33.157" UserId="776" />
  <row Id="1872" PostId="1559" Text="They are not the same if one represents a probability and the other a proportion." CreationDate="2010-08-17T02:00:28.800" UserId="776" />
  <row Id="1873" PostId="1176" Text="Just for the protocol, Frank's talk (in video) is now online: http://www.r-bloggers.com/RUG/2010/08/user-2010-conference-videos/" CreationDate="2010-08-17T02:43:58.303" UserId="253" />
  <row Id="1874" PostId="1758" Text="Sorry, but I don't understand the point of derivations" CreationDate="2010-08-17T03:30:40.437" UserId="511" />
  <row Id="1875" PostId="1674" Text="It seems from the question, the entropy/volume connection is established, similarly we know of the volume/surface area connection. So the point here is to connect the trace of the FI to entropy. Or am I missing the point of your question? " CreationDate="2010-08-17T04:26:15.600" UserId="251" />
  <row Id="1876" PostId="1763" Text="Great list Jeromy, thank you :)" CreationDate="2010-08-17T04:47:44.670" UserId="253" />
  <row Id="1877" PostId="1708" Text="Yes, I am working with SNP data from SNP arrays.&#xA;But the problem is much simpler. I am looking for a statistical test that will give an estimate of max. variation along 3 eigenvalues." CreationDate="2010-08-17T05:16:29.570" UserId="952" />
  <row Id="1878" PostId="1755" Text="I don't know... Paul provided xy coordinates but not SD for all the points. We should see a graph with 5 points with perhaps some error bars (assuming we know the SD)." CreationDate="2010-08-17T05:27:37.150" UserId="144" />
  <row Id="1879" PostId="1764" Text="Those Excel guides look pretty spooky..." CreationDate="2010-08-17T05:41:55.500" UserId="88" />
  <row Id="1880" PostId="1737" Text="Question is misleading as you ask about unique combinations of factors and then in details you ask about summary by unique combinations." CreationDate="2010-08-17T05:59:00.613" UserId="419" />
  <row Id="1881" PostId="1665" Text="It rather does not change your knowledge about distribution of other variable. Definition of independence: P(Y|X)=P(Y). We know value of other variable only in rare cases of perfect prediction (correlation equals 1 or -1). " CreationDate="2010-08-17T06:10:20.527" UserId="419" />
  <row Id="1882" PostId="1764" Score="3" Text="Can you imagine how a tutorial on R looks to a person who has never seen a line of code in his or her life? :)" CreationDate="2010-08-17T06:50:39.917" UserId="144" />
  <row Id="1883" PostId="262" Text="For my part graphs that are black and white plots without shaded backgrounds that use different icons or line types are important.  Also an easy way to do confidence intervals (where appropriate) would be nice too." CreationDate="2010-08-17T07:10:38.873" UserId="196" />
  <row Id="1884" PostId="1676" Score="2" Text="I would close as a duplicate; also this question is within the scope of the site." CreationDate="2010-08-17T07:13:50.033" UserId="196" />
  <row Id="1885" PostId="1483" Text="Where TP is Test Positive and TN is Test Negative and FP is False Positive and FN is False Negative?  " CreationDate="2010-08-17T07:20:04.357" UserId="196" />
  <row Id="1886" PostId="1762" Text="About the chronographs' +/- 4%: A friend and I setup our two chronographs in tandem so that a single bullet fired would be measured by both. In the shade, his would measure a consistent 2% lower than mine: 940.0 fps on mine would record about 921.2 fps on his. When we moved the chronos to brighter lighting conditions, his would consistently record about 1.5% faster than mine, but both of our chronos were also recording about 1% higher than in the shade." CreationDate="2010-08-17T07:40:01.063" UserId="937" />
  <row Id="1887" PostId="1483" Text="How would the question asker obtain SE^2_1 and SE^2_2?" CreationDate="2010-08-17T07:52:09.567" UserId="196" />
  <row Id="1888" PostId="1758" Text="Perhaps, I misinterpreted what you wanted. Did you not want to express $\epsilon$ as a function of $m$? The above does not exactly get you there but I thought it is a start towards that goal." CreationDate="2010-08-17T08:19:50.873" />
  <row Id="1889" PostId="1764" Score="1" Text="@romunov Ok, but I can also imagine all those people drawing manually dozens of bar breaks in Excel and believing that it is an only (and thus easiest and fastest) way to do this. Or people spending hours trying to unify the formatting in a large Word document." CreationDate="2010-08-17T08:54:25.050" UserId="88" />
  <row Id="1891" PostId="1774" Text="I really do not know the rate. To be honest all I know is that my program crashed with a division-by-zero and that I need to handle that case somehow." CreationDate="2010-08-17T09:54:40.183" UserId="977" />
  <row Id="1892" PostId="1773" Text="I don't think we need limit tag." CreationDate="2010-08-17T11:08:27.627" UserId="88" />
  <row Id="1893" PostId="1773" Text="Presumably you're attempting to quantify performance of some diagnostic procedure; is there any reason you're not using a proper signal detection theory metric like d', A', or area under the ROC curve?" CreationDate="2010-08-17T11:19:07.750" UserId="364" />
  <row Id="1894" PostId="1775" Score="3" Text="Extending your answer: If TP=0 (as in both cases), recall is 1, since the method has discovered all of none true positives; precision is 0 if there is any FP and 1 otherwise." CreationDate="2010-08-17T11:44:27.600" UserId="88" />
  <row Id="1895" PostId="1658" Text="first one, I'm not famous... yet ;o)&#xA;&#xA;BTW, the second one is intended as a complement, just in case there was any doubt." CreationDate="2010-08-17T11:50:40.957" UserId="887" />
  <row Id="1896" PostId="1778" Text="+1 for pointing R^2." CreationDate="2010-08-17T11:59:01.190" UserId="88" />
  <row Id="1897" PostId="1334" Text="shame they often only look better... :-(" CreationDate="2010-08-17T12:05:19.850" UserId="887" />
  <row Id="1898" PostId="1764" Text="&quot;Father, forgive them, for they do not know what they are doing.&quot; comes to mind. :)" CreationDate="2010-08-17T12:19:12.163" UserId="144" />
  <row Id="1899" PostId="1776" Text="Cool, butterfly data. If you would be willing to discuss about your work, drop me an email. I'm registered at GMail." CreationDate="2010-08-17T12:30:12.967" UserId="144" />
  <row Id="1901" PostId="485" Text="for more stats 101 videos, see: http://stats.stackexchange.com/questions/1761/statistics-probability-videos-for-beginners" CreationDate="2010-08-17T13:54:19.727" UserId="183" />
  <row Id="1902" PostId="1776" Score="1" Text="@downvoters: Could you please leave a comment as to how the question can be improved? I am not sure if downvoting without leaving a comment is helping the OP.&#xA;&#xA;@Elaine I think you should fix the qn to give the correct value for R^2 and any other edits that you feel would help others understand your question." CreationDate="2010-08-17T14:07:54.247" />
  <row Id="1903" PostId="1781" Text="Just so I understand correctly the censoring issue: When you dilute a sample the concentration of a compound falls so low that the test instrument can fail to detect its presence. Is that an accurate re-phrasing of the censoring problem?" CreationDate="2010-08-17T14:34:17.977" />
  <row Id="1904" PostId="1771" Text="Thanks for the help.  This is where show my novice.  I , simply have been working through the basic examples in the r packages and aggregated the data to a point where I created my own graph.  Up to this point, it's been an exercise to get started in network analysis and data visualization.  This is a huge step forward though, thanks." CreationDate="2010-08-17T14:34:31.827" UserId="569" />
  <row Id="1905" PostId="1754" Text="Er... don't envy, I meant." CreationDate="2010-08-17T14:43:19.733" UserId="71" />
  <row Id="1906" PostId="1758" Text="Yes, as function of $m$ and $P$. Since $P$ is given, it doesn't make sense to rewrite it" CreationDate="2010-08-17T16:41:00.963" UserId="511" />
  <row Id="1907" PostId="1723" Text="Thanks for a great answer, and a great question afterwards. It's crucial to get an insight about the background of the problem. Well, so many times I've seen people doing t-test, Pearson's r or ANOVA without getting any idea about the shape of distribution (which is often heavy-skewed) - parametric techniques &quot;need&quot; satisfied normality assumption. In psychology (which is my field of interest), we often deal with small samples, therefore I need appropriate normality test." CreationDate="2010-08-17T16:41:53.863" UserId="1356" />
  <row Id="1908" PostId="1791" Text="thanks! however - which one would I use under which circumstance, and how would I deal with &gt;2 methods?" CreationDate="2010-08-17T16:56:14.407" UserId="979" />
  <row Id="1909" PostId="1758" Text="Oh, I understand." CreationDate="2010-08-17T17:18:02.640" />
  <row Id="1910" PostId="1787" Score="3" Text="Any reason not to like the the standard errors produced by glm?" CreationDate="2010-08-17T17:21:16.363" UserId="279" />
  <row Id="1911" PostId="1791" Text="Good question; test will only handle two methods, because it can only say if they are not equal. Still, you can convert scores for each sample to ranks and there take mean rank per method and select this with smallest mean. Then you can use test to justify if the best one is significantly better than the second." CreationDate="2010-08-17T17:32:00.323" UserId="88" />
  <row Id="1912" PostId="1782" Text="Didn't know pscal handled classical NB models (besides zero inflated and hurdle).  Thanks." CreationDate="2010-08-17T17:45:03.203" UserId="251" />
  <row Id="1913" PostId="1773" Score="2" Text="@Mike, precision and recall are common evaluation metrics in, e.g., information retrieval where ROC, or in particular specificity is awkward to use because you already expect a high number of false positives." CreationDate="2010-08-17T18:02:36.650" UserId="979" />
  <row Id="1914" PostId="1791" Score="1" Text="You may also look at http://en.wikipedia.org/wiki/MaxDiff" CreationDate="2010-08-17T18:09:25.747" UserId="88" />
  <row Id="1915" PostId="878" Score="12" Text="Good quote, but it's not true!  Absence of evidence is not *proof* of absence, but it certainly is *evidence*.  Why do we think magnetic monopoles (or unicorns, for that matter) don't exist?  Because we've looked and haven't found any." CreationDate="2010-08-17T18:15:29.727" UserId="319" />
  <row Id="1916" PostId="1781" Text="Yes, that is correct: dilution by a factor of D increases all detection limits by a factor of D as well.  (The matrix interference issue is more difficult to quantify and the general situation is extremely complex.  To simplify this, the conventional model is that a suite of tests on one sample yields a vector (x[1], ..., x[k]) where the x[i] are either real numbers or are intervals of reals, typically with left endpoint at -infinity; an interval identifies a set in which the true value is assumed to lie.)" CreationDate="2010-08-17T18:20:52.273" UserId="919" />
  <row Id="1917" PostId="1632" Score="1" Text="All answers were both helpful and useful, and would all deserve to be accepted. This one, however, does a very good job at answering the question: with Python, you have to put together lots of pieces to do what you want. These pointers will no doubt be very useful for anyone wanting to do statistics/modeling/etc. with Python. Thanks to everyone!" CreationDate="2010-08-17T18:42:30.127" UserId="890" />
  <row Id="1918" PostId="1787" Score="4" Text="Standard error is a measure of the expected quality of your estimate while a standard deviation is a measure of variability.  The former decreases with n while the latter does not." CreationDate="2010-08-17T18:53:33.487" UserId="601" />
  <row Id="1919" PostId="1483" Text="@Jay: didn't see your comment sooner, but the above is applicable if the DOR are independent.  @drknexus: updated the answer for how to obtain SE." CreationDate="2010-08-17T18:59:43.803" UserId="251" />
  <row Id="1920" PostId="1764" Score="1" Text="Official band of stats.stackexchange.com: The Broken Axes." CreationDate="2010-08-17T19:09:18.933" UserId="71" />
  <row Id="1921" PostId="1762" Text="Sorry if this is a dumb question but what do the lighting conditions have to do with chronograph accuracy?" CreationDate="2010-08-17T19:40:22.167" UserId="666" />
  <row Id="1922" PostId="1762" Text="Old chronos worked by shooting metal screens at start and stop gates which would break a circuit to mark start and stop times. Modern chronos use a light/IR sensor to detect the bullet passing over the gates. These sensors are affected by different ambient light levels. At major matches, the staff will typically build a box with lights inside around a pair of tandem chronos to provide consistent lighting to eliminate this factor." CreationDate="2010-08-17T20:21:07.453" UserId="937" />
  <row Id="1923" PostId="1796" Text="But the variability of results among one method is dominated by the variability of tests, so I am afraid the spread may be negligibly meaningful. " CreationDate="2010-08-17T21:22:41.067" UserId="88" />
  <row Id="1924" PostId="1781" Text="Why would the detection limits go up? Are they not a feature of the test instrument rather than that of the sample being tested?" CreationDate="2010-08-17T21:33:50.633" />
  <row Id="1925" PostId="1796" Text="But, that is the point. If a method's performance is not robust to diversity of test situations then I would tend to prefer it less than a method that is more robust assuming of course that on average they are comparable. Of course, this is all abstract as we do not know if the OP really cares about reliability/robust methods." CreationDate="2010-08-17T21:39:00.617" />
  <row Id="1926" PostId="1804" Text="this procedure results in an estimate of $M^T M$ that may not be positive semidefinite, which would be bad." CreationDate="2010-08-18T00:40:16.183" UserId="795" />
  <row Id="1927" PostId="1723" Score="1" Text="But normality is *never* satisfied. It's sometimes a reasonable description of the data, but they're not actually normal.While it's sensible to check for non-normality when you assume it, it's not particularly useful to test it (for the reasons I described above). I do a qq-plot, for example, but a hypothesis test answers the wrong question in this situation. t-tests and anova usually work reasonably well if the distributions aren't heavily skew. A better approach might be to use procedures that don't assume normality - perhaps resampling techniques." CreationDate="2010-08-18T00:50:33.493" UserId="805" />
  <row Id="1929" PostId="1723" Text="Or you can use non-parametric tests, at cost of having less power. And nothing is absolutely satisfied in statistics, it's not solely a normality issue. However, bootstrapping or jackknifing are not a solution when introducing someone to t-test and/or ANOVA assumptions. I doubt that resampling techniques solve normality issues at all. One should check normality both graphically (density plot, boxplot, QQplot, histogram) and &quot;numerically&quot; (normality tests, skewness, kurtosis, etc.). What do you suggest? This is completely off topic, but how would you check, say, ANOVA normality assumptions?" CreationDate="2010-08-18T01:55:18.857" UserId="1356" />
  <row Id="1930" PostId="1809" Text="Sure. I added a sample problem." CreationDate="2010-08-18T02:58:40.480" UserId="850" />
  <row Id="1932" PostId="1815" Text="see this similar existing question: http://stats.stackexchange.com/questions/1352/references-for-how-to-plan-a-study" CreationDate="2010-08-18T09:27:20.310" UserId="183" />
  <row Id="1933" PostId="1796" Text="Hi, thanks for the input. You're right about mean/variance tradeoff one has to look at. However, in the given situation, mbq's right, that's not so much the issue. The point is that results almost never looks like your [0.80, 0.60] / [0.71, 0.69] example: The first one would mostly stay better, just on a different scale. Still, this is a helpful reference, since I might use this at another point (I also have different families of sample generators which might need to be compared), and also I'll update my question." CreationDate="2010-08-18T10:15:24.213" UserId="979" />
  <row Id="1934" PostId="1816" Text="How would I go about using the information from all generations?" CreationDate="2010-08-18T10:37:23.367" UserId="986" />
  <row Id="1935" PostId="1816" Text="The easiest way is to do multiple tests, i.e. test at every generation, then use a Bonferroni or fdr correction." CreationDate="2010-08-18T10:43:58.960" UserId="8" />
  <row Id="1936" PostId="1332" Text="The best antidote to &quot;...lies, damned lies, and statistics.&quot;" CreationDate="2010-08-18T10:44:57.120" UserId="521" />
  <row Id="1937" PostId="328" Text="@Shane: Yes, absolutely. I was being slightly facetious in my comment, but I do believe important insights are being withheld by the private-sector community as competetive advantages, which is ultimately to the detriment of an efficient global economy." CreationDate="2010-08-18T11:18:07.490" UserId="229" />
  <row Id="1938" PostId="1816" Text="When comparing at every generation, I would have to test at a significance level of 1/1000 * 0.05 ? Isn't that a bit harsh?" CreationDate="2010-08-18T11:18:28.283" UserId="986" />
  <row Id="1939" PostId="1821" Score="1" Text="It is the variance which is an unbiased estimator. S is not an unbiased estimate of the population standard deviation." CreationDate="2010-08-18T12:13:50.883" UserId="159" />
  <row Id="1940" PostId="1816" Text="True, but you're also doing lots of testing - can't have everything ;) You could rank the p-values, use them as a guide to see where possible errors may occur." CreationDate="2010-08-18T12:26:57.970" UserId="8" />
  <row Id="1941" PostId="1821" Text="Thanks Rob, I stand corrected! This is a subtlety that I was previousy unaware of. Others may wish to see:&#xA;&#xA;http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation" CreationDate="2010-08-18T12:41:14.290" UserId="521" />
  <row Id="1942" PostId="1824" Text="sounds reasonable, any more details?" CreationDate="2010-08-18T12:59:49.713" UserId="986" />
  <row Id="1943" PostId="1823" Text="Thank you for the answer Jeromy.  Did you happen to find something similar in R ?" CreationDate="2010-08-18T13:01:36.480" UserId="253" />
  <row Id="1945" PostId="1815" Text="@Jeromy Fair enough, but all the answers there so far have managed to avoid recommending even a single book!" CreationDate="2010-08-18T13:14:37.717" UserId="174" />
  <row Id="1946" PostId="1826" Text="This question is to help with any confusion over the site name proposal: http://meta.stats.stackexchange.com/questions/21/what-should-our-site-be-called-what-should-our-domain-name-be/53#53." CreationDate="2010-08-18T13:23:01.910" UserId="5" />
  <row Id="1947" PostId="1827" Text="@srikant thanks! i'll take a look at it." CreationDate="2010-08-18T13:30:04.733" UserId="283" />
  <row Id="1948" PostId="1830" Text="Technically, this is holdout validation but one can imagine extending the subway example to a cross-validation context. If it helps I will re-write the example and the rest of the text to be specific to cross-validation." CreationDate="2010-08-18T13:42:31.487" />
  <row Id="1949" PostId="1831" Text="@shane that book looks pretty nice! i just installed the 'arm' package, too. thanks shane!" CreationDate="2010-08-18T13:52:25.473" UserId="283" />
  <row Id="1950" PostId="1815" Score="1" Text="@walkytalky: I agree more discussion from a different angle would be great. I was just posting the other question as related supplementary reading." CreationDate="2010-08-18T13:53:58.497" UserId="183" />
  <row Id="1951" PostId="1831" Text="@mropa: That is my favorite statistics text: enjoy it! " CreationDate="2010-08-18T13:54:29.773" UserId="5" />
  <row Id="1952" PostId="1781" Text="As an example, suppose an instrument's detection limit is 1 microgram per Liter (ug/L).  A sample is diluted 10:1 (with great precision, so we don't worry about error here) and the instrument reads &quot;&lt;1&quot;; that is, nondetectable, for the diluted sample.  The laboratory infers that the concentration in the sample is less than 10*1 = 10 ug/L and reports it as such; that is, as &quot;&lt;10&quot;." CreationDate="2010-08-18T14:06:17.390" UserId="919" />
  <row Id="1953" PostId="1804" Text="That's a good point, but the result might not be so bad.  What one hopes is that the estimate of M*M is close enough to the true value that the perturbation of eigenvalues is reasonably small.  Thus, by projecting to the eigenspace corresponding to the largest eigenvalues, you achieve only a slight perturbation of the correct solution, still achieving the sought-after dimension reduction.  Perhaps the biggest problem may be algorithmic: since you can no longer assume semidefiniteness, you might need to use a more general-purpose algorithm to find the eigensystem." CreationDate="2010-08-18T14:09:39.020" UserId="919" />
  <row Id="1954" PostId="1828" Score="2" Text="It seems given discussions elsewhere on this site that k-fold cross validation is just one type of cross validation and describing it does not do the general job of describing what cross validation is." CreationDate="2010-08-18T14:31:12.633" UserId="196" />
  <row Id="1955" PostId="1836" Text="That is counter-intuitive!" CreationDate="2010-08-18T14:41:53.250" />
  <row Id="1956" PostId="1836" Score="1" Text="Yes it is!  That's why it's well worth studying: challenges to our intuition are exceptionally educational.  I first learned of this from a clear paper on Carlos Rodriguez' (SUNY Albany) Web page but I couldn't find it this morning: it appears the server is down.  Try Googling &quot;carlos rogriguez statistics&quot; later.  (His paper is supposed to be at http://omega.albany.edu/8008/confint.html , but this might be an old URL.)" CreationDate="2010-08-18T14:51:48.607" UserId="919" />
  <row Id="1957" PostId="1797" Text="Since in essence the nesting in a lmer makes it a repeated measures design is there a way in which your question about the appropriate confidence interval around the effect size is related to the question in repeated-measures ANOVA about which measure of effect size to report?  Specifically, it is unclear whether the error term should include subject variance or not (etc)?" CreationDate="2010-08-18T14:55:54.510" UserId="196" />
  <row Id="1958" PostId="1813" Text="Just as a clarification: isn't it the case that a genetic algorithm searches randomly for a solution, so that the initial segment of any run is unlikely to produce any worthwhile solution?  Also, what exactly do you mean by &quot;the minimum error in the population&quot;?  If you mean the minimum difference between a known true value and any solution out of the 1000 values in a run, then isn't that a biased indication of the run's result?  After all, in practice you would accept the final solution in each run and reject everything that precedes it, right?" CreationDate="2010-08-18T14:57:26.877" UserId="919" />
  <row Id="1959" PostId="1823" Text="@Tal Quick-R lists a few power analysis procedures in R: http://www.statmethods.net/stats/power.html ; or you can use R to run a simulation for custom power analysis. &#xA;The lme4 package is good for multilevel modelling." CreationDate="2010-08-18T15:01:28.740" UserId="183" />
  <row Id="1960" PostId="1311" Text="Suppose y=2, n=4, and there are just two birthdays.  Your formula, adapted by replacing 365 by 2, seems to say the probability that exactly 2 people share a birthday is Comb(4,2)*(2/2)^2*(1-1/2)*(1-2/2) = 0.&#xA;&#xA;(In fact, it's easy to see--by brute force enumeration if you like--that the probabilities that 2, 3, or 4 people share a &quot;birthday&quot; are 6/16, 8/16, and 2/16, respectively.)&#xA;&#xA;Indeed, whenever n-y &gt;= 365, your formula yields 0, whereas as n gets large and y is fixed the probability should increase to a non-zero maximum before n reaches 365*y and then decrease, but never down to 0." CreationDate="2010-08-18T15:05:43.567" UserId="919" />
  <row Id="1961" PostId="1797" Text="Nevermind - I didn't think that all the way through." CreationDate="2010-08-18T15:06:22.877" UserId="196" />
  <row Id="1963" PostId="1813" Text="By error I basically mean 1/fitness, so I'm talking about the value of the best individual in a generation.&#xA;&#xA;I've recorded the fitness value of the best individual for every generation.&#xA;&#xA;So I have 1000*20*2 numbers, each corresponding to the &quot;fitness&quot; of the best individual in a particular generation of a particular run." CreationDate="2010-08-18T15:12:47.640" UserId="986" />
  <row Id="1964" PostId="1828" Score="2" Text="@drknexus: That's fair, but I mention that it's k-fold and I wanted to provide a visualization of the process to help explain it." CreationDate="2010-08-18T15:16:04.913" UserId="5" />
  <row Id="1965" PostId="1839" Text="By algorithm, I mean roughly the software implementation used for fitting a line to model the mean of a distribution." CreationDate="2010-08-18T15:18:39.907" UserId="988" />
  <row Id="1967" PostId="1834" Text="Right, that's why I consider OLS to be an &quot;algorithm&quot; used in linear regression..." CreationDate="2010-08-18T15:22:58.910" UserId="988" />
  <row Id="1968" PostId="1813" Text="I guess the initial question was ill-posed, I've added some clarifications .." CreationDate="2010-08-18T15:23:11.033" UserId="986" />
  <row Id="1969" PostId="1834" Score="3" Text="Ordinary least squares is an estimator. There are a variety of  algorithms for computing the estimate: usually some sort of orthogonal matrix decomposition, such as QR, is used. See http://en.wikipedia.org/wiki/Numerical_methods_for_linear_least_squares" CreationDate="2010-08-18T15:34:31.490" UserId="495" />
  <row Id="1970" PostId="1842" Text="Uh, I tried setting up df = data.frame(t=days, values=c(data2,cum), type=rep(c(&quot;Bytes&quot;, &quot;Changes&quot;), each=1001)), but it gives an Error in rbind.zoo(...) : indexes overlap " CreationDate="2010-08-18T15:36:42.490" UserId="990" />
  <row Id="1971" PostId="1311" Text="Why you are replacing 365 by $n$? The probability that 2 people share a birthday is computed as: 1 - Prob(they have unique birthday). Prob(that they have unique birthday) = (364/365). The logic is as follows: Pick a person. This person can have any day of the 365 days as a birthday. The second person can then only have a birthday on one of the remaining 364 days. Thus, the prob that they have a unique birthday is 364/365. I am not sure how you are calculating 6/16." CreationDate="2010-08-18T15:39:58.903" />
  <row Id="1972" PostId="1842" Text="That's because data2 and cum are zoo objects. Use as.vector(data2) to get the raw values. Also, I used 1001 because I had 1001 observations. You will need something different." CreationDate="2010-08-18T15:42:29.480" UserId="8" />
  <row Id="1973" PostId="1843" Text="That's interesting, but how do we tell the reader which scale corresponds to which line?" CreationDate="2010-08-18T15:52:35.470" UserId="990" />
  <row Id="1974" PostId="1842" Text="Noob R user here: Error in data.frame(t = days, values = c(as.vector(data2), as.vector(cum)), : arguments imply differing number of rows: 1063, 1300, 2&#xA;" CreationDate="2010-08-18T16:01:35.510" UserId="990" />
  <row Id="1975" PostId="1842" Text="Type &quot;days&quot;, &quot;data2&quot; and &quot;cum&quot; to look at your data. Then look at &quot;length(days)&quot;, etc. You need to match up the time points with the values." CreationDate="2010-08-18T16:05:46.333" UserId="8" />
  <row Id="1976" PostId="1843" Text="Have a look at this graph: http://imgur.com/K8BCr.png&#xA;There, we present y-axis labels and ticks only where they apply to the data (i.e., for the left axis on the top of the graph, as the corresponding data, and for the right axis on the bottom of the graph, as the correspoding data).&#xA;Additionally we used different colors (as in the example above) and line types and explained it in the caption.&#xA;You could also use a line chart on the left and a bar chart on the right axis to make the distinction clearer.&#xA;" CreationDate="2010-08-18T16:11:38.543" UserId="442" />
  <row Id="1977" PostId="1816" Score="1" Text="Instead of bonferroni correction, you could always use the more powerful bonferroni holm. See my anyswer here: http://stats.stackexchange.com/questions/575/post-hocs-for-within-subjects-tests/760#760" CreationDate="2010-08-18T16:19:04.000" UserId="442" />
  <row Id="1979" PostId="1832" Text="+1 for Maxwell &amp; Delaney!" CreationDate="2010-08-18T16:40:08.380" UserId="442" />
  <row Id="1980" PostId="1844" Text="How do these estimates relate to the one given by Cox regression? That's gotta be the gold standard for estimating HR." CreationDate="2010-08-18T16:41:19.750" UserId="279" />
  <row Id="1983" PostId="1636" Text="A similar technique proceeds by computing the $\phi$ statistic of the contingency table. Since $\phi$ is actually the correlation coefficient of two 0-1 vectors (one based on prediction, the other on reality), one can use the Fisher R-Z transform (just $\atanh$) to get an approximately normal statistic. Comparing $\atanh{(\phi_1)}$ to $\atanh{(\phi_2)}$ becomes an exercise in comparing independent normals with known variances. A similar technique for tetrachoric coefficient probably exists as well. Regrettably, these techniques are fairly low power..." CreationDate="2010-08-18T17:05:15.583" UserId="795" />
  <row Id="1985" PostId="1720" Text="A nice example of a need to make inferences about the means, no matter what, is afforded by some environmental risk assessments. To simplify greatly, imagine you are planning to develop land into a park. You test the soils for some compound of concern and, as is often the case, find its concentration is approximately lognormally distributed.  Nevertheless, people using the park--who might directly become exposed to these soils--will effectively &quot;sample&quot; the soils uniformly at random as they move around. Their exposure over time will be the arithmetic mean concentration, not its geometric mean." CreationDate="2010-08-18T18:42:14.327" UserId="919" />
  <row Id="1986" PostId="1851" Text="(1)  Are these clusters in *L* alone or clusters in *N* -dimensional space?&#xA;&#xA;(2)  The penultimate remark about &quot;a function of *L* &quot; may be confusing, because it seems to conflict with what a &quot;parametric representation ... in ...v[i]&quot; would be (namely, a smooth map from an *M* &lt;&lt; *N* dimensional space into *N* dimensional space whose image closely approximates most of the records)." CreationDate="2010-08-18T18:53:39.840" UserId="919" />
  <row Id="1987" PostId="1851" Text="@whuber: (1) The points are in both in L and the N-dimensional space of the v_i. However since I have more than a few variables most points lie near the surface and using simple box cuts e.g. v_i&gt;0.5 would miss a lot of points that would have a &quot;good enough&quot; L.&#xA;&#xA;(2) Think of L as the onion layer I would be sitting on." CreationDate="2010-08-18T19:47:38.173" UserId="56" />
  <row Id="1988" PostId="1840" Score="1" Text="I don't think I'd call that &quot;plain&quot; English :)" CreationDate="2010-08-18T19:53:40.687" UserId="74" />
  <row Id="1989" PostId="1828" Text="This is very simple and useful" CreationDate="2010-08-18T19:54:32.167" UserId="74" />
  <row Id="1990" PostId="1840" Text="I think it's not so bad, generally I don't plan to change it." CreationDate="2010-08-18T20:12:03.390" UserId="88" />
  <row Id="1991" PostId="1830" Text="Could the downvoters explain why they feel the answer does not help? " CreationDate="2010-08-18T20:12:08.723" />
  <row Id="1992" PostId="1830" Text="I agree with Srikant. Would any downvoters please explain what's wrong." CreationDate="2010-08-18T20:25:41.110" UserId="8" />
  <row Id="1993" PostId="1830" Score="1" Text="By the way, if something can be done to improve the answer, feel free to offer suggestions. I would be more than happy to edit the answer in response to reasonable, constructive feedback." CreationDate="2010-08-18T20:37:32.690" />
  <row Id="1994" PostId="1825" Text="Hi Mike, thank you for your feedback. Temperature levels X1....X5 are specific degree values (i.e., heat stimuli = 42C, 44C, 46C, 48C, 50C; cold stimuli = 8C, 6C, 4C, 2C, 0C); however, they can also be conceptualized as stimuli ranging from &quot;likely to cause minimal discomfort (i.e., 42C or 8C)&quot; to &quot;likely to cause maximum discomfort permissible by my research ethics board (i.e., 50C, 0C)&quot;. With this being the case do you think that a series of univariate repeated measures or a multivariate analysis is ok?&lt;br/&gt;&#xA;I too am am interested to hear suggestions from any mixed effects/multivariate gurus" CreationDate="2010-08-18T20:39:53.807" UserId="835" />
  <row Id="1995" PostId="1856" Text="Just to be clear: your question is about the size of the data, not about the setting, correct?" CreationDate="2010-08-18T20:40:34.477" UserId="5" />
  <row Id="1996" PostId="1856" Text="Exactly, I wonder if there are any references about the &quot;smallest&quot; n (wrt. to a high number of variables), or more precisely if any cross-validation techniques (or resampling strategy like in RFs) remain valid in such an extreme case." CreationDate="2010-08-18T20:45:10.597" UserId="930" />
  <row Id="1997" PostId="1843" Text="The example you've given is very good... How did you managed to vertical offset each axis?" CreationDate="2010-08-18T20:55:13.987" UserId="990" />
  <row Id="1998" PostId="1454" Text="You can find more ideas and code in &quot;Separating Two Populations in a Sample,&quot; http://stats.stackexchange.com/questions/899/separating-two-populations-from-the-sample.  (Is there perhaps some better way of cross-referencing threads than I've done here?)" CreationDate="2010-08-18T21:49:36.920" UserId="919" />
  <row Id="1999" PostId="1859" Text="What do you mean by &quot;reducing&quot; a contingency table? Do you have to define the categories in advance? Do you know the number of observations you will have?" CreationDate="2010-08-18T23:01:55.330" UserId="279" />
  <row Id="2000" PostId="1326" Text="Forgotten bonus: unbalanced data are still useful!" CreationDate="2010-08-18T23:17:53.043" UserId="71" />
  <row Id="2001" PostId="1855" Text="That's my second scenario, the confidence interval is too large to have any inferential value within the design of the experiment since differences between conditions are based on effects with between S variability removed.  It seems it always has a compromise meaning and needs it's own special name because you can't use it like a regular CI." CreationDate="2010-08-18T23:19:21.933" UserId="601" />
  <row Id="2002" PostId="1855" Text="Blouin &amp; Riopelle (2005) called them narrow and broad inference confidence intervals but given that the general scientific populace outside stats has a hard enough time with regular ones..." CreationDate="2010-08-18T23:25:54.043" UserId="601" />
  <row Id="2003" PostId="1836" Score="1" Text="Amazing. I didn't know that. Thanks for the reference." CreationDate="2010-08-19T00:29:41.940" UserId="159" />
  <row Id="2004" PostId="1809" Text="This should be a comment on the original post, not an answer." CreationDate="2010-08-19T00:33:33.523" UserId="159" />
  <row Id="2005" PostId="1823" Text="Thanks Jeromy.  I think I'll just ask here on how to do that.  I have wrote code in the past for power analysis, but they tended to get complex - I would be curious to see how betteR coder then me would solve this." CreationDate="2010-08-19T01:32:30.810" UserId="253" />
  <row Id="2006" PostId="1865" Text="Apparently this has an easy solution." CreationDate="2010-08-19T01:38:51.183" UserId="994" />
  <row Id="2007" PostId="1847" Text="I am guessing you mean this:  http://www.stat.columbia.edu/~gelman/arm/  I just read a good review about it here: http://www.r-bloggers.com/bookshelf-remodelling/" CreationDate="2010-08-19T02:02:15.573" UserId="253" />
  <row Id="2008" PostId="1865" Score="1" Text="Want to post it ?" CreationDate="2010-08-19T02:28:26.027" UserId="253" />
  <row Id="2009" PostId="1862" Text="How about PCA/FA? You would shrink correlated variables into factors and work from there..." CreationDate="2010-08-19T05:46:16.097" UserId="144" />
  <row Id="2010" PostId="1560" Text="according to the paper I quote both James stein and soft/hard thresholding satisfy oracle inequalities. I guess James Stein is more difficult to manipulate og to understand intuitively than hard thresholding but the why question is a good question!  " CreationDate="2010-08-19T06:32:30.733" UserId="223" />
  <row Id="2011" PostId="1874" Text="you mean a a good 3D image reconstruction algorithm? maybe you only need an algorithm to estimate 3-D connex region? something like level set segmentation? " CreationDate="2010-08-19T06:47:33.400" UserId="223" />
  <row Id="2012" PostId="1850" Text="In case it's useful for a potential answerer formulas are listed here: http://en.wikipedia.org/wiki/Effect_size" CreationDate="2010-08-19T06:58:32.677" UserId="183" />
  <row Id="2013" PostId="1883" Text="The only way you can keep this question open is by making it community wiki so please tick the case. " CreationDate="2010-08-19T07:03:02.727" UserId="223" />
  <row Id="2014" PostId="1883" Score="2" Text="However, I have the feeling that this is subjective, argumentative and will require extended discussion please read http://stats.stackexchange.com/faq&#xA;I vote to close but encourage you to ask a more specific question (since the idea of the question is good but way too wide). " CreationDate="2010-08-19T07:05:59.710" UserId="223" />
  <row Id="2015" PostId="1883" Text="one of the extended discussion that could start: are you sure that Prof Rob Hyndman was a reasearcher when parzen and Rozenblatt proposed exponential smoothing :) ? " CreationDate="2010-08-19T07:08:13.167" UserId="223" />
  <row Id="2016" PostId="1879" Text="Very helpful answer!. &quot;Discovering Statistics Using SPSS&quot; is the one I was also using. Perhaps I'll also have to review SPSS in the year to come. :-(" CreationDate="2010-08-19T07:17:32.367" UserId="339" />
  <row Id="2017" PostId="1883" Score="1" Text="I think with the availability of more powerful computers, different kinds of methods suddenly become practical and important (would one use e.g. boosted decision trees without fast computers ?)" CreationDate="2010-08-19T07:32:31.487" UserId="961" />
  <row Id="2018" PostId="1857" Text="Thanks! I have Hastie's book and that of C. Bishop (Pattern Recognition and Machine Learning). I know that such a small n would lead to spurious or unreliable (see Jeromy Anglim's comment) association. However, the RF algorithm as implemented by Breiman allows to cope with a limited number of features each time a tree is grown (in my case, 3 or 4) and although OOB error rate is rather high (but this should be expected), analyzing variable importance lead me to conclude that I would reach similar conclusion using bivariate tests (with permutation test)." CreationDate="2010-08-19T07:35:00.717" UserId="930" />
  <row Id="2019" PostId="1883" Text="To add to Robin's comment, your question seems more appropriate for a blog or a discussion forum. " CreationDate="2010-08-19T07:36:07.750" />
  <row Id="2020" PostId="1858" Text="Thanks! I was attending the IVth EAM-SMABS conference last month, and one of the speaker presented an application of ML in a biomedical study; unfortunately, this was a somewhat &quot;standard&quot; study with N~300 subjects and p=10 predictors. He is about to submit a paper to *Statistics in Medicine*. What I am looking for is merely articles/references wrt. standard clinical study with, e.g. outpatients, where generalizability of the results is not so much an issue." CreationDate="2010-08-19T07:39:58.490" UserId="930" />
  <row Id="2021" PostId="1423" Text="Prediction about the past can also be surprisingly tricky!" CreationDate="2010-08-19T07:58:29.943" UserId="174" />
  <row Id="2022" PostId="1850" Text="A simulation in R with varying n1, n2, s1, s2, and population difference would make a nice exercise. Anyone?" CreationDate="2010-08-19T08:04:16.043" UserId="183" />
  <row Id="2023" PostId="1843" Score="2" Text="Really good example. The only issue with your graph, is that both Y variable names are overlapping. In this case you would want one on the left and the other on the right (possibly even in a vertical position). To upgrade your example from &quot;really good&quot; to &quot;perfect&quot;, you might wanna use the mtext function from R to do the variable names" CreationDate="2010-08-19T09:18:15.950" UserId="447" />
  <row Id="2024" PostId="1883" Text="To subjective. I would vote close." CreationDate="2010-08-19T09:25:47.103" UserId="8" />
  <row Id="2025" PostId="1877" Text="Yes in this case you should go for binomial test instead of using asymptotic chi squared test." CreationDate="2010-08-19T09:28:02.877" UserId="994" />
  <row Id="2026" PostId="813" Text="By God, he was right!" CreationDate="2010-08-19T09:32:41.740" UserId="994" />
  <row Id="2027" PostId="729" Score="3" Text="God must bring data too." CreationDate="2010-08-19T09:33:49.293" UserId="994" />
  <row Id="2028" PostId="1809" Score="1" Text="Sorry. Couldn't comment, as I had &lt; 50 points. However, I wanted to try to help the poster, and that was the only mechanism I had." CreationDate="2010-08-19T09:54:28.110" UserId="561" />
  <row Id="2029" PostId="1843" Text="@Hugo @Dave: See my update for an incorporation of both comments." CreationDate="2010-08-19T10:00:12.460" UserId="442" />
  <row Id="2030" PostId="750" Score="3" Text="Whoever said this had no basic understanding of Statistics, or he was joking." CreationDate="2010-08-19T10:21:44.893" UserId="994" />
  <row Id="2031" PostId="835" Text="I would say data is the bullet, Statistics is the gun." CreationDate="2010-08-19T10:23:50.663" UserId="994" />
  <row Id="2032" PostId="1406" Text="All physical experiments I have ever seen have a standard deviation attached to it in some form (most commonly a +/- range). So he must have been joking." CreationDate="2010-08-19T10:25:27.590" UserId="994" />
  <row Id="2033" PostId="1888" Text="The Clason and Dormody article looks good.&#xA;Your response distribution comments are interesting to contemplate. I agree that differences in distributions might be of interest. But if you were only interested in whether population group means were different, it would not necessarily matter what distributions gave rise to such equality." CreationDate="2010-08-19T10:33:16.880" UserId="183" />
  <row Id="2034" PostId="1889" Text="@Dirk like so ;-)" CreationDate="2010-08-19T10:44:24.840" UserId="88" />
  <row Id="2035" PostId="1849" Score="1" Text="@Aniko. I agree. Also, depending on the domain a weighted composite of categories might be a more appropriate index of an overall rating." CreationDate="2010-08-19T10:55:16.013" UserId="183" />
  <row Id="2036" PostId="1848" Score="1" Text="Interesting question. Initially, I had some trouble understanding your pseudo-code, and then I read: http://www.thebroth.com/blog/118/bayesian-rating" CreationDate="2010-08-19T10:57:16.523" UserId="183" />
  <row Id="2037" PostId="1888" Text="In this case, you are assuming that your Likert scale (in other words, the perceived difference between, e.g. much satisfied and &quot;just&quot; satisfied) behaves ideally and is perceived as having the same meaning in both population. Thus you are implicitly making the assumption that this is a numeric scale, but I agree that this is often considered as such in applied research, especially if participants come from the same country. My point was just to emphasize the categorical data analysis perspective, as usually found in the Factor Analysis tradition, like in my reply to Question #10." CreationDate="2010-08-19T11:01:26.003" UserId="930" />
  <row Id="2038" PostId="1889" Text="Thanks -- I once knew that trick." CreationDate="2010-08-19T11:23:24.813" UserId="334" />
  <row Id="2039" PostId="1892" Text="What about SAS and spss?" CreationDate="2010-08-19T11:29:12.243" UserId="5" />
  <row Id="2040" PostId="1883" Text="There are answers from three persons that can vote for closing and only csgillepsie and I voted for closing. I guess this means people want it open. It is important to discuss that. I have openned a discussion about that on meta http://meta.stats.stackexchange.com/questions/336/what-is-the-limit-in-subjective-argumentative-questions&#xA;" CreationDate="2010-08-19T11:40:28.403" UserId="223" />
  <row Id="2041" PostId="1890" Text="is turing the site into a discussion forum a revolution ;) ? " CreationDate="2010-08-19T11:41:47.620" UserId="223" />
  <row Id="2042" PostId="1896" Text="I would go with variance and/or standard deviation (sd is just the square root of the variance) instead of the absolute deviation, because variance and sd punish extreme outliers stronger. This seems desirable in your case." CreationDate="2010-08-19T12:13:10.383" UserId="442" />
  <row Id="2043" PostId="1896" Text="When I go with standard deviation or variance, I come up with the problem of how to combine the results for the subbenchmarks into a whole. Absolute variance (assuming that's what I did) gives me a single number." CreationDate="2010-08-19T12:17:38.690" UserId="1001" />
  <row Id="2044" PostId="1896" Text="Note that I tried to combine my sub-variances using &quot;the variance of the total is the mean of the variances of the subgroups + variance of the means of the subgroups&quot;, which I got from http://en.wikipedia.org/wiki/Variance." CreationDate="2010-08-19T12:18:48.187" UserId="1001" />
  <row Id="2045" PostId="1892" Text="And don't forget Stata." CreationDate="2010-08-19T12:38:20.613" UserId="521" />
  <row Id="2046" PostId="1892" Text="Probably deserving of its own question: Which statistical package has made the most revolutionary contribution to the science and practice of data analysis and statistics?" CreationDate="2010-08-19T12:44:34.500" UserId="183" />
  <row Id="2047" PostId="1897" Text="I'm told that the results of the benchmarks (the times in ms) are not normally distributed. Will the variances be normally distributed? How does that affect the choice of 1.96?" CreationDate="2010-08-19T12:49:45.737" UserId="1001" />
  <row Id="2048" PostId="1892" Text="That would be overly argumentative.  I think having an answer here that acknowledges all statistical software is on point." CreationDate="2010-08-19T13:06:15.627" UserId="5" />
  <row Id="2049" PostId="1883" Text="I would vote to close, but when moderators vote to close, it's closed immediately.  I think that Colin and I are showing restraint and letting the community decide." CreationDate="2010-08-19T13:07:50.927" UserId="5" />
  <row Id="2050" PostId="1883" Score="1" Text="Answering the question is not really a clear indication that you would vote to close. People saw my comment, they saw your answer, ... 10 very fast heterogeneous answers in less than an hour ! looks like a chat room ;) " CreationDate="2010-08-19T13:25:07.753" UserId="223" />
  <row Id="2051" PostId="1883" Text="That's fair, although so long as the question is open, I think that it should be answered correctly.  :)  If the close votes get to 4, then I will vote to close." CreationDate="2010-08-19T13:28:31.897" UserId="5" />
  <row Id="2052" PostId="1892" Text="@Shane. Fair enough. I used to use SPSS. Now I use R. R revolutionised the way that I think about and conduct data analysis. It made data analysis fun. I can't speak too much about Stata and SAS, so I'll leave it for others to justify why they might be revolutionary." CreationDate="2010-08-19T13:32:03.713" UserId="183" />
  <row Id="2053" PostId="1896" Text="A strong case can be made for robust techniques that do the opposite of &quot;punishing&quot; outliers, because outliers in benchmarks often occur unavoidably due to other processes and external perturbations causing a process to wait from time to time.  This could be the reason the benchmarks do not appear to be normally distributed, too.  However, as cgillespie's answer points out, you can go a long way with the standard hypothesis testing machinery applied to SDs.  As a compromise, a lightly Winsorized version of the SD might be a good choice.  (http://en.wikipedia.org/wiki/Winsorising )" CreationDate="2010-08-19T13:47:27.700" UserId="919" />
  <row Id="2054" PostId="1895" Score="2" Text="An amusing observation: all your benchmark times are multiples of 2^-12 (=1/4096)." CreationDate="2010-08-19T13:48:59.023" UserId="919" />
  <row Id="2055" PostId="1909" Score="1" Text="The www.r-project.org/useR-20xx  URLs are preferable." CreationDate="2010-08-19T13:56:00.550" UserId="334" />
  <row Id="2056" PostId="1912" Text="sorry if it is my english (unfortunatly I'm french) but is it proper english ? what do you mean by &quot;losing power&quot; is it about testing? if yes, testing what ? " CreationDate="2010-08-19T14:22:29.410" UserId="223" />
  <row Id="2057" PostId="1912" Score="1" Text="There's nothing unfortunate about being French.  :)  Regretfully, I don't entirely know what Gary was talking about (hence the question).  I might assume that he meant &quot;loss of information&quot;, but would be happy to get guidance in that respect.  I would say that this is correct english, but questionable statistics." CreationDate="2010-08-19T14:28:46.447" UserId="5" />
  <row Id="2058" PostId="1901" Text="Interesting... I'm a bit busy at the moment, but I'll definitely have a look at that!" CreationDate="2010-08-19T14:35:21.343" UserId="582" />
  <row Id="2059" PostId="1912" Text="Perhaps, you can email him and ask? You could use the opportunity to point to this thread seeking his permission to post his answer here. (indirect promotion of the site)" CreationDate="2010-08-19T14:41:44.793" />
  <row Id="2060" PostId="1912" Text="@Srikant: Done; we'll see what he says." CreationDate="2010-08-19T14:54:49.207" UserId="5" />
  <row Id="2061" PostId="1636" Text="Just for those of you not familiar with Phi (\phi), http://en.wikipedia.org/wiki/Contingency_table#Measures_of_association" CreationDate="2010-08-19T15:14:29.700" UserId="196" />
  <row Id="2062" PostId="1915" Text="sorry for the graphical-techniques tag, but my brand new account doesn't let me tag with 'graph' and 'partitioning' as I intended to." CreationDate="2010-08-19T16:10:09.153" UserId="1007" />
  <row Id="2063" PostId="1871" Text="Thanks all.&#xA;&#xA;The behavior that lead to the accuracy and time measures was very controlled. I'm looking at the results from an exploratory perspective, while trying to ask why the inaccuracies occurred.&#xA;&#xA;I'm unsure about &quot;Each participant provides 6 values, therefore it is not unbalanced&quot;? When subjects are correct, I have a different number of time measures between the levels, e.g. subject A got 23 correct in level 1, 16 in level 2, 10 in level 3. If I take the mean correct per level per subject can I just ignore the number of correct answers &amp; the difference in the number of correct answers?" CreationDate="2010-08-19T16:25:38.313" UserId="993" />
  <row Id="2064" PostId="1915" Text="@csgillespie: thank you for retagging it :)" CreationDate="2010-08-19T16:55:17.843" UserId="1007" />
  <row Id="2065" PostId="1852" Score="1" Text="(1) I think you meant to write that the *sample*, not the *population* mean, is 112 with 95% *probability*, not *confidence.*&#xA;&#xA;(2) Your point is well taken--it could apply to any question--but isn't it stated a little extremely? First, the question doesn't ask for an inference about the population mean: we are told it is $200. Thus, conditional on the assumptions, we can estimate the population mean with certainty! Second, even if we were asked to estimate the population mean from the sample, there's still some trivial information we could offer (e.g., it doesn't exceed $10^11 per annum)." CreationDate="2010-08-19T16:56:09.480" UserId="919" />
  <row Id="2066" PostId="1852" Text="(1) good catch. (2), yes, I can make the problem setup asymptotically perverse for fixed results, _post hoc_. my bad.&#xA;however, I am no longer sure what the OP is trying to test. If they know the population mean is 200, why are they trying to test it?" CreationDate="2010-08-19T17:13:03.993" UserId="795" />
  <row Id="2067" PostId="1852" Text="BTW, evidently a CEO salary/least paid salary ratio of 400 is not considered extreme in the US. 800 is a bit perverse, though." CreationDate="2010-08-19T17:15:45.953" UserId="795" />
  <row Id="2068" PostId="1919" Text="May fit better under: http://stats.stackexchange.com/questions/1906/data-mining-conferences?" CreationDate="2010-08-19T17:16:13.207" UserId="5" />
  <row Id="2069" PostId="1920" Text="Already noted here: http://stats.stackexchange.com/questions/1906/data-mining-conferences" CreationDate="2010-08-19T17:21:43.103" UserId="5" />
  <row Id="2070" PostId="1919" Text="@Shane, I guess so. I hadn't seen the other two conference questions before I answered." CreationDate="2010-08-19T17:28:27.027" UserId="11" />
  <row Id="2071" PostId="1912" Text="I saw this as well, and assumed it was a warning against blindly studentizing data." CreationDate="2010-08-19T17:47:36.353" UserId="795" />
  <row Id="2072" PostId="1916" Text="Thanks. I fitted a no nugget model to the likelihood using proc mixed, I guess I should compare the AICC with that of a model with nugget." CreationDate="2010-08-19T18:08:06.140" UserId="1004" />
  <row Id="2073" PostId="1844" Text="Cox model incorporates covariates. The Kaplan-Meier, Nelson-Aalen, Mantel-Haenszel methods model hazard as a function only of age." CreationDate="2010-08-19T18:54:53.113" UserId="795" />
  <row Id="2074" PostId="1871" Text="those raw answers... 23, 16, etc. cannot go into a repeated measures ANOVA.  You need to get their means and enter one value of each (6 total) for each S." CreationDate="2010-08-19T19:52:32.937" UserId="601" />
  <row Id="2075" PostId="1924" Text="Hm...following the derivations under asymptotic normality link I'm a bit confused...what's the difference between I and H?" CreationDate="2010-08-19T20:14:36.283" UserId="511" />
  <row Id="2076" PostId="1924" Text="There is no difference. See the [definition of fisher information i.e., I](http://en.wikipedia.org/wiki/Fisher_information#Definition) where it is re-written as H. I guess the variance expression can be simplified on the wiki as $H^{-1} I$ is an identity matrix." CreationDate="2010-08-19T20:32:39.393" />
  <row Id="2077" PostId="1924" Text="Hm....it says that H becomes I when the model is correctly specified, this makes me wonder what's the difference when it is not" CreationDate="2010-08-19T20:50:44.980" UserId="511" />
  <row Id="2078" PostId="1928" Text="Nice reference: thanks.  It comprehensively covers everything mentioned here." CreationDate="2010-08-19T21:03:18.633" UserId="919" />
  <row Id="2079" PostId="1884" Text="I am not convinced -- is P(X_n != X_(n+1)) independent from n?" CreationDate="2010-08-19T21:36:15.600" UserId="573" />
  <row Id="2080" PostId="1874" Text="Just a good image reconstruction algorithm(The regions involved are often not convex). It's just that there are a lot of different ones, and I'm not familiar which ones are best in the context I'm approaching the problem in. (Easy to parametrize estimated surfaces, easy to compute volume, and easy to find minimum distance from a given point)" CreationDate="2010-08-19T21:55:29.840" UserId="996" />
  <row Id="2081" PostId="1924" Text="Where does it say that? I am a bit unsure what the model correctness has got to do with the relationship between H and I." CreationDate="2010-08-19T22:36:29.543" />
  <row Id="2082" PostId="1791" Text="Well that's a bunch of hints. I'll have a look at those, thx!" CreationDate="2010-08-19T23:02:21.863" UserId="979" />
  <row Id="2083" PostId="1924" Text="At the bottom of the asymptotic normality derivation from your link. It says &quot;Finally, the information equality guarantees that when the model is correctly specified, matrix H will be equal to the Fisher information I, so that the variance expression simplifies to just I^−1&quot; " CreationDate="2010-08-19T23:19:04.473" UserId="511" />
  <row Id="2084" PostId="1924" Text="I thought about it and I fail to see the connection to the model being the correct one for the data. I do not have a standard textbook to see what the issue is. If I can think of something I will update my answer or add a comment." CreationDate="2010-08-20T01:03:51.620" />
  <row Id="2085" PostId="1909" Text="Thanks Dirk. I added the link to R conferences on R-project" CreationDate="2010-08-20T01:44:32.523" UserId="183" />
  <row Id="2086" PostId="1940" Text="Great article - thanks!" CreationDate="2010-08-20T01:48:14.233" UserId="561" />
  <row Id="2087" PostId="1925" Text="Would you like to explain your reasons?&#xA;I can see how such a model might provide a more precise model of observed responses. However, in the typical practical research situations that I have seen, researchers are interested in whether the two groups differ in terms of the mean (e.g., did the training group report greater performance than the control; was student satisfaction higher one year to the next). The proportional odds ratio model does not test this question exactly as far as I am aware." CreationDate="2010-08-20T01:53:50.553" UserId="183" />
  <row Id="2088" PostId="1888" Text="I assume that the mean of the sample responding to a Likert item is generally a meaningful summary of the group's position on the underlying dimension.&#xA;It's interesting to think about when the meaning of a Likert item would vary systematically between groups. Of course, this issue extends beyond just Likert items, probably to any subjective measurement procedure." CreationDate="2010-08-20T02:03:10.717" UserId="183" />
  <row Id="2089" PostId="1836" Score="1" Text="@whuber: Thanks - any chance this is the Rodriguez paper you're thinking of?  http://arxiv.org/abs/bayes-an/9504001" CreationDate="2010-08-20T06:07:24.607" UserId="251" />
  <row Id="2090" PostId="1849" Text="Thanks for the thoughts Aniko. I'll take a look at this in the next few days." CreationDate="2010-08-20T06:18:29.590" UserId="991" />
  <row Id="2091" PostId="1844" Text="@shabbychef: with Cox PH, use a single binary covariate, i.e. coded 0/1 for reference/comparison groups, then exp(beta) = HR." CreationDate="2010-08-20T06:47:38.413" UserId="251" />
  <row Id="2092" PostId="1944" Score="2" Text="maybe `standardized` ?" CreationDate="2010-08-20T08:34:49.683" UserId="339" />
  <row Id="2093" PostId="1945" Text="wtf. you are to fast. i was still typing. nevertheless +1 for the correct answer." CreationDate="2010-08-20T08:45:05.027" UserId="442" />
  <row Id="2094" PostId="1946" Text="no, it is called z-score (small letter), big letter Z-scores refer to mean = 100 and sd = 10." CreationDate="2010-08-20T08:46:51.370" UserId="442" />
  <row Id="2095" PostId="1945" Text="Thank you ars! :)" CreationDate="2010-08-20T08:49:46.013" UserId="582" />
  <row Id="2096" PostId="1946" Text="@Henrik I have never heard of something like this; nevertheless I've made the change, it seems lowercase z is used more often. " CreationDate="2010-08-20T09:25:39.137" UserId="88" />
  <row Id="2097" PostId="1946" Text="@mbq could be that it is a psychology specific distinction. browsing the web i realized that it could even be specific German. At least I only found it in the German wikipedia (not surprisingly i learned it in a German university): http://de.wikipedia.org/wiki/Normskala" CreationDate="2010-08-20T09:50:21.083" UserId="442" />
  <row Id="2099" PostId="1853" Text="Multivariate or univariate ?(it makes a huge difference in this context)" CreationDate="2010-08-20T10:09:52.970" UserId="603" />
  <row Id="2100" PostId="1944" Text="Just a comment on the tag. Right now we have one question tagged &quot;nomenclature&quot; and another question tagged &quot;terminology&quot;. I personally prefer &quot;terminology&quot; - it's much more common in everyday language than nomenclature, but I see nomenclature potentially being used. Could someone consider a tag synonym or just a rename one way or the other?" CreationDate="2010-08-20T10:18:43.617" UserId="110" />
  <row Id="2101" PostId="1944" Text="@Thomas Owens: you are right, terminology is better, I changed it.&#xA;I guess &quot;nomenclature&quot; sounds better to people in biology :)" CreationDate="2010-08-20T11:05:26.677" UserId="582" />
  <row Id="2102" PostId="1922" Text="upvoted for the reference to the paper and the book, I'll read them " CreationDate="2010-08-20T11:54:36.750" UserId="840" />
  <row Id="2103" PostId="1862" Text="this might be too much, if management asks 'how did you get the aggregated numbers?' they'll want a simpler technique so they can (feel they) understand it. Alas, the real world :-( Thanks, though." CreationDate="2010-08-20T11:57:37.980" UserId="840" />
  <row Id="2104" PostId="1869" Text="I'll mark this as the answer; there are several good suggestions in it so I'll think how to apply them." CreationDate="2010-08-20T11:58:40.817" UserId="840" />
  <row Id="2105" PostId="1943" Text="Having had a brief look at that paper i'm not sure the estimates they consider are the same as those in the questioner's equations. I agree with the comments under the question - maybe in 1981 approximate methods were useful but these days there's no obvious reason not to use Cox regression." CreationDate="2010-08-20T12:42:10.483" UserId="449" />
  <row Id="2106" PostId="1844" Text="The log-rank is a more powerful test than Cox PH when the proportionbal Hazards assumption is satisfied. So with a single 2-level covariate, a log-rank or Mantel-Haenszel test is preferable." CreationDate="2010-08-20T12:54:43.443" UserId="521" />
  <row Id="2107" PostId="1844" Text="see below for answer..." CreationDate="2010-08-20T13:11:51.667" UserId="521" />
  <row Id="2108" PostId="1949" Text="Interesting... I guess analytical chemists are very much behind the times! Mind telling me how both of these became discredited? I will look into your reference and see how the algorithms for these look like." CreationDate="2010-08-20T13:31:16.067" UserId="830" />
  <row Id="2109" PostId="1836" Text="@ars: Yes, it is: the URL in the subtitle confirms it.  Thanks for finding it." CreationDate="2010-08-20T14:16:48.697" UserId="919" />
  <row Id="2110" PostId="1941" Text="Thank you very much for this thought.  Indeed, this is a standard and well-documented approach to multiple censoring.  One difficulty lies in its intractability: those integrals are notoriously difficult to compute.  There's a modeling problem lurking here, too: the value of *d* is usually positively correlated with *Y*, as implied by the first paragraph of my description." CreationDate="2010-08-20T14:22:58.607" UserId="919" />
  <row Id="2111" PostId="1872" Text="Thank you Kevin - I went with this option, went a head and ordered his book." CreationDate="2010-08-20T14:37:13.133" UserId="253" />
  <row Id="2112" PostId="1879" Text="Great answer Jeromy, I will make use of the resources you offered. thanks!" CreationDate="2010-08-20T14:38:01.977" UserId="253" />
  <row Id="2113" PostId="1912" Text="He says:&#xA;&quot;not stat'l power, but scale invariance loses the power that can be extracted from knowledge of the substance&quot;&#xA;http://twitter.com/kinggary/status/21624260888" CreationDate="2010-08-20T14:40:01.723" UserId="449" />
  <row Id="2116" PostId="1912" Text="Thanks @onestop: now I understand what he meant.  Even though I know the answer, I'll update the question for future searches." CreationDate="2010-08-20T15:21:20.860" UserId="5" />
  <row Id="2117" PostId="1956" Score="1" Text="You can use the HMM package: http://cran.r-project.org/web/packages/HMM/index.html." CreationDate="2010-08-20T15:26:12.730" UserId="5" />
  <row Id="2118" PostId="1943" Text="@onestop: hmm, think definition of O/E == LR with the log forgotten above? I agree with what you say about Cox PH -- that's not the question I was trying to answer, but your advice is better in the broader context." CreationDate="2010-08-20T15:27:21.653" UserId="251" />
  <row Id="2119" PostId="1946" Text="Let's not confuse the z-score with the process of standardization.  The former is described in ars' answer: it is a statistic.  Standardization consists of replacing each of the X[i] by (X[i]-m)/s for further analysis.  That's what nico was asking about." CreationDate="2010-08-20T15:58:49.197" UserId="919" />
  <row Id="2120" PostId="1844" Text="@Thylacoleo: the log-rank test is equivalent to the Score test under Cox PH. Think you're confusing PH *tests* with HR *estimators* here." CreationDate="2010-08-20T16:10:02.463" UserId="251" />
  <row Id="2121" PostId="1961" Text="In the context of categorical variables, one usually talks of a &quot;measure of association&quot; instead of &quot;correlation&quot;. Searching for this term should lead you to lots of options." CreationDate="2010-08-20T17:20:03.630" UserId="279" />
  <row Id="2122" PostId="1958" Text="Thanks for the suggestions. I'll have to look into them in more detail, but one thing that occurs immediately is that the series do not share a time frame, so cointegration and VAR are probably not relevant. I've edited the question to try to clarify this, apologies for being misleading before. Noise filtering brings up some other issues, but those are for another question and another day!" CreationDate="2010-08-20T17:44:04.463" UserId="174" />
  <row Id="2123" PostId="1958" Text="Ok.  Well, looking at the distributions may be the way to go.  You can also try a distribution test such as Anderson–Darling (as in this question: http://stats.stackexchange.com/questions/1645/appropriate-normality-tests-for-small-samples)." CreationDate="2010-08-20T18:01:16.977" UserId="5" />
  <row Id="2124" PostId="1961" Text="Thanks Aniko. I'll correct my question here (and also look for it)" CreationDate="2010-08-20T18:04:46.707" UserId="253" />
  <row Id="2125" PostId="1947" Text="Nice answer, Robin!  " CreationDate="2010-08-20T18:07:46.560" UserId="251" />
  <row Id="2129" PostId="1895" Text="@whuber: must be the maximum resolution - a quarter of a microsecond ain't bad! 22-bits resolution-ish?" CreationDate="2010-08-20T19:51:40.860" UserId="1001" />
  <row Id="2130" PostId="1963" Text="Thank you @shane for creating the meta-analysis tag: I wanted to but couldn't." CreationDate="2010-08-20T20:38:57.143" UserId="919" />
  <row Id="2133" PostId="1965" Text="You're correct for the case beta = 0, but not for positive values of beta: you lost everything by throwing away the -beta*n*log(n) term." CreationDate="2010-08-20T21:41:12.633" UserId="919" />
  <row Id="2134" PostId="498" Text="I'm with Peter and Srikant. While I generally support the policy change of being more broadly accepting of computing questions, this one seems far removed from statistical content and more like a question for tech support. One here and there doesn't bother me, but I think it would be seriously detrimental if the main page became cluttered by such questions." CreationDate="2010-08-20T22:32:00.287" UserId="71" />
  <row Id="2135" PostId="498" Text="Keeping the majority of questions about statistical analysis will encourage experts to check out questions that might lie outside of the immediate domain.  If we get to the point where tags have to be used to sort past all of the questions about interfacing with statistical programs, I suspect we'll lose valuable cross-pollination." CreationDate="2010-08-20T22:33:43.843" UserId="71" />
  <row Id="2136" PostId="498" Text="For example: the GIS Exchange (http://gis.stackexchange.com/) is very permissive about what kinds of questions can be asked there, and as a result, the main page is almost completely dominated by technical questions.  That's fine and clearly what the community there wants, but it makes it tough to find questions that grapple with interesting problems or applications - so I largely ignore most of the questions and overall participate much less than I do here.  Interesting theoretical questions and applications are the majority of what I see here, and I'd like to keep it that way." CreationDate="2010-08-20T22:47:36.910" UserId="71" />
  <row Id="2137" PostId="498" Text="(Sorry for the big block o' text)" CreationDate="2010-08-20T22:48:20.417" UserId="71" />
  <row Id="2139" PostId="1965" Text="uhh, no, not really. I assumed the OP was interested in an asymptotic (in $n$) bound. the worst case for my lower bound is $\beta = 1$. Then for, say, $n &gt; 35$, we have $\log{n} &lt; 0.1 n$, and thus $\log{L} = -n\log{n} + n^2 &gt; 0.9 n^2$. Using L'Hopital's rule, you can show that $\log{L} \ge (1 - \epsilon) n^2$, asymptotically in $n$, for every $\epsilon &gt;= 0$." CreationDate="2010-08-21T00:47:02.553" UserId="795" />
  <row Id="2140" PostId="1972" Text="Articles directly about this would be hard to find... perhaps look up least significant difference scores... which is equivalent to what you ware trying to do." CreationDate="2010-08-21T02:09:24.930" UserId="601" />
  <row Id="2141" PostId="1961" Text="After a second thought - what I am looking at here is ordered, so looking at three levels as categorical would not fit here..." CreationDate="2010-08-21T03:40:05.790" UserId="253" />
  <row Id="2142" PostId="1962" Score="2" Text="Quick note: Phi and Pearson correlation give the same value. Phi is just a simpler calculation procedure." CreationDate="2010-08-21T05:10:33.107" UserId="183" />
  <row Id="2143" PostId="1962" Text="@Jeromy: thanks." CreationDate="2010-08-21T05:50:03.713" UserId="251" />
  <row Id="2144" PostId="1982" Text="These look like some good examples. Cheers." CreationDate="2010-08-21T07:02:06.553" UserId="183" />
  <row Id="2145" PostId="1844" Text="ars, you are probably correct. I wrote my comment from memory and it doesn't seem correct on reflection." CreationDate="2010-08-21T09:48:28.893" UserId="521" />
  <row Id="2146" PostId="1962" Score="1" Text="And Cramer's V gives the same value as Phi for 2x2 tables." CreationDate="2010-08-21T11:02:27.203" UserId="1356" />
  <row Id="2147" PostId="1984" Text="No, not formally -- LaTeX submission is encourages but if you look at the [instructions page](http://www.jstatsoft.org/instructions) it does not contain the word Sweave.  Authors do use it and/or ship the R code with the paper, but to me this echos Shane's point about package vignettes." CreationDate="2010-08-21T11:44:07.067" UserId="334" />
  <row Id="2148" PostId="1984" Text="Ok, still most submitters do use it (also journal style includes Swave.sty); the main problem is that there are no Rnws published, still papers made by Sweave come with Stangle output." CreationDate="2010-08-21T13:21:08.277" UserId="88" />
  <row Id="2149" PostId="1965" Text="oops, should be $\epsilon &gt; 0$; letting $\epsilon = 0$ is right out." CreationDate="2010-08-21T16:03:09.020" UserId="795" />
  <row Id="2150" PostId="1884" Score="5" Text="@KalEl: You ought to credit the person who *really* answered this question (George Lowther), especially because you copied his answer with only trivial changes.  See http://math.stackexchange.com/questions/2763/what-is-the-expected-number-of-runs-of-same-color-in-a-standard-deck-of-cards ." CreationDate="2010-08-21T17:17:44.323" UserId="919" />
  <row Id="2152" PostId="1359" Score="1" Text="@Brett There are a few book recommendations on my [related question](http://stats.stackexchange.com/questions/1815/recommended-books-on-experiment-design) and some others may accrue in time. Not sure how well they fit your criteria, but might be worth a glance." CreationDate="2010-08-21T18:18:12.117" UserId="174" />
  <row Id="2154" PostId="1986" Text="I know about Tukey's HSD. I think I was unclear in my question. While i'm doing ANCOVA, i'm not trying to generate CIs from post-hoc pairwise comparisons. I'm just generating them by bootstrapping the samples in each level of the factor (also tried using the t-distribution). The CIs are for the mean of each individual group (factor level). I agree with Tal's comment that this does not allow differences between groups to be assessed. It's easy to do a Bonferroni correction on these CIs, is it possible to do the Hochberg (or other) method? (p.adjust requires a vector of p values I don't have)." CreationDate="2010-08-21T23:26:12.157" UserId="1029" />
  <row Id="2155" PostId="1868" Text="Your blog is chock full of useful nuggets; thanks!" CreationDate="2010-08-22T00:21:34.117" UserId="251" />
  <row Id="2156" PostId="1998" Text="The following question on [&quot;intuitive explanation of multicollinearity&quot;](http://stats.stackexchange.com/questions/1149/is-there-an-intuitive-explanation-why-multicollinearity-is-a-problem-in-linear-re) may be of some help in the above context." CreationDate="2010-08-22T01:30:24.227" />
  <row Id="2157" PostId="1997" Score="1" Text="I second this answer and would just like to add another great reference on this topic:  Singer's Applied Longitudinal Data Analysis text &lt;http://gseacademic.harvard.edu/alda/&gt;.  Though it is specific to longitudinal analysis, it gives a nice overview of MLM in general.  I also found Snidjers and Bosker's Multilevel Analysis good and readable &lt;http://stat.gamma.rug.nl/multilevel.htm &gt;.  John Fox also provides a nice intro to these models in R here &lt;http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf&gt;." CreationDate="2010-08-22T03:17:00.480" UserId="485" />
  <row Id="2158" PostId="2002" Text="Notice that the span measure would mean different window size for different number of observations." CreationDate="2010-08-22T04:10:17.800" UserId="253" />
  <row Id="2159" PostId="2007" Score="1" Text="I advise this to be a community wiki.  " CreationDate="2010-08-22T04:13:00.603" UserId="253" />
  <row Id="2160" PostId="2007" Text="sounds like a homework question" CreationDate="2010-08-22T06:15:00.160" UserId="74" />
  <row Id="2161" PostId="2008" Score="4" Text="If your aim is to judge how much that citing article is &quot;important&quot; or &quot;good&quot; by itself, you could always try reading it..." CreationDate="2010-08-22T07:47:05.030" UserId="449" />
  <row Id="2163" PostId="2007" Text="@Tal Certainly, now converted." CreationDate="2010-08-22T09:48:01.200" UserId="88" />
  <row Id="2164" PostId="1868" Text="@ars. Thanks for that." CreationDate="2010-08-22T09:57:07.883" UserId="183" />
  <row Id="2165" PostId="2008" Score="1" Text="Well, reading takes time and you may not even have the article available to begin with, so trying to estimate the relevance and quality vs time and maybe even money investment required is crucial." CreationDate="2010-08-22T10:40:55.350" UserId="979" />
  <row Id="2166" PostId="1763" Text="stat579: Where exactly are the video lectures?" CreationDate="2010-08-22T11:52:22.617" UserId="339" />
  <row Id="2167" PostId="1763" Text="@gd047 That's sad. stat579 videos appear to have been removed. I've removed the link from my answer." CreationDate="2010-08-22T12:07:15.790" UserId="183" />
  <row Id="2168" PostId="1884" Text="@whuber Yes definitely, it's not my credit. I merely reproduced it here. Thanks for putting up that note.&#xA;@Karsten It is shown by a counting argument - once you have fixed $X_n$ in 52 ways, there are 26 ways you can select non-matching $X_{n+1}$. So $P(X_n\ne X_{n+1})=\frac{52*26}{52*51}$." CreationDate="2010-08-22T12:35:10.773" UserId="994" />
  <row Id="2169" PostId="1884" Text="@Karsten If you are asking if the events $X_n \ne X_{n+1}$ are independent of $n$, they are not. But that doesn't matter since expectation of sum = sum of expectations even if the quantities are dependent." CreationDate="2010-08-22T12:38:02.077" UserId="994" />
  <row Id="2170" PostId="2007" Text="@el chief - It's a very broad and general question... but I'm afraid it's not a homework question." CreationDate="2010-08-22T13:16:01.720" UserId="1026" />
  <row Id="2172" PostId="1763" Text="Perhaps because this &quot;newer version&quot; is available. http://www.public.iastate.edu/~hofmann/stat480/" CreationDate="2010-08-22T14:32:49.147" UserId="339" />
  <row Id="2173" PostId="2023" Text="Thanks. I don't have easy access to the full-text, but from the opening page it does look useful regarding point 1 (i.e., combining two correlations)." CreationDate="2010-08-22T14:55:24.623" UserId="183" />
  <row Id="2174" PostId="2024" Text="I alfred.  What model (on your data) do you use to estimate the mean and SE ?  " CreationDate="2010-08-22T14:59:10.047" UserId="253" />
  <row Id="2175" PostId="1763" Text="The videos appear to still be available online. I added the links." CreationDate="2010-08-22T15:36:08.213" UserId="183" />
  <row Id="2176" PostId="2024" Text="Can you post an example graph?" CreationDate="2010-08-22T16:10:03.547" UserId="88" />
  <row Id="2177" PostId="1763" Text="@Jeromy Anglim Ahhh! You did a great job! (I wonder how). Is there any video from stat480 online also?" CreationDate="2010-08-22T16:20:18.073" UserId="339" />
  <row Id="2179" PostId="2013" Score="1" Text="you could add Red-R to the list (kind of a clone of Orange in R): http://www.red-r.org/" CreationDate="2010-08-22T19:10:44.310" UserId="170" />
  <row Id="2180" PostId="2027" Text="Thank you John &amp; Tai&#xA;&#xA;I guess that covers any potential avenue of my poorly formulated question.&#xA;&#xA;regards&#xA;&#xA;Alfredo&#xA;&#xA;&#xA;&#xA;&#xA;" CreationDate="2010-08-22T19:23:31.523" />
  <row Id="2181" PostId="2032" Text="Please make this community wiki." CreationDate="2010-08-22T22:42:42.153" UserId="5" />
  <row Id="2182" PostId="2032" Text="Possible duplicate: http://stats.stackexchange.com/questions/652/best-books-for-an-introduction-to-statistical-data-analysis" CreationDate="2010-08-22T22:43:40.670" UserId="5" />
  <row Id="2183" PostId="2027" Text="You're welcome Alfred.  Please choose Johns answer so to verify you found your answer :)  Cheers,&#xA;Tal" CreationDate="2010-08-22T23:12:32.077" UserId="253" />
  <row Id="2185" PostId="949" Text="I assume you mean &quot;when are the MLEs of the parameters in closed form?&quot;" CreationDate="2010-08-23T00:43:52.180" UserId="805" />
  <row Id="2186" PostId="203" Text="A related question: People often use the nonparametric Mann-Whitney test for this kind of data. Since there are only five possible values, there will be lots of tied ranks. The Mann-Whitney test adjusts for tied ranks, but does this adjustment work when there are a huge number of ties?" CreationDate="2010-08-23T01:22:52.757" UserId="25" />
  <row Id="2188" PostId="707" Text="@mbq: I say the &quot;usual&quot; choice. Does not mean every choice" CreationDate="2010-08-23T02:52:22.497" UserId="188" />
  <row Id="2189" PostId="2034" Text="Good point. However, in research settings this may or may not be possible. Means and standard deviations are typically known. However, in some cases a different measurement tool is used in the two studies. E.g., Imagine a study looking at the correlation between intelligence and job performance across jobs and two studies varied in the measurement of job performance or in the particular intelligence test used. Thus, the observed variables are on different metrics, even though the theorised latent variables are the same." CreationDate="2010-08-23T03:40:13.407" UserId="183" />
  <row Id="2190" PostId="1853" Text="univariate. but now you've made me curious about the multivariate case. ;)" CreationDate="2010-08-23T03:49:11.310" UserId="795" />
  <row Id="2191" PostId="2032" Text="I collected some links here: http://meta.stats.stackexchange.com/questions/6/what-should-our-faq-contain/361#361" CreationDate="2010-08-23T04:47:35.687" UserId="251" />
  <row Id="2193" PostId="2038" Text="I'd change the title to something like: &quot;Interpretation of biplots in principal components analysis in R&quot;" CreationDate="2010-08-23T10:23:14.007" UserId="183" />
  <row Id="2195" PostId="2013" Text="I've downloaded R and I am playing with it now." CreationDate="2010-08-23T11:52:45.863" UserId="1026" />
  <row Id="2196" PostId="2036" Text="Thanks!, let me check it out!" CreationDate="2010-08-23T12:05:31.753" UserId="1043" />
  <row Id="2197" PostId="2032" Text="Community-wiki'ed." CreationDate="2010-08-23T13:39:32.273" UserId="1049" />
  <row Id="2198" PostId="2034" Text="A necessary condition for combining these would be some quantitative relationship between the two forms of measurement, Jeromy.  Absent that, it's difficult to see how any combination of the two study results could be adequately defended." CreationDate="2010-08-23T13:59:41.453" UserId="919" />
  <row Id="2199" PostId="2037" Text="I changed the tag from &quot;truncated-gaussian&quot; to &quot;truncation&quot; because most answers will be potentially useful in situations involving other distributions." CreationDate="2010-08-23T14:21:10.707" UserId="919" />
  <row Id="2202" PostId="1997" Text="Thank you all for your responses :) As a follow up question, couldn't most data be conceptualized as being naturally hierarchical/nested? For example, in most psychological studies the there are a number of dependent variables (questionnaires, stimuli responses, etc...) nested within individuals, which are further nested within two or more groups (randomly or non-randomly assigned). Would you agree that this represents a naturally hierarchical and/or nested data structure?" CreationDate="2010-08-23T16:20:47.613" UserId="835" />
  <row Id="2203" PostId="1988" Text="Carlos, thank you for your contribution, but the level of scientific validation that I need should, at least, give me a &quot;probability&quot; value of that claim. Thanks, anyway. " CreationDate="2010-08-23T16:20:59.517" UserId="990" />
  <row Id="2204" PostId="1999" Text="Thank you for your detailed reply Srikant :) I am not currently familiar with Bayesian analyses, but i is one of the topics that I have been meaning to investigate. Is Hierarchical Bayesian analysis different from the other multilevel/hierarchical analyses discussed on this page? If so do you have a recommended resource for interested parties to learn more?" CreationDate="2010-08-23T16:24:56.717" UserId="835" />
  <row Id="2205" PostId="1997" Text="If any of you multilevel/hierarchical gurus could spare a few minutes I would be very grateful if you could weigh in on the analysis questions posed in a different post (http://stats.stackexchange.com/questions/1799/recommendations-or-best-practices-for-analyzing-non-independent-data-specifi). Specifically, do you think that the pain perception data outlined in that post would be better analyzed by hierarchical analyses than non-hierarchical analyses? Or would it not make a difference or even be inappropriate? Thanks :D" CreationDate="2010-08-23T16:31:20.017" UserId="835" />
  <row Id="2206" PostId="1999" Text="From an analytical perspective HB analysis = multi-level models. However, the term multi-level models is used when you have different levels that occur naturally (See the example of @ars). The term HB models is used when you do not necessarily have different levels in the situation. For example, if you are modeling a consumer's response to various marketing variables (e.g., price, adv spend etc) then you may have the following structure at the consumer level: $β_i \sim N(\bar{\beta},\Sigma)$ and $\bar{\beta} \sim N(.,.)$ at the population level. For references: See the other answers." CreationDate="2010-08-23T16:35:38.700" />
  <row Id="2207" PostId="1943" Text="Bernstein et. al. show some reasons (small n, ties) that cause the two methods to be inaccurate or different. But all of the discrepancies they showed are small. So I don't think anything in that paper explains the three fold discrepancy I saw that prompted this question. See below for the answer I came up with.&#xA;" CreationDate="2010-08-23T17:38:22.250" UserId="25" />
  <row Id="2208" PostId="2038" Text="@Jeromy Good idea." CreationDate="2010-08-23T17:40:28.283" UserId="88" />
  <row Id="2209" PostId="1965" Text="The point is that O(n^2) is too crude.  You obtained the next term, -beta*n*log(n), but then threw that away.  So your answer, O(exp(n^2)), is correct but it's not best." CreationDate="2010-08-23T17:57:47.570" UserId="919" />
  <row Id="2210" PostId="2013" Text="@Amro Thanks! However, it is not available on Mac platform, unless I'm mistaking?" CreationDate="2010-08-23T18:37:46.093" UserId="930" />
  <row Id="2213" PostId="2036" Text="According to reference, the distribution function is: $g(y) = C\sum_{k=1}^\infty\frac{\delta_k y^{\rho+k-1}e^{-\frac{y}{\beta_1}}}{\Gamma(\rho+k)\beta_i^{\rho+k}} $" CreationDate="2010-08-23T20:53:17.600" UserId="1043" />
  <row Id="2216" PostId="2036" Text="where $C = \sum_{i=1}^n\left(\frac{\beta_1}{\beta_i}\right)^{\alpha_i}$" CreationDate="2010-08-23T20:59:32.750" UserId="1043" />
  <row Id="2217" PostId="2036" Text="And $\gamma_k = \sum_{i=1}^n\alpha_i\frac{(1 - \frac{\beta_1}{\beta_i})^k}{k}$&#xA;" CreationDate="2010-08-23T21:19:17.980" UserId="1043" />
  <row Id="2218" PostId="2036" Text="with&#xA;$\rho = \sum_{i=1}^n\alpha_i$" CreationDate="2010-08-23T21:19:36.560" UserId="1043" />
  <row Id="2219" PostId="2036" Text="And finally &#xA;$\delta_k = \frac{1}{k+1}\sum_{i=1}^{k+1}i\gamma_i\delta_{k+1-i}; \delta_0=1, k=0,1,2,3,\ldots$" CreationDate="2010-08-23T21:19:54.530" UserId="1043" />
  <row Id="2220" PostId="386" Text="@Robin: This is a nice idea, but if I have understood it correctly, it seems to enlarge the idea of &quot;outlier&quot; to include *any* value in a dataset that is distant from the others.  For example, in the batch {-110[1]-90, 0, 90[1]110} of 43 integers, wouldn't your procedure identify the 0 (which is the *median* of these numbers!) as the unique &quot;outlier&quot;? " CreationDate="2010-08-23T22:36:55.787" UserId="919" />
  <row Id="2221" PostId="1965" Text="It is not clear what kind of bound the OP is looking for. Because the free variable was given as $n$, I assume a bound asymptotic in $n$ is desired. In my answer I note that this is an asymptotic bound. (the big-O notation is another hint that this is the case.) The bound that I showed is an asymptotic *lower bound*. I am not 'throwing away' the log term: $\Omega{(n^2 - n\log{n})}$ is $\Omega{(n^2)}$. Then since this lower bound is the same as the trivial upper bound, the trivial upper bound is optimal." CreationDate="2010-08-23T22:47:04.807" UserId="795" />
  <row Id="2222" PostId="2059" Score="2" Text="EM is _not_ a heuristic, it is an algortihm for maximizing likelihood with known properties. The error you are looking for probably has nothing to do with EM, but rather the efficiency of the maximum likelihood estimate. This will surely depend on the unknown parameters, and not just the sample size." CreationDate="2010-08-24T00:00:20.510" UserId="279" />
  <row Id="2223" PostId="2059" Text="Yep, my question refers to those &quot;known properties&quot;. It is used as a heuristic in the sense that I see people apply it on &quot;good faith&quot; beyond what is explained by provable bounds. I changed the wording of the question, since this is really not the issue I'm interested in." CreationDate="2010-08-24T00:59:35.717" UserId="1072" />
  <row Id="2224" PostId="2060" Text="@Moritz : In reply to your comment about EM being a heuristic, EM does have strong theoretical foundations and it is known to provably converge to a local maximum." CreationDate="2010-08-24T01:13:51.697" UserId="881" />
  <row Id="2225" PostId="2061" Text="Great question!!" CreationDate="2010-08-24T02:54:22.507" UserId="253" />
  <row Id="2226" PostId="2062" Text="@Tal This looks good. Thanks." CreationDate="2010-08-24T03:14:02.160" UserId="183" />
  <row Id="2229" PostId="1762" Text="I think I'm going to simply write a simulator and tabulate results." CreationDate="2010-08-24T06:20:45.883" UserId="937" />
  <row Id="2230" PostId="2066" Score="1" Text="Do you want to know the mathematics or simply a code solution in R or some such?" CreationDate="2010-08-24T10:40:50.777" UserId="601" />
  <row Id="2231" PostId="2066" Text="I need to implement this in C#, so the mathematics would be good. A code sample would be fine too, if it doesn't rely on some builtin R/Matlab/Mathematica function I can't translate to C#." CreationDate="2010-08-24T10:55:38.190" UserId="956" />
  <row Id="2232" PostId="1873" Score="4" Text="He's clumsy enough to change his name in the middle of a question at least! ;)" CreationDate="2010-08-24T11:10:19.190" UserId="229" />
  <row Id="2233" PostId="2056" Text="Interesting, I'll have to look that reference up. Are you sure you mean &quot;fourth&quot; approach? The first approach I list seems to describe &quot;simply resampling the subjects&quot;." CreationDate="2010-08-24T11:17:48.503" UserId="364" />
  <row Id="2234" PostId="2067" Score="1" Text="Can you clarify the predicted variable, &quot;mean endorsement&quot;?. Is this a 0-100 scale that participants used for response, or is this a measure of the proportion of trials on which participants said &quot;yes, I endorse&quot; (vs. &quot;no, I do not endorse&quot;). If the latter, then it is inappropriate to analyse this data as proportions. Instead, you should be analyzing the raw, trial-by-trial data using a mixed effects model with a binomial link function." CreationDate="2010-08-24T11:36:06.837" UserId="364" />
  <row Id="2235" PostId="2067" Text="Sorry, for omitting this: it is a 0-100 response scale." CreationDate="2010-08-24T11:39:40.877" UserId="442" />
  <row Id="2236" PostId="1898" Text="what is FWE and FDR? I suppose FWE to be family wise error, but the other??" CreationDate="2010-08-24T11:56:34.820" UserId="442" />
  <row Id="2237" PostId="2069" Text="I am not quite sure I understand your recommendation in 1. As actual SE [i.e., SD/sqrt(n)] and estimated SE are both model based, you recommend using the model-based. So which one?&#xA;Or do you mean: go with the more complicated model (here: ANOVA) cause both models are reasonable." CreationDate="2010-08-24T12:07:14.580" UserId="442" />
  <row Id="2238" PostId="2069" Text="agree with point 1 completely" CreationDate="2010-08-24T12:09:29.227" UserId="601" />
  <row Id="2239" PostId="2067" Text="Do you have many 0's or 100's? If not, I'd consider dividing by 100 and performing a logit transform to take into account the restriction of range at the extremes. This is essentially what is achieved by the binomial link function when you have binary data, but is useful if you only have proportion-like data as you seem to have here. However, you can't logit transform 1 or 0, so you'd have to toss any responses of 100 or 0." CreationDate="2010-08-24T12:16:05.193" UserId="364" />
  <row Id="2240" PostId="2068" Text="I have good (non-statistical) reasons to plot the graph as it is: You directly see the answer to the research question. Furthermore, I am not looking for a error bars for inferential purposes as I know about the within-between problems. But, thanks to pinpointing me back to Mason &amp; Loftus, I must have forgotten that they had a mixed example. I have to think on whether or not it serves my purpose." CreationDate="2010-08-24T12:16:29.943" UserId="442" />
  <row Id="2241" PostId="2067" Text="Oops, just realized that my first comment was not 100% correct. Each plotted mean represents the mean of two responses on a 0-100 scale. In this data there are a lot values very near to 100, and some directly on 100, but actually very little at 0 and around 0. You have some literature for justifing your recommendation?" CreationDate="2010-08-24T12:28:31.263" UserId="442" />
  <row Id="2242" PostId="2066" Text="PDF, CDF or inverse CDF?" CreationDate="2010-08-24T12:29:48.187" UserId="830" />
  <row Id="2243" PostId="604" Text="@Srikant Because the RSS would vanish in this case, attached is a toy example: http://gist.github.com/547494 (I did not investigate the DA results though)." CreationDate="2010-08-24T12:59:06.567" UserId="930" />
  <row Id="2244" PostId="2013" Text="@chl: I'm not a Mac user, but I think the Linux build could work for you (you need to manually install all python dependencies): http://www.red-r.org/forum/topic.php?id=22#post-76" CreationDate="2010-08-24T13:03:57.150" UserId="170" />
  <row Id="2245" PostId="2067" Text="As a side note, because the conditions are not varying along continuous dimensions some data visualization people might claim this data is better represented by bar graphs." CreationDate="2010-08-24T13:25:06.667" UserId="196" />
  <row Id="2246" PostId="2070" Text="Mike, in the package languageR the pvals.fnc function does an MCMC to evaluate the hypotheses of the lmer model - however it does not handle designs with random slopes - that lead me to suspect that there was some reason doing MCMC with random slopes was in someway problematic, do you know definitively that there is no such problem?" CreationDate="2010-08-24T13:32:04.997" UserId="196" />
  <row Id="2247" PostId="1898" Text="Well, this thread is subjective, so who knows...  Now seriously - FDR stands for false discovery rate (wikipedia it)" CreationDate="2010-08-24T13:45:03.633" UserId="253" />
  <row Id="2248" PostId="2070" Text="I have to admit I still haven't figured out how MCMC works, which is one of the reasons I opted for bootstrapping instead. While bootstrapping should be possible with random slopes, as you intimated, it may be that  pvals.fnc doesn't let you do CIs for models with random slopes because this is for some reason invalid, and further it may be that this invalidity extends to bootstrapping such models. I don't intuitively think there would be any problem with bootstrapping, but that may be a function of my limited expertise." CreationDate="2010-08-24T14:28:02.140" UserId="364" />
  <row Id="2249" PostId="2073" Text="When alpha and beta are not too far apart (i.e., alpha/beta are bounded above and below), the SD of Beta[alpha, beta] is proportional to 1/Sqrt(alpha).  E.g., for alpha = beta = 10^6, the SD is very close to 1/Sqrt(8) / 1000.  I think there will be no problem with the representation of l and r even if you're only using single precision floats." CreationDate="2010-08-24T15:25:46.810" UserId="919" />
  <row Id="2250" PostId="2071" Text="Stupid question: How did you do that graphical experiment? I tried to plot the distribution for alpha/beta around 100, but I couldn't see anything due to underflow errors." CreationDate="2010-08-24T15:57:41.523" UserId="956" />
  <row Id="2251" PostId="2073" Text="which is to say that $10^6$ is not 'sufficiently large' ;)" CreationDate="2010-08-24T16:25:59.780" UserId="795" />
  <row Id="2252" PostId="931" Text="No, although there are plug-ins (for Firefox, at least) that help with downloading Flash.  Not easy in all cases, but possible in most." CreationDate="2010-08-24T16:57:58.680" UserId="71" />
  <row Id="2253" PostId="2073" Score="1" Text="Yeah, it's a crazy number for a beta application.&#xA;BTW, those inequalities won't produce good intervals at all, because they are extremes over all distributions (satisfying certain constraints)." CreationDate="2010-08-24T17:19:34.647" UserId="919" />
  <row Id="2254" PostId="2071" Text="You don't want to plot the integrand: you want to plot the integral.  However, you can get the integrand in many ways.  One is to enter &quot;plot D (beta(x, 1000000, 2000000), x) / beta(1, 1000000, 2000000) from 0.3325 to 0.334&quot; at the Wolfram Alpha site.  The integral itself is seen with &quot;Plot beta(x, 1000000, 2000000) / beta(1, 1000000, 2000000) from 0.3325 to 0.334&quot;." CreationDate="2010-08-24T17:27:44.750" UserId="919" />
  <row Id="2255" PostId="2074" Text="Perfect! I had the NR book on my desk all the time, but never thought to look there. Thanks a lot." CreationDate="2010-08-24T17:34:37.830" UserId="956" />
  <row Id="2256" PostId="2067" Text="Other data visualization people might claim that bar graphs are a crime against humanity :Op" CreationDate="2010-08-24T17:52:21.827" UserId="364" />
  <row Id="2257" PostId="2065" Text="I figured out the answer two days ago, and posted it here as a new answer. I also then expanded and updated the web page at graphpad.com that you found. I just edited that page again to include a link to an Excel file with the problem data (http://www.graphpad.com/faq/file/1226.xls). I couldn't do that until I got permission from the guy who generated the data (he wants to be anonymous, and the data is vaguely labeled). " CreationDate="2010-08-24T18:28:08.800" UserId="25" />
  <row Id="2258" PostId="1944" Text="btw, this is special case of &quot;whitening&quot; which takes data and makes it's empirical covariance matrix identity" CreationDate="2010-08-24T19:30:40.407" UserId="511" />
  <row Id="2259" PostId="1931" Text="Any measure that coincides with the usual median when restricted to R1 is a candidate generalization. There must be a lot of them." CreationDate="2010-08-24T19:37:24.870" UserId="1011" />
  <row Id="2260" PostId="2073" Text="@whuber: You're right, they are crazy numbers. With my naive algorithm, the &quot;sane&quot; numbers were easy and worked well, but I couldn't imagine how to calculate it for &quot;crazy&quot; parameters. Hence the question." CreationDate="2010-08-24T19:52:59.853" UserId="956" />
  <row Id="2261" PostId="1963" Text="Thank you @propofol and @Jeromy Anglim for your excellent contributions: after skimming DeCoster and looking over the Amazon reviews of the books, I have passed them all along to my colleague.  It is a real shame one cannot checkmark multiple answers; in this case I literally flipped a coin: tails, it went to propofol." CreationDate="2010-08-24T22:29:17.460" UserId="919" />
  <row Id="2262" PostId="2081" Text="Yes, it looks like it needs to be solved numerically in general. I really just needed a practical way of plotting epsilon as a function of n, and ended up using a version of Chernoff inequality for fair coin from which expression for epsilon can be obtained in closed form" CreationDate="2010-08-24T23:00:01.933" UserId="511" />
  <row Id="2263" PostId="2069" Text="Hi Henrik,&#xA;Simple example - compare two groups (x1, x2) assumed ND.&#xA;Assumptions &amp; models:&#xA;&#xA;1) Independently sampled, different variance. SEs for x1, x2 estimated separately. This is implicitly the assumption in many graphical presentations. The estimated SEs differ.&#xA; &#xA;2) Indep., same var. Usual ANOVA assumption. Estimate SEs using pooled RSS. Estimate is more robust IF assumptions correct.&#xA;&#xA;3) Each x1 has an x2 pair. SEs estimated from x1-x2. To effectively plot them you need to plot the difference x1-x2.&#xA;&#xA;Once you mix 1) and 2) you have a real problem plotting meaningful SEs or CIs.&#xA;" CreationDate="2010-08-25T00:42:31.133" UserId="521" />
  <row Id="2264" PostId="2065" Text="Thanks Harvey,&#xA;I thought your answer was very good." CreationDate="2010-08-25T00:47:26.537" UserId="521" />
  <row Id="2265" PostId="2069" Text="Henrik, a comment on the plot. How many subjects do you have? I would strongly recommend plotting the data individually and use line segments to link individuals. (Line segments linking means is deceptive.) There is no need to plot SEs. The idea is to visually support your statistical analysis. Provided the plot does not become too cluttered, a reader should see (for example) that the clear majority of scores go up from MP-valid-implaus to AC-inval-plaus for the Inductive group &amp; down for the Deductive group. See:&#xA;http://www.jstor.org/stable/2685323?seq=1&#xA;Especially Figs 1 &amp; 9 bottom panels. " CreationDate="2010-08-25T02:01:21.397" UserId="521" />
  <row Id="2266" PostId="2087" Score="1" Text="Thanks Tal, I'll try corrplot now. I also wish I knew how to simplify your solution (which I linked to in the question) but I'm just a newbie in R so you know more than me. I'll update the question to clarify the solution looks complicated *to me*" CreationDate="2010-08-25T04:04:52.400" UserId="840" />
  <row Id="2267" PostId="2086" Score="1" Text="You sound like R is inferior to propriety software. :)" CreationDate="2010-08-25T05:49:51.640" UserId="144" />
  <row Id="2268" PostId="2076" Text="Consider this posting as a stand-alone question. I would be very interested what gurus have to say about this." CreationDate="2010-08-25T06:02:31.277" UserId="144" />
  <row Id="2269" PostId="2093" Text="Do you want to generate normal random variables truncated to range between 0 and 1 or uniform random variables between 0 and 1? The Marsagla polar method will return numbers over the entire real line as it generates standard normal random variables." CreationDate="2010-08-25T09:46:57.313" />
  <row Id="2271" PostId="2086" Text="For me it sounds totally reasonable to use the pearson product-moment-correlation (assuming continuous data) in your case (assuming enough points on your scale and not a don't know midpoint). Whole fields within psychology (e.g., personality or social psychology) rest (successfully) on the assumption that answers to a single item on an e.g., five-point (or seven-point) scale ranging from very un-X to very X can be treated as continuous. See also this thread: http://stats.stackexchange.com/questions/539/does-it-ever-make-sense-to-treat-categorical-data-as-continuous " CreationDate="2010-08-25T09:59:53.510" UserId="442" />
  <row Id="2272" PostId="2093" Text="I don't know if this solves your problem, but you might wanna take a look at this question where some transformations that squash everything between 0 and 1 are discussed: http://stats.stackexchange.com/questions/1112/how-to-represent-an-unbounded-variable-as-number-between-0-and-1/1113#1113" CreationDate="2010-08-25T10:40:47.987" UserId="442" />
  <row Id="2273" PostId="2086" Text="@romunov: Not sure how you got the impression that I believe R is inferior to other s/w. But it's not the case at all." CreationDate="2010-08-25T12:45:32.550" UserId="840" />
  <row Id="2274" PostId="2071" Text="I plotted the integrand, i.e. the pdf of the beta distribution, in Stata - it has a builtin function for the pdf. For large alpha and beta you need to restrict the range of the plot to see it's close to normal.&#xA;&#xA;&#xA;If I was programming it myself i'd compute its logarithm then exponentiate at the end. That should help with the underflow problems. The beta function in the denominator is defined in terms of gamma functions, equivalent to factorials for integer alpha and beta, and many packages/libraries include lngamma() or lnfactorial() instead/as well as gamma() and factorial() functions." CreationDate="2010-08-25T12:51:07.750" UserId="449" />
  <row Id="2275" PostId="2090" Text="Although most of them don't seem relevant for the OP questions, still, this is a nice link - thanks Jeromy!" CreationDate="2010-08-25T13:20:17.817" UserId="253" />
  <row Id="2276" PostId="2057" Text="That was a great insight! All your assumptions can be considered, and your rationale is sound! I do have some doubts though, if you don't mind: &#xA;&#xA;(I) Shouldn't d2[j,i] = h[j,i+2] ... ?&#xA;&#xA;(II) Why do you consider the increment as y[j,i+2] - 2*y[j,i+1] + y[j,i] and not just y[j,i+1]-y[j,i] ? Is it a kind of derivative (i.e., to calculate the &quot;slope&quot;)?" CreationDate="2010-08-25T13:25:00.443" UserId="990" />
  <row Id="2277" PostId="2088" Text="The Sunflower is a fun solution.  Using a jitter is what I tried when first I looked at the topic, but I found it do be not effective enough for the plotting of correlation matrixs... " CreationDate="2010-08-25T13:26:17.820" UserId="253" />
  <row Id="2278" PostId="2057" Text="(1) Yes, good catch.  I fixed that typo.&#xA;&#xA;(2) We're really looking at *second* derivatives: the increments d[j,i] = h[j,i+1] - h[j,i] act like slopes (and are slopes when times are equally spaced).  I am proposing to distinguish &quot;exponential&quot; from &quot;sublinear&quot; behavior in terms of changes in the slopes.  Whence we want to consider the signs of&#xA;&#xA;d2[j,i] := d[j,i+1] - d[j,i] = (h[j,i+2] - h[j,i+1]) - (h[j,i+1] - h[j,i]) etc." CreationDate="2010-08-25T13:32:12.270" UserId="919" />
  <row Id="2279" PostId="2087" Text="The corrplot looks good. It gives a great visual snapshot of size and direction of correlations. In the case of 5-point ordered categorical variables, it might be useful to supply some other measure of association besides Pearson's correlation: e.g., polychoric correlations. The size of standard Pearson's correlations of ordered categorical variables is influenced somewhat by the mean of the two variables." CreationDate="2010-08-25T13:38:35.783" UserId="183" />
  <row Id="2280" PostId="2088" Text="Yeah, jitter could get pretty messy with a scattermatrix with lots of variables. I suppose the benefit of jitter and sunflower is that you get to see the raw data (albeit perturbed in the jitter case)." CreationDate="2010-08-25T13:40:47.777" UserId="183" />
  <row Id="2282" PostId="2085" Score="2" Text="R and S-Plus are very similar, but definitely not the same (I know from painful experience!)." CreationDate="2010-08-25T13:44:10.357" UserId="5" />
  <row Id="2283" PostId="30" Text="Retagged &quot;random-variate&quot; to the more popular (and more appropriate) &quot;random-variable&quot;." CreationDate="2010-08-25T14:13:18.617" UserId="919" />
  <row Id="2284" PostId="40" Text="Retagged &quot;random-variate&quot; to &quot;random-variable&quot; for consistency with similar questions." CreationDate="2010-08-25T14:14:06.740" UserId="919" />
  <row Id="2285" PostId="111" Text="Are you sure about your second point, generating normal random variables by inverting the CDF?  The inverse of the normal CDF is a fairly expensive function to evaluate.  I imagine Box-Muller's method would be faster.  Faster still would be Marsaglia's ziggurat method for generating normals." CreationDate="2010-08-25T14:53:59.127" UserId="319" />
  <row Id="2286" PostId="2073" Score="2" Text="@shabbychef: OK, you're right: once alpha+beta exceeds 10^30 or so, you will have difficulties with doubles :-).  (But if you represent l and r as differences from the mean of alpha/(alpha+beta), you'll be fine until alpha or beta exceed about 10^303.)" CreationDate="2010-08-25T15:34:21.457" UserId="919" />
  <row Id="2287" PostId="2106" Text="Thank you for taking the time to answer my question. however, your response doesn't answer my question. I already mentioned in my post that I already came across the ezANOVA function. I tried it without success. I am also already familiar with the aov notation you metnion." CreationDate="2010-08-25T15:55:28.673" UserId="1084" />
  <row Id="2289" PostId="2106" Score="1" Text="Do you get any error messages when using ezANOVA? Whenever I use it, I get a nice table of sphericity tests and GG/HF corrections.&#xA;It does not work for ANCOVA models, for this you will need to use the lme4 package, and there are other things to account for variance." CreationDate="2010-08-25T16:05:43.847" UserId="966" />
  <row Id="2290" PostId="2106" Text="It would probably help if you put the output of &quot;str(p12bl)&quot; or &quot;str(subset(p12bl, exp2==1))&quot; here." CreationDate="2010-08-25T16:16:23.930" UserId="966" />
  <row Id="2291" PostId="111" Text="I also find this suspicious. Marsaglia's Ziggurat is the default in Matlab, and I can't imagine Matlab being better than R in the field of random number generation." CreationDate="2010-08-25T16:21:20.663" UserId="795" />
  <row Id="2292" PostId="2088" Text="Agreed (I love jitter, simply not for this :) )" CreationDate="2010-08-25T16:32:54.063" UserId="253" />
  <row Id="2293" PostId="2103" Text="+1 A better answer than mine as it explicitly takes into consideration the range over which the pdf is non-zero." CreationDate="2010-08-25T16:36:18.603" />
  <row Id="2295" PostId="2104" Text="Do you want to look at sphericity or a factor to correct F-values because of violations in sphericity?" CreationDate="2010-08-25T16:50:57.357" UserId="601" />
  <row Id="2296" PostId="2108" Text="This may be a question more appropriately asked on the stackoverflow forum, also you may be able to get a more concise answer if you describe what type of file the url is pointing to." CreationDate="2010-08-25T17:38:06.030" UserId="1036" />
  <row Id="2297" PostId="2103" Text="@Srikant: Thanks; you are generous.  Did you notice the connection with the theory of copulas?  Maybe we should even add a &quot;copula&quot; tag." CreationDate="2010-08-25T17:39:22.930" UserId="919" />
  <row Id="2298" PostId="2094" Text="Ok this makes it clear. Exponential pdf can be used to model waiting times between any two successive poisson hits while poisson models the probability of number of hits. Poisson is discrete while exponential is continuous distribution. It would be interesting to see a real life example where the two come into play at the same time." CreationDate="2010-08-25T18:03:50.560" UserId="862" />
  <row Id="2299" PostId="2093" Text="@Ants: &quot;Scaling down&quot; from the real line to an interval will necessarily make the resulting generator non-Normal." CreationDate="2010-08-25T18:14:27.713" UserId="919" />
  <row Id="2300" PostId="2112" Score="1" Text="I was just about to recommend this.  The ez package relies on car for some analysis." CreationDate="2010-08-25T18:47:42.030" UserId="601" />
  <row Id="2303" PostId="2104" Text="@John I need to see if I need the correction" CreationDate="2010-08-25T18:55:22.780" UserId="1084" />
  <row Id="2304" PostId="2112" Text="Thank you, the R newsletter is where I picked the mlm direction from. alas, I am still figuring how to choose idata idesign for a nested within subject design. I will try to figure out Anova()." CreationDate="2010-08-25T19:14:54.643" UserId="1084" />
  <row Id="2305" PostId="2111" Score="1" Text="Are you refering to the following paper?&#xA;Kuk, A.Y.C. (1984) All subsets regression in a proportional hazards model. Biometrika, 71, 587-592" CreationDate="2010-08-25T19:22:05.237" UserId="930" />
  <row Id="2306" PostId="2013" Text="@Amro I'll give it a try; in the past I've been testing RAnalyticFlow (http://j.mp/bYF8xs) but did not get convinced: I am basically a CLI user :-)" CreationDate="2010-08-25T19:52:42.210" UserId="930" />
  <row Id="2307" PostId="2114" Score="1" Text="BTW, the penalized R package (http://j.mp/bdQ0Rp) includes l1/l2 penalized estimation for Generalized Linear and Cox models." CreationDate="2010-08-25T20:07:49.780" UserId="930" />
  <row Id="2308" PostId="2105" Text="I definitely want a normally distributed variable. I'll update the question with the specific problem I'm trying to solve." CreationDate="2010-08-25T20:08:50.327" UserId="937" />
  <row Id="2309" PostId="2098" Text="For rejection sampling, what will I use as my g(x) and M value to simulate the normal distribution? For inverse transform sampling, do I use D Ibbetson's Algorithm 209 (http://old.sigchi.org/~perlman/stat/doc/z.c) to compute the cdf? What do I use as the inverse?" CreationDate="2010-08-25T20:21:40.643" UserId="937" />
  <row Id="2310" PostId="2117" Text="Not bad at all. The format is static, but the information is very useful." CreationDate="2010-08-25T20:32:56.373" UserId="840" />
  <row Id="2311" PostId="2102" Text="I want to get a &quot;normal&quot; distribution with the appropriate hump at 0.5." CreationDate="2010-08-25T21:03:41.687" UserId="937" />
  <row Id="2312" PostId="219" Text="Maybe also L'Ecuyer's RNG with multiple streams (http://j.mp/bzJSlm)?" CreationDate="2010-08-25T21:08:09.673" UserId="930" />
  <row Id="2313" PostId="2098" Text="@Ants If I am right, you don't need a cannon like ITS." CreationDate="2010-08-25T21:25:22.547" UserId="88" />
  <row Id="2314" PostId="111" Text="@John Indeed, the polar method is available in R, see the setRNG package." CreationDate="2010-08-25T21:28:18.160" UserId="930" />
  <row Id="2315" PostId="2098" Text="For rejection sampling set f(x) ~ standard normal pdf I(0&lt;x&lt;1), g(x) ~ standard normal and and M=1. Your condition will then become select x if u &lt; I(0&lt;x&lt;1) which is basically what I stated in my one line sentence above under rejection sampling." CreationDate="2010-08-25T21:34:16.550" />
  <row Id="2316" PostId="45" Text="e.g., Python (http://j.mp/9TVyhv), Perl (http://j.mp/by85El), Octave (http://j.mp/aVQ5Xz), Clojure (http://j.mp/dkj9Z9, not the default one), Haskell (http://j.mp/aWK7kK), lua (http://j.mp/bSD2vO), or even SQL (http://j.mp/aOPJMW, for an overview) :-)" CreationDate="2010-08-25T21:41:15.657" UserId="930" />
  <row Id="2317" PostId="2103" Text="@whuber My familiarity with copulas does not go very far. In any case, I am not sure how many will see the connection. So, a copula tag may just confuse some." CreationDate="2010-08-25T21:56:16.427" />
  <row Id="2318" PostId="2111" Text="yes indeed. I guess I'll have to dig up that paper somehow. It seems old, however." CreationDate="2010-08-25T22:56:20.283" UserId="795" />
  <row Id="2319" PostId="2114" Text="stuck in matlab land, implementing it myself..." CreationDate="2010-08-25T23:23:39.040" UserId="795" />
  <row Id="2321" PostId="2098" Text="@mbq: I don't get the ITS reference. I've been missing out on a lot of pop culture and current events references the past few months." CreationDate="2010-08-26T00:37:51.657" UserId="937" />
  <row Id="2322" PostId="2098" Text="@Srikant: Oh, okay. Thanks!" CreationDate="2010-08-26T00:39:19.937" UserId="937" />
  <row Id="2323" PostId="2093" Text="@Henrik: Thanks. I'll have to remember some of those tricks for future reference." CreationDate="2010-08-26T00:41:23.443" UserId="937" />
  <row Id="2324" PostId="2097" Text="Thanks for the pointer. LOL! My new implementation for NormalDistribution looks a lot like I'd taken a slice of that code project and focused on just normal random numbers. I've got the same abstract classes for RNG's and Distribution's. I've got just one concrete RNG that uses the .NET Random class, and a concrete NormalDistribution class." CreationDate="2010-08-26T00:48:57.593" UserId="937" />
  <row Id="2325" PostId="2103" Text="@Srikant: Granted.  But isn't the point of tags to facilitate searches and create semantic ties among threads?  I'm sure plenty of existing tags also have the potential to confuse the uninitiated already.  Maybe this is a discussion for the meta site?" CreationDate="2010-08-26T01:05:29.610" UserId="919" />
  <row Id="2326" PostId="2102" Text="Then you won't be able to restrict it to the interval [0, 1).  As a practical matter, normal probabilities further out than about seven standard deviations are too small to be worth computing, anyway, so you could set the standard deviation to 1/2 / 7 = 1/14 = about 0.07.  This isn't too far from the value you found empirically.  With your value and enough draws, you are likely to get a number outside [0, 1); with 1/14, it's extremely unlikely.  Thus, my recommendation is to take a value from the Marsaglia algorithm, divide by 14, and add 1/2.  Adjust the 1/14 to suit your application." CreationDate="2010-08-26T01:08:43.993" UserId="919" />
  <row Id="2327" PostId="1991" Text="But what does this have to do with statistical &quot;power&quot; in the King quotation?  (I would ask the same question of stormygeorge, too.)" CreationDate="2010-08-26T01:15:10.923" UserId="919" />
  <row Id="2328" PostId="2125" Text="I have no idea what this question is about." CreationDate="2010-08-26T01:35:53.393" UserId="352" />
  <row Id="2330" PostId="2125" Score="4" Text="It sounds like it is about the difference between correlation and simple linear regression." CreationDate="2010-08-26T02:19:02.163" UserId="485" />
  <row Id="2331" PostId="2098" Text="@Ants ITS=Inverse Transform Sampling" CreationDate="2010-08-26T06:53:50.287" UserId="88" />
  <row Id="2332" PostId="2097" Text="But as I can see they are not overloading Microsoft's original, so they can break the [0,1) limit." CreationDate="2010-08-26T06:55:11.193" UserId="88" />
  <row Id="2333" PostId="2131" Text="This asymmetry does not look *so* bad; are you sure you don't have an error somewhere else? Also can you give some more details about the data? There is not a general solution to this." CreationDate="2010-08-26T07:05:53.020" UserId="88" />
  <row Id="2334" PostId="2089" Text="Another great answer! PyIMSL Studio sounds interesting, too bad it isn't open source. There's some overlap with NumPy/SciPy, though. In any case, I think these were good tips for anyone wanting to assemble their own Python statistics workbench!" CreationDate="2010-08-26T07:29:27.057" UserId="890" />
  <row Id="2335" PostId="2097" Text="Yes, that is correct." CreationDate="2010-08-26T08:30:42.673" UserId="937" />
  <row Id="2337" PostId="2110" Text="Thanks Shane, it's actually an XML web service. I had a quick look at the XML package documentation, am I right in thinking that it just does the parsing? I'm still unsure of how to actually request the data, from the web service. Thanks" CreationDate="2010-08-26T10:07:01.410" UserId="428" />
  <row Id="2338" PostId="2136" Score="1" Text="Actually, &quot;conditional logit&quot; is a very ambiguous term. It some contexts (mainly when dealing with panel data), it is equivalent to Chamberlain's estimator, but it is very unfrequent. Most of the times, it refers to a cross-sectional model where the outcome variable can take more than 2 values. All your proposals actually refers to packages that consider this last possibility. The same with mixed-logit: it is not a fixed-effect logit. I've already taken a look at Farnsworth's overview, but it is not exhaustive enough to speak about this estimator. Anyway, thank you for your answer ! " CreationDate="2010-08-26T10:11:31.553" UserId="1091" />
  <row Id="2339" PostId="2110" Text="@Matt As Shane wrote; connect with RCurl, get the output and parse it with XML. We would be able to give you the code if you'd give more details about this web service." CreationDate="2010-08-26T10:21:34.030" UserId="88" />
  <row Id="2340" PostId="2135" Score="2" Text="+1; still ksvm from kernlab seems to be the preferred R SVM implementation. For instance it scales variables on its own and has a nice (=working very well) heuristic procedure for picking good $\gamma$ for RBF." CreationDate="2010-08-26T10:49:51.093" UserId="88" />
  <row Id="2341" PostId="2123" Text="Thanks, that's all I've wanted to hear. Now I can permalink to your answer! =)" CreationDate="2010-08-26T12:27:31.327" UserId="1356" />
  <row Id="2342" PostId="2138" Text="Excellent Henrik. I also prefer the &quot;points with means dislocated&quot;. Linking subjects with line segments may look too cluttered. Pity. As to homogeneity of variance I am a little more sanguine. The variance problem may not be as bad as it looks in the raw data. For the most part I suspect you will be comparing contrasts - within group differences. Contrast variances will be more homogeneous that the variances of the raw data. If raw measures with different variances are  compared (eg. Inductive vs Deductive in the MP-valiad &amp; plausible group) a non-parametric test could be used as a back-up." CreationDate="2010-08-26T12:58:52.157" UserId="521" />
  <row Id="2343" PostId="2093" Text="Your clarification completely confused me. You plan to generate a random normal variable `x1` truncated to the 0-1 range, then plug it into the `x1 * w * stdDev + mean` formula (what is `w`?) to get a normal with mean `mean` and standard deviation `stdDev`? If so, that's totally wrong. Just take a standard normal `z`, and calculate `mean + z*stdDev` to get a non-standard normal variable." CreationDate="2010-08-26T13:01:13.707" UserId="279" />
  <row Id="2344" PostId="2136" Text="&quot;Conditional logit&quot; does *not* refer to having more than two outcome levels. Some functions might extend it to that situation, but that is not the point." CreationDate="2010-08-26T13:09:34.070" UserId="279" />
  <row Id="2345" PostId="2112" Text="@ John Anova() is supposed to take my aov() object as a model, but for some reason I am having problems making it work." CreationDate="2010-08-26T13:21:49.427" UserId="1084" />
  <row Id="2346" PostId="2141" Text="Thanks, that looks like exactly what I'm after." CreationDate="2010-08-26T13:26:28.503" UserId="1094" />
  <row Id="2347" PostId="733" Text="This remind me of a poster I saw at the Human Brain Mapping 2010 conference. Tom Nichols used parametric bootstrap to assess the stability of the cophenetic correlation and silhouette in hierarchical clustering, but see his poster: http://j.mp/9yXObA." CreationDate="2010-08-26T13:27:22.553" UserId="930" />
  <row Id="2348" PostId="2112" Score="1" Text="@Adam In the on-line help for Anova(), you will find a working example with within- and between-subjects factors; the main caveats of Anova() is that you need to pass idata/idesign parameters in addition to your lm object. Also see this thread on the r-help mailing list, http://j.mp/9lphud." CreationDate="2010-08-26T13:39:37.537" UserId="930" />
  <row Id="2349" PostId="2103" Text="@whuber True. But, right now the connection to copulas is a bit obscure for those who do not know what they are. Perhaps, you can add a line or two to make the connection in which case the tag 'copula' may make sense. " CreationDate="2010-08-26T13:54:52.310" />
  <row Id="2350" PostId="2141" Text="In R that's var.test(q1, q2)" CreationDate="2010-08-26T14:08:55.743" UserId="601" />
  <row Id="2352" PostId="1991" Score="1" Text="@whuber Please see in the question where Gary King explicitly says that he is not talking about statistical power, but about &quot;the power that can be extracted from knowledge of the substance&quot;." CreationDate="2010-08-26T14:23:36.120" UserId="666" />
  <row Id="2353" PostId="2100" Text="Thanks, Jeromy. Actually this is even better than what I asked." CreationDate="2010-08-26T14:25:10.797" UserId="666" />
  <row Id="2354" PostId="2093" Text="@Aniko: I was just using 'w' because that's what code that I grabbed had. I should have renamed it 'r2' to represent r-squared during the looping, and then added a new variable 'transform' for the 'w' in the second to the last line in the original code. When I gave up trying to get a return value from 0-1, I resorted to &lt;code&gt;x1 * w * stdDev + mean&lt;code&gt; but the code is not used in the context of overriding Random.Sample() anymore. How do I get &lt;code&gt;z&lt;code&gt; given the Marsaglia polar method?" CreationDate="2010-08-26T14:39:12.727" UserId="937" />
  <row Id="2355" PostId="2145" Text="Does that mean we can always get &#xA;$$\frac{d^kf(a)}{dx^k} = \gamma_k$$?" CreationDate="2010-08-26T14:57:54.883" UserId="1043" />
  <row Id="2357" PostId="2093" Text="Sorry, I did not scroll up to the code. Now I see what `x1` and `w` are. Mea culpa. Then using `x1 * w * stdDev + mean` is correct (`z` is `x1*w`). I still don't get the part about having to have the result to be between 0 and 1, but I see that you are happy with mbq's answer, so I am happy to live with my confusion." CreationDate="2010-08-26T15:40:23.063" UserId="279" />
  <row Id="2358" PostId="2145" Text="Almost: replace *a* by 0 and divide by *k!*." CreationDate="2010-08-26T15:49:48.783" UserId="919" />
  <row Id="2359" PostId="733" Text="@chl Thanks; indeed recently I have seen similar thing done in my lab; the result was that the clusters are not significant though :-/" CreationDate="2010-08-26T15:50:29.943" UserId="88" />
  <row Id="2360" PostId="1991" Text="@Carlos: Ah, thanks: I was focusing on the first quotation only.  But that leaves us wondering specifically what King means by &quot;power.&quot;  Evidently it's something that is gained by not &quot;shirking responsibility&quot; ;-)." CreationDate="2010-08-26T15:52:58.737" UserId="919" />
  <row Id="2361" PostId="2131" Text="my dataset have 17 predictors(3 continuous and 14 categorical), obviously 1 class variable, and total 1000 obsrvations. The frequency dist. for class var. with train and test is train(bad) = 197, test(bad)= 103, trian(good)= 446, test(good)= 254 " CreationDate="2010-08-26T15:57:34.693" UserId="861" />
  <row Id="2363" PostId="2136" Score="1" Text="Yup, but the conditional logit model can (as I said) take more than 2 values, which differentiates it easily from Chamberlain's model, just like the fact that Chamberlain is designed specifically for panel data. This is thus a relevant information ; precise description of the usual conditional logit isn't (and the description of both would take more than 600 chars)." CreationDate="2010-08-26T16:07:32.973" UserId="1091" />
  <row Id="2364" PostId="2127" Text="1) yes, the question is somewhat ambiguous; there are, as you mention, many definitions of 'optimal': via information criterion, cross validation, etc. Most of the heuristic approaches I have seen to the problem proceed by stepwise predictor addition/removal: single pass forward addition or subtraction, etc. However, Hosmer &amp; Lemeshow make reference to this method (a variant of work by Lawless &amp; Singhal), which somehow 'magically' selects predictors by a single computation of an MLR (modulo some other stuff). I am very curious about this method..." CreationDate="2010-08-26T16:55:58.027" UserId="795" />
  <row Id="2365" PostId="2114" Text="LARS is great, BTW. very cool stuff. not sure how I can jam it into the framework of Cox Proportional Hazards model, tho..." CreationDate="2010-08-26T16:56:36.527" UserId="795" />
  <row Id="2366" PostId="2146" Score="1" Text="Can you make your question more concise?" CreationDate="2010-08-26T17:53:25.587" UserId="88" />
  <row Id="2367" PostId="2089" Text="It is free as in beer (for non-commercial use), but alas, not free as in speech." CreationDate="2010-08-26T18:08:07.593" UserId="1080" />
  <row Id="2368" PostId="2146" Score="1" Text="Please try your question again in clear and simple language" CreationDate="2010-08-26T21:04:07.457" UserId="74" />
  <row Id="2370" PostId="2151" Text="Is dimension reduction an option?" CreationDate="2010-08-27T01:59:57.453" UserId="5" />
  <row Id="2371" PostId="2151" Text="we have 4 classes, not really" CreationDate="2010-08-27T03:03:06.800" />
  <row Id="2372" PostId="2152" Score="1" Text="I would say the same providing another link http://www.google.gr/url?sa=t&amp;source=web&amp;cd=1&amp;ved=0CB8QFjAA&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.98.4088%26rep%3Drep1%26type%3Dpdf&amp;ei=nlx3TIq-LJCk4Abn_Li3Bg&amp;usg=AFQjCNHO-_yjWAJrRVnJms7MbcqaJkd8eg&amp;sig2=sqERq2v68UvOhJDMviOklg" CreationDate="2010-08-27T06:40:29.287" UserId="339" />
  <row Id="2373" PostId="2152" Score="1" Text="And here is another one, directly related to multi-class problem: Multi-class ROC analysis from a multi-objective optimisation perspective, Pattern Recognition Letters 2006 27(8): 918-927 (http://j.mp/9AMgzq)." CreationDate="2010-08-27T07:38:55.643" UserId="930" />
  <row Id="2374" PostId="2112" Text="@Adam Did you mean the example included in the help file for Anova? In this case, I grabbed the R transcript and put it there, http://j.mp/9pGtKY. HTH" CreationDate="2010-08-27T08:30:41.030" UserId="930" />
  <row Id="2375" PostId="2154" Text="Better than naming it Part 2 (or n), make a link to the previous question inside this." CreationDate="2010-08-27T09:10:00.560" UserId="88" />
  <row Id="2376" PostId="2114" Score="1" Text="The Glmnet software has a lasso'd Cox PH model:&#xA;http://cran.r-project.org/web/packages/glmnet/index.html&#xA;there is also a MATLAB version (not sure if it does a cox model though):&#xA;http://www-stat.stanford.edu/~tibs/glmnet-matlab/" CreationDate="2010-08-27T09:29:41.577" UserId="495" />
  <row Id="2377" PostId="1817" Text="I had a look at this, and it's perfectly fine but has a bit too much of the feel of an undergraduate text book for my tastes. Not that there's anything wrong with that..." CreationDate="2010-08-27T09:44:20.223" UserId="174" />
  <row Id="2378" PostId="2086" Text="I was just being a smart ass. I hope there's no hard feelings. :)" CreationDate="2010-08-27T10:41:42.997" UserId="144" />
  <row Id="2379" PostId="2126" Text="Note that in actual computations, one should not be forming $A^T A$ directly; exploit the QR or singular value decomposition of $A$ for this." CreationDate="2010-08-27T10:45:22.963" UserId="830" />
  <row Id="2380" PostId="1817" Text="Very nice book indeed! I've started long ago to reproduce his analysis (with Design Expert and SAS) using R, but never find time to finish it. If you like to check it out, http://www.aliquote.org/articles/tech/dae/" CreationDate="2010-08-27T11:17:11.820" UserId="930" />
  <row Id="2381" PostId="2115" Text="@chl: Thanks. It worked - this way I can plot both the Class random effects (level=1), and the student (level=2). " CreationDate="2010-08-27T11:30:46.447" UserId="108" />
  <row Id="2382" PostId="1544" Text="Thanks for your response John.  I'm not sure how I would go about using mixed-effects modelling to test from equivalence - do you have any examples?  Thanks!  Sam.&#xA;" CreationDate="2010-08-27T12:27:35.097" UserId="867" />
  <row Id="2383" PostId="2152" Score="1" Text="Thanks for the other links, apparently that whole issue is popular, and its archive on science direct can be found here&#xA;http://www.sciencedirect.com/science?_ob=PublicationURL&amp;_tockey=%23TOC%235665%232006%23999729991%23621242%23FLA%23&amp;_cdi=5665&amp;_pubType=J&amp;view=c&amp;_auth=y&amp;_acct=C000050221&amp;_version=1&amp;_urlVersion=0&amp;_userid=10&amp;md5=ece739626e7c50b228e85480cb2407e3" CreationDate="2010-08-27T13:17:58.810" UserId="1036" />
  <row Id="2384" PostId="2146" Text="I think it highly unlikely that you will get answered without clarifying this question. " CreationDate="2010-08-27T13:48:09.280" UserId="5" />
  <row Id="2385" PostId="1233" Text="I agree that this is the simplest method and will probably do the job. I think I would just suggest a minimum level for the c-line. If one or two cases are raising too many false positive alerts, just change the threshold to something higher." CreationDate="2010-08-27T13:54:44.227" UserId="1036" />
  <row Id="2386" PostId="2157" Score="1" Text="P-values are not appropriate measures for control charts, because of the serial multiple comparisons involved and the possibility (likelihood, in this case) of serial correlations.  Control chart properties are better assessed in terms of expected lengths of out-of-control (OOC) runs and in-control runs under the null and alternative hypotheses, respectively.  On this basis, letting even one case trigger an OOC is almost never a good procedure." CreationDate="2010-08-27T14:12:07.123" UserId="919" />
  <row Id="2387" PostId="1959" Text="Could you tone down the criticism in this a little?  Your point is fair, but I would love to see Gary get involved in the site (if at all possible)..." CreationDate="2010-08-27T14:13:51.740" UserId="5" />
  <row Id="2388" PostId="2112" Text="@chl thank you! for some reason I couldn't find an on-line help for car Anova(). but this example is already in the R off-line help at ?Anova. I am starting to understand this method." CreationDate="2010-08-27T14:19:11.353" UserId="1084" />
  <row Id="2389" PostId="2112" Text="@Adam Ah ok, my answer was thus a little bit out... What I call on-line is indeed the help() or ? call under R. For external references, you can try http://www.rseek.org/ or http://search.r-project.org/ (much better than directly goggling about R). Also I just this post with a nice illustration: http://j.mp/dnJgW0." CreationDate="2010-08-27T14:42:32.210" UserId="930" />
  <row Id="2390" PostId="1934" Text="Ditto with your second paragraph. If you know 2 infections is too low to be bothered with you can probably reasonably form a minimum estimate in which you want an alert raised." CreationDate="2010-08-27T14:49:44.803" UserId="1036" />
  <row Id="2392" PostId="2155" Text="THIS IS FANTASTIC THANK YOU" CreationDate="2010-08-27T15:03:41.550" />
  <row Id="2393" PostId="2157" Text="I was talking about the p-value of one case, so I think it is correct. Of course I agree with a general claim that this can (and should) be done better, still from the voice of the article lockhart cited I can conclude he seeks for statistics &quot;light&quot;. " CreationDate="2010-08-27T15:04:17.417" UserId="88" />
  <row Id="2394" PostId="2154" Score="2" Text="The answer I supplied to your previous question suggests a version of the &quot;c-chart&quot; that is more rigorously justified, quite simple, and accordingly is easy to interpret. (It also raises the logically earlier question of whether control charting is an appropriate technique for you.) You did not respond to this, but now you re-raise the question, so I am wondering why: why do you think this solution will not work for you?" CreationDate="2010-08-27T15:10:33.780" UserId="919" />
  <row Id="2395" PostId="2157" Text="I agree that your p-value calculation is correct, but the issue I am raising concerns whether that's the right way, or even a good way, to think about and design control charts.  For those familiar with adjusting for large numbers of multiple comparisons, a single p-value can be an excellent guide, but for others this way of thinking might be misleading." CreationDate="2010-08-27T15:14:16.603" UserId="919" />
  <row Id="2398" PostId="1959" Text="Normally I am reluctant to criticize (for example, I have yet to vote down any contribution in these forums), but the criticism seems warranted in light of the tone of that tweet.  Nevertheless, I share your interests and will delete the unnecessary parts of the response." CreationDate="2010-08-27T15:19:51.207" UserId="919" />
  <row Id="2399" PostId="1959" Text="Thanks.  I'll delete my comments here too now so that this all just looks like a coherent train of thought.  :)" CreationDate="2010-08-27T15:22:12.710" UserId="5" />
  <row Id="2400" PostId="2112" Score="1" Text="Here is the correct link, http://j.mp/c6nDvn." CreationDate="2010-08-27T15:30:48.977" UserId="930" />
  <row Id="2401" PostId="2164" Text="Thanks! When I looked through par in an attempt to figure this out myself I didn't realize the relevance of xpd." CreationDate="2010-08-27T15:37:10.983" UserId="1084" />
  <row Id="2402" PostId="2164" Text="Yeah, I find the par man page pretty impenetrable. I'm not sure how much of my life I've spent staring at it but it's a significant proportion. I've got this over my desk: http://cran.r-project.org/doc/contrib/Short-refcard.pdf It doesn't get as deep as xpd but it usually gives me a starting point. " CreationDate="2010-08-27T16:03:12.123" UserId="1076" />
  <row Id="2403" PostId="2164" Text="Check also the mtext() function for putting text in the margins. Also you can do quite everything you want with the grid interface, but it is rather difficult to grasp at first; a good starting point is Paul Murrell's website: http://j.mp/a9fplf." CreationDate="2010-08-27T16:15:09.340" UserId="930" />
  <row Id="2404" PostId="1544" Text="equivalence of what?  Your question doesn't mention this.  Perhaps edit the question to be more precise." CreationDate="2010-08-27T16:52:11.227" UserId="601" />
  <row Id="2405" PostId="2112" Text="@chl Thanks, I saw this one before. " CreationDate="2010-08-27T17:10:11.010" UserId="1084" />
  <row Id="2407" PostId="2167" Score="1" Text="BTW, kernels are not really essential for SVM's. The &quot;heart&quot; of SVM is the principle of soft margin maximization. Going to kernel representation makes your problem dimensionality O(m^2) instead of O(d) where m is the number of examples and d is the dimension of your feature space, so if m^2 is more than d, you might be better off doing away with kernels&#xA;http://jmlr.csail.mit.edu/papers/v6/keerthi05a.html" CreationDate="2010-08-27T17:53:21.080" UserId="511" />
  <row Id="2409" PostId="2168" Text="+1 I was going to post the same answer." CreationDate="2010-08-27T18:11:39.647" UserId="88" />
  <row Id="2410" PostId="2168" Text="Thanks for the answer.  @mbq @ebony1: IMO we need to make more of an effort to post more serious machine learning questions on the site, in order to attract more of that community.  " CreationDate="2010-08-27T18:23:58.813" UserId="5" />
  <row Id="2411" PostId="2169" Text="What is the distribution of $x$? I suppose we can assume that the weights are known." CreationDate="2010-08-27T18:34:51.940" />
  <row Id="2412" PostId="2168" Text="@Shane I completely agree, but what about other SO site like http://metaoptimize.com/qa/?" CreationDate="2010-08-27T18:54:26.600" UserId="930" />
  <row Id="2413" PostId="2169" Text="right, sorry, the weights are independent from $x$. The restrictions on the weights characterize, evidently, the class of 'coherent' risk measures. I have not seen any suggestions about how to set them, though, and this appears to be a black art. However, perhaps one would choose them to get the best discriminatory power (i.e. smallest standard error)..." CreationDate="2010-08-27T18:56:19.233" UserId="795" />
  <row Id="2414" PostId="2168" Text="@chl: That's also an option, but it's not part of the StackExchange (it's controlled by one person, and on different software) and I personally would rather have these different data analysis communities mingle in one place." CreationDate="2010-08-27T19:01:41.343" UserId="5" />
  <row Id="2415" PostId="2168" Text="@Shane Well, ok it makes sense." CreationDate="2010-08-27T20:05:15.083" UserId="930" />
  <row Id="2416" PostId="2168" Text="there's also Machine Learning stack exchange proposal http://area51.stackexchange.com/proposals/7607/machine-learning" CreationDate="2010-08-27T21:11:22.253" UserId="511" />
  <row Id="2417" PostId="2168" Text="@Yaroslav, @chl, @Sahne I've started this topic on chat, I think it is more comfortable place to discuss than comments ;-) http://chat.meta.stackoverflow.com/rooms/170/stats" CreationDate="2010-08-27T22:39:08.553" UserId="88" />
  <row Id="2418" PostId="2171" Score="4" Text="Should probably be community wiki?" CreationDate="2010-08-27T22:46:02.860" UserId="5" />
  <row Id="2419" PostId="2146" Text="Cautious +1 to the edit. In truth, I think the question is still not perfectly clear, but I know I'd love to read the answers!" CreationDate="2010-08-27T23:14:00.557" UserId="174" />
  <row Id="2420" PostId="2169" Text="could you answer Srikant's first question? I don't see how you're going to get anywhere unless you specify that, without using some form of resampling." CreationDate="2010-08-27T23:28:07.677" UserId="805" />
  <row Id="2421" PostId="2168" Text="chatroom requires 20 meta.stackoverflow reputation, and I have 1" CreationDate="2010-08-27T23:29:10.673" UserId="511" />
  <row Id="2422" PostId="2169" Text="@Glen_b: I am looking for a non-parametric estimator. I just found 2 papers by Stigler which, though very dense, appear to give a form of the asymptotic variance for the parametric case.  I am now hunting down papers by Downton, Stillito, as referred to in Elamir &amp; Seheult's papers on L-moment standard errors." CreationDate="2010-08-27T23:35:58.400" UserId="795" />
  <row Id="2423" PostId="2173" Score="3" Text="1. IMHO time spent with R is well invested if you plan to do anything serious.&#xA;2. Also consider KNIME http://www.knime.org/ as RapidMiner alternative." CreationDate="2010-08-28T00:01:20.573" UserId="22" />
  <row Id="2424" PostId="2172" Score="1" Text="+1 for R and ggplot2" CreationDate="2010-08-28T00:01:46.813" UserId="22" />
  <row Id="2425" PostId="233" Text="Moreover, outliers often have high leverage and therefore might actually have *small* residuals (and cause non-outlying values to have large residuals)." CreationDate="2010-08-28T01:36:51.560" UserId="919" />
  <row Id="2426" PostId="2169" Text="Oh, okay, you mean asymptotically. But you will still need certain assumptions about the distribution for the results to apply, and the results will depend on characteristics of the distribution that are unknown (such as the variance)." CreationDate="2010-08-28T03:24:53.343" UserId="805" />
  <row Id="2427" PostId="2172" Text="Awesome, great links! I already use R and ggplot2, but the visualizations from there seem more of the &quot;graphics for a report&quot;-variety, than of the &quot;eye candy/visualization interesting in of itself&quot;-variety I'm looking for. (ggplot2 is super beautiful, but it's not really meant to allow unlimited creativity.) Am I wrong?, or do you sometimes use R/ggplot2 as an input into another visualization tool?" CreationDate="2010-08-28T04:28:46.970" UserId="1106" />
  <row Id="2428" PostId="2159" Text="Again some interesting tools. I knew about Numpy, Matplotlib and ReportLab, but Pyspread seems like an interesting idea. At least I would like to type Python expressions in spreadsheet cells. While it doesn't solve all possible problems, it could be good for prototyping and playing around with data." CreationDate="2010-08-28T06:21:35.973" UserId="890" />
  <row Id="2429" PostId="2168" Text="When you say &quot;can only be applied to linear models&quot; do you mean that you can prove that no application exit to something else than linear models?   " CreationDate="2010-08-28T08:14:56.387" UserId="223" />
  <row Id="2430" PostId="2168" Text="What do you mean by &quot;examples&quot; ? as you state it is seems to be something that can be interpreted as a dot product. " CreationDate="2010-08-28T08:16:42.597" UserId="223" />
  <row Id="2431" PostId="2168" Text="for me it is -1 until you answer my questions :) " CreationDate="2010-08-28T08:17:33.237" UserId="223" />
  <row Id="2432" PostId="2174" Text="`igraph` works in R also; for 3D openGL accelerated vis in R, use `rgl` &amp; `misc3d` packages." CreationDate="2010-08-28T09:41:15.630" UserId="88" />
  <row Id="2433" PostId="2106" Text="@Matt thank you for the very helpful edit. I was trying other things after the ~ except 1 - as in your answer. Thanks again!" CreationDate="2010-08-28T16:41:11.930" UserId="1084" />
  <row Id="2434" PostId="2176" Text="Thank you for a very thoughtful and instructive answer!" CreationDate="2010-08-28T16:42:30.257" UserId="1084" />
  <row Id="2435" PostId="2106" Text="Ah, it seems that there's a bug in the current version of ez (v1.6.1). I've tracked it down to the fact that in converting to the formats that Anova() likes, ez was failing to turn something into a factor. I'm not sure whether this was always a bug, or whether Anova() recently changed in a way that broke ez. Regardless, I'm preparing ez v2.0, and this is one of the bug fixes. Stay tuned..." CreationDate="2010-08-28T17:16:34.920" UserId="364" />
  <row Id="2436" PostId="2183" Text="@Srikvant.  Assume 5% is acceptable significance.  I'm seeking a more precise answer, one which exposes the idea that &quot;A leads B&quot; is a new statistic, the difference of pA and pB, and that it's corresponding confidence interval is not simply 2*MOE." CreationDate="2010-08-28T23:39:37.843" />
  <row Id="2438" PostId="2183" Text="@somebody I fixed the answer. I hope it is what you want." CreationDate="2010-08-29T00:34:31.930" />
  <row Id="2439" PostId="2106" Text="Thanks Mike, it seems to have happened fairly recently. I just finished an analysis last month using your package and haven't noticed any problems until this question came up." CreationDate="2010-08-29T07:12:59.430" UserId="966" />
  <row Id="2440" PostId="1720" Text="Sometimes we are interested in problems that come from from formulations of the total quantity of something. If you know the mean, you can go from the mean to the total (multiplying by the number of the observations). There is no way to go from the median to the total!  " CreationDate="2010-08-29T08:22:41.133" UserId="339" />
  <row Id="2441" PostId="2180" Score="1" Text="Has any one of those algos been implemented in R or other software? " CreationDate="2010-08-29T09:26:32.740" UserId="339" />
  <row Id="2442" PostId="2180" Score="1" Text="Yes, take a look at the R penalizedSVM package. Other packages of interest are : penalized, elasticnet, ppls, lars, or more generally: http://cran.r-project.org/web/views/MachineLearning.html" CreationDate="2010-08-29T09:33:19.063" UserId="930" />
  <row Id="2443" PostId="2187" Text="do you recommend to read all of them ? :) " CreationDate="2010-08-29T10:41:03.577" UserId="223" />
  <row Id="2444" PostId="2187" Score="3" Text="In Psychology the Tabachnik &amp; Fidell Book has a pretty good reputation. It is very understandable and applied and not too mathematic. However, examples are only in SPSS or SAS (no R!). But if your problem is covered in there, you will definitely solve it with the book. I recommend it as a good starting point. I don't like the Hair book (same level as Tabachnik &amp; Fidell, but worse). And you gotta LOVE the Gelman. However, it is more complicated." CreationDate="2010-08-29T12:00:38.657" UserId="442" />
  <row Id="2445" PostId="2187" Score="3" Text="HAIR et al is good if you don't like math and you want a step by step process. It's popular in management and business schools. If you can handle math, Hair et al can seem verbose.&#xA;&#xA;Tabachnick and Fidell is popular in psychology. It's clearly written and does contain some mathematics. However, if you want a rigorous mathematical treatment, I'd look for an additional book to complement it." CreationDate="2010-08-29T13:09:37.017" UserId="183" />
  <row Id="2446" PostId="2181" Text="To what extent do you want: (a) mathematical rigour; (b) applications in particular software (e.g., R, SPSS, SAS, etc.); (c) domain-specific applications?" CreationDate="2010-08-29T13:12:27.157" UserId="183" />
  <row Id="2447" PostId="2181" Text="Jeromy, allow me to take all of these with one blow: I'm a psychology student. And I reckon you're familiar with required statistical background... So there... =)&#xA;(I'm good with R and SPSS... but R has a greater priority)" CreationDate="2010-08-29T15:26:47.500" UserId="1356" />
  <row Id="2448" PostId="2187" Text="Thanks for giving accent to statistics in psychology." CreationDate="2010-08-29T15:31:06.757" UserId="1356" />
  <row Id="2449" PostId="2190" Text="Can you expand on this question a little?  What else to you know about the &quot;great prediction&quot;?" CreationDate="2010-08-29T17:00:13.033" UserId="5" />
  <row Id="2451" PostId="2190" Score="2" Text="If you mean reading the forest by looking at each tree and trying to understand what is going on, don't do it, you'll fail to see anything in such a complex, randomized structure. RF is just a black box method, its models are not intended to describe data (as usual in machine learning)." CreationDate="2010-08-29T17:33:21.067" UserId="88" />
  <row Id="2452" PostId="2138" Text="I like the points with mean centrally.  It has a truer representation of the lines.  You could make the points smaller." CreationDate="2010-08-29T19:18:56.073" UserId="601" />
  <row Id="2453" PostId="2193" Text="As I understand it is possible to reconstruct the structure but it is impossible to assign class specific lineage of nodes. Here is my problem. I have a binary problem (presence/absence) of variable that can be predictive of one of 4 classes (as single and as joint for example combination of 5 variables). I am getting high prediction relying on top 20 by VIMP out of 1500 variables). I have to extract the specific combinations at least some of them" CreationDate="2010-08-29T23:18:37.163" />
  <row Id="2454" PostId="2177" Text="Thanks for the response, I did not include specific examples of methodologies such as LOESS smoothing line, or model selection types (like BIC or using an F-test) because they are non-central to my question. Thanks for the clear up and I was definitely referring to partial regression plots (the first article I had been exposed to it in had called them added variable plots). Although it is not really central to my question either so I just referred to the wikipedia page. I also edited multi-variate as indeed I am only predicting 1 variable. Thanks again for the response." CreationDate="2010-08-30T12:32:58.930" UserId="1036" />
  <row Id="2455" PostId="783" Text="Anyway, I can give a citation of Lehman about that: &quot;There is some convenience in such standardization since it permits a reduction in certain tables needed for carrying out various tests&quot;. " CreationDate="2010-08-30T12:37:28.153" UserId="223" />
  <row Id="2456" PostId="1332" Text="depending on what you call 'the thruth'... I don't think I give so much value to statistical truth :) " CreationDate="2010-08-30T12:39:31.627" UserId="223" />
  <row Id="2457" PostId="2201" Text="This is a good idea, thanks for the links." CreationDate="2010-08-30T12:41:35.273" UserId="1117" />
  <row Id="2458" PostId="1360" Text="Do you think that by &quot;those&quot; he means &quot;everyone&quot; ? what was the context of this citation ? it seems a bit strong like this :) " CreationDate="2010-08-30T12:42:22.667" UserId="223" />
  <row Id="2459" PostId="2072" Text="Can you clarify what you mean by irregular? I had initially taken it to mean that you had two series of different discrete time intervals." CreationDate="2010-08-30T12:42:39.313" UserId="1036" />
  <row Id="2460" PostId="2206" Text="I'll run with the idea that it's Monte Carlo -- thanks for the response." CreationDate="2010-08-30T12:44:02.773" UserId="1117" />
  <row Id="2461" PostId="2072" Text="Yes, I mean two time series with different random arrival times (not regularly sampled).  " CreationDate="2010-08-30T12:56:30.667" UserId="5" />
  <row Id="2462" PostId="2177" Text="Also I think my last statement sufficiently summarizes what I am interested in, and I don't see it as being entirely consumed by formal inference." CreationDate="2010-08-30T13:37:58.507" UserId="1036" />
  <row Id="2463" PostId="2201" Text="Another useful link: http://stats.stackexchange.com/questions/1385/techniques-for-handling-incomplete-missing-data &#xA;&#xA;" CreationDate="2010-08-30T13:49:03.360" UserId="339" />
  <row Id="2465" PostId="646" Text="I'm not sure how many of these will have the scope of &quot;marketing&quot; research, but I would also suggest the OP check out ICPSR. It is also good to know the quality of the sampling and data record keeping, which I imagine could be haphazard if you get data from a proprietary, non-education related source." CreationDate="2010-08-30T15:32:19.930" UserId="1036" />
  <row Id="2466" PostId="2200" Text="what? the question says 'blah blah blah would be $min(a,b)$, how do I find the min?' I answered the question." CreationDate="2010-08-30T16:38:00.527" UserId="795" />
  <row Id="2467" PostId="2145" Text="In fact, I got another form&#xA;&#xA;$$ (n+1)!\delta_{n+1} = \gamma_{n+1}+\sum_{k=1}^nk\gamma_k\cdot[(n-k)!\delta_{n-k}] $$" CreationDate="2010-08-30T17:27:09.630" UserId="1043" />
  <row Id="2468" PostId="2215" Text="Re: &quot;I think of Monte Carlo as any method that uses probabilistic sampling of data as input into a computation...&quot; this is a useful way for me to think of the problem. Thank you." CreationDate="2010-08-30T17:46:11.527" UserId="1117" />
  <row Id="2469" PostId="766" Text="In addition, `n!` = `Gamma(n+1)` for n &gt;= 0. So try to look for a function called `Gamma` if you need to calculate the factorial (or log Gamma if you're calculating the log likelihood)" CreationDate="2010-08-30T18:43:09.860" UserId="961" />
  <row Id="2470" PostId="2221" Score="1" Text="This is also why the area of a circle is *not* equal to $\int_0^{2\pi} \int_0^R 1 d\theta dr$." CreationDate="2010-08-30T19:03:38.453" UserId="795" />
  <row Id="2471" PostId="2213" Text="Although I didn't ask the question, I think it would be great to have some basic literature with the answers. Please." CreationDate="2010-08-30T19:19:52.830" UserId="442" />
  <row Id="2472" PostId="2213" Text="@Henrik: You want basic literature on neural networks?  You might consider posting another question?" CreationDate="2010-08-30T19:39:20.523" UserId="5" />
  <row Id="2474" PostId="2223" Score="4" Text="This seems like a duplicate: http://stats.stackexchange.com/questions/1963/looking-for-good-introductory-treatment-of-meta-analysis.  @Andrew: You need to differentiate this question from the other one, or else it will be closed." CreationDate="2010-08-30T20:36:24.217" UserId="5" />
  <row Id="2475" PostId="2222" Text="Thank you. Is it ok to generate the samples using `sample` and `replace=TRUE`? Is there any reason to use a package like `boot`?" CreationDate="2010-08-30T20:36:33.120" UserId="339" />
  <row Id="2476" PostId="2222" Text="Typically it's done with replacement so you want to set that to TRUE.  As for why... the package is optimized so it should run faster.... never timed it.  That can be a problem if you set R large.  And, as you can see, the code is nice and concise.  It also has lots of features you wouldn't get easily rolling your own." CreationDate="2010-08-30T23:10:18.733" UserId="601" />
  <row Id="2479" PostId="2202" Text="your last explanation on finding the smallest intersection can be applied to all cases. Min `A∩B` is `max(a+b-1,0)`" CreationDate="2010-08-31T03:57:00.773" UserId="862" />
  <row Id="2480" PostId="2224" Text="I think it is better to put a comment and vote for closing the question. Indeed it may be a bit confusing to have an answer wich is a link to a question on the same site... I voted to close the question. " CreationDate="2010-08-31T07:38:31.393" UserId="223" />
  <row Id="2481" PostId="2222" Text="boot.ci returns the confidence interval. Is there any (boot) function that gives the p.value? (as the ratio of the number of differences at least as high as the one observed, over the total number of generated samples)" CreationDate="2010-08-31T08:49:08.873" UserId="339" />
  <row Id="2482" PostId="2222" Text="ok, I found a way to get it: `sum(b$t&gt;=b$t0)/b$R`" CreationDate="2010-08-31T09:02:29.920" UserId="339" />
  <row Id="2483" PostId="2234" Text="Do we have to assume that you are considering a fixed set of predictors, i.e. you are interested in getting a reliable prediction given $k$ predictors, or are you also interested in some kind of penalization on the $X_j\quad (j=1\dots k)$?" CreationDate="2010-08-31T11:29:14.603" UserId="930" />
  <row Id="2484" PostId="2234" Text="I admit that for my personal interest, penalization wouldn't be necessary, and for the sake of knowledge here I would say both are relevant answers :)" CreationDate="2010-08-31T11:46:39.227" UserId="253" />
  <row Id="2485" PostId="2234" Text="@Tal: For future reference: you might have been able to phrase this question in such a way that we would have allowed it as a non-CW question.  See http://meta.stats.stackexchange.com/questions/290/what-is-community-wiki" CreationDate="2010-08-31T13:07:31.287" UserId="5" />
  <row Id="2486" PostId="2234" Text="Thank you for the link Shane.  A very interesting discussion you opened there.  After reading Thomasas answer, I believe this should still be a community wiki, since my intention was to find as many alternatives as possible (something i doubt any one person would be able to supply).&#xA;   Yet, again, thank you for directing me to that thread!" CreationDate="2010-08-31T13:22:00.373" UserId="253" />
  <row Id="2487" PostId="2200" Text="@shabbychef I agree; bad questions should be punished with no answers. " CreationDate="2010-08-31T14:21:09.380" UserId="88" />
  <row Id="2488" PostId="2224" Text="@Robin: Thank you for the suggestion.  I do not see how it would be any more or less confusing to place a link in an answer or a comment, though.  I, too, have voted to close the question, but I first waited a day for the poster to respond to Shane's suggestion that this question be edited." CreationDate="2010-08-31T14:22:26.660" UserId="919" />
  <row Id="2489" PostId="2174" Text="Also `matplotlib` plots are ugly; they may be nice for a long-years gnuplot user." CreationDate="2010-08-31T14:23:30.763" UserId="88" />
  <row Id="2490" PostId="2106" Text="@ Matt, I still get the same EZanova warning msg. for Anova I get another error msg: &#xA;Note: model has only an intercept; equivalent type-III tests substituted.&#xA;Error in linearHypothesis.mlm(mod, hyp.matrix, SSPE = SSPE, idata = idata, : The error SSP matrix is apparently of deficient rank = 0 &lt; 1      " CreationDate="2010-08-31T14:43:45.307" UserId="1084" />
  <row Id="2491" PostId="2227" Text="What's the source?" CreationDate="2010-08-31T15:02:25.087" UserId="5" />
  <row Id="2492" PostId="2236" Text="+1 I was about to post the same answer!" CreationDate="2010-08-31T15:05:46.110" UserId="930" />
  <row Id="2493" PostId="2224" Score="1" Text="@whuber my view is that answering a question is a step toward accepting it in the site. " CreationDate="2010-08-31T15:16:04.040" UserId="223" />
  <row Id="2494" PostId="2106" Text="This may help with the type-III thing https://stat.ethz.ch/pipermail/r-help/2007-October/144240.html ; and I've edited my answer to reflect the updated version of ez, which has changed &quot;sid&quot; to &quot;wid&quot;. I don't get your problem with the data you've given me." CreationDate="2010-08-31T15:24:33.373" UserId="966" />
  <row Id="2495" PostId="2106" Text="#Try this with the code I've written for the car Anova&#xA;#transfer data back to long to perform ezANOVA    &#xA;df2&lt;-melt(df1, id.vars=&quot;observ&quot;, measure.vars=names(df1)[6:21])&#xA;df2$suc&lt;-substr(df2$variable,1,2)&#xA;df2$cit&lt;-substr(df2$variable,3,4)&#xA;df2$observ&lt;-factor(df2$observ)&#xA;ez1&lt;-ezANOVA(data=df2,&#xA;  within=.(suc, cit),&#xA;  wid=.(observ),&#xA;  dv=.(value)&#xA;  )&#xA;" CreationDate="2010-08-31T15:28:08.107" UserId="966" />
  <row Id="2496" PostId="2222" Text="@gd047 : take into account that this is a one-sided p-value you're calculating. " CreationDate="2010-08-31T15:58:54.067" UserId="1124" />
  <row Id="2497" PostId="2227" Text="It was taken from Mike West's website: http://www.stat.duke.edu/~mw/fineart.html" CreationDate="2010-08-31T16:23:32.740" UserId="881" />
  <row Id="2498" PostId="2240" Text="Can you give an explanation of why permutation tests should not be used on datasets that you can't calculate all possible permutations?" CreationDate="2010-08-31T16:27:10.973" UserId="1036" />
  <row Id="2499" PostId="2240" Text="@Andy W : First define &quot;permutation test&quot;. for me, permutation tests are exact tests, using every permutation possible. That is impossible on larger datasets. The &quot;approximate permutation tests&quot; are in fact the simples Monte Carlo method around, and should be adressed that way. Next to that, the central limit theorem assures in most cases that the assumptions regarding the distribution of the test statistics are met when using large datasets. In complex testing, the use of permutation tests on large datasets makes calculation times unbearably long without adding any significant value. my2cents" CreationDate="2010-08-31T16:42:08.420" UserId="1124" />
  <row Id="2500" PostId="2200" Text="No, that's not what he's asking for. He asked for the min of the intersection, as can be deducted by careful reading of the first of the two sentences in the opening post." CreationDate="2010-08-31T16:51:40.677" UserId="1124" />
  <row Id="2501" PostId="2242" Text="I may be a bit confused but I think the above answer implicitly interprets confidence intervals as credible intervals. I am not sure if you can reason about the Prob(true polling percentage &lt;= 34.5) using the classical concept of a confidence interval." CreationDate="2010-08-31T17:04:38.783" />
  <row Id="2502" PostId="2224" Text="@Robin: Thank you; that makes sense and it's a principle that will help in future cases." CreationDate="2010-08-31T17:12:55.590" UserId="919" />
  <row Id="2503" PostId="2222" Text="@John Christie : Could you show me how your last bootstrap code should be modified in order to draw resamples of size 5 with replacement from the first sample and separate resamples of size 6 from the second sample (instead of resampling from the total number of 11 values). As I read on page 18 of the linked pdf, this should be the correct way to draw the bootstrap samples." CreationDate="2010-08-31T18:12:56.903" UserId="339" />
  <row Id="2504" PostId="2240" Text="@Joris Meys: I didn't say anything like bootstrapping a permutation test. I came into this question after reading the last paragraph of [SECTION 14.5 | Summary], in the linked pdf." CreationDate="2010-08-31T18:33:03.053" UserId="339" />
  <row Id="2505" PostId="2238" Text="+1 it is always nice to bootstrap something. On the other hand every challenge is easy with R: http://stat.ethz.ch/R-manual/R-devel/library/boot/html/tsboot.html" CreationDate="2010-08-31T19:47:47.827" UserId="88" />
  <row Id="2506" PostId="2249" Text="Thank you very much for that.  I thought of the Mann-Whitney test, but many (most?) of the values will result in ties.  Won't this reduce the power of the analysis? " CreationDate="2010-08-31T21:38:04.753" />
  <row Id="2507" PostId="2249" Score="2" Text="Yes, of course it will; but as you're working with 10 observations per group, don't expect very much from this setting; check for yourself at http://statpages.org/#Power" CreationDate="2010-08-31T21:53:56.843" UserId="930" />
  <row Id="2508" PostId="2186" Text="why the Tinsley book? No reviews on Amazon suggests it ain't a big seller or particularly good. " CreationDate="2010-08-31T23:18:05.303" UserId="74" />
  <row Id="2509" PostId="2238" Text="Thanks Dirk and mbq. I have one time series I collected. I am not using a data generating process to produce different series. So in this case I understand that I cannot get std deviations? Please correct me if I'm wrong. Thanks again!" CreationDate="2010-08-31T23:51:38.377" UserId="443" />
  <row Id="2510" PostId="2238" Text="You could try 'chunked' resampling. Google for block bootstrap." CreationDate="2010-09-01T01:14:32.647" UserId="334" />
  <row Id="2511" PostId="2245" Text="great question: see also this related question on correlation and causation http://stats.stackexchange.com/questions/534/under-what-conditions-does-correlation-imply-causation" CreationDate="2010-09-01T05:14:43.280" UserId="183" />
  <row Id="2512" PostId="2200" Text="@mbq - whats wrong with the question?" CreationDate="2010-09-01T06:00:17.303" UserId="862" />
  <row Id="2513" PostId="2257" Text="thanks for this. I would like to higlight the groups using color such that all points from the same group have the same color. Can quite figure out what the convert command does, let me check the documentation though a layman explanation can also help. &#xA;Thanks a lot!" CreationDate="2010-09-01T06:37:16.597" UserId="201" />
  <row Id="2514" PostId="2258" Text="LaTeX works, you must just stop editing for a while so that it will be applied to preview window.&#xA;" CreationDate="2010-09-01T07:26:03.240" UserId="88" />
  <row Id="2515" PostId="2258" Text="For LaTeX, enclose terms in $ signs." CreationDate="2010-09-01T07:26:12.747" UserId="159" />
  <row Id="2516" PostId="2258" Text="Got it. Which tag can I use for multiline latex block?" CreationDate="2010-09-01T07:33:25.310" UserId="862" />
  <row Id="2517" PostId="2256" Score="1" Text="You may also check out `rgl`." CreationDate="2010-09-01T07:35:39.570" UserId="88" />
  <row Id="2518" PostId="2258" Text="The align environment will work, but use \newline instead of \\." CreationDate="2010-09-01T07:45:25.050" UserId="159" />
  <row Id="2519" PostId="2200" Text="@saminny You could have formulated it better so that it would be less ambiguous and easier to understand. shabbychef just pointed it out with his answer." CreationDate="2010-09-01T07:46:30.797" UserId="88" />
  <row Id="2520" PostId="1305" Text="Could you be a bit more specific about the values you have available and the values you want to be able to output.  From the current information it is hard to tell what you need." CreationDate="2010-09-01T07:48:30.957" UserId="196" />
  <row Id="2521" PostId="2240" Text="@gd047 Then I have misread your question. But you should really keep confidence intervals and p.values strictly separated. The confidence interval is estimated based on bootstrapping within each sample (although it is biased by definition), the permutation test is done by permutations over the complete dataset. Those are two completely different things." CreationDate="2010-09-01T08:15:40.293" UserId="1124" />
  <row Id="2522" PostId="2242" Text="We're not working in a Bayesian framework, so one can hardly speak of a posterior probability. I simply use the definition of a confidence interval, which is said to contain the true estimate with a probability of 95%. So it would be correct to say that the chance of having a true A (not the estimate) smaller than 34.5 is only 2.5%, by definition of the confidence interval." CreationDate="2010-09-01T08:22:41.603" UserId="1124" />
  <row Id="2523" PostId="2260" Score="2" Text="Interesting point about the means. I suppose differences between means would suggest that the groups are different in some respect. And empirically I'd expect that where there's large differences between means, there's more likely to be differences in factor structure. Differences in means might also have implications for how the findings regarding the factor structure could be generalised. It might also increase correlations between items in general due to an effect the opposite of range restriction." CreationDate="2010-09-01T08:35:28.633" UserId="183" />
  <row Id="2524" PostId="2258" Text="how do I use align environment? Is it a tag?" CreationDate="2010-09-01T08:41:23.433" UserId="862" />
  <row Id="2525" PostId="2262" Score="1" Text="Could you explain what a 'peak' stand for in the context of your data?" CreationDate="2010-09-01T10:03:25.763" UserId="930" />
  <row Id="2526" PostId="2262" Score="1" Text="Also do you have access to the underlying data? How are the curves drawn?&#xA;" CreationDate="2010-09-01T10:09:50.720" UserId="8" />
  <row Id="2527" PostId="2264" Text="Please may we have an expected value tag?" CreationDate="2010-09-01T10:22:01.693" UserId="1134" />
  <row Id="2528" PostId="2258" Text="Several points: 1. See my edits as to how to use Latex. 2. You may also want to see this thread: [Vote early, vote often](http://meta.stats.stackexchange.com/questions/314/vote-early-vote-often). 3. If this is homework could you please tag it as such so that people can provide hints instead of complete answers." CreationDate="2010-09-01T11:00:54.530" />
  <row Id="2529" PostId="2265" Text="Thanks for the comprehensive response. OpenMX does look like a good open source R package option for multi-group CFA. The IRT options also look interesting. Good point about the problems of hypothesis testing in CFA. Thanks for the references." CreationDate="2010-09-01T11:12:05.397" UserId="183" />
  <row Id="2530" PostId="2257" Text="`s3d &lt;- scatterplot3d(x.mds$points[,1:3], color=as.numeric(gl(2,50)))` would highlight the first 50 points in black, the remaining ones in red; you can pass any logical vector or factor to suit your data. HTH" CreationDate="2010-09-01T11:12:43.320" UserId="930" />
  <row Id="2531" PostId="2265" Text="I'd recommend lavaan as well, stupid I didn't think of it earlier.&#xA;" CreationDate="2010-09-01T11:28:17.313" UserId="1124" />
  <row Id="2532" PostId="2250" Text="With only 10 observations in each group, ordinal logistic regression is very unlikely to fit, let alone give correct estimates of the variances. I'd suggest the permutation test." CreationDate="2010-09-01T11:54:28.100" UserId="1124" />
  <row Id="2533" PostId="2264" Text="although not the same, you may want to see this related post: http://stats.stackexchange.com/questions/1848/bayesian-rating-system-with-multiple-categories-for-each-rating" CreationDate="2010-09-01T11:55:58.223" UserId="183" />
  <row Id="2534" PostId="2269" Text="This is an inappropriate place to ask this question. You need to deal with SAS directly or other forums directly related to utilizing SAS software." CreationDate="2010-09-01T12:15:22.400" UserId="1036" />
  <row Id="2535" PostId="2253" Text="Thanks everyone.  I went back and tried the Mann-Whitney test, using the method of listing the score of each individual as a column.  (FWIW, I use GraphPad Prism for the analysis.)  The results do appear reasonable, given what we already know about the biology.  Thanks very much for your help." CreationDate="2010-09-01T12:15:46.770" />
  <row Id="2536" PostId="2269" Text="Agree, now closed." CreationDate="2010-09-01T12:42:06.853" UserId="88" />
  <row Id="2537" PostId="2268" Text="Thanks for the link. I agree that the number of ratings for the product compared to average number for all products is significant. But the variance must be significant too. Suppose every product had zero variance in its ratings, then a single product rating would be sufficient to provide the ev. " CreationDate="2010-09-01T12:50:30.447" UserId="1134" />
  <row Id="2538" PostId="2266" Text="As a newbie student maybe I shouldn't be too picky, but I'm not very happy with this answer. Why can't an ev be calculated? Can you prove that statement? Also 95% CI seems plain arbitrary.  " CreationDate="2010-09-01T12:57:18.917" UserId="1134" />
  <row Id="2539" PostId="849" Text="+1 cause it's a great answer that helped my a lot today. Sad that it only got one point so far. Go Brett, go!" CreationDate="2010-09-01T13:26:14.213" UserId="442" />
  <row Id="2540" PostId="2263" Text="Hello Joris.&#xA;&#xA;Thanks for the input.&#xA;&#xA;All the peaks are gaussians - having a nice bell shape. The only thing that varies is the amplitude (height) and the SD (width).&#xA;&#xA;I actually don't really know about this density function. COuld you explain it more or give me link towards it?&#xA;&#xA;Thanks." CreationDate="2010-09-01T13:48:46.960" UserId="1133" />
  <row Id="2541" PostId="2263" Text="@chl: 'peak' is simply any area above my baseline. All peaks have gaussian distributions.&#xA;&#xA;@csgillespie: I have access to the original data that plotted the histogram and the curves are drawn in GraphPad Prism using a 'Sum of three gaussians' fit." CreationDate="2010-09-01T13:52:02.093" UserId="1133" />
  <row Id="2542" PostId="2263" Text="This is how the whole graph looks like: http://imgur.com/ipBiL.png" CreationDate="2010-09-01T13:53:16.993" UserId="1133" />
  <row Id="2543" PostId="2273" Text="That doesn't produce a scatterplot, does it?" CreationDate="2010-09-01T13:58:16.093" UserId="5" />
  <row Id="2544" PostId="2263" Text="@Cadu Ok, so this does not resemble what is commonly refered to as AUC, since this term is more often used in reliability studies or when assessing performance of classifier, i.e. when you plot sensibility against (1-specificity)." CreationDate="2010-09-01T13:58:26.763" UserId="930" />
  <row Id="2545" PostId="2263" Text="In full, the probability density function or pdf : http://en.wikipedia.org/wiki/Probability_density_function . The one for the normal distribution, which I use, is thoroughly described. Google will give you tons of hits on that. As you used the sum of gaussians, the approach I gave you should work. Mind you, it gives you the total area for each gaussian curve. So if you add the AUC's for the three curves, you end up with a higher area than under the combined curve." CreationDate="2010-09-01T14:03:54.363" UserId="1124" />
  <row Id="2546" PostId="2262" Score="1" Text="@Cadu Maybe change your title because AUC is definitively misleading there..." CreationDate="2010-09-01T14:10:20.980" UserId="930" />
  <row Id="2547" PostId="2273" Text="@Shane no, it's just a variation of the boxplot with an added kernel density estimate" CreationDate="2010-09-01T14:13:42.087" UserId="930" />
  <row Id="2548" PostId="2273" Score="1" Text="@Shane @Tal BTW, Box-percentile plot are better (`bpplot` in the `Hmisc` package)." CreationDate="2010-09-01T14:16:28.777" UserId="930" />
  <row Id="2549" PostId="2273" Text="Hi chl.  Thank you for the answer.  I remember coming by that function, but as Shane said - it doesn't produce the scatter plot element.  I'll +1 for the good intentions - but will keep this question open :).  Cheers, Tal" CreationDate="2010-09-01T14:16:49.010" UserId="253" />
  <row Id="2550" PostId="2232" Text="Thanks, I like the convergence of infinite series point-of-view!" CreationDate="2010-09-01T14:20:24.957" UserId="1106" />
  <row Id="2551" PostId="2263" Text="Sounds pretty good for me, Joris.&#xA;&#xA;However i am still unsure which one of those formulas is the correct one. If it is not too much, can you point it out which one you think would be the best one for this situation?&#xA;&#xA;And, yes, i'll keep in mind that it is not the total AUC.&#xA;&#xA;Thanks for your time again." CreationDate="2010-09-01T14:23:11.117" UserId="1133" />
  <row Id="2552" PostId="2250" Text="@Joris Good point! I forgot the sample size issue when I initially wrote my answer. The GLM would certainly yields poor estimates... Re-randomization test should be better, you're right. Thanks!" CreationDate="2010-09-01T14:26:30.210" UserId="930" />
  <row Id="2553" PostId="2274" Text="Hey Chl. I will try this mclust package and see what it can do.&#xA;&#xA;Thanks for the hint :)" CreationDate="2010-09-01T14:32:47.650" UserId="1133" />
  <row Id="2555" PostId="2273" Score="1" Text="@Tal Well, I'll try to figure out myself how to make it in R; I think it would not be so difficult using `stripchart()` or a jittering procedure." CreationDate="2010-09-01T14:46:14.957" UserId="930" />
  <row Id="2556" PostId="2273" Text="I liked that colour... " CreationDate="2010-09-01T14:50:32.057" UserId="88" />
  <row Id="2557" PostId="2271" Score="5" Text="This is maybe a stupid comment, but what does the position of dots suppose to mean?" CreationDate="2010-09-01T14:52:09.283" UserId="88" />
  <row Id="2558" PostId="2253" Score="1" Text="Mann-Whitney can be a good choice, but be aware that it makes some specific assumptions about the relationships among the two groups.  It is good when results appear reasonable, but that cannot be the sole reason to adopt a statistical test: that would be getting into a circular argument (&quot;we chose this test because it gives reasonable answers; our answers are significant because this test says so&quot;).  At a minimum, make sure to graph all the data in a way that promotes visual comparison of the groups." CreationDate="2010-09-01T14:56:58.837" UserId="919" />
  <row Id="2559" PostId="2249" Text="@John: I suspect the KS test I proposed would be more powerful than Mann-Whitney because it can detect various alternatives that M-W cannot (such as a change in spread) and otherwise is doing a mathematically similar calculation.  This is difficult to study in full generality--in your case, each group's results has four degrees of freedom, so there are eight dimensions of differences to explore--but if you could be more specific about exactly *how* the two groups might differ (which is matter for science, not statistics, to decide), you would have a better basis for selecting a test." CreationDate="2010-09-01T15:01:46.097" UserId="919" />
  <row Id="2560" PostId="2260" Text="Good point I hadn't thought of it that way.  Thank you.  FYI I'm not sure how common it is to use this term, but Rosenthal (of Rosenthal and Rosnow, 2007) refers to the effect opposite of restriction of range as hypertrophy of range." CreationDate="2010-09-01T15:08:25.563" UserId="196" />
  <row Id="2562" PostId="2271" Text="You mean in the X or Y axis ?" CreationDate="2010-09-01T15:18:41.810" UserId="253" />
  <row Id="2563" PostId="2273" Text="Hi chl, thanks for giving this a shot.  The code used here:   http://www.cl.cam.ac.uk/~sjm217/projects/graphics/  Might be a good start (since it succeeds in superimposing dots, instead of just jittering them)" CreationDate="2010-09-01T15:21:30.793" UserId="253" />
  <row Id="2565" PostId="2167" Text="@Yaroslav: Thanks for the reference.  Are you aware of any implementations of that &quot;Modified Finite Newton Method&quot;?" CreationDate="2010-09-01T15:23:11.533" UserId="5" />
  <row Id="2566" PostId="2266" Text="E[X] is a property of the pop that you are drawing inferences about.  x-bar is an unbiased estimator of that value.  You don't have to use 95% -- adjust t accordingly.  But note the SE of x-bar is s/sqrt(n) not s/n. I'll see if I can edit this." CreationDate="2010-09-01T15:31:26.750" UserId="1107" />
  <row Id="2567" PostId="2266" Text="@Kingsford : sorry, my mistake. it's edited." CreationDate="2010-09-01T15:39:35.643" UserId="1124" />
  <row Id="2568" PostId="2266" Text="@Bart : an expected value is about the population, and a theoretical value. It can be seen as the limit of the sample mean when the sample size goes to infinity. You need to estimate it using the mean, but you can't calculate it unless you know the complete population. Which you don't. See : http://en.wikipedia.org/wiki/Expected_value " CreationDate="2010-09-01T15:40:19.450" UserId="1124" />
  <row Id="2569" PostId="2263" Text="@Cadu : What's the software you're using? If it's merely for calculation, you shouldn't use the theoretical functions. In R, there is a function called dnorm() that does the calculation for you. On this wiki page you get more information on the probability density function for the normal distribution : http://en.wikipedia.org/wiki/Normal_distribution" CreationDate="2010-09-01T15:46:02.810" UserId="1124" />
  <row Id="2570" PostId="2277" Text="Hmm, you got a point there, whuber.&#xA;&#xA;In the graph that i posted i got the mean, which indeed it nothing more than the position in the X axis of the peak. The SD deviation should be half of the width of the peak. And i also got the amplitude of these peaks which is the height.&#xA;&#xA;If i got it right, your suggestion is that i could calculate the area of the peaks simply by multiplying the height (amplitude) by the width (2*SD). And due to the fact that i am only interested in the ratios between the peaks, this should be enough.&#xA;&#xA;Did i get you right?" CreationDate="2010-09-01T15:51:11.850" UserId="1133" />
  <row Id="2571" PostId="2263" Text="@chi: I think you are mixing up two acronyms. &#xA;ROC curve = Receiver Operator Characteristic curve&#xA;AUC = Area Under Curve" CreationDate="2010-09-01T16:03:26.103" UserId="25" />
  <row Id="2572" PostId="2277" Text="Yes, you are correct.  You don't need to worry about the factor of 2, either--it's meaningless--because you just want ratios.  The problem with this approach is that when the peaks are close together, the tails of a peak's neighbors contribute somewhat to the peak's own height.  Thus, it's best to use a peak-estimation procedure that compensates for this.  That's what chl, for example, was referring to by a &quot;mixture of gaussians.&quot;" CreationDate="2010-09-01T16:06:28.773" UserId="919" />
  <row Id="2573" PostId="2282" Score="1" Text="@Tal: I am basically happy to allow any R questions on here, but I just don't see how this has any relevance to the subject of data analysis.  This is a meta question about R itself, not even about using R.  We'll see what others think, but this seems more appropriate for stackoverflow, r-help, twitter, or a blog post." CreationDate="2010-09-01T16:12:52.953" UserId="5" />
  <row Id="2574" PostId="2275" Text="I think it would be ideal if you could present some of the data you finally have. At least for me, that would help a lot in thinking about an answer." CreationDate="2010-09-01T16:13:12.703" UserId="442" />
  <row Id="2575" PostId="2282" Text="Regarding how to do this: I would suggest making your own version of the `library` function that would track each package when you load it." CreationDate="2010-09-01T16:13:56.367" UserId="5" />
  <row Id="2576" PostId="2282" Text="Related meta questions: http://meta.stats.stackexchange.com/questions/1/how-to-answer-r-questions and http://meta.stats.stackexchange.com/questions/169/is-it-possible-to-integrate-data-from-so-com-questions-tagged-r-here." CreationDate="2010-09-01T16:16:57.347" UserId="5" />
  <row Id="2577" PostId="2277" Text="Okay then :)&#xA;&#xA;Actually i was trying to use chl's approach but nothing so far.&#xA;&#xA;On the other hand, i used Jori's approach to calculate the area and then i calculated the ratios between them. Afterward i did the same thing with this easy procedure of yours - and i must say that although the areas were different, the ratio was pretty much similar.&#xA;&#xA;I guess this solves my issues.&#xA;&#xA;Thanks everybody for the help :)" CreationDate="2010-09-01T16:18:06.783" UserId="1133" />
  <row Id="2578" PostId="2282" Text="Lastly: regarding how to ftp; not sure how to do this from within an R function, but you can do it easily with `system(&quot;ftp ...&quot;)`." CreationDate="2010-09-01T16:20:45.077" UserId="5" />
  <row Id="2579" PostId="2266" Text="@Joris The wiki entry is much too advanced for me. Maybe I'm not asking for the right quantity. If by definition you need the entire population to calculate ev, I'd like a similar quantity that allows me to incorporate priors when my data is incomplete. So I can update the question, what would I call that?" CreationDate="2010-09-01T16:22:26.057" UserId="1134" />
  <row Id="2580" PostId="2271" Score="1" Text="It wasn't a stupid comment because the answer to how to plot that is plot(x,y).  I'm sure mbq was trying to get at the idea that what you're trying to do may be something other than a simple scatter plot." CreationDate="2010-09-01T16:41:29.060" UserId="601" />
  <row Id="2581" PostId="2264" Text="@bart your entreaties have been heard and accepted ;-)" CreationDate="2010-09-01T16:43:02.340" UserId="88" />
  <row Id="2582" PostId="2282" Text="Thanks Shane. I'll go through the threads.  You are correct, this is more of a R community topic then a statistical one - I'll vote to close the question (I don't erase it since it is an example of a &quot;bad&quot; question for this website)" CreationDate="2010-09-01T16:56:23.747" UserId="253" />
  <row Id="2583" PostId="2282" Text="Thanks @Tal.  This is similar in nature to this question: http://stats.stackexchange.com/questions/2269/access-tables-created-in-sas-enterprise-guide-client-into-sas-enterprise-miner-cl, which was also closed.  " CreationDate="2010-09-01T17:00:20.163" UserId="5" />
  <row Id="2584" PostId="2271" Score="1" Text="It's also something other than a simple violin plot since that's supposed to be symmetric around the vertical axis." CreationDate="2010-09-01T17:01:03.093" UserId="601" />
  <row Id="2585" PostId="2282" Text="You should also reach out to @johnmyleswhite: http://twitter.com/johnmyleswhite/status/22395563278" CreationDate="2010-09-01T17:03:41.573" UserId="5" />
  <row Id="2586" PostId="2282" Text="Thanks for the pointer Shane!" CreationDate="2010-09-01T17:08:56.797" UserId="253" />
  <row Id="2587" PostId="2282" Score="1" Text="If you were looking for gathering data on _installed_ packages, the crantastic package with its function crantastic.submitInstalledPackages would be worth looking at.." CreationDate="2010-09-01T17:09:46.867" UserId="573" />
  <row Id="2588" PostId="2282" Text="Thank you Karsten - this is good to know." CreationDate="2010-09-01T17:15:29.067" UserId="253" />
  <row Id="2589" PostId="2275" Text="The actual probability after the fact is either 1 or 0.  So for example For test 1: I say 75% of a 1, and ends up a 1, test 2: 35% ends up 0, etc.  How can I quantify the accuracy of my estimate?" CreationDate="2010-09-01T18:14:36.757" UserId="1137" />
  <row Id="2590" PostId="2285" Text="Yes this looks like it might be what I am looking for.&#xA;" CreationDate="2010-09-01T18:16:22.650" UserId="1137" />
  <row Id="2591" PostId="2284" Text="The Federalist paper is very interested and describes similar to above answer.  Thanks." CreationDate="2010-09-01T18:30:29.947" UserId="1137" />
  <row Id="2592" PostId="2269" Score="4" Text="I wholeheartedly disagree. We should cover statistics software here too. How many R questions are there?" CreationDate="2010-09-01T18:38:27.443" UserId="74" />
  <row Id="2593" PostId="2280" Text="@Kingsford Thanks. I've looked at some of your suggestions and they seem to be along the lines I was thinking. But can you recommend anything more specific and suitable for the lay person that I am." CreationDate="2010-09-01T18:56:59.057" UserId="1134" />
  <row Id="2594" PostId="2266" Score="2" Text="@bart, think of this question: what is the expected value of the height of an American? The only way to know this value exactly would be to ask every single person his height and take the average. That's the expected value---it's a property of the full population. That's really hard. Instead, we get a sample, a subset of our population, ask them their heights and take the average. This is the sample mean---a property of our sample and an estimate of the expected value. As our sample gets larger, we should get a sample mean that gets closer to the true expected value (the law of large numbers)." CreationDate="2010-09-01T19:03:47.900" UserId="401" />
  <row Id="2595" PostId="2263" Text="@Joris: Since dnorm(avg, avg, sd) = C * 1/sd, this formula for &quot;AUC&quot; simply proposes computing heights * sd / C (where of course C = 1/sqrt(2 pi) for Gaussians but can be some other constant for other scale-location families).  The constant factor disappears when computing ratios (which is what the question asks for), whence one might just as well compute heights * sd and be done with it." CreationDate="2010-09-01T19:32:58.400" UserId="919" />
  <row Id="2596" PostId="2277" Text="@Cadu: the ratios shouldn't be &quot;pretty much similar&quot;; they should be *identical* (to within floating point error)." CreationDate="2010-09-01T19:34:48.583" UserId="919" />
  <row Id="2597" PostId="2271" Score="2" Text="@Tal, @John -- I know how standard vioplot works, but I can't figure out how those points were obtained (and as I see not only me, while it is crucial for producing good answer) -- some kind of stem? Or maybe someone just thought that filling vioplot with distorted polka dots is a good idea?" CreationDate="2010-09-01T19:51:03.503" UserId="88" />
  <row Id="2598" PostId="2285" Text="@Simon: could you elaborate on how the bin counts would be compared to the probabilities?  I presume you have something quantitative in mind." CreationDate="2010-09-01T20:35:54.763" UserId="919" />
  <row Id="2599" PostId="2262" Text="The title should say &quot;peak height&quot; rather than &quot;peak mean&quot;. " CreationDate="2010-09-01T20:46:40.303" UserId="25" />
  <row Id="2600" PostId="2271" Text="Hello John and mbq.  John, I didn't say (nor think) that the comment was stupid.  Now regarding the plot, I came across it in a recent article.  The article stated they used SAS to create it. More then this I can not offer at this time." CreationDate="2010-09-01T20:55:50.677" UserId="253" />
  <row Id="2601" PostId="2285" Score="1" Text="@whuber: the probabilities can be plotted against the empirical frequencies to get what's called a calibration plot; the points should be roughly along the diagonal. I don't know if one could do a more quantitative analysis, this isn't an area I know a whole lot about, just a few things i've picked up here and there." CreationDate="2010-09-01T21:10:28.863" UserId="495" />
  <row Id="2602" PostId="2280" Score="1" Text="@bart -- I wish I could offer a safe automated way to do this but I don't think it exits.  See above for edits w/ a little more detail." CreationDate="2010-09-01T22:54:38.690" UserId="1107" />
  <row Id="2603" PostId="2271" Text="I'm guessing that the dots are simply jittered observations with jitter distance determined by the width of the vioplot." CreationDate="2010-09-02T00:16:43.083" UserId="159" />
  <row Id="2604" PostId="2292" Text="Point taken.&#xA;&#xA;The problem appeared because the estimates of the smoothing parameters of the best fitting model (an ETS(M,A,N) ) are 0.0001 and 0.9999 which I think are the set limits for the possible values the estimates can take. So I was trying to find out what was going on. &#xA;&#xA;I was looking for standard errors because I'm used to finding out about a model's parameters' distributions whenever I have to estimate one..:) thinking that if they vary too much they are not too reliable." CreationDate="2010-09-02T01:06:57.013" UserId="443" />
  <row Id="2605" PostId="2292" Text="I did bootstrap as advised earlier. I found that the standard deviation was not that useful because the distribution was very skewed. " CreationDate="2010-09-02T01:08:07.847" UserId="443" />
  <row Id="2606" PostId="2279" Text="Yes its trivial. I was able to come up with a simple proof using sigma algebra." CreationDate="2010-09-02T01:39:48.033" UserId="862" />
  <row Id="2607" PostId="2298" Text="p.s. maybe this should be a community wiki? I'm not sure how that works..." CreationDate="2010-09-02T03:32:49.573" UserId="795" />
  <row Id="2608" PostId="2167" Text="no, but Keerthi and Langford's pages have links to some software that may be related, since they both worked at Yahoo Research" CreationDate="2010-09-02T03:53:17.880" UserId="511" />
  <row Id="2609" PostId="2292" Text="@noworries. Yes, they are the boundaries of the parameter region. A value close to zero is not a problem -- it indicates the associated state variable is extremely stable and essentially does not change over time. A value close to one usually indicates the model is trying to adapt quickly to each new observation which probably means the model is mis-specified." CreationDate="2010-09-02T05:24:17.683" UserId="159" />
  <row Id="2610" PostId="2298" Text="if you don't give us more info on $f$ and on what type of characteristic then it seems to be community wiki but I am not sure neither :) " CreationDate="2010-09-02T05:47:49.420" UserId="223" />
  <row Id="2611" PostId="2298" Text="The fact that $f$ in undefined makes it a feed of a different multivariate visualization methods, so it should be CW. And now it is." CreationDate="2010-09-02T07:20:09.280" UserId="88" />
  <row Id="2612" PostId="2271" Text="Hello Rob - I suspect that the dots are attempting to simply get distributed evenly by the width of the violin. Otherwise, I wouldn't understand how no dots in the image overlap each other (except for the ones at the bottom, where there is no room for all the dots)" CreationDate="2010-09-02T08:26:27.007" UserId="253" />
  <row Id="2613" PostId="2261" Text="Regarding your last point about GLM, this is basically how IRT models work: they can be viewed as generalized mixed-effects model with a logit function link and individuals treated as random effets (not items!). This yields the so-called marginal likelihood approach, where subjects' parameters are assumed to follow a standard normal distribution, for model identification purpose. The problem is that it has to be done on each subscale which are hypothesized to measure a single construct." CreationDate="2010-09-02T08:32:10.840" UserId="930" />
  <row Id="2614" PostId="2268" Text="Good point. I just wanted to flag that the Bayesian option seems like the way to go. Of course @Kingsford has now provided a more rigorous explanation." CreationDate="2010-09-02T08:49:29.270" UserId="183" />
  <row Id="2615" PostId="338" Text="Still, none of those books claim that there is more than one entropy." CreationDate="2010-09-02T10:25:52.500" UserId="88" />
  <row Id="2616" PostId="2306" Text="I should say for full disclosure that I have already posted this to the bioconductor list" CreationDate="2010-09-02T10:26:34.950" UserId="1150" />
  <row Id="2617" PostId="2280" Text="@Kingsford. You forget that some products don't have a score, or only very few. Lme is very likely to get convergence problems, and the estimates of the standard errors cannot be trusted. " CreationDate="2010-09-02T11:00:59.217" UserId="1124" />
  <row Id="2618" PostId="2261" Score="1" Text="Good point. And in fact, It would be better to use GEE models or mixed effect models instead of GLM, a standard GLM doesn't allow for random effects." CreationDate="2010-09-02T11:04:04.780" UserId="1124" />
  <row Id="2619" PostId="2266" Text="@Bart : You want to estimate the expected value, and you do that by modelling the sample means in any way. It sounds trivial, but it is crucial to understand what exactly you're doing. I see far too many students struggle with statistics because they don't understand the underlying concepts. I don't want to be rude, but it might be a good idea to look for a thorough introduction into statistics. It will help you tremendously in understanding what is going on." CreationDate="2010-09-02T11:09:12.697" UserId="1124" />
  <row Id="2620" PostId="2269" Text="There is an ongoing discussion about this question on meta: http://meta.stats.stackexchange.com/questions/400/is-this-question-really-not-related-to-statistical-analysis" CreationDate="2010-09-02T11:11:45.983" />
  <row Id="2621" PostId="2307" Text="Thanks for that.  I'm not particular sold on SVM, just using that as an example.  So if you used random trees, you don't have to do cross-validation? Is that right. " CreationDate="2010-09-02T11:13:59.020" UserId="1150" />
  <row Id="2622" PostId="2269" Text="of course you need some software/tool and can't do all statistics by just using calculator. If this is not a valid question then why do tags like &quot;R&quot;, &quot;SAS&quot; exists here?&#xA;&#xA;Are you guys planning a new site for &quot;Statistical Analysis Softwares&quot;?" CreationDate="2010-09-02T11:26:25.350" UserId="263" />
  <row Id="2623" PostId="2307" Score="4" Text="yes, RFs include a random sampling of variables (typically $\sqrt{p}$) when growing a tree and each tree is based on a boostraped sample of the individuals; variable importance is computed on so-called out-of-bag samples (those not used for building the decision tree) using a permutation technique. The algorithm is repeated for m trees (default m=500) and results are averaged to compensate uncertainty at the tree level (boosting)." CreationDate="2010-09-02T11:34:20.303" UserId="930" />
  <row Id="2624" PostId="2281" Text="A 95% confidence interval by definition covers the true parameter value in 95% of the cases, as you indicated correctly. Thus, the chance that your interval covers the true parameter value is 95%. You can sometimes say something about the chance that the parameter is larger or smaller than any of the boundaries, based on the assumptions you make when constructing the interval (pretty often the normal distribution of your estimate). You can calculate P(theta&gt;ub), or  P(ub&lt;theta). The statement is about the boundary, indeed, but you can make it." CreationDate="2010-09-02T11:44:24.477" UserId="1124" />
  <row Id="2625" PostId="2307" Score="2" Text="It is important that it is called Random Forest not Random Trees; you may have problems with Google." CreationDate="2010-09-02T11:52:11.267" UserId="88" />
  <row Id="2626" PostId="2263" Text="@Harvey Yes, you're right: these are ROC that are used to show classification accuracy, but we use their corresponding AUC (or, AUC wrt. first bisector) to compare one classifier to another (or one predictive instrument to another). Now let's try &quot;AUC&quot; as a keyword in Pubmed and look at the results: How much deal with clinical evidence-based medicine or biomedical studies?" CreationDate="2010-09-02T12:12:39.073" UserId="930" />
  <row Id="2627" PostId="2261" Text="FYI, this article from Tuerlinckx et al. 2006 provides a nice review of GLMMs: http://bit.ly/9k8vJs. Also, about the use of GEE as an alternative marginal model, it was proposed by Feddag, Grama, and Mesbah (2003) to estimate parameters of the Rasch Model (one-parameter IRT model), or loglinear multidimensional IRT models as defined by Kelderman and Rijkes (1994). It is currently available in the Stata `raschtest` package." CreationDate="2010-09-02T12:33:14.857" UserId="930" />
  <row Id="2628" PostId="2261" Text="@chl : thx for the reference, very nice reading material." CreationDate="2010-09-02T12:48:52.963" UserId="1124" />
  <row Id="2629" PostId="2311" Text="Found one problem with it: If cut returns an empty level, you get an error. " CreationDate="2010-09-02T13:03:27.577" UserId="1124" />
  <row Id="2630" PostId="2271" Score="1" Text="I've seen that type of graph before, can't remember which software was used, but the x position was just &quot;random&quot; jitter (proportional to the density at that point anyways). Personally, I find those a bit though to read, I'm more of a box-and-whiskers plot I guess :)" CreationDate="2010-09-02T13:06:49.673" UserId="582" />
  <row Id="2631" PostId="2311" Text="Nice plot anyway!" CreationDate="2010-09-02T13:10:28.527" UserId="930" />
  <row Id="2632" PostId="2271" Text="@Tal: Please go find the original reference to this chart.  Where did you get the picture from?  At this point, we have no idea what it's supposed to be or mean.  It isn't a violin or box chart (at least not in any traditional sense)." CreationDate="2010-09-02T13:12:05.683" UserId="5" />
  <row Id="2633" PostId="2311" Text="+1 Good job! Still I think something is missing -- that original plot is asymmetric. " CreationDate="2010-09-02T13:17:48.643" UserId="88" />
  <row Id="2634" PostId="2311" Text="@mbq? Something missing? I just optimized that original plot. It's not a bug, it's a feature! ;-)" CreationDate="2010-09-02T13:20:59.673" UserId="1124" />
  <row Id="2635" PostId="2242" Score="1" Text="Let me correct myself: It would be correct to say that the chance of your lower limit to be bigger than the true value is only 2.5%, assuming normality of your sample mean. Which is a different phrasing (the chance is about the limit, not about A), but boils down to the same." CreationDate="2010-09-02T13:27:35.740" UserId="1124" />
  <row Id="2636" PostId="2311" Score="1" Text="@Joris Maybe try using `cut2` from Hmisc instead of `cut`?" CreationDate="2010-09-02T13:35:45.003" UserId="930" />
  <row Id="2637" PostId="2311" Text="@chl If I don't have to load other libraries, I rather avoid it. I just used the wrong number in the for-loop, that has been corrected now." CreationDate="2010-09-02T13:54:01.817" UserId="1124" />
  <row Id="2638" PostId="2146" Text="Do I understand it right that you have two sets of time series, and you want to check whether or not they differ significantly?" CreationDate="2010-09-02T13:59:52.023" UserId="1124" />
  <row Id="2639" PostId="2281" Score="2" Text="Joris, I can't agree. Yes, for any value of the parameter, there will be &gt;95% probability that the resulting interval will cover the true value. That doesn't mean that after taking a particular observation and calculating the interval, there still is 95% conditional probability given the data that THAT interval covers the true value.&#xA;&#xA;As I said below, formally it would be perfectly acceptable for a confidence interval to spit out [0, 1] 95% of the time and the empty set the other 5%. The occasions you got the empty set as the interval, there ain't 95% probability the true value is within!" CreationDate="2010-09-02T14:21:34.080" UserId="1122" />
  <row Id="2640" PostId="2297" Text="Perfect! Thanks!" CreationDate="2010-09-02T14:33:45.217" UserId="1146" />
  <row Id="2641" PostId="2269" Text="Reopened the question per this discussion: http://meta.stats.stackexchange.com/questions/400/is-this-question-really-not-related-to-statistical-analysis" CreationDate="2010-09-02T14:36:22.687" UserId="5" />
  <row Id="2643" PostId="2281" Text="@ Keith : I see your point, although an empty set is not an interval by definition. The probability of a confidence interval is also not conditional on the data, in contrary. Every confidence interval comes from a different random sample, so the chance that your sample is drawn such that the 95%CI on which it is based does not cover the true parameter value, is only 5%, regardless of the data. " CreationDate="2010-09-02T15:02:43.540" UserId="1124" />
  <row Id="2644" PostId="2311" Text="@Joris so you know the original plot's definition?" CreationDate="2010-09-02T15:19:35.963" UserId="88" />
  <row Id="2645" PostId="2280" Text="@Joris-Without a prior you can't include products with 0 information, but because there are thousands of products, having few observations per product is not a problem (even many with 1 obs will be OK). But still, without being familiar with the data, the context of the problem, and desired output/inferences we are only speculating as to what is appropriate." CreationDate="2010-09-02T15:44:48.880" UserId="1107" />
  <row Id="2646" PostId="2306" Text="Please summarize any bioconductor results back here?  " CreationDate="2010-09-02T16:01:50.647" UserId="5" />
  <row Id="2648" PostId="2271" Score="2" Text="OK, I found what the software was. It's a &quot;column scatter plot&quot; made in GraphPad Prism. See for instance http://graphpad.com/help/prism5/prism5help.html?what_is_a_frequency_distribution.htm . I found some reference to those also here: http://www.originlab.com/www/products/GraphGallery.aspx?GID=104&amp;s=8&amp;lm=215 " CreationDate="2010-09-02T16:38:29.387" UserId="582" />
  <row Id="2649" PostId="2266" Text="@Joris You are not being rude. I've tried studying maths and stats in a conventional way and I have to say it hasn't been very successful. What seems to work for me is to reason the problems out in my own special way and then see how that matches up with what others do. " CreationDate="2010-09-02T17:07:36.433" UserId="1134" />
  <row Id="2650" PostId="2314" Text="I think there's still a generalization issue when using the same sample (1) to assess the classifier classification/prediction performance while tuning its parameters (eventually, with feature selection) and (2) use in turn its predictions on the whole data set. In fact, you are breaking the control exerted on overfitting that was elaborated using cross-validation. Hastie et al. provide a nice illustration of CV pitfalls, esp. wrt. feature selection, in their ESL book, § 7.10.2 in the 2nd edition." CreationDate="2010-09-02T17:16:12.673" UserId="930" />
  <row Id="2651" PostId="2298" Text="it's fairly easy for me to fit $f$ to linear model, so we should assume that has already been done, and we are looking at the _residual_ from that fit. As I mentioned in the question, I am looking for interesting facts about the function, so I don't yet know what I might find. Because of this open-ended nature, I thought maybe it should be a CW. I am still not sure on how one decides whether a Q should be CW." CreationDate="2010-09-02T17:25:09.440" UserId="795" />
  <row Id="2652" PostId="2311" Text="This will add a little bit of randomness:&#xA;`x.plot &lt;- sample(seq(-outline[i],outline[i],outline[i]/25), n, replace=T)`" CreationDate="2010-09-02T17:44:14.367" UserId="582" />
  <row Id="2653" PostId="2281" Text="Joris, I was using &quot;data&quot; as a synonym for &quot;sample,&quot; so I think we agree. My point is that it's possible to be in situations, after you take the sample, where you can prove with absolute certainty that your interval is wrong -- that it does not cover the true value. This does not mean that it is not a valid 95% confidence interval.&#xA;&#xA;So you can't say that the confidence parameter (the 95%) tells you anything about the probability of coverage of a particular interval after you've done the experiment and got the interval. Only an a posteriori probability, informed by a prior, can speak to that." CreationDate="2010-09-02T17:46:47.777" UserId="1122" />
  <row Id="2654" PostId="2316" Text="You should delete this 'answer' and add it to your question as you have not really found the answer as evident by the last line." CreationDate="2010-09-02T17:48:29.817" />
  <row Id="2655" PostId="2316" Text="Nice find, but when Simon writes&#xA;&#xA;&quot;view that single observation as a draw from a true probability distribution who's average you want...and you can view that true average itself as having been drawn from a probability distribution of averages&quot;&#xA;&#xA;he is describing a mixed or multilevel model.  I don't think there's a need to guess that &quot;K=25 seems to work well&quot; because the best linear unbiased predictor equations were worked out more than 60 years ago (BLUP). Excellent Bayesian estimators exist as well.  As Brad Efron said, &quot;Those who ignore Statistics are condemned to reinvent it&quot;" CreationDate="2010-09-02T18:00:39.740" UserId="1107" />
  <row Id="2656" PostId="2318" Text="I would even recommend the entire book from I. Guyon and coworkers, http://j.mp/anblwx. The ESL book from Hastie et al., http://j.mp/bW3Hr4, provides also interesting discussions around this 'hot' topic." CreationDate="2010-09-02T18:24:46.450" UserId="930" />
  <row Id="2657" PostId="2318" Text="I disagree with your claim; FS is interesting on its own for some explanatory information that it delivers (marker/SNPs selection is an example when it is a main aim of analysis). The feature selection overfit is of course a problem, but there are methods to omit it." CreationDate="2010-09-02T18:25:58.000" UserId="88" />
  <row Id="2658" PostId="2316" Text="@Kingsford What's worong with K = Vb/Va?" CreationDate="2010-09-02T18:57:20.117" UserId="1134" />
  <row Id="2659" PostId="2266" Text="@Joris How do I calculate the ev from the CI? How does this approach make use of the prior information provided by the other products?" CreationDate="2010-09-02T19:06:21.267" UserId="1134" />
  <row Id="2660" PostId="2316" Text="@Srikant It's the best &quot;answer&quot; I've got so far. I think I'll be trying it out because:&#xA;1. As currently stated Jorly's doesn't actually provide the ev, and there is no indication of how prior information is utilised.&#xA;2. Kingsford's answer could suffer convergence problems apparently, and there is a lot of work for me to figure out what it all means.&#xA;3. Simon's solution worked well for him, and I trust Simon.&#xA;4. I think in time I'll get a proof of his method&#xA;5. The method uses all the available data and is correct at the extrema when individual variance is small and large" CreationDate="2010-09-02T19:28:35.520" UserId="1134" />
  <row Id="2661" PostId="2298" Text="At first blush this question seems purely mathematical: visualizing the function has nothing to do with the distribution of its arguments.  However, the reference to &quot;heteroskedasticity&quot; suggests you're really trying to visualize an object that is more explicitly represented as $f(x_1, ..., x_k) + \epsilon$ or more generally as  $g(f(x_1, ..., x_k), \epsilon)$ where $epsilon$ is a zero-mean random variable and $g(z,0) = z$ for all z.  Isn't this just a response-surface analysis?" CreationDate="2010-09-02T19:42:16.897" UserId="919" />
  <row Id="2662" PostId="2314" Text="@chl: who said anything about tuning parameters? If additional things are performed, they should be repeated during cross-validation as well. Clearly modifying your algorithm until you get good cross-validated error rates is &quot;cheating&quot;. BTW, I agree that cross-validation, especially leave-one-out, is not all that great." CreationDate="2010-09-02T20:39:28.647" UserId="279" />
  <row Id="2663" PostId="2316" Text="@bart - I don't think you'll run into convergence problems, but you're right that it takes awhile to learn the methods.  As for what's wrong with Vb/Va, I'm not sure how to answer because it's not clear to me how those values are being calculated.  But under the nested Gaussian assumption described, one good way to define 'best' is in terms of minimizing mean squared error (MSE).  This is what the BLUP equations do.  See, for example, [here](http://tiny.cc/4zfl1)." CreationDate="2010-09-02T21:00:07.593" UserId="1107" />
  <row Id="2664" PostId="2321" Score="1" Text="The inputs are assumed to be iid Gaussian, implying there is no direction of maximal variance.  But perhaps I have misinterpreted this assumption?" CreationDate="2010-09-02T22:07:06.990" UserId="919" />
  <row Id="2665" PostId="2298" Text="@whuber I don't know enough RSM to say for sure (and the wikipedia page is not helping me). I am not necessarily trying to optimize the function (gasp), but am trying to find interesting facts about it. What makes this a statistical question is that I'm not terribly interested in the case where any of the $x_i$ are far from zero (assumed standard normal), and so what constitutes 'interesting' should somehow be weighted by relevance." CreationDate="2010-09-02T22:54:51.437" UserId="795" />
  <row Id="2666" PostId="2304" Text="ggobi looks interesting, but I am repelled by the `xml` data format. maybe I'll get over that, but it is not high priority." CreationDate="2010-09-02T22:55:42.977" UserId="795" />
  <row Id="2667" PostId="707" Text="@mbq: I have read the paper; Shao reports on a simulation study with only 40 observations, and shows that LOOCV underperforms the Monte-Carlo CV except in the case where no subselection is appropriate (the full feature set is optimal). 100 is way more than enough, at least for subset selection in linear models." CreationDate="2010-09-02T23:14:06.780" UserId="795" />
  <row Id="2668" PostId="2322" Text="Oh my, yet again; have you read the title of this article? " CreationDate="2010-09-02T23:49:54.067" UserId="88" />
  <row Id="2669" PostId="2322" Text="Ok, to be clear -- I'm not saying LOOCV is a good idea for a big number of objects; obviously it is not, but Shao is not applicable here. Indeed in most of cases rules for LMs does not hold for ML." CreationDate="2010-09-03T00:17:35.580" UserId="88" />
  <row Id="2670" PostId="2316" Text="@bart It is not driven by sound statistical principles as is evident by the hand waving he does. If you want a statistically sound solution you should really explore multi-level models that Kingsford refers to in his comments. See my [answer](http://stats.stackexchange.com/questions/1822/test-for-poolability-of-individual-data-series/1827#1827) to another question which illustrates the basic idea. " CreationDate="2010-09-03T00:22:41.180" />
  <row Id="2671" PostId="538" Text="Er... I'm not familiar with real stats by a long shot... but isn't &quot;exposing all hidden variables&quot; by definition impossible? How do you know when there's no more &quot;hidden&quot; variables?" CreationDate="2010-09-03T01:54:54.713" UserId="1170" />
  <row Id="2672" PostId="1287" Text="Chapter 17 covers:&#xA;17 An overview of the GLM&#xA;17.1 linearity and the GLM&#xA;17.2 bivariate to multivariate statistics and overview&#xA;17.2.1 bivariate form&#xA;17.2.2 simple multivariate form&#xA;17.2.3 full multivariate form&#xA;17.3 Alternative research strategies&#xA;&#xA;I hope this can help&#xA;Regards&#xA;" CreationDate="2010-09-03T02:01:23.580" UserId="694" />
  <row Id="2673" PostId="2304" Score="1" Text="@shabbychef Then check out the interface from R called rggobi http://www.ggobi.org/rggobi/" CreationDate="2010-09-03T02:30:47.927" UserId="183" />
  <row Id="2674" PostId="2326" Score="1" Text="Many questions: 1. Is it an experiment or an observational study? 2. State clearly what are your predictors and what is your outcome variable. 3. Try to be more specific. 4. What do you mean by sorting? 5. What is your sample? Is it n=100 pieces?" CreationDate="2010-09-03T03:44:30.893" UserId="183" />
  <row Id="2675" PostId="1136" Text="A good model is never solely based on statistical tests and criteria. It should be a combination of literature review, experience, statistics and common sense." CreationDate="2010-09-03T03:46:11.643" UserId="694" />
  <row Id="2676" PostId="2316" Text="@Kingsford Thanks for your help and time. " CreationDate="2010-09-03T06:07:07.153" UserId="1134" />
  <row Id="2677" PostId="2328" Text="Do you really need NN? This method is rather considered obsolete (partially because it is very hard to generally answer your questions)." CreationDate="2010-09-03T06:36:53.673" UserId="88" />
  <row Id="2678" PostId="707" Text="@shabbychef You've got me here; the second argument in my first comment is of course a junk, I had some other works in mind and overgeneralized. Nevertheless, I will still argue that Shao's paper is not a good reference for general &quot;LOO fails for large N&quot; since its scope is reduced to linear models." CreationDate="2010-09-03T06:43:08.900" UserId="88" />
  <row Id="2679" PostId="2314" Text="@Aniko -- not it is not cheating, since CV shows you the approximation how algorithm will perform on new data. You only need to be sure that you haven't settled on something based on the whole set (this is a leak of information about the structure of the full set, so it can immediately bias all train parts)." CreationDate="2010-09-03T07:15:11.827" UserId="88" />
  <row Id="2680" PostId="2318" Text="I was making the point that FS doesn't necessarily improve predictive importance, and can make it worse.  If finding the informative features is of intrinsic importance, then of course FS should be used, but it may well be that predictive performance is compromised if over-fitting the feature selection criterion occurrs (which happens rather easily).  For tasks like micro-array analysis, I would use (bagged) ridge regression for predictions and something like the LASSO for determing the key features (for gaining understanding of the biology).  There is no need to do both in the same model." CreationDate="2010-09-03T07:28:20.430" UserId="887" />
  <row Id="2681" PostId="2314" Text="@mbq - Ankino is correct, tuning your model to minimise a CV statistic is &quot;cheating&quot; and the CV statistic of the final model will have a substantial optimistic bias. The reason for this is that the CV statistic has a non-negligible variance (as it is evaluated on a finite set of data) and thus if you directly optimise the CV statistic you can over-fit it and you can end up with a model that generalises less well than the one you started with.  For a demonstration of this, in a machine learning context, see&#xA;&#xA;http://jmlr.csail.mit.edu/papers/v11/cawley10a.html&#xA;&#xA;Solution: Use nested XVAL" CreationDate="2010-09-03T07:40:01.317" UserId="887" />
  <row Id="2683" PostId="2337" Text="Ideally, the question should be self-contained. Could you please give the relevant formulas as well? By the way, I may not have understood your question properly but won't euclidean distance give the same weight to all the variables? That metric seems to be one of the distance measures offered by the library you are using." CreationDate="2010-09-03T10:42:31.263" />
  <row Id="2684" PostId="2326" Score="1" Text="To add to Jeromy's comment. It would be nice if you could state your research question. So far I am pretty confused." CreationDate="2010-09-03T10:50:09.453" UserId="442" />
  <row Id="2685" PostId="2311" Text="@mbq. No, I just have too much time on my hands, the academic year has yet to start." CreationDate="2010-09-03T10:52:10.773" UserId="1124" />
  <row Id="2686" PostId="2338" Text="Hello Mike. Thank you for the answer, and for your package - it is really great! " CreationDate="2010-09-03T11:05:41.337" UserId="253" />
  <row Id="2687" PostId="2271" Text="Thank you Nico - great job." CreationDate="2010-09-03T11:07:36.520" UserId="253" />
  <row Id="2688" PostId="2311" Text="Joris, as always, you rock." CreationDate="2010-09-03T11:23:17.413" UserId="253" />
  <row Id="2689" PostId="2281" Text="@ Keith : I see your point. So in the Bayesian approach, I take a diffuse prior to construct the same interval and call it a credible interval. In a Frequentist approach, if I can prove with absolute certainty that the interval is wrong, I have either violated assumptions, or I know the true value. In either case, the 95% confidence interval is not valid any more. The assumptions involved imply a diffuse prior, i.e. a complete lack of knowledge about the true parameter. If I have prior knowledge I shouldn't calculate a confidence interval in the first place. " CreationDate="2010-09-03T11:23:47.263" UserId="1124" />
  <row Id="2690" PostId="2313" Text="Thank you mbq, I was wondering who's answer to pick.  I choose Joris, simply since he wrapped it up.  Either way - both answers are great and won my +1 vote.  Cheers - Tal" CreationDate="2010-09-03T11:24:11.053" UserId="253" />
  <row Id="2691" PostId="2266" Text="@Bart. The expected value is a theoretical concept. Statistics is used to approximate it. If you can calculate the true expected value, there is no need for a confidence/credibility interval any more, because (Frequentist) the chance the true expected value is the true expected value, is 1, or (Bayesian) your true expected value is not a random variable. I gave you the frequentist approach, no strict priors involved. You use the information of the other products by calculating a common standard deviation. So if you want a Bayesian approach, my answer is indeed not what you're looking for." CreationDate="2010-09-03T11:50:30.047" UserId="1124" />
  <row Id="2692" PostId="2335" Text="Could you be more precise about what you mean by cross-tabulating the ORs? Your factors have more than two levels?" CreationDate="2010-09-03T12:50:06.580" UserId="930" />
  <row Id="2693" PostId="2335" Text="Yes, the factors have 3 and 6 levels respectively. I'm wanting a table of what the predicted odds are for each possible combination of `fac1` and `fac2`." CreationDate="2010-09-03T13:22:42.680" UserId="229" />
  <row Id="2694" PostId="2313" Text="@Tal I agree...." CreationDate="2010-09-03T13:35:50.600" UserId="88" />
  <row Id="2695" PostId="2342" Text="Excellent! This is exactly what I was looking for, thanks!" CreationDate="2010-09-03T14:01:30.963" UserId="229" />
  <row Id="2696" PostId="2298" Text="@Shabbychef: Right, your stipulation of standard normals establishes you want to know $f$ in a neighborhood of the origin that doesn't extend more than a few units away. RSM occurs to me because it includes methods that *efficiently* explore the behavior of a function, so if evaluating $f$ is expensive, RSM methods may be helpful. If evaluating it is not expensive, you have a purely mathematical problem. If each evaluation for a given set of arguments can produce a variety of results, you need the $\epsilon$ term--but maybe you can eliminate it by averaging multiple evaluations at each point?" CreationDate="2010-09-03T14:15:19.337" UserId="919" />
  <row Id="2697" PostId="2337" Text="@Space_C0wb0y: instead of &quot;(multiply each dimension with 1/average of that dimension)&quot; don't you mean to multiply by the reciprocal of the *standard deviation* of the corresponding coordinates?  This is not a bad choice but it's sensitive to outliers.  Rescaling by a resistant measure of spread, such as the IRQ, would protect you from outliers and still accomplish what you intend." CreationDate="2010-09-03T14:19:10.310" UserId="919" />
  <row Id="2698" PostId="2345" Score="2" Text="Maybe it should be a comment? (even if it is a good comment :) ) " CreationDate="2010-09-03T14:24:08.980" UserId="223" />
  <row Id="2699" PostId="2344" Score="2" Text="There is not a single tree... But see @Shane's response for plotting one of them for illustrative purpose. " CreationDate="2010-09-03T14:48:41.973" UserId="930" />
  <row Id="2700" PostId="2335" Text="Ok, @Bernd's answer is fine with me. Maybe have a look at the `Design` package from Franck Harrell; it has very nice functions along `lrm()` for GLMs and related stuff." CreationDate="2010-09-03T14:52:41.900" UserId="930" />
  <row Id="2701" PostId="2340" Text="Thanks for your answer. I just found a useful paper on that issue: http://dx.doi.org/10.1007/BF01897163. It compares the usefulness of different normalization approaches with each other and with no normalization." CreationDate="2010-09-03T15:05:22.917" UserId="977" />
  <row Id="2702" PostId="2337" Text="@Srikant Vadali: I wanted to post the formulas, but I couldn't find a way to embed them here, and ASCII-formulas would have made things worse instead of better. Also, the book I referenced is freely available as an ebook, so everyone can look it up." CreationDate="2010-09-03T15:07:17.560" UserId="977" />
  <row Id="2703" PostId="2337" Text="@Space_C0wb0y Check out http://meta.stats.stackexchange.com/q/386/930 for LaTeX typesetting" CreationDate="2010-09-03T15:09:58.353" UserId="930" />
  <row Id="2704" PostId="2348" Text="Perhaps, CW? The question seems like a poll of what people use. How would you define the 'best' answer? I admit that I am a bit fuzzy when to apply CW. So, feel free to ignore this comment if you feel otherwise." CreationDate="2010-09-03T15:22:57.730" />
  <row Id="2705" PostId="2281" Text="No, I'm afraid you still haven't quite got it. There is no requirement for a &quot;diffuse prior&quot; in either case. It is fine to calculate a confidence interval whether you have prior knowledge or not -- the point is that the confidence interval just doesn't care. A confidence interval guarantees its coverage probability absolutely, even in the worst case. It will not be &quot;the same interval&quot; as a credibility interval informed by a prior, at least not in general." CreationDate="2010-09-03T15:47:27.160" UserId="1122" />
  <row Id="2706" PostId="2281" Text="And as I said, it is perfectly acceptable, formally speaking, that at the end of your experiment you arrive at a particular confidence interval that you can prove does not cover the true value. This does NOT mean the interval was invalid or that it's not a 95% confidence interval. Of course if you reran the same experiment 100 times you must expect to get such a nonsense result less than 5 of those times, but the fact that you get provable nonsense 5% of the runs is formally okay as long as the confidence interval covers the value the other 95% of the outcomes." CreationDate="2010-09-03T15:50:52.437" UserId="1122" />
  <row Id="2707" PostId="2281" Text="And the transpose is true for a credibility interval -- it is perfectly acceptable to have values of the parameter that produce a credibility interval that's always wrong! As long as your prior says those values are rare.&#xA;&#xA;Imagine a bag containing a trillion weighted coins -- one of which has heads probability 10%, and the rest are fair coins. Your experiment is: draw a coin from this distribution, flip it ten times, count the discrete # of heads, then state 95% credible interval on heads prob. If you get the &quot;10%&quot; coin the interval will ALWAYS FAIL TO COVER. Again, doesn't make it invalid." CreationDate="2010-09-03T15:57:56.950" UserId="1122" />
  <row Id="2708" PostId="2348" Score="1" Text="I wouldn't mind leaving this as non-CW, especially if Colin can rephrase it slightly to allow for the possibility of one best answer.  That said, I can't envision how to do that..." CreationDate="2010-09-03T16:05:09.193" UserId="5" />
  <row Id="2709" PostId="707" Text="@mbq I'm curious enough about this that I think it should be a question in its own right. I would guess that Shao's work extends to variable selection under logistic regression or Fisher LDA. possibly to classification with kernel methods for finite kernel space (e.g. polynomial kernels), b/c the model is linear in the kernel space. but I have no good intuition for whether it is appropriate for comparing models, using different kernels, say. I hope somebody has tested the results in these arenas. The fact that LOO is _impractical_ for large $n$ is another issue..." CreationDate="2010-09-03T16:07:06.473" UserId="795" />
  <row Id="2710" PostId="2281" Text="In one of Jaynes papers&#xA;&#xA;http://bayes.wustl.edu/etj/articles/confidence.pdf&#xA;&#xA;He constructs a confidence interval and then shows that for the particular sample you can be 100% sure that the true value does not lie in the &quot;confidence interval&quot;.  That doesn't mean that the CI is &quot;wrong&quot;, it is just that a frequentist confidence interval is not an answer to the question &quot;what is the interval that contains the true value of the statistic with probability 95%&quot;.  Sadly that is the question we would like to ask, which is why the CI is often interpreted as if it was an answer to that question. :-(" CreationDate="2010-09-03T16:24:03.463" UserId="887" />
  <row Id="2712" PostId="2200" Text="before the question was edited to have proper LaTeX, when I answered it, it was very hard to parse. Look at this in plaintext: 'Max value of intersection would be min(a,b). how do I find the min?' &#xA;With the proper markup, it is much easier to parse. (Though it looks like a homework question to me...)" CreationDate="2010-09-03T16:30:58.677" UserId="795" />
  <row Id="2713" PostId="2228" Text="Many many thanks! However, installation fails in my system. Who should I report this to?" CreationDate="2010-09-03T16:33:42.900" UserId="1084" />
  <row Id="2714" PostId="2298" Text="@whuber: as these things go, eventually $k$ will be sufficiently large, and the code will be sufficiently encumbered with bells and whistles that some sampling technique will have to be employed. I am not there yet, though." CreationDate="2010-09-03T17:25:45.633" UserId="795" />
  <row Id="2715" PostId="2298" Text="@Shabbychef: That points away from answers proffered by eng-carlin (because dimensionality reduction won't accomplish anything) and Josh Hermann (because a tornado chart looks purely at one coordinate at a time and further reduces the graph of $f$ from one dimension to zero) and favors answers akin to Jeromy Anglim's.  Along those lines, Mathematica offers great utilities to interpolate and visualize functions, given either as formulas or as sampled data." CreationDate="2010-09-03T17:37:41.017" UserId="919" />
  <row Id="2716" PostId="2324" Text="nice work on the free link!" CreationDate="2010-09-03T17:44:58.777" UserId="74" />
  <row Id="2717" PostId="2352" Text="There must be something in chemometrics books; the only man I know that uses LOO is also doing it." CreationDate="2010-09-03T17:51:55.360" UserId="88" />
  <row Id="2718" PostId="42" Text="how's the performance of this? (I run screaming whenever I see the word 'Java'...)" CreationDate="2010-09-03T18:01:28.507" UserId="795" />
  <row Id="2719" PostId="2350" Score="1" Text="Just to be clear: Are you applying RFs in an iterative manner, that is by selecting the top-ranked features (according to Gini index or decrease in MSE) from the whole input space? I know that RFs may not necessarily improve with increasing trees number, but what you are depicting seems to make sense (the best features allow to predict with 100% accuracy the OOB samples), although there is clearly a risk of overfitting when proceeding this way." CreationDate="2010-09-03T18:08:07.913" UserId="930" />
  <row Id="2720" PostId="2326" Text="Okay, more detail - you got it ;)" CreationDate="2010-09-03T18:10:36.163" UserId="1114" />
  <row Id="2721" PostId="2345" Text="@robin-girard Perhaps it should. Actually I started it as a comment, but then decided that it does answer one of the questions: is the probability of the tree 0.25 or 0.5." CreationDate="2010-09-03T18:12:36.690" UserId="279" />
  <row Id="2722" PostId="42" Text="@shabbychef Seem quite good from what I've heard, but generally Weka is used as a first step to test several algorithms and look at their performance on given data sets (or a subset thereof), then one recode part of the core program to optimize its efficiency (e.g. with high-dimensional data calling for cross-validation or bootstraping), sometimes in C or Python." CreationDate="2010-09-03T18:14:12.297" UserId="930" />
  <row Id="2723" PostId="9" Text="@James + http://j.mp/9fVmtV, http://j.mp/aNDyf2, http://j.mp/9Gzzri :-)" CreationDate="2010-09-03T18:19:39.867" UserId="930" />
  <row Id="2724" PostId="2304" Text="Bump for rggobi. I just downloaded it after reading @shabbychef's comment and it changed/rocked my world." CreationDate="2010-09-03T18:39:20.043" UserId="1076" />
  <row Id="2725" PostId="2356" Score="10" Text="Under rather mild assumptions, (a) Bayesian estimation procedures are admissible and (b) all, or almost all, admissible estimators are Bayesian with respect to some prior.  Thus it's no surprise that a Bayesian confidence interval &quot;yields the same or better results.&quot;  Note that my statements (a) and (b) are part of the *frequentist* analysis of rational decision theory.  Where frequentists part company with Bayesians is not over the mathematics or even the statistical procedures, but concerns the meaning, justification, and correct use of a prior for any particular problem." CreationDate="2010-09-03T18:48:20.173" UserId="919" />
  <row Id="2726" PostId="2272" Text="I think the answers we have so far are very good, but I'd like to get some more votes and maybe a few more answers, so I'm placing a bounty.  Please vote!  And if you have another way of explaining it, post it - more perspectives are always welcome. (for some reason, this comment didn't go through when I placed it originally...)" CreationDate="2010-09-03T18:48:42.387" UserId="71" />
  <row Id="2727" PostId="2251" Score="1" Text="A huge plus one for point 2.  Other than going through human subjects protection training, I've never received the tiniest bit of training on data collection and storage.  Getting the data collection right is vastly more important than the analysis." CreationDate="2010-09-03T18:54:34.447" UserId="71" />
  <row Id="2728" PostId="2346" Score="2" Text="Small extension about plots: `plot.randomForest` shows how OOB error and in-class OOB error evolved with increasing number of trees; `varImpPlot` shows attribute importance measures for top attributes and `MDSplot` all objects plotted on the 2D projection of RF object proximity measure." CreationDate="2010-09-03T19:25:18.740" UserId="88" />
  <row Id="2729" PostId="2362" Score="1" Text="+1 Great point about setting the seed values." CreationDate="2010-09-03T19:32:24.090" UserId="5" />
  <row Id="2730" PostId="2356" Score="1" Text="So, does the above comment imply that the answer to the OP's question is 'No such examples can be constructed.'? Or perhaps, some pathological example exists which violates the assumptions behind admissibility? " CreationDate="2010-09-03T19:44:08.477" />
  <row Id="2731" PostId="2346" Text="+1 for citing the `MDSplot()` function. I must admit I often use RFs as a way to highlight clusters of individuals (based on the RF proximity measure) rather than selecting the best features. Clinicians often read much easily such plots than dotplot of var. importance..." CreationDate="2010-09-03T20:07:49.980" UserId="930" />
  <row Id="2732" PostId="2365" Score="2" Text="It is possible to be interested in the differences between Bayesian and frequentist statistics without it being a quarrel. It is important to know the flaws as well as benefits of ones preferred approach.&#xA;&#xA;I specifically excluded priors as that is not a problem with the framework, per se, but just a matter of GIGO.  The same thing applies to frequentists statistics, for example by assuming and incorrect parametric distribution for the data.  That wouldn't be a criticism of frequentist methodology, just the particular method.&#xA;&#xA;BTW, I have no particular problem with improper priors." CreationDate="2010-09-03T20:52:49.227" UserId="887" />
  <row Id="2733" PostId="2221" Text="Thanks for this response. To follow-up: is the solution to the the &quot;other formulation&quot; (wind vectors), 1/3, correct given the issues that you raise?" CreationDate="2010-09-03T21:56:55.473" UserId="401" />
  <row Id="2734" PostId="2365" Score="1" Text="Jaynes first example: Not one statistician in his right mind will ever use an F-test and a T-test on that dataset. Apart from that, he compares a two-tailed test to P(b&gt;a), which is not the same hypothesis tested. So his example is not fair, which he essentially admits later on. Next to that, you can't compare &quot;the frameworks&quot;. What are we talking about then? ML, REML, LS, penalized methods,...? intervals for coefficients, statistics, predictions,...? You can as well ask whether Lutheran service is equivalent or superior to Shiite services. They talk about the same God." CreationDate="2010-09-03T22:22:11.210" UserId="1124" />
  <row Id="2735" PostId="2365" Text="Could you clarify what is your data and what are the parameters you  would be estimating in your model? I am a bit confused on this point. Also, could you please use $$ instead of $ to center the formula? The font size is very small right now." CreationDate="2010-09-03T22:23:09.330" />
  <row Id="2736" PostId="2356" Text="@Srikant: Good question.  I think the place to begin investigating is a situation where there are non-Bayes admissible estimators--not necessarily a &quot;pathological&quot; one, but at least one that provides some opportunity to find a &quot;contrary example.&quot;" CreationDate="2010-09-03T22:29:08.203" UserId="919" />
  <row Id="2737" PostId="2365" Text="@Srikant: The example in Felsensteins book is based on a Jukes-Cantor model for DNA evolution. Data is DNA sequences. You want to estimate the probability of change in your sequence, which is related to your branch length based on the mentioned formula. Branch lengths are defined as time of evolution : the higher the chance for changes, the more time that passed between the ancestor and the current state. Sorry, but I can't summarize the whole theory behind ML and Bayesian phylogenetic inference in just one post. Felsenstein needed half a book for that." CreationDate="2010-09-03T22:29:34.647" UserId="1124" />
  <row Id="2738" PostId="2365" Text="I guess I just wanted you to clarify what variables in your equation was data and which ones were the parameter as it was not clear from your post especially to someone like me who is an outsider. I am still lost but I guess I would need to read the book to find out more." CreationDate="2010-09-03T22:44:20.240" />
  <row Id="2739" PostId="2365" Text="@Srikant : I tried to clarify a bit more. Actually, P is the parameter that is used in the likelihood function for optimization, and the formula merely gives its relation to t, which can alternatively be used in the likelihood function. Sorry I can't be more clear. If Phylogeny interests you, I can surely recommend Felsensteins book, it's a gem. http://www.sinauer.com/detail.php?id=1775" CreationDate="2010-09-03T23:04:10.970" UserId="1124" />
  <row Id="2740" PostId="2281" Text="@Keith: I'm not getting it. If you mean that the 10% coin only gives head 1 in 10 times, and you end up with 0 heads, you cannot compute a confidence interval. If you have 1 head in ten times, your interval will indeed not cover 50%. But I never claimed it covered. I just claimed it is unlikely it doesn't cover. I do NOT know the true value. Plus, all CI (Wald, score,Pearson,...) have a bad coverage on the edges of the probability space, definitely with only 10 cases. So I wouldn't state anything based on that CI. I'd use probability calculation to come to a conclusion. Like Bayes did." CreationDate="2010-09-03T23:35:53.847" UserId="1124" />
  <row Id="2741" PostId="2365" Text="It isn't clear to me why it is a problem that a flat prior on p implies an exponentially decreasing prior on t.  If that is inconsistent with biological knowledge, it simply means that a flat prior on p does not reflect actual prior knowledge.&#xA;&#xA;I also don't see why it is a problem to use an improper flat prior on t (other than I would have thought it inconsistent with prior knowledge; the branch time can't be say a billion years, if it were we wouldn't be here yet, so it is inappropriate to use a flat prior).&#xA;&#xA;Note that flat priors don't necessarily imply ignorance." CreationDate="2010-09-03T23:45:34.603" UserId="887" />
  <row Id="2742" PostId="2281" Text="@Keith : but I got your point - the true value is not a random variable - I agree. My bad." CreationDate="2010-09-03T23:47:27.230" UserId="1124" />
  <row Id="2743" PostId="2365" Text="@Dikran : it's not a problem. It is a fact. The problem is that p and t are strictly related, and hence should give exactly the same model. That happens in an ML approach, but that doesn't happen in the Bayesian approach. In Felsensteins example, a truncation of the t-prior at 700 or larger makes that the credible interval doesn't cover the true value any more. In this particular case, i.e. the lack of prior knowledge, Bayesian inference just isn't feasible. There is no sensible &quot;uninformative&quot; prior that can be used." CreationDate="2010-09-04T00:08:47.533" UserId="1124" />
  <row Id="2744" PostId="2365" Text="@Dikran : Regarding the flat t-prior: the prior gets truncated. When truncated at 5(!), most of the mass of the prior on p is concentrated around the maximum p-value. With larger truncation values, this effect is even more pronounced. The point is-again- that it's impossible to find a sensible prior when you have no prior knowledge in phylogenetic inference." CreationDate="2010-09-04T00:11:10.677" UserId="1124" />
  <row Id="2745" PostId="2281" Text="Joris, my last comment was about a &quot;95% credible interval&quot; -- not confidence interval! If you have a bag with one trillion fair coins and a single 10%-heads coin, and your experiment has you draw a coin uniformly at random from the bag, flip it ten times and then state a credibility interval on the heads probability, your credibility interval will always be [0.5, 0.5] no matter what. Thus if you happened to draw the unfair coin, the credibility interval will always be wrong." CreationDate="2010-09-04T03:59:07.460" UserId="1122" />
  <row Id="2746" PostId="2281" Text="Also I can't agree that &quot;all CI&quot; have bad coverage on the edges. Any exact CI, and some approximate CIs, will guarantee that the coverage is always greater than the confidence parameter (e.g. the 95%), even in the worst case. This is true of the Blyth-Still-Casella and Clopper-Pearson intervals for a proportion." CreationDate="2010-09-04T04:02:57.267" UserId="1122" />
  <row Id="2747" PostId="2367" Text="I believe Shao showed that k-fold CV is inconsistent if $k$ is fixed while $n$ grows." CreationDate="2010-09-04T05:18:10.490" UserId="795" />
  <row Id="2748" PostId="2367" Text="The BIC has k growing with n." CreationDate="2010-09-04T05:56:26.920" UserId="159" />
  <row Id="2749" PostId="2365" Text="Joris, I think you are missing the point, a flat prior is not necessarily non-informative.  It is completely reasonable for the same state of knowledge/ignorance to be expressed by a flat prior on p and (say) a flat prior on log(t) (which is a very common Jeffrey's prior) rather than a flat prior on t.  Does the book investigate ideas of MAXENT and transformation groups for this problem? &#xA;&#xA;There isn't enough detail in your example, but from what I can tell even a truncated flat prior on t is likely to be inconsistent with prior knowedge about t." CreationDate="2010-09-04T08:39:52.407" UserId="887" />
  <row Id="2750" PostId="2365" Text="@Joris, also in your original comment you suggest the flat prior on t must be truncated, because otherwise the area under the density function is infinite.  This is not true, there are plenty of problems where improper priors work very well, so there is not necessarily a need to truncate the flat prior.  " CreationDate="2010-09-04T08:49:51.587" UserId="887" />
  <row Id="2751" PostId="2287" Text="+1 That was a great example." CreationDate="2010-09-04T09:01:13.590" />
  <row Id="2752" PostId="2369" Score="1" Text="Food for thought, however the particular example is unfair as the frequentist approach is allowed to consider the relative costs of false-positive and false-negative costs, but the Bayesian approach isn't.  The correct thing to do according to Bayesian decision theory is to give an interval of [0,1] as there is no penalty associated with false-negatives.  Thus in a like-for-like comparison of frameworks, none of the Bayesians would ever be beheaded either.&#xA;&#xA;The issue about bounding false-positives though gives me a direction in which to look for an answer to Jaynes' challenge." CreationDate="2010-09-04T09:10:21.210" UserId="887" />
  <row Id="2753" PostId="2369" Text="Note also that if the selected coin is flipped often enough, then eventually the Bayesian confidence interval will be centered on the long run frequency of heads for the particular coin rather than on the prior.  If my life depended on the interval containing the true probability of a head I wouldn't flip the coin just once!" CreationDate="2010-09-04T09:57:59.440" UserId="887" />
  <row Id="2754" PostId="2348" Text="I've tried to change the question to make it less CW - not sure if I have succeeded :( @Shane @Srikant if you still think that it should be a CW, feel free to change it." CreationDate="2010-09-04T10:50:46.140" UserId="8" />
  <row Id="2755" PostId="2374" Text="If I understand correctly, your question encompasses at least two different topics: (1) use of FA in attitude or motivational scales, and (2) how to handle 'extreme' patterns of responses (ceiling/floor effects) in such scales? " CreationDate="2010-09-04T11:32:56.387" UserId="930" />
  <row Id="2756" PostId="2372" Text="Technically, you are correct but do note that the confidence interval gives the set of parameter values for which the null hypothesis is true. Thus, &quot;are the observed data x reasonable given our hypothesis about theta?&quot; can be re-phrased as &quot;What true values of theta would be a compatible hypothesis given the observed data x?&quot; Note that the re-phrased question does not necessarily imply that theta is being assumed to be a random variable. The re-phrased question exploits the fact that we perform null hypothesis tests by inspecting if the hypothesized value falls in the confidence interval." CreationDate="2010-09-04T15:32:22.633" />
  <row Id="2757" PostId="2186" Text="Just because it is the only book I know which combines exploratory MV analysis, statistical modeling, and psychometrics. Maybe not the best one actually, but interesting on its own." CreationDate="2010-09-04T16:59:16.127" UserId="930" />
  <row Id="2758" PostId="2367" Score="1" Text="I will just silently remind that *IC &lt;--&gt; *CV correspondence from Shao paper works **only** for linear models, and BIC is equivalent only to k-fold CV with certain k." CreationDate="2010-09-04T17:26:08.060" UserId="88" />
  <row Id="2759" PostId="2111" Score="1" Text="Find this article in the meantime, The lasso method for variable selection in the cox model, from Tibshirani (Stat. Med. 1997 16: 385-395), http://j.mp/bw0mB9. HTH" CreationDate="2010-09-04T21:30:44.700" UserId="930" />
  <row Id="2760" PostId="2111" Text="and this more recent one (closely linked to the `penalized` R package), http://j.mp/cooIT3. Maybe this one too, http://j.mp/bkDQUj. Cheers" CreationDate="2010-09-05T08:29:41.960" UserId="930" />
  <row Id="2761" PostId="2385" Score="2" Text="The aim of visualization is not to show the data, but to show a claim that this data supports -- so first try to focus on what you want to show, to make this question answerable. Or, if you just want a pool of visualization ideas, make it a community wiki." CreationDate="2010-09-05T13:08:37.680" UserId="88" />
  <row Id="2762" PostId="2385" Text="@mbq I don't think it is that ambiguous. He wants to compare means of the scales between different groups, as well as the response rates between different groups. Visualization can be exploratory (the fact that he is making a report to senior management doesn't mean to me that it can't be exploratory)." CreationDate="2010-09-05T13:45:01.203" UserId="1036" />
  <row Id="2763" PostId="2380" Text="May I suggest that you edit your main question to include this information?" CreationDate="2010-09-05T15:50:49.080" UserId="196" />
  <row Id="2764" PostId="2377" Text="Do you require that the standard deviation remain constant with this transformation as well?" CreationDate="2010-09-05T16:04:35.903" UserId="196" />
  <row Id="2765" PostId="2385" Text="@Andy so my suggestion for CW; even exploratory visualization is made to check *something*. " CreationDate="2010-09-05T19:32:13.847" UserId="88" />
  <row Id="2766" PostId="2365" Text="@Dikran : Guess you are missing the point : using the same uninformative prior gives two different models with Bayesian statistics on the same dataset. Not so with ML. The Bayesian can be very biased due to the nature of the model and the incompatibility of that model with infinite priors. You don't have to believe me. Felsenstein is the authority on phylogenetic inference, and his book explains you better than I will be able to. Reference in a previous comment." CreationDate="2010-09-05T20:02:47.387" UserId="1124" />
  <row Id="2767" PostId="2397" Text="What exactly do you mean by limiting distribution in this case?" CreationDate="2010-09-05T21:24:39.593" UserId="352" />
  <row Id="2768" PostId="2397" Text="ie, the one you get from Central Limit Theorem, let me update details" CreationDate="2010-09-05T21:28:01.960" UserId="511" />
  <row Id="2769" PostId="2398" Score="1" Text="but the covariance matrix of multinomial in question is singular, you showed it yourself..." CreationDate="2010-09-05T22:26:32.703" UserId="511" />
  <row Id="2770" PostId="2398" Text="Oh, I see your problem! One of the elements, say the $d$th is completely dependent on the others. Probably if you chop off the last row and column of $C$ you will get that the $[p_1,p_2,\dots,p_{d-1}]$ are normally distributed, but I'll have to have a think about it.  Surely this is solved somewhere already!" CreationDate="2010-09-05T22:41:08.137" UserId="352" />
  <row Id="2772" PostId="2365" Text="@Joris, as I said a flat prior is NOT NECCESSARILY UNINFORMATIVE. Consider this, if two priors give different results, then the must logically represent a different state of prior knowledge (see early chapters of Jaynes book that set out desiderata for Baysian inference).  Therefore the &quot;flat p&quot; prior and &quot;flat t&quot; prior cannot both be uninformative.  Felsenstein may be an expert on phylogenetic inference, but it is possible that he is not an expert on Bayesian inference. If he states that two priors giving different results are both uninformative, he is at odds with Jaynes (who certailny was)." CreationDate="2010-09-05T23:09:15.037" UserId="887" />
  <row Id="2773" PostId="2398" Text="One suggestion I found is to still use a Gaussian, but use pseudo-inverse instead of inverse and &quot;product of non-zero eigenvalues&quot; in place of determinant. For d=2 this seems to give the correct density form, but the normalization factor is off" CreationDate="2010-09-05T23:26:10.103" UserId="511" />
  <row Id="2775" PostId="2400" Text="So then, sigma is the entropy of N(0,sigma) corresponding to squared error, and min(p,1-p) is the entropy of Bernoulli(p) corresponding to 0,1 prediction loss? Seems like quite a generalization!" CreationDate="2010-09-06T00:03:45.560" UserId="511" />
  <row Id="2776" PostId="2228" Text="What error did you get? maybe try the following: install.packages(c('lme4','car','ggplot2','stringr'));install.packages('ez',dependencies=F,type='source')" CreationDate="2010-09-06T00:27:27.360" UserId="364" />
  <row Id="2777" PostId="2401" Score="2" Text="Do you know of any published research that looks at the relative merits of using likert scales as intervals vs ordinal data? Perhaps, the flaws of treating them as interval level scales are not serious enough to warrant a complex approach. If that is the case then your approach may simply be a wild goose chase." CreationDate="2010-09-06T01:12:06.877" />
  <row Id="2778" PostId="2377" Text="no, I expect it will not, but the excess kurtosis should remain fixed. I would expect the transform to be monotonic, however, and preferably deterministic." CreationDate="2010-09-06T02:23:41.060" UserId="795" />
  <row Id="2779" PostId="2392" Text="I had been doing something like this: rank, then use the g-and-h transform to get a fixed kurtosis and skew. However, this technique assumes I actually know the population kurtosis, which I can estimate, but I am interested, philosophically, if there is a transform which preserves the kurtosis without me having to know what it is..." CreationDate="2010-09-06T02:26:04.253" UserId="795" />
  <row Id="2780" PostId="2392" Text="@shabbychef:  Oh, well then sorry for not adding anything new. However, you added something new, I hadn't heard of the g-and-h formula before.  Do you have a freely accessible citation that provides it?  I stumbled onto one paper with it spelled out (http://fic.wharton.upenn.edu/fic/papers/02/0225.pdf) but the notion is a bit foreign to me (in particular is that e^Z^g or something else)?  I tried it like this... but the results seemed odd... a+b*(e^g^z-1)*(exp((h*z^2)/2)/g)." CreationDate="2010-09-06T05:04:41.457" UserId="196" />
  <row Id="2781" PostId="2377" Score="1" Text="Yikes - woe unto the person that wants to prove a non-deterministic function is monotonic." CreationDate="2010-09-06T05:05:54.120" UserId="196" />
  <row Id="2782" PostId="2386" Text="There is nothing wrong with a scientist &quot;believing&quot; in something, especially as a Bayesian probability represents the degree of belief or knowledge regarding the truth of some proposition." CreationDate="2010-09-06T07:25:56.310" UserId="887" />
  <row Id="2783" PostId="2386" Text="...The problem arises only when a scientist cannot distinguish between a belief and a fact.  There is nothing unscientific in the belief that Bayesian or frequentist statistics are superior, as there is no objective test that can decide the answer (AFAIK), so the choice is largely subjective and/or a matter of &quot;horses for courses&quot;." CreationDate="2010-09-06T07:36:00.367" UserId="887" />
  <row Id="2785" PostId="2378" Text="I realise that there are countless variants of this algorithm. What I was interested in was which ones do people routinely use." CreationDate="2010-09-06T07:50:24.937" UserId="8" />
  <row Id="2788" PostId="2369" Text="Having though about this a bit more, this example is invalid as the criterion used to measure success is not the same as that implied by the question posed by the king.  The problem is in the &quot;no matter which coin is drawn&quot;, a clause that is designed to trip up any method that uses the prior knowledge about the rarity of the biased coin.  As it happens, Bayesains can derive bounds as well (e.g. PAC bounds) and if asked would have done so, and I suspect the answer would be the same as the Clopper-Pearson interval.  To be a fair test, the same information must be given to both approaches.  " CreationDate="2010-09-06T08:29:18.367" UserId="887" />
  <row Id="2789" PostId="2345" Text="Thanks, I added some suggestion on one possible equivalence relation such that formula (2) would apply (I would have liked to add this as a comment but the number of characters was too high)." CreationDate="2010-09-06T08:46:29.947" UserId="1185" />
  <row Id="2790" PostId="2406" Score="1" Text="I would put it: &quot;Of the X% American/Australian/French/etc. adults who intend to buy a television, Y% intends to buy brand A, Z% intend to buy brand B,...&quot; So you immediately see both information in one sentence. " CreationDate="2010-09-06T08:49:23.627" UserId="442" />
  <row Id="2791" PostId="2350" Text="Yes that's what I am doing" CreationDate="2010-09-06T08:56:44.447" UserId="1150" />
  <row Id="2793" PostId="2397" Score="1" Text="What you're referring to is the asymptotic distribution of the **maximum likelihood estimator** of a multinomial. Also, the first equation should be n^{-1}, not n^{-1/2}." CreationDate="2010-09-06T10:12:10.567" UserId="495" />
  <row Id="2794" PostId="2406" Text="@Henrik. Thanks. I agree. I guess I was trying to put it in slightly more general terms. " CreationDate="2010-09-06T10:16:50.120" UserId="183" />
  <row Id="2796" PostId="2388" Text="The use of box-plots seems doubtful there. I just try to plot 190 bxp on the same page and this doesn't seem to carry out so much information as can be seen here: http://j.mp/925AY8. P.S. I didn't even use the R base graphics but relied on the `grid` package for the layout. @Shane's suggestion might be better." CreationDate="2010-09-06T14:23:22.810" UserId="930" />
  <row Id="2797" PostId="2388" Text="@chl Your right that is difficult to envision. The thinner you make the plots the more it just turns into a trend (like a time series). I don't think that would be a problem except that there is likely no logical ordering to the groups in this posters problem." CreationDate="2010-09-06T14:47:28.517" UserId="1036" />
  <row Id="2798" PostId="1297" Text="Also, isn't it possible to get CI (for the predictions) using bootstrapping ?" CreationDate="2010-09-06T14:58:08.747" UserId="253" />
  <row Id="2800" PostId="2412" Text="Not to spoil the party, but how are random forest supposed to provided robustness to contamination by outliers is a mystery." CreationDate="2010-09-06T15:27:04.850" UserId="603" />
  <row Id="2801" PostId="2411" Text="Thank you kwak.  This article seems to be talking about boosting methods.  Do the results they present hold for the simple classifier case of a CART model? (on the surface it sounds like it, but I didn't go through the article enough to really know)" CreationDate="2010-09-06T15:28:18.840" UserId="253" />
  <row Id="2802" PostId="2411" Text="The result they present holds for any convex loss function, and was initially discussed by Tukey. To sum things up, the measure of spread (Gini or entropy) used to quantify the quality of a node is sensitive to contamination by outliers (i.e. observations that are miss-labeled in the dataset).  This problem affects both the building and the prunning stage. Contamination of a dataset by observation with wrongly imputed label will typically cause the resulting tree to be much too complex (you can check this rather easely by yourself)." CreationDate="2010-09-06T15:38:22.293" UserId="603" />
  <row Id="2803" PostId="2362" Text="So @mbq when doing a CV is it valid to firstly do a random forest with all samples selected; doing it twice once with all and secondly with the top 10 variables (which can be quoted in a paper).  Then do a leave-one out cross-validation (selecting the 10 top genes each try) and quote the CV error from that?" CreationDate="2010-09-06T15:47:14.057" UserId="1150" />
  <row Id="2804" PostId="2365" Text="@Dikran : The point is not whether a flat prior is uninformative. The point is that an satisfying uninformative prior cannot be defined due to the nature of the model. Hence rendering the whole method unusable if you don't have prior information, and thus leading to the conclusion that Bayesian inference in this case is inferior to the ML approach. Felsenstein never said a flat prior was uninformative. He just illustrated why an uninformative prior cannot be determined, using the example of a flat prior. " CreationDate="2010-09-06T16:03:44.807" UserId="1124" />
  <row Id="2805" PostId="2411" Text="Thank you Kwak!  And is there no loss function that is robust?" CreationDate="2010-09-06T16:11:35.113" UserId="253" />
  <row Id="2806" PostId="2411" Text="no *convex* loss function. See this article &quot;A fast algorithm for the minimum covariance determinant estimator&quot; for an example of what can be done with non-convex loss functions (although not related to classification, the article is worth a read)." CreationDate="2010-09-06T16:14:31.003" UserId="603" />
  <row Id="2808" PostId="2365" Text="@Joris - it may be that an uninformative prior cannot be constructed in this case, but nothing that you have written so far establishes that to be the case.  What does Felsenstein write about MAXENT and transformation groups (the two main techniques used to determine an uninformative prior for a particular problem)?  If he has not investigated those methods, how can he know an uninformative prior is impossible?  It looks to me that a flat prior on p corresponds a flat prior on log(t), which is a well known Jeffreys' prior.  Can you demonstrate that the flat log(t) prior is informative?" CreationDate="2010-09-06T16:22:09.630" UserId="887" />
  <row Id="2809" PostId="2411" Text="It is very enjoyable to ask questions when getting answers like the ones you give - thank you!  Last questions: Do you know of an easy way to implement such a loss function in R's CART? (rpart)" CreationDate="2010-09-06T16:29:18.837" UserId="253" />
  <row Id="2810" PostId="2411" Text="The problem you point out seems easier to solve for regression than classification trees. I do not know how to change the spread criterion (from s.d. to Qn) in R but since Qn (and MAD) are implemented it should not be difficult. Best (let us know whether you had any success with a 'robustified' version)." CreationDate="2010-09-06T16:40:08.137" UserId="603" />
  <row Id="2811" PostId="2394" Score="2" Text="Good answer, John.  Note that kurtosis usually isn't worth looking at when a distribution is skewed: the skew already tells you there's going to be substantial kurtosis.  Thus, good applications are with datasets known a priori to be (approximately) symmetric.  Residuals often are this way: the kurtosis can be a quick numerical means to assess the shape of their distribution.  An additional point is that the sampling *variance* of the standard deviation is a function of the kurtosis, so if you're estimating SDs or variances, you might want to glance at the kurtosis to assess the precision." CreationDate="2010-09-06T17:11:40.827" UserId="919" />
  <row Id="2812" PostId="2379" Text="Thank you for posting this.  It's an important (and potentially inspiring) discussion to have." CreationDate="2010-09-06T17:27:30.127" UserId="919" />
  <row Id="2813" PostId="2362" Score="1" Text="@danielsbrewer I would do this in some other way (paying more attention to feature selection), but this is correct; yet it is more on the topic of benchmarking RF feature selection than on selecting best markers for your biological problem. " CreationDate="2010-09-06T17:39:35.890" UserId="88" />
  <row Id="2814" PostId="2411" Score="2" Text="@Tal CART is an equivalent to boosting, of a &quot;pivot classifier&quot; (the criterion that sits in each tree node, like some attribute grater than something or some attribute value in set something)." CreationDate="2010-09-06T17:47:23.223" UserId="88" />
  <row Id="2815" PostId="2411" Text="True.  Since I was intending to try this on regression trees, I'll have a go at it in the next few weeks and will come back to report if something nice will be formalized.  Best." CreationDate="2010-09-06T17:49:10.917" UserId="253" />
  <row Id="2816" PostId="2412" Score="3" Text="@kwak Still, this is a good answer; trees in RF don't see the entire set, so many of them will not be contaminated. Even better -- tracking in which leafs do OOB cases land can be used to find mislabeled objects and eliminate them. (As I recall now, this is mentioned in Breiman's paper about RF)." CreationDate="2010-09-06T17:50:57.683" UserId="88" />
  <row Id="2817" PostId="2397" Score="1" Text="In notation above, for d=2, X_n is the number of heads after n coin throws, so it's X_n/sqrt(n) that approaches Normal, not X_n/n, no?" CreationDate="2010-09-06T18:11:51.080" UserId="511" />
  <row Id="2819" PostId="2397" Score="1" Text="Yes, you're right. I was just confusing myself." CreationDate="2010-09-06T18:37:09.383" UserId="495" />
  <row Id="2820" PostId="2408" Text="What's a bit unsatisfactory is the freedom of choice, to get a valid density I need to ask for distribution of A x where A is some d-1 rank (d)x(d-1) matrix. Will the error of CLT approximation for finite n be equivalent for all choices of A? That's not clear to me" CreationDate="2010-09-06T18:39:07.037" UserId="511" />
  <row Id="2821" PostId="2406" Text="He should make sure the sample size from the new group is not too small however" CreationDate="2010-09-06T18:45:49.710" UserId="74" />
  <row Id="2822" PostId="2408" Score="1" Text="Yes, the error should always be the same. Keep in mind that the last element of the vector is functionally dependent on the other (d-1) elements (in both the finite sample and asymptotic cases)." CreationDate="2010-09-06T18:46:39.127" UserId="495" />
  <row Id="2823" PostId="2382" Text="I think statistics is a separate area from probability theory, though." CreationDate="2010-09-06T19:31:01.530" UserId="1106" />
  <row Id="2824" PostId="2419" Score="2" Text="Rpy, of course ;-)" CreationDate="2010-09-06T19:31:49.947" UserId="88" />
  <row Id="2825" PostId="2415" Text="What are ways in which people are trying to solve these problems? " CreationDate="2010-09-06T19:42:19.960" UserId="1106" />
  <row Id="2826" PostId="2408" Text="@Simon: It's not that the `last' element is dependent, Yaroslav's problem is that he doesn't like the idea of getting to choose which element to drop.  I agree with the answer you have given but I also think that a bit more thought and care is required here." CreationDate="2010-09-06T21:30:33.580" UserId="352" />
  <row Id="2827" PostId="2408" Text="@Yaroslav: Perhaps it would be good to have an idea of what application you have in mind here, because at this stage there are potentially lots of answers to your question. " CreationDate="2010-09-06T21:33:57.927" UserId="352" />
  <row Id="2828" PostId="2415" Text="@grautur: That's four excellent questions (plus many more, because your response applies to every answer in this thread).  They all deserve elaborate answers, but obviously there's no space for that here: one question at a time, please!" CreationDate="2010-09-06T21:58:17.927" UserId="919" />
  <row Id="2829" PostId="2427" Score="4" Text="Monte Carlo methods can give very accurate answers provided you do enough simulations." CreationDate="2010-09-06T22:59:43.747" UserId="159" />
  <row Id="2830" PostId="2056" Text="Yes, it does, but it describes resampling the subject means.  D&amp;H advocate resampling the subjects and fitting the original model.  " CreationDate="2010-09-06T23:00:55.163" UserId="187" />
  <row Id="2832" PostId="2408" Score="1" Text="Robby -- application I had in mind is here http://mathoverflow.net/questions/37582/how-big-is-the-sum-of-smallest-multinomial-coefficients  Basically integrals of Gaussian suggested by CLT give extremely good approximation to sums of binomial coefficients (for small n, even better than integrating Gamma representation directly!), so I was seeing if I can do something similar to get approximate sums of multinomial coefficients, which I need to get non-asymptotic error bounds for various fitters (like, maximum likelihood)" CreationDate="2010-09-06T23:23:45.367" UserId="511" />
  <row Id="2833" PostId="2423" Score="2" Text="CW perhaps?...." CreationDate="2010-09-06T23:34:02.847" />
  <row Id="2834" PostId="2427" Score="3" Text="If you want a programming solution a monte carlo is the only approach. I do not see any reason why you will not get accurate answers using monte carlo. A mathematical/analytical solution may not be easy." CreationDate="2010-09-06T23:49:37.943" />
  <row Id="2835" PostId="2427" Text="I have seen discussion about Monte Carlo and people said if you want to achieve 6 decimal places, it will take too long, or perhaps I'm confused with other similar problems. Since it's fairly easy to code a Monte Carlo approach, I guess it will be worthwhile to give it a try first." CreationDate="2010-09-07T01:14:38.150" UserId="18" />
  <row Id="2836" PostId="2420" Score="2" Text="Technically speaking, a histogram is used to show summary information about a frequency distribution. I think you want to ask &quot;Is a bar chart satisfactory?&quot;, to which I would answer Yes. If there are repeated measures on the categorical variables, then you can capture variability with a box plot, as mentioned in csgillespie's comment." CreationDate="2010-09-07T02:22:19.663" UserId="1080" />
  <row Id="2837" PostId="2406" Text="@Neil. Good point." CreationDate="2010-09-07T03:15:54.313" UserId="183" />
  <row Id="2838" PostId="2423" Text="see also question on mathoverflow regarding books on mathematical statistics http://mathoverflow.net/questions/31655/statistics-for-mathematicians" CreationDate="2010-09-07T03:18:37.553" UserId="183" />
  <row Id="2839" PostId="2431" Text="I don't think that's been around for a while, and it's not very comprehensive either...one reason I started this post was because I found an error there, and also couldn't find some things I needed, like density formula for functions of random vectors" CreationDate="2010-09-07T03:30:55.853" UserId="511" />
  <row Id="2840" PostId="2369" Score="1" Text="Dikran, there needn't be &quot;Bayesians&quot; and &quot;Frequentists.&quot; They're not incompatible schools of philosophy to which one may subscribe to only one! They are mathematical tools whose efficacy can be demonstrated in the common framework of probability theory. My point is that IF the requirement is an absolute bound on false positives no matter the true value of the parameter, THEN a confidence interval is the method that accomplishes that. Of course we all agree on the same axioms of probability and the same answer can be derived many ways." CreationDate="2010-09-07T04:55:38.120" UserId="1122" />
  <row Id="2841" PostId="2414" Text="maybe I've misunderstood, but I think a scale change does *not* change the kurtosis. It does change the fourth moment, but the kurtosis is adjusted by the variance squared, and thus is invariant to scale (and location) changes." CreationDate="2010-09-07T05:02:33.917" UserId="795" />
  <row Id="2842" PostId="2432" Text="discontinuities at known x-values, or at unknown x-values? (known x is easy enough)" CreationDate="2010-09-07T07:04:27.317" UserId="805" />
  <row Id="2843" PostId="2432" Text="@glen I updated the question: I'm interested in situations where timing of discontinuities is not known apriori." CreationDate="2010-09-07T07:08:14.553" UserId="183" />
  <row Id="2844" PostId="2433" Text="@robin Re LOESS: Perhaps my terminology is not quite right. By LOESS I'm referring to models that predict Y from X using some form of localised curve fitting. e.g., as seen in most of these graphs: http://www.google.com/images?um=1&amp;hl=en&amp;biw=1076&amp;bih=569&amp;tbs=isch:1&amp;sa=1&amp;q=lowess+graph&amp;aq=f&amp;aqi=&amp;aql=&amp;oq=&amp;gs_rfai=" CreationDate="2010-09-07T07:12:49.097" UserId="183" />
  <row Id="2845" PostId="755" Text="Like it, though it does remind me of http://en.wikipedia.org/wiki/Swiss_Toni" CreationDate="2010-09-07T07:18:02.573" UserId="449" />
  <row Id="2846" PostId="2420" Text="@Josh - see updated question -&gt; aren't bar charts for displaying quantities? : are dates quantities ? -" CreationDate="2010-09-07T07:51:35.607" UserId="414" />
  <row Id="2847" PostId="750" Score="1" Text="I think this quote is more of a cynical but realistic view of how statistical data is mostly used in debates (i.e. selected to support a preconceived notion rather than produced to test a hypothesis)" CreationDate="2010-09-07T09:12:43.227" UserId="127" />
  <row Id="2848" PostId="2362" Text="@mbq, sorry to go on, but I am very new to the machine leaning classification area.  So if you wanted to select the best discriminating markers for my sort of problem how would you go about it? and what error would you report?  I appreciate all the input." CreationDate="2010-09-07T10:18:10.480" UserId="1150" />
  <row Id="2849" PostId="2375" Text="I just UPDATED the references list and add some comments on other approaches that I now think are better suited to handle your question about extreme answering pattern. HTH" CreationDate="2010-09-07T10:19:46.703" UserId="930" />
  <row Id="2850" PostId="2375" Text="Thanks a lot... Very complete... " CreationDate="2010-09-07T10:34:57.687" UserId="1154" />
  <row Id="2852" PostId="2434" Score="1" Text="Your answering a slightly different question than the question asks. The question is asking the expected number of cells that would be empty after 50 jumps. Correct me if I am wrong, but I see no direct path from the probability a flea ends up in a certain square after 50 jumps to the answer how many cells would be expected to be empty." CreationDate="2010-09-07T13:24:19.570" UserId="1036" />
  <row Id="2853" PostId="2431" Text="It may be useful if you email the author. Perhaps, it is a misunderstanding and if not it will help in correcting the error in the next edition." CreationDate="2010-09-07T13:53:27.133" />
  <row Id="2854" PostId="2419" Text="I agree with mbq. Is there a very good reason why you have to do that in Python? Otherwise I'd use the workhorse R as a back-end as well." CreationDate="2010-09-07T13:54:16.430" UserId="1124" />
  <row Id="2855" PostId="2281" Text="@Keith. I should specify &quot;bad&quot; coverage. Too much coverage is also bad coverage. I'll state it differently : on the edges, the exact coverage does not coincide with the chosen coverage. " CreationDate="2010-09-07T13:57:23.110" UserId="1124" />
  <row Id="2856" PostId="2414" Text="You are correct, but I'm not proposing a mere scale change: when f is nonlinear, the kurtosis will change too.  (As a simple example, consider the change from a normal to a lognormal distribution.)  However, this point does not seem relevant: aren't you asking for transformations that leave the kurtosis *unchanged* while changing the skewness?" CreationDate="2010-09-07T13:59:35.623" UserId="919" />
  <row Id="2857" PostId="2434" Text="@Andy W -- great comment; yet Monte Carlo can be used to do this last step ;-)" CreationDate="2010-09-07T14:19:51.443" UserId="88" />
  <row Id="2858" PostId="2423" Score="1" Text="You could specify whether you need an introduction on applied statistics, or one on (theoretical) statistical inference. I.e., do you want the framework of testing, regression and ANOVA explained or do you want to know what the central limit theorem and the inequality of Chebiyshev have to do with the weak law of large numbers?" CreationDate="2010-09-07T15:11:25.727" UserId="1124" />
  <row Id="2859" PostId="2430" Score="2" Text="You should keep in mind that achieving what you want might be impossible." CreationDate="2010-09-07T15:33:30.707" UserId="279" />
  <row Id="2860" PostId="2420" Score="1" Text="@Tom - Yes, bar charts are used to display quantities, so you could use a simple dot plot or line chart too. Some would argue that bar charts have a low &quot;information-to-ink&quot; ratio, but it is a perfectly reasonable choice here. And yes, I would consider dates quantities, especially in this setting where the dates are &quot;days since some event&quot;. My original comment was admittedly nitpicking: your original question asked &quot;For example is a histogram satisfactory...&quot;, but histograms are for when the independent variable is discrete/continuous over some interval. " CreationDate="2010-09-07T15:41:06.107" UserId="1080" />
  <row Id="2861" PostId="2228" Text="I tried, but  ERROR: dependencies ‘lme4’ are not available for package ‘ez’ &#xA;* removing ‘/Library/Frameworks/R.framework/Versions/2.11/Resources/library/ez’" CreationDate="2010-09-07T15:47:22.230" UserId="1084" />
  <row Id="2862" PostId="2427" Score="1" Text="I don't dispute any of the three previous answers, but the (simple) analysis in the answer I have offered puts these remarks in perspective: if you want six decimal place accuracy for an estimate of a number that will be in the hundreds, the Monte Carlo simulation will take at least a year on a machine with 10,000 CPUs running in parallel." CreationDate="2010-09-07T15:53:14.987" UserId="919" />
  <row Id="2863" PostId="2429" Text="+1: i too learned from Lehman (very good reference), but this one is not mentionned nearly enough" CreationDate="2010-09-07T16:05:51.123" UserId="603" />
  <row Id="2864" PostId="2228" Text="looks like lme4 isn't installing properly. try: install.packages(&quot;lme4&quot;, repos=&quot;http://R-Forge.R-project.org&quot;)" CreationDate="2010-09-07T16:09:07.760" UserId="364" />
  <row Id="2865" PostId="2420" Text="One more thing: you have some votes for box plots but can you clarify if you have a single date for each category, or if you have multiple dates? " CreationDate="2010-09-07T16:10:03.143" UserId="1080" />
  <row Id="2867" PostId="2430" Score="1" Text="'You should keep in mind that achieving what you want might be impossible.' I will stick this quote on the door of my office." CreationDate="2010-09-07T16:21:08.647" UserId="603" />
  <row Id="2868" PostId="2414" Text="The problem is that all $Z_{\lambda}$ will have the _same_ kurtosis, as long as $\lambda &gt; 0$, because $Z_{\lambda}$ is an affine transformation of $Y$ (I assume $\nu_1$ is a constant).  How do I get $\kappa$ out of this?  " CreationDate="2010-09-07T16:25:23.287" UserId="795" />
  <row Id="2869" PostId="2392" Text="@drnexus: I did not want to bias the results by mentioning my technique. I learned about the g-and-h and the g-and-k distributions from Haynes et. al, http://dx.doi.org/10.1016/S0378-3758(97)00050-5 , and Fisher &amp; Klein, http://econstor.eu/bitstream/10419/29578/1/614055873.pdf " CreationDate="2010-09-07T16:32:29.343" UserId="795" />
  <row Id="2870" PostId="2431" Text="I have confirmation from author, so it'll probably be corrected in next edition" CreationDate="2010-09-07T16:40:39.233" UserId="511" />
  <row Id="2871" PostId="2367" Text="Actually, I believe Shao shows that CV is inconsistent unless $n_v/n \to 1$ as $n \to \inf$, where $n_v$ is the number of samples in the test set. Thus $k$-fold CV is always inconsistent for variable selection. Have I misunderstood? By $k$-fold CV I mean dividing the sample into $k$ groups and training on $k-1$ of them, and testing on 1 of them, then repeating $k$ times. Then $n_v/n = 1/k$ for $k$-fold CV, which never approaches 1. " CreationDate="2010-09-07T16:46:57.707" UserId="795" />
  <row Id="2872" PostId="2431" Score="1" Text="Could you, with the permission of the author, post the error briefly as a comment here? That will be useful to everyone." CreationDate="2010-09-07T17:04:10.053" />
  <row Id="2873" PostId="2423" Text="see also question http://stats.stackexchange.com/questions/414/introduction-to-statistics-for-mathematicians" CreationDate="2010-09-07T17:29:02.283" UserId="223" />
  <row Id="2874" PostId="2228" Text="Still doesn't work. During EZ instalation Error in dyn.load(file, DLLpath = DLLpath, ...) : &#xA;  unable to load shared library '/Library/Frameworks/R.framework/Versions/2.11/Resources/library/lme4/libs/i386/lme4.so':&#xA;  dlopen(/Library/Frameworks/R.framework/Versions/2.11/Resources/library/lme4/libs/i386/lme4.so, 6): Library not loaded: /usr/local/lib/libgfortran.2.dylib&#xA;  Referenced from: /Library/Frameworks/R.framework/Versions/2.11/Resources/library/lme4/libs/i386/lme4.so&#xA;  Reason: image not found&#xA;Error : package 'lme4' could not be loaded&#xA; " CreationDate="2010-09-07T18:05:56.620" UserId="1084" />
  <row Id="2875" PostId="2414" Text="I see your point: we need to incorporate \lambda within f, for example as a Box-Cox parameter.  Thus my response is premature (although perhaps not invalid) and needs more computation to support it." CreationDate="2010-09-07T18:35:01.657" UserId="919" />
  <row Id="2876" PostId="2228" Text="It may be related to the error Attaching package: 'Matrix'&#xA;&#xA;The following object(s) are masked from 'package:base':&#xA;&#xA;    det&#xA;" CreationDate="2010-09-07T18:41:12.383" UserId="1084" />
  <row Id="2877" PostId="2442" Text="Just for fun, I did a brute-force calculation in Mathematica.  Its answer is a ratio of a 21,574 digit integer to a 21,571 digit integer; as a decimal it's comfortably close to 900/e as expected (but, since we're asked not to post a solution, I'll not give any more details)." CreationDate="2010-09-07T18:43:28.077" UserId="919" />
  <row Id="2878" PostId="1136" Text="You can not eliminate human judgment -- nor would you want to. That said, making a model explicit does help reveal assumptions and open them up for discussion." CreationDate="2010-09-07T19:12:07.600" UserId="660" />
  <row Id="2879" PostId="2369" Text="I agree with the first point, it is a matter of &quot;horses for courses&quot;, but examples which show where the boundaries lie are interesting and provide insight into the &quot;courses&quot; best suited to each &quot;horse&quot;.  However, the examples must be fair, so that the criterion for success matches the question as posed (Jaynes is perhaps not completely immune to that criticism, which I will address in my answer which I will post later).  " CreationDate="2010-09-07T19:47:20.120" UserId="887" />
  <row Id="2880" PostId="2369" Text="The confidence interval only provides a bound on the *expected* number of false positives, it is not possible to put an absolute bound on the number of false positives for a particular sample (neglecting a trivial interval of [0,1]).  A Bayesian would determine an interval such that the probability of of more than five beheadings is less than some threshold value (e.g. 10^-6).  This seems at least as useful as a bound on the expected number of beheadings and has the advantage of being a (probabilistic) bound on what happens to the actual sample of courtiers.  I'd say this one was a clear draw." CreationDate="2010-09-07T19:57:02.983" UserId="887" />
  <row Id="2881" PostId="2447" Text="That's a neat idea: thanks for the reference.  However, the residuals to that particular fit look pretty bad, which makes me wonder how well it identifies potential changepoints." CreationDate="2010-09-07T20:40:12.780" UserId="919" />
  <row Id="2882" PostId="2447" Text="whuber: i do not know how much you are familiar with the theory of quantile regression. These lines have a major advantage over splines: they do not assume any error distribution (i.e. they do not assume the residuals to be Gaussian). " CreationDate="2010-09-07T20:58:39.387" UserId="603" />
  <row Id="2883" PostId="2420" Text="@Josh - no problem - I understood your distinction - I think you are right about dates being a quantity and in fact that's what I was struggling to see and hence the original question." CreationDate="2010-09-07T21:33:26.767" UserId="414" />
  <row Id="2884" PostId="2420" Text="Yes, I have single date (last access time) per category" CreationDate="2010-09-07T21:35:16.523" UserId="414" />
  <row Id="2885" PostId="2431" Text="Theorem 14.6, Sigma is singular so density isn't defined, this is fixed by instead considering distribution of $\hat{p_i}$ which is $\hat{p}$ with $i$'th component dropped (covariance matrix will be $\Sigma$ with i'th row/column removed)" CreationDate="2010-09-07T21:50:26.463" UserId="511" />
  <row Id="2886" PostId="2423" Text="Joris: well, internet is already pretty good for explanations, my motivation is having something to check against when I need a statistics related formula. For instance, recently I needed a formula for P(X=x|v'x=a) where X is multivariate gaussian and v is some vector, and none of my statistics books had it" CreationDate="2010-09-07T21:57:05.253" UserId="511" />
  <row Id="2887" PostId="2412" Score="1" Text="The problem is that outliers will make some 'bad' (i.e. contaminated) tree look better than good (uncontaminated) ones. This is called, masking effect and is easy to replicate with simulated data. The problem comes about because the criterion you use to evaluate trees is not in itself robust to outliers. I know i'm starting to sound like a fundamentalist mullah, but unless each and every tool you use is made robust, your procedure can be shown to be sensitive (at one level or another) to outliers (and hence not robust)." CreationDate="2010-09-07T22:20:44.310" UserId="603" />
  <row Id="2888" PostId="2434" Text="@Andy W: Actually, the hard part was getting all those probabilities.  Instead of adding them at each cell, multiply their complements: that's the probability that the cell will be empty.  The sum of these values over all cells gives the answer.  Glen_b's approach beats simulation by seven or eight orders of magnitude ;-)." CreationDate="2010-09-07T22:36:18.643" UserId="919" />
  <row Id="2889" PostId="2446" Score="1" Text="I don't think this is quite the same thing as ecological regression - that's about trying to infer something about the within-group slopes when you only have the group means. Here you have the within-group slopes." CreationDate="2010-09-07T22:42:22.317" UserId="449" />
  <row Id="2890" PostId="2056" Score="1" Text="You also might like to see the recently-published: Ren, Shiquan , Lai, Hong , Tong, Wenjing , Aminzadeh, Mostafa , Hou, Xuezhang and Lai, Shenghan(2010) 'Nonparametric bootstrapping for hierarchical data', Journal of Applied Statistics, 37: 9, 1487 — 1498" CreationDate="2010-09-07T23:28:17.007" UserId="187" />
  <row Id="2891" PostId="2362" Score="1" Text="The main problem is that it is really hard to compare two models (model=learning method+feature selection method), but for simplicity you can just assume something (like I'll use RF and select top 10 attributes) and admit that you know that this may be suboptimal, but you agree on that while you are for instance satisfied with the accuracy. In that case your only problem is to remove bias of attribute selection. tbc." CreationDate="2010-09-07T23:53:19.617" UserId="88" />
  <row Id="2892" PostId="2362" Score="1" Text="So, I would do a simple bagging: you create 10 (or 30 if you have a good computer) random subsamples of objects (let's say by random picking with replacement), train RF on each, get it's importance and return a rank of each attribute averaged over all repetitions (best attribute gets rank 1, second best 2 and so on; it can be averaged so the attribute that was 12 times 1st and 18 times 2nd have rank of 1.6), finally select 10 with best ranks and call them your markers. Then use a CV (LOO, 10-fold or preferably random sampling) to obtain an error approximation of RF using your markers. tbc. " CreationDate="2010-09-08T00:00:02.157" UserId="88" />
  <row Id="2893" PostId="2362" Score="1" Text="Report the ranks (hopefully they should be pretty near 1,2,3...), CV error with its deviation (just count standard deviation of results of each CV round) and OOB error (probably will be identical to CV error). DISCLAIMER: This is not a method for selecting optimal number of attributes -- you need RFE and nested CV to do that. DISCLAIMER2: I haven't worked with such a data, so I don't guarantee that your referees will be happy with it (though I believe they should)." CreationDate="2010-09-08T00:11:27.630" UserId="88" />
  <row Id="2894" PostId="2456" Text="Splendid, thanks." CreationDate="2010-09-08T00:45:41.743" UserId="88" />
  <row Id="2895" PostId="2446" Text="onestep:&gt; Good point                                            Adam&gt;: do you have access to the underlying data ? Or at least to the standard errors arround each slope  ?" CreationDate="2010-09-08T02:08:40.083" UserId="603" />
  <row Id="2896" PostId="2463" Text="excellent: +1 for the details " CreationDate="2010-09-08T02:30:48.797" UserId="603" />
  <row Id="2897" PostId="2445" Text="@whuber Thanks. I'll have a read." CreationDate="2010-09-08T02:35:42.107" UserId="183" />
  <row Id="2898" PostId="2447" Text="@kwak This looks interesting. Not assuming a normal error distribution would be useful for one of my applications." CreationDate="2010-09-08T02:36:46.173" UserId="183" />
  <row Id="2899" PostId="2445" Score="2" Text="@whuber I found the bcp R package http://www.jstatsoft.org/v23/i03/paper which implements the Barry &amp; Hartigan algorithm" CreationDate="2010-09-08T02:39:44.990" UserId="183" />
  <row Id="2900" PostId="2447" Text="Indeed, what you get out of this estimation are actual conditionnal quantiles: in a nutshell, these are to splines/LOESS-regressions what boxplots are to the couple  (mean, s.d.): a much richer view of your data. They also retain there validity in non gaussian context (such as assymetric errors,...)." CreationDate="2010-09-08T02:51:52.027" UserId="603" />
  <row Id="2901" PostId="4" Text="Are you interested in whether the distributions are of a different form (e.g., normal, poisson, etc.) or whether parameters are different (e.g., mean or sd of a normal distribution) or both?" CreationDate="2010-09-08T03:16:45.867" UserId="183" />
  <row Id="2902" PostId="2448" Text="Hi Kwak, The Huettmann paper is interesting but I have a few questions. 1) Under Assessing the link Functions, what is meant by the &quot;mean coefficients derived from the 1000 comparisons&quot; (p44)? 2) What do the &quot;Predictions&quot; in Table2 refer to? Predicted probability or predicted number of nesting sites, or what? 3) The marbled murrelet data are claimed to be public domain. Do you know the source? Thanks." CreationDate="2010-09-08T03:44:33.633" UserId="521" />
  <row Id="2903" PostId="2454" Text="@Andrew: Thanks for that. A customised solution might work best." CreationDate="2010-09-08T06:03:08.310" UserId="183" />
  <row Id="2904" PostId="2419" Text="the only reason being that I have used R only very few times a year or so ago and python I'm using every day..." CreationDate="2010-09-08T06:03:21.287" UserId="961" />
  <row Id="2905" PostId="2228" Text="Are you using Linux or Mac? I can recall glimpsing over some problems installing the Mac version of lme4. There's quite a few posts on R-sig-mixed-models about this." CreationDate="2010-09-08T06:25:25.350" UserId="966" />
  <row Id="2906" PostId="2443" Text="@Keith Thanks for this refined illustration." CreationDate="2010-09-08T07:19:14.940" UserId="930" />
  <row Id="2907" PostId="2472" Text="awesome. thanks." CreationDate="2010-09-08T07:45:28.713" UserId="183" />
  <row Id="2908" PostId="2461" Score="2" Text="It's `file.choose()`" CreationDate="2010-09-08T07:56:31.650" UserId="339" />
  <row Id="2909" PostId="2471" Text="+1 I wanted to post something similar." CreationDate="2010-09-08T08:22:34.380" UserId="88" />
  <row Id="2910" PostId="2193" Text="Yes, so you just look at the top 20 you rely on, and find the patterns of similarity for the joint. The joint combinations giving you the prediction contain the information of the variable associations. " CreationDate="2010-09-08T10:22:18.927" UserId="1098" />
  <row Id="2911" PostId="2193" Text="Oh yeah and you just do correlation analysis like a" CreationDate="2010-09-08T10:23:08.177" UserId="1098" />
  <row Id="2912" PostId="2146" Text="@Joris Meys- exactly" CreationDate="2010-09-08T10:25:40.360" UserId="1098" />
  <row Id="2913" PostId="2475" Score="2" Text="This seems more like a comment than an answer..." CreationDate="2010-09-08T10:38:38.153" UserId="5" />
  <row Id="2914" PostId="2478" Text="Thanks. Does LDA require you to know the topics upfront? In our case we do not know to which topic each Document belongs and we will be using the similarity measure to perform clustering (EM- G-Means, or GAAC) " CreationDate="2010-09-08T10:59:12.803" UserId="1212" />
  <row Id="2915" PostId="2478" Text="@ebony1 Nice reference to LSA, I made a similar answer some time ago at http://stats.stackexchange.com/questions/369/working-through-a-clustering-problem/2196#2196" CreationDate="2010-09-08T11:00:17.393" UserId="930" />
  <row Id="2916" PostId="135" Text="In SAS, KS test is available in `proc npar1way`. In R, in addition to `ks.test()`, there is the `nortest` package which provides several other adjustment tests." CreationDate="2010-09-08T11:15:20.670" UserId="930" />
  <row Id="2917" PostId="2367" Text="I probably shouldn't have said k-fold. The BIC is equivalent to leave-v-out CV where v=n[1-1/(log(n)-1)] and so it satisfies the Shao consistency criterion. I've updated my answer accordingly." CreationDate="2010-09-08T11:23:26.587" UserId="159" />
  <row Id="2918" PostId="2478" Score="1" Text="@Joel: No, LDA doesn't assume that you know the topics for each document beforehand. BTW, just to be clear, LDA represents each document as a mixture of topics, not by just a single topic. So each topic will contribute to some fraction in the documents (and the individual fractions will sum to 1). Basically, LDA assumes that each word in the document is generated by some topic." CreationDate="2010-09-08T11:37:44.257" UserId="881" />
  <row Id="2920" PostId="2367" Text="@mbq. I've now added the linear caveat. " CreationDate="2010-09-08T11:42:56.870" UserId="159" />
  <row Id="2921" PostId="2482" Score="3" Text="Highlight your code and click on the button that looks like binary numbers..." CreationDate="2010-09-08T11:52:22.867" UserId="5" />
  <row Id="2922" PostId="2478" Text="@ebony - thanks! At risk of a rephrasing the question and repeating myself, does LDA require you to know the number of discreet topics?" CreationDate="2010-09-08T12:02:01.140" UserId="1212" />
  <row Id="2925" PostId="2461" Text="Thank you -- corrected." CreationDate="2010-09-08T12:21:21.250" UserId="334" />
  <row Id="2927" PostId="2228" Text="Looks like you're on a mac, in which case you might try building lme4 from source ( install.packages('lme4',type='source') ), but to do that you'll first need to install gfortran: http://cran.r-project.org/bin/macosx/tools/ " CreationDate="2010-09-08T12:26:01.550" UserId="364" />
  <row Id="2930" PostId="2209" Score="1" Text="And it is available for free download at the authors page: http://www.inference.phy.cam.ac.uk/mackay/itila/book.html" CreationDate="2010-09-08T12:37:47.323" UserId="247" />
  <row Id="2931" PostId="2483" Text="This is not an answer, but the question reminds me of a presentation by Max Khesin on &quot;graphical models&quot; that he gave at a recent NY meetup: http://www.slideshare.net/xamdam/graphical-models-4dummies" CreationDate="2010-09-08T12:47:15.413" UserId="5" />
  <row Id="2932" PostId="2367" Text="@Rob Thanks. [more characters]" CreationDate="2010-09-08T12:49:21.357" UserId="88" />
  <row Id="2933" PostId="26" Text="Agreed, it's a valid question. It's also well stated as it asks for example usage and calculation. Surely the purpose of the site is to create a repository for ALL questions statistical." CreationDate="2010-09-08T13:03:15.660" UserId="1212" />
  <row Id="2934" PostId="2481" Text="Kolmogorov test is for distributions hence object lying in infinite dimensional space... not obvious to get asymptotic power=1. t-test is about testing the mean of a real valued random variable?" CreationDate="2010-09-08T13:10:01.210" UserId="223" />
  <row Id="2935" PostId="2481" Text="I haven't claimed that it is obvious, I just said that I have found this statement. So the natural question for me would be: what about other statistical tests - no matter what they are designed for." CreationDate="2010-09-08T13:33:17.817" UserId="1215" />
  <row Id="2936" PostId="2486" Text="Thank you, Joris. So &quot;my definition&quot; was just partially correct. ;) Do you probably know the asyptotic power of Friedman test (non-parametric repeated measures ANOVA)? " CreationDate="2010-09-08T13:43:04.720" UserId="1215" />
  <row Id="2941" PostId="2459" Text="That's a nice observation.  However, I found it more bother than it's worth to exploit this explicitly.  Most of the programming amounts to setting up the transition matrix.  Once you do that, just square it and work with that.  By using sparse matrices, removing half the zeros doesn't save any time anyway." CreationDate="2010-09-08T14:17:25.013" UserId="919" />
  <row Id="2942" PostId="321" Text="@kwak: Could you please delete your comment and post it as an answer so it can be accepted?" CreationDate="2010-09-08T14:18:02.387" UserId="220" />
  <row Id="2945" PostId="2447" Text="@kwak: The residuals are heavily correlated with the x-coordinate.  For example, there are long runs of negative or small positive residuals.  Whether they have a Gaussian distribution or not, then, is immaterial (as well as irrelevant in any exploratory analysis): this correlation shows that the fit is poor." CreationDate="2010-09-08T14:30:58.283" UserId="919" />
  <row Id="2946" PostId="2445" Text="@Jeromy: Thank you for the R package and for inserting the links to the references." CreationDate="2010-09-08T14:32:07.280" UserId="919" />
  <row Id="2951" PostId="2478" Text="Yes. But there are variants of LDA (HDP-LDA) that do not require specifying the number of topics. See this paper: http://www.cse.buffalo.edu/faculty/mbeal/papers/hdp.pdf" CreationDate="2010-09-08T14:55:48.960" UserId="881" />
  <row Id="2952" PostId="2473" Text="Thanks. Nice idea cross classification. " CreationDate="2010-09-08T15:17:22.743" UserId="1154" />
  <row Id="2953" PostId="2473" Text="I'll come back with results." CreationDate="2010-09-08T15:18:49.043" UserId="1154" />
  <row Id="2954" PostId="2474" Text="Thanks for your answer and reference. I will check differents models and do cross validation. Bayesian is a good idea." CreationDate="2010-09-08T15:19:33.783" UserId="1154" />
  <row Id="2955" PostId="2460" Text="We should also note for the sake of new users of R in Windows that the directories are provided to setwd() using forward slashes e.g. setwd(&quot;c:/temp&quot;) not backslashes (as is typical in environments with a DOS lineage)." CreationDate="2010-09-08T15:34:06.437" UserId="196" />
  <row Id="2956" PostId="2465" Text="Its a within subject design. With 4 slope values for each subject. Each slope value was calculated, in a different condition, over 5 independent variable values - and there are actually 4 measurements for each of these iv values. " CreationDate="2010-09-08T15:44:02.333" UserId="1084" />
  <row Id="2957" PostId="2446" Text="kwak:&gt; I have the data! Not sure what is the best way to share data here." CreationDate="2010-09-08T15:46:51.223" UserId="1084" />
  <row Id="2958" PostId="2460" Text="Or with double back slashes: `c:\\temp`." CreationDate="2010-09-08T15:50:10.047" UserId="5" />
  <row Id="2959" PostId="2441" Text="I like the idea of creating a chain of indicators, but in your description it sounds like you still have to maintain full counts of fleas in all cells and run that, too, as a Markov chain.  So what is accomplished with the additional complexity of tracking the $n_{ij}$?" CreationDate="2010-09-08T15:53:12.723" UserId="919" />
  <row Id="2960" PostId="2394" Text="@whuber: actually the range of possible values of skew and _excess kurtosis_ allows distributions with some skew and zero excess kurtosis. " CreationDate="2010-09-08T16:49:40.733" UserId="795" />
  <row Id="2961" PostId="2272" Text="Whew... great answers, all around.  The marked best answer is the one that worked best, on its own, for me, but the collection of answers is most helpful of all.  Thanks, everyone." CreationDate="2010-09-08T16:59:24.990" UserId="71" />
  <row Id="2962" PostId="2367" Text="ah, OK. much more sensible now." CreationDate="2010-09-08T17:27:11.190" UserId="795" />
  <row Id="2963" PostId="2355" Text="Beyond proofs, I'm wondering if there have been simulation studies of any of the five cases I list, for example. " CreationDate="2010-09-08T17:27:57.630" UserId="795" />
  <row Id="2964" PostId="2459" Text="@whuber: I suspect the point of these problems is to learn problem solving techniques, rather than consume a lot of computational cycles. Symmetry, parity, etc, are classic techniques from Larson's book on problem solving." CreationDate="2010-09-08T17:31:12.197" UserId="795" />
  <row Id="2965" PostId="2447" Text="Whuber: the line you see on the plot is an estimate of the 90% conditionnal quantile (i.e. the function $f(x):p(y_i&lt;f(x_i))\approx 0.9\; forall i$. This is true assymptotically for $f(x)$, not for each of the piecewise linear function taken individually. " CreationDate="2010-09-08T17:39:55.470" UserId="603" />
  <row Id="2966" PostId="2492" Score="3" Text="For reference: I don't think that this needed to be community wiki." CreationDate="2010-09-08T17:57:46.223" UserId="5" />
  <row Id="2967" PostId="2355" Text="Wanna make some?" CreationDate="2010-09-08T17:58:26.340" UserId="88" />
  <row Id="2968" PostId="2492" Text="I wasn't sure there was a 'right answer'..." CreationDate="2010-09-08T18:01:40.223" UserId="795" />
  <row Id="2969" PostId="2492" Text="See http://meta.stats.stackexchange.com/questions/290/what-is-community-wiki" CreationDate="2010-09-08T18:03:57.193" UserId="5" />
  <row Id="2970" PostId="2492" Text="In a certain sense, this is true of all test of a finite number of parameters. With $k$ fixed (the number of parameters on which the test is caried) and $n$ growthing without bounds, any difference between the two groups (no matter how small) will always break the null at some point. Actually, this is an argument in favor of bayesian tests." CreationDate="2010-09-08T18:07:28.977" UserId="603" />
  <row Id="2971" PostId="2473" Text="@Andrej Do you know of any papers comparing scores computed based on PLS-DA or PLS-1/2 with regression scores coming from a factor model (which account for measurement errors)?" CreationDate="2010-09-08T18:42:31.890" UserId="930" />
  <row Id="2972" PostId="2492" Text="For me, it is not a valid argument. Anyway, before giving any answer you need to formalize things a little bit. You may be wrong and you may not be but now what you have is nothing more than an intuition: for me the sentence &quot;In the era of cheap memory, big data, and fast processors, normality tests should always reject the null of normal &quot; needs clarifications :) I think that if you try giving more formal precision the answer will be simple. " CreationDate="2010-09-08T19:01:08.107" UserId="223" />
  <row Id="2973" PostId="2441" Text="@whuber No, you need not maintain a flea position as a markov chain. Think of what I am proposing as a random walk for a cell. A cell initially is at position '1' from where it can go to 0, 1, 2, 3, 4, or 5. The probability of state transition depend on the states of the adjacent cells. Thus, the proposed chain is a on a re-defined state space (that of cell counts for each cell) rather than on the flea position itself. Does that make sense?" CreationDate="2010-09-08T19:41:35.480" />
  <row Id="2974" PostId="2483" Text="Insights into the warning process would help in precise answers: Are your events of interest (i.e., weather in your example) continuous variables which are discretized or are they truly discrete? How does the warning system work? In other words, what triggers a warning? " CreationDate="2010-09-08T19:58:44.217" />
  <row Id="2975" PostId="2493" Score="1" Text="Do you have a the raw data from the GPS devices?  If so, you might consider sharing it here and we might be able to give some more explicit answers." CreationDate="2010-09-08T19:59:54.037" UserId="5" />
  <row Id="2977" PostId="2446" Text="Ok Adam, i don't need the data, but if you have access to all 150 data points, then, it is a completely different problem statistically (you should metion it in the question). The good news is this becomes much simpler to solve. I will respond in the answer zone." CreationDate="2010-09-08T21:17:19.227" UserId="603" />
  <row Id="2978" PostId="2446" Text="I erased my previous comment, as i had missunderstood the nature of your question." CreationDate="2010-09-08T21:24:18.250" UserId="603" />
  <row Id="2979" PostId="1433" Score="3" Text="In statistics, bias is not the same as error. Error is purely random, bias is not. You have bias when you know that the expected value of your estimate is not equal to the true value. " CreationDate="2010-09-08T21:30:57.787" UserId="1124" />
  <row Id="2980" PostId="2251" Text="I'd love to answer too, but I'm afraid there's nothing left to add to what Kingsford said." CreationDate="2010-09-08T21:43:55.307" UserId="1124" />
  <row Id="2981" PostId="2441" Text="It makes sense, but it seems like a step backwards, because isn't the number of states now much larger? In one model there are 900 states--the position of a single flea--and no more than four transitions out of each one. The computation only needs to be done for a single flea because they all move independently.  In yours it seems a state is described by a cell's occupancy along with the occupancy of its up to four neighbors. That would be an extremely large number of states and also a very large number of transitions among the states. I must be misunderstanding what your new state space is." CreationDate="2010-09-08T21:50:10.693" UserId="919" />
  <row Id="2982" PostId="2459" Text="That's a good point.  Ultimately some judgment is needed.  Project Euler appears to emphasize tradeoffs between mathematical insight and computational efficiency.  Glen_b mentioned symmetries that are worth exploiting first because there is more to be gained from them.  Moreover, by using sparse matrix arithmetic you will achieve the twofold gain automatically (whether you're aware of the parity or not!)." CreationDate="2010-09-08T21:56:59.060" UserId="919" />
  <row Id="2983" PostId="2498" Text="this is great! I'm slapping myself for not doing the experiments myself..." CreationDate="2010-09-08T22:35:17.087" UserId="795" />
  <row Id="2984" PostId="2458" Text="Questions about pure programming aspects of R might better be asked at stackoverflow, there's a larger userbase for now." CreationDate="2010-09-08T23:01:02.787" UserId="1124" />
  <row Id="2985" PostId="2498" Score="4" Text="On a side note, the central limit theorem makes the formal normality check unnecessary in many cases when n is large." CreationDate="2010-09-08T23:19:31.450" UserId="1124" />
  <row Id="2986" PostId="2497" Text="Firstly, would you be able to point me at the GIS forums where you've seen this dicussed? When it comes to sub-sampling I take it you mean sampling from the existing (sampled) data. By extrapolating a fit what kind of fit are you talking about? Lastly, I don't think the 2D kernel smooth approach you describe would work well for me as I would like the process to be automated. Is it not possible to directly combine both datasets and then kernel smooth them into one line?" CreationDate="2010-09-09T01:21:20.200" UserId="1227" />
  <row Id="2987" PostId="2434" Text="@whuber , Thanks for the explanation. Indeed getting those probabilities in under a minute would be challenging. It's a fun puzzle and thanks for your input. " CreationDate="2010-09-09T01:33:24.747" UserId="1036" />
  <row Id="2988" PostId="2493" Score="1" Text="+1 great question." CreationDate="2010-09-09T01:42:55.147" UserId="352" />
  <row Id="2989" PostId="2482" Text="Thanks for the detailed answer Bruce!" CreationDate="2010-09-09T03:25:04.647" UserId="1210" />
  <row Id="2993" PostId="515" Text="I think your first example refers to Bertrand's paradox. Very nice illustration of the different ways to define a probabilistic space!" CreationDate="2010-09-09T07:48:50.480" UserId="930" />
  <row Id="2994" PostId="2473" Score="1" Text="Sorry for delay. You might find useful the paper &quot;Importance of data structure in comparing two dimension reduction methods for classification of microarray gene expression data&quot; from Caroline Truntzer (BMC Bioinformatics). Actually, it is not about factor model, but PCA model." CreationDate="2010-09-09T08:17:17.417" UserId="609" />
  <row Id="2995" PostId="2499" Score="1" Text="Not to be rude, but isn't this a question that is already answered ad nausea on Wikipedia and the likes? Google gave me the answer within 15 seconds..." CreationDate="2010-09-09T08:32:42.190" UserId="1124" />
  <row Id="2996" PostId="694" Score="2" Text="That's definitely my favorite. I always have to stop on this and laugh when scrolling over this page. It's just so bad!!" CreationDate="2010-09-09T09:02:48.803" UserId="442" />
  <row Id="2997" PostId="2498" Score="1" Text="yes, the real question is not whether the data are actually distributed normally but are they sufficiently normal for the underlying assumption of normality to be reasonable for the practical purpose of the analysis, and I would have thought the CLT based argument is normally [sic] sufficient for that." CreationDate="2010-09-09T09:37:22.440" UserId="887" />
  <row Id="2998" PostId="2509" Text="I should point out that I have left out any scale factors (e.g. $\sqrt{N}$ is common in these problems) but they can be added without trouble. What you choose for a scale factor will depend on the distribution of $g'(p_0)$." CreationDate="2010-09-09T12:31:31.280" UserId="352" />
  <row Id="2999" PostId="2510" Text="It would help immensely to clarify what you mean by &quot;globally&quot; and &quot;approximate&quot;!  For example, h itself is a fine approximation to g according to many measures.  If you require that g and its approximator be relatively close pointwise for all of R^2, then you're not going to get any better than that." CreationDate="2010-09-09T12:54:22.783" UserId="919" />
  <row Id="3000" PostId="2497" Text="Check out the forums (both the old and new) at esri.com.  By &quot;fit&quot; I mean any reasonable (nonlinear) fit to the points.  A direct combination of two paths is highly problematic.  It would be tempting to just intersperse the union of all GPS readings in the two tracks, but there is no obvious unique way of connecting them up or of averaging them out somehow.  That's the basic problem to be solved here and I'm not aware that anybody actually has a general-purpose solution." CreationDate="2010-09-09T13:03:13.443" UserId="919" />
  <row Id="3002" PostId="2456" Text="Ok, indeed I rewrote it completely to have different scales on both sides, but it seems there is nothing in R that can do it natively. " CreationDate="2010-09-09T13:24:54.540" UserId="88" />
  <row Id="3003" PostId="2456" Text="I'm sure that you could do it with ggplot2, but it would require a little effort." CreationDate="2010-09-09T13:44:54.943" UserId="5" />
  <row Id="3004" PostId="1646" Text="I'm having some trouble understanding the dependent measure in your question.  How is team effectiveness being measured?  When you say &quot;the measure of team effectiveness&quot; it sounds as though you are talking about your measure of &quot;team functioning&quot;.  If so, isn't the dependent measure is necessarily criteria contaminated?" CreationDate="2010-09-09T14:43:33.053" UserId="196" />
  <row Id="3005" PostId="2513" Text="Please may I suggest a new tag along the lines of &quot;artefacts&quot;? (Or &quot;artifacts&quot; if you happen to be of the American persuasion)." CreationDate="2010-09-09T15:06:00.053" UserId="266" />
  <row Id="3007" PostId="2507" Text="I presume you are referring to a situation where one first does a normality test, and then uses the result of that test to decide which test to perform next." CreationDate="2010-09-09T16:07:23.570" UserId="25" />
  <row Id="3008" PostId="2511" Text="this is a very good point. however, don't most hypothesis tests have arbitrarily low power against some alternative? _e.g._ a test for zero mean will have very low power when given a sample from a population with mean $\epsilon$ for $0 \lt |\epsilon|$ small.  I am still left wondering whether such a test can be sanely constructed at all, much less whether it has low power in some cases." CreationDate="2010-09-09T16:11:25.107" UserId="795" />
  <row Id="3009" PostId="2511" Text="also, 'polluted' distributions like the one you cite always seemed to me to be at odds with the idea of being 'identically distributed'. Perhaps you would agree. It seems that saying samples are drawn i.i.d. from some distribution _without stating the distribution_ is meaningless (well, the 'independently' part of i.i.d. is meaningful)." CreationDate="2010-09-09T16:14:21.530" UserId="795" />
  <row Id="3010" PostId="2506" Text="This is a nice way of putting it. I am wondering if you can generalize this description to also apply to the kernel of 'kernel density estimation'." CreationDate="2010-09-09T16:18:52.630" UserId="795" />
  <row Id="3012" PostId="2510" Text="I may be missing something but I agree with @whuber. I do not see why you need an approximation g(x) when you have h(x)^+ and in what sense will any other function g(x) be better than h(x)^+?" CreationDate="2010-09-09T16:36:47.557" />
  <row Id="3017" PostId="2506" Text="In a way, yes. One way to understand kernel density estimation is that you approximate the density of a point from some distribution as a weighted average of its similarities with a set of points from the distribution. So the notion of similarity does play a role here as well." CreationDate="2010-09-09T17:14:44.687" UserId="881" />
  <row Id="3020" PostId="2511" Text="(1) You're right about the low power, but the problem here (it seems to me) is that there is no gradual step from &quot;finite&quot; to &quot;infinite&quot;: the problem seems not to have a natural scale to tell us what constitutes a &quot;small&quot; departure from the null compared to a &quot;large&quot; departure. (2) The distributional form is independent of considerations of iid.  I don't mean that, say, 1% of the data will come from a Cauchy and 99% from a Normal.  I mean that 100% of the data come from a distribution that is almost normal but has Cauchy tails. In this sense the data can be iid for a contaminated distribution." CreationDate="2010-09-09T17:39:07.823" UserId="919" />
  <row Id="3022" PostId="2510" Text="@kwak: In light of your clarification, are you asking for a polynomial approximation to the entire function g or to the data (x_i, g(x_i))?  (The solutions to those questions are somewhat different...)" CreationDate="2010-09-09T17:42:00.877" UserId="919" />
  <row Id="3024" PostId="2499" Text="I absolutely hate wikipedia answers for stats. There are rambling, symbolic messes. I am looking for a gem of an answer that can explain the answer in plain English, as I believe that that shows a deeper level of understanding than a math equation. There are many popular &quot;plain English&quot; questions on here, and for good reason. " CreationDate="2010-09-09T18:04:48.773" UserId="74" />
  <row Id="3025" PostId="2510" Text="@ Sri. Indeed i do have h(x) (the alpha are known) and therefore h(x)^+. But h(x)^+ is not a polynome. For reason pertaining to computational issues and ease of derivation of some properties, i need to approximate this function by a polynomial. Since h(x) [and therefore h(x)^+] are convex functions, i though, maybe, there should be a way to approximate them by some polynome, over the entire domain (R^2)." CreationDate="2010-09-09T18:43:51.903" UserId="603" />
  <row Id="3027" PostId="2510" Text="@ Whuber: Im really looking for an approximation to the entire function g(x_i). The criterion is such that we have somethin practical to work with (i.e. the criterion is essentially L2 distance between g and \hat{g}). See also my response to Sri's comment above." CreationDate="2010-09-09T18:45:31.370" UserId="603" />
  <row Id="3029" PostId="2519" Score="1" Text="+1: indeed, all three answers here are logically consistent with one another." CreationDate="2010-09-09T19:03:23.623" UserId="603" />
  <row Id="3030" PostId="2516" Score="3" Text="+1: this question usually exposes some interesting point of views." CreationDate="2010-09-09T19:05:58.943" UserId="603" />
  <row Id="3031" PostId="2520" Text="This will get the ball rolling. $g(x)$ has indeed as simple analytical form, but not a polynomial one. It will make my problem simpler if i could re-express $g(x)$ as a (or even several) polynome(s), even very complicated one(s). Wiggly is okay, so long as i can make it more precise by adding more terms.  " CreationDate="2010-09-09T19:12:43.580" UserId="603" />
  <row Id="3032" PostId="2518" Text="That's weird... intuitively, this seems to contradict the Law of Large Numbers." CreationDate="2010-09-09T19:12:50.343" UserId="666" />
  <row Id="3033" PostId="2518" Text="Carlos:&gt; can you be more specific ?" CreationDate="2010-09-09T19:25:45.360" UserId="603" />
  <row Id="3034" PostId="2521" Score="1" Text="Fantastic. Although your contribution answers my problem, i would like to use it to ask the following question: assuming now that we only consider a compact $B\in\mathbb{R}^2$. What is not the non-trivial answer?" CreationDate="2010-09-09T19:31:59.050" UserId="603" />
  <row Id="3035" PostId="2441" Text="@whuber I see the problem now. I think my proposal amounts to defining a markov chain on the state of the grid. Define the state of the grid as $\{n_{ij}\}$. Then we have a markov chain on the grid's state as the cell counts of all the cells at any one time period are dependent on the cell counts of all the cells in the previous time period. I agree with you this is an impractical proposal as the state space increases dramatically." CreationDate="2010-09-09T19:45:14.293" />
  <row Id="3036" PostId="2521" Score="2" Text="@kwak: Least squares!  The standard example of a compact subset is a finite set of points and the L^2 norm sums the squares of residuals.  With measurable compact subsets you need to *integrate* the squares of residuals.  There's also a classical L-infinity (sup norm) theory of approximating functions by polynomials (Weierstrass' Theorem) or even with sets of arbitrary real powers (Muntz-Szazs Theorem).  Rudin's textbook on real and complex analysis has an accessible account (I'm looking at a 1974 printing).  I believe these extend from R^1 to higher dimensions." CreationDate="2010-09-09T19:49:03.323" UserId="919" />
  <row Id="3037" PostId="2521" Text="@kwak (continued): The L^2 theory constructs orthonormal sets of polynomial functions (such as Legendre Polynomials and Chebyshev Polynomials).  You obtain the coefficients of good approximations simply by taking the inner product with the basis elements, exactly as in finite-dimensional linear algebra.  Different bases have different approximation properties.  Arbitrary (measurable) subsets B likely require variants of these systems adapted to their specific geometry." CreationDate="2010-09-09T19:51:26.597" UserId="919" />
  <row Id="3038" PostId="2521" Text="Ok, thanks very much. In some odd way this solves my problem (limits what i can do)." CreationDate="2010-09-09T19:53:31.580" UserId="603" />
  <row Id="3039" PostId="2518" Text="The LLN basically states that the larger your sample is, the better it represents the &quot;real&quot; probability distribution. In your example, the more house numbers I examine, the closer to 50% the number of odd-numbered houses will be. So it sounds weird that it becomes easier for you to break through the band, since it shrinks in proportion to the square root of $n$. (Am I making sense here?)" CreationDate="2010-09-09T19:54:59.470" UserId="666" />
  <row Id="3040" PostId="2518" Text="You do make sense in a theoretical framework. This is addressed by Andre's comment below. For matters of measurement errors, incomplete models, ect... you will nonetheless observe differences (even minute ones) in measured income between the two groups that will not shrink to 0, regardless of sample size. " CreationDate="2010-09-09T20:01:01.710" UserId="603" />
  <row Id="3044" PostId="2492" Text="The thread at &quot;Are large datasets inappropriate for hypothesis testing&quot; discusses a generalization of this question.  (http://stats.stackexchange.com/questions/2516/are-large-data-sets-inappropriate-for-hypothesis-testing )" CreationDate="2010-09-09T20:17:48.403" UserId="919" />
  <row Id="3046" PostId="2473" Text="@Andrej Thx, I will look at that paper." CreationDate="2010-09-09T20:52:41.140" UserId="930" />
  <row Id="3047" PostId="1537" Text="Because (n+k)/2 is not necessarily integral, consider rewriting the probability as Pr(S_n = 2^{2k-n}) = 2^-n Comb(n,k), 0 &lt;= k &lt;= n.  (There's also something fishy about your equating 2^k x = k.)" CreationDate="2010-09-09T21:08:15.373" UserId="919" />
  <row Id="3048" PostId="2355" Score="1" Text="I do; I'm going to have to learn a lot more R, though, to share the results here, though." CreationDate="2010-09-09T21:16:44.830" UserId="795" />
  <row Id="3052" PostId="56" Text="It should be pointed out that, from the frequentists point of view, there is no reason that you can't incorporate the prior knowledge *into* the model. In this sense, the frequentist view is simpler, you only have a model and some data. There is no need to separate the prior information from the model." CreationDate="2010-09-09T22:29:26.867" UserId="352" />
  <row Id="3055" PostId="471" Score="1" Text="Back when this was a \$3.95 and then a \$4.95 paperback, I bought copies by the dozen and gave them away to friends, clients, and anyone else who might be interested." CreationDate="2010-09-09T22:54:01.843" UserId="919" />
  <row Id="3056" PostId="370" Score="1" Text="Both are worth a periodic re-reading, maybe once a decade, just to refresh the ideas.  Concerning Tukey: it's great to sit down just with pencil and paper once in a while and do a deep analysis of an interesting dataset." CreationDate="2010-09-09T22:55:49.750" UserId="919" />
  <row Id="3057" PostId="2518" Text="@Carlos -- but convergence does not mean equality; this is guaranteed only for unreachable limit of infinity. So there is no contradiction ;-)" CreationDate="2010-09-09T23:32:56.983" UserId="88" />
  <row Id="3058" PostId="93" Text="when you say very small how small are you talking about? Unfortunately is is common to see people using sophisticated methods applied to a handful of data. If the problem is not having a good study to start with one should look at alternatives and make a case for a better study or data collection" CreationDate="2010-09-09T23:58:29.500" UserId="694" />
  <row Id="3059" PostId="2526" Text="This demonstration seems to assume the expected amount of money is zero, but it's not.  Furthermore, you appear to assume that the net of doublings minus halvings must be nonnegative, which is also incorrect.  Collectively these errors yield the correct limit, but that's an accident." CreationDate="2010-09-10T02:15:23.513" UserId="919" />
  <row Id="3060" PostId="2447" Text="@Kwak: Thank you for the clarification.  But if the estimate is true only asymptotically and globally, why should we expect it to perform at all well in identifying changepoints?  Is there some additional property the broken line regression has, of which I am unaware, that provides some comfort in this respect?" CreationDate="2010-09-10T02:36:03.867" UserId="919" />
  <row Id="3061" PostId="720" Text="@Harvey: I am confused by the remark about changing units, because by definition AIC is unitless (it's an adjusted maximum log likelihood).  A change in the data units would not change the maximized likelihood at all and therefore would not change the AIC either.  (Regardless, your recommendation to pay attention only to the difference is not in question.)" CreationDate="2010-09-10T02:41:22.407" UserId="919" />
  <row Id="3062" PostId="1646" Text="@drknexus In typical team studies, each team member is asked about for example their level of agreement with statements like &quot;my team is performing well&quot;, &quot;team members get along well with each other&quot;, &quot;my team handles conflict constructively&quot;... The question is thus the degree to which such beliefs are shared by team members." CreationDate="2010-09-10T02:52:57.690" UserId="183" />
  <row Id="3063" PostId="1537" Text="should your variable `money` be `x`?" CreationDate="2010-09-10T03:15:35.293" UserId="183" />
  <row Id="3064" PostId="2034" Text="The situation I mention above is standard practice in meta analytic studies of correlations. The &quot;combining apples and oranges&quot; critique is of course commonly raised regarding meta analysis. However, in defence of the practice, there is often substantial theoretical and empirical evidence that different measurements of the same construct (e.g., different intelligence measures) are highly correlated." CreationDate="2010-09-10T03:31:32.300" UserId="183" />
  <row Id="3065" PostId="2034" Text="@Jeromy: Let's see whether I have understood. X and Y could be different measures of &quot;intelligence.&quot; In study A, 100 people randomly drawn from a special school are tested. They all have comparable (low) levels of intelligence and r = 0.50. In study B, 50 gifted students are tested; they all have comparable (high) levels of intelligence and r = 0.45. If the raw data were to be combined, r would be almost 1.00. In study C, 50 more low-level people are studied and r = 0.45. The r from the combination of studies A and C would lie close to 0.45 or 0.50. How would a meta-analysis handle this?" CreationDate="2010-09-10T04:02:06.290" UserId="919" />
  <row Id="3066" PostId="2511" Text="I don't follow (2). It seems you are flipping a heavily biased coin, and usually drawing a normal random variate, and very rarely a Cauchy. Since you specified this distribution, the words 'identically distributed' are meaningful. However, to say that some observations are identically distributed _without_ giving the distribution seems meaningless, because that distribution could be: first uniformly select an integer $i &lt;= 1000$, then draw a random variable from distribution $i$ out of 1000 different distributions. my Q: how could one have a sample which is *not* 'identically distributed'?" CreationDate="2010-09-10T04:36:19.893" UserId="795" />
  <row Id="3068" PostId="2034" Text="@whuber typically X would be one variable (e.g., intelligence) and Y would be a different variable (e.g., average grade at university). Each study might use a slightly different measure of intelligence and a slightly different measure of average grade. Some meta analytic studies apply corrections for range restriction,  reliability of measurement, and other factors. Of course in the ideal situation the same measure would be used in all studies, and the raw data would be available, but meta analysis aims to synthesise results in the published literature as best as possible." CreationDate="2010-09-10T07:14:41.730" UserId="183" />
  <row Id="3069" PostId="2517" Text="I do not know if Mark and Shern had your view in mind but just to re-phrase your point- if the model for the data under the null is 'wrong' then you will reject the null hypothesis for a large enough data." CreationDate="2010-09-10T08:40:53.683" />
  <row Id="3070" PostId="1537" Text="@Jeromy: Yes, I've changed it. @whuber: You're correct, I've tried to make the probability a bit clearer. BTW, integral-&gt;integer in your comment." CreationDate="2010-09-10T08:43:23.247" UserId="8" />
  <row Id="3071" PostId="2513" Text="Thank you, mbq." CreationDate="2010-09-10T09:05:07.817" UserId="266" />
  <row Id="3072" PostId="2513" Text="Could you add the calculated T as formula using latex?" CreationDate="2010-09-10T09:10:56.707" UserId="1124" />
  <row Id="3073" PostId="2507" Score="2" Text="I refer to the general utility of normality tests when used as  method to determine whether or not it is appropriate to use a certain method. If you apply them in these cases, it is, in terms of probability of committing an alpha error, better to perform a more robust test to avoid the alpha error accumulation." CreationDate="2010-09-10T10:42:59.330" UserId="442" />
  <row Id="3074" PostId="2447" Text="@Whuber:&gt; i appended my answer with some mathematical details and a much needed scientific citation. Best." CreationDate="2010-09-10T11:24:13.183" UserId="603" />
  <row Id="3075" PostId="2537" Text="Are you working with R or Stata? Are the items locally independant (i.e. the probability of choosing any one item does not depend on response to other items)? Could you confirm that what you call &quot;conditional fractions&quot; are just the % of males/females that choosed a given item, adjusting for other effects in your model?" CreationDate="2010-09-10T12:56:18.630" UserId="930" />
  <row Id="3077" PostId="2537" Text="@chl : that's exactly what I want to achieve." CreationDate="2010-09-10T13:37:41.697" UserId="1124" />
  <row Id="3078" PostId="2500" Text="Very nice! I'm going to use your example with the circle to explain kernel methods, as it is the best visualization I met up til now. Thanks!" CreationDate="2010-09-10T13:47:06.480" UserId="1124" />
  <row Id="3079" PostId="2447" Text="@Kwak: Thank you!  I see how the case \tau = 0.5 might be effective.  (BTW, you probably meant to write \tau instead of \alpha in the formula.)  It would be interesting to compare its performance to methods specifically designed to identify changepoints." CreationDate="2010-09-10T14:02:16.820" UserId="919" />
  <row Id="3080" PostId="2483" Text="I think it would be ideal if you tell us your real question. Then we can give a lot better answers. The changed example also needs more clarification which will be specific to this example and will probably lead us away from your real question. I know that revealing your real question comes with the potential cost of somebody &quot;stealing&quot; it. But I think that is the price for getting good answers here." CreationDate="2010-09-10T14:24:10.020" UserId="442" />
  <row Id="3082" PostId="2538" Text="@Joris Maybe I will have to rework my response because it seems to me I'm not entirely answering your question; indeed, I just talked about estimating main effects of your covariates, but we can also imagine model with random slope or random intercept, or a combination thereof to derive proper estimates for each gender." CreationDate="2010-09-10T15:01:07.663" UserId="930" />
  <row Id="3083" PostId="2538" Text="@chl : this is already more than I hoped for. I've seen a comparison between gee and glmm before, and the output differed sometimes quite substantially, hence my choice for gee. I'll check out the gee package. I suppose I could use the same approach more or less for the interaction terms, as I want the odds per item and per gender (hence the interaction terms). Or am I wrong there?" CreationDate="2010-09-10T15:25:13.743" UserId="1124" />
  <row Id="3084" PostId="2538" Text="@Joris Ok, I've somewhat UPDATED my response. Under the GLMM, you can fit separate intercept and/or slope for both gender, which is not exactly the same as an interaction term (which as formulated there would also be viewed as a fixed effect). Whatever the solution you choose, you need to account for a different probability of endorsing any one of your 5 items depending on the gender of the respondent. More generally, you will useful references by just looking at *Differential Item Functioning* (DIF) on the web. I just remember that the `difR` package has interesting functionalities for this." CreationDate="2010-09-10T15:43:54.527" UserId="930" />
  <row Id="3085" PostId="2538" Text="@chl, indeed, I want to see gender and item as a fixed effect. The research hypotheses are formulated that way. The only randomness in the whole thing is the personId. But I'll read in on the topic using your links. Thanks for the valuable help." CreationDate="2010-09-10T15:47:28.390" UserId="1124" />
  <row Id="3086" PostId="2538" Score="1" Text="@Joris Other links that might be useful: http://j.mp/bzetkQ, http://j.mp/ac20UQ, http://j.mp/a1UNRb. HTH" CreationDate="2010-09-10T15:52:20.437" UserId="930" />
  <row Id="3087" PostId="2507" Text="Hello Henrik, you bring an interesting case of multiple comparisons which I never thought of in this case - thanks.  (+1)" CreationDate="2010-09-10T16:59:19.503" UserId="253" />
  <row Id="3088" PostId="2511" Text="*Q: how could one have a sample which is not 'identically distributed'?* an example of non-iid sample would be in time series (GARCH), where the observations are generated by a mixture of Gaussian distributions whose parameter are themselves random variables. In the context of the example above, a non iid sample would be one generated by \alpha percent Gaussian + (1-\alpha) percent Cauchy, where \alpha is a r.v. in (0,1). Then draws from this mixture would be non-iid. " CreationDate="2010-09-10T17:48:27.150" UserId="603" />
  <row Id="3089" PostId="2447" Text="@Whuber:&gt; with the caveat that this is not an 'online' algorithm. If you add observations, the locations of the change points will vary (although this behavior is fairly easy to 'fix', for some reasons, the author have never explored that direction)." CreationDate="2010-09-10T17:59:16.093" UserId="603" />
  <row Id="3091" PostId="2541" Text="Without reference to the number of parameters you try to estimate, or equivalently the kind of model your are working with, it seems rather difficult to give you a clear answer." CreationDate="2010-09-10T18:36:26.123" UserId="930" />
  <row Id="3093" PostId="2509" Text="Yup, and page 51 of van der Vaart's &quot;Asymptotic Statistics&quot; gives the distribution of that quantity as Gaussian with covariance matrix $$E[g'^2]/E[g'']^2$$ I don't see why consistency is that important here...Taylor expanding around the biased estimate p* instead of p0 seems to produce Gaussian with the same variance by this argument" CreationDate="2010-09-10T18:43:49.833" UserId="511" />
  <row Id="3094" PostId="2535" Score="1" Text="Cronbach's alpha is essentially a measure of internal consistency, that it is of inter-items correlations. It suffers from many pitfalls though, especially the fact that it increases with increasing the # of items (it may even be negative, although it is a ratio of variances!). I think we shall focus here at describing variations/homogeneity at the level of individuals within teams, so thinking of mixed-effects modeling or hierarchical models is a good starting point. " CreationDate="2010-09-10T19:10:51.390" UserId="930" />
  <row Id="3095" PostId="1198" Text="Thank you, Carlos, for the recommendation.  It is indeed a great read, despite (or perhaps because of) its age.  I'm especially impressed that the authors (Box, Hunter, &amp; Hunter) appeal to *permutation* distributions, rather than arguing for normality, as the &quot;ultimate&quot; justification for the classical tests (t, F, etc.)." CreationDate="2010-09-10T19:50:39.127" UserId="919" />
  <row Id="3096" PostId="2531" Text="isn't there a copy/past typo in the first equation ?" CreationDate="2010-09-10T19:53:23.747" UserId="603" />
  <row Id="3097" PostId="730" Score="1" Text="This sentence itself is a model (an epistemological one)" CreationDate="2010-09-10T20:00:18.917" UserId="603" />
  <row Id="3098" PostId="2547" Score="3" Text="It seems like you already had a reasonable answer on the other site?" CreationDate="2010-09-10T20:40:51.857" UserId="5" />
  <row Id="3099" PostId="2548" Text="+1 Beat me....." CreationDate="2010-09-10T20:50:17.087" UserId="88" />
  <row Id="3100" PostId="1520" Text="Probability of observing value above median is one half, but your quantity is log-normally distributed for large n, which has different mean and median, so you shouldn't expect that probability to approach 1/2." CreationDate="2010-09-10T21:33:02.290" UserId="511" />
  <row Id="3102" PostId="2548" Score="5" Text="Breakdown isn't an issue for a descriptive statistic of an entire population." CreationDate="2010-09-10T22:06:33.870" UserId="919" />
  <row Id="3103" PostId="2546" Text="Beautiful reference--and spot on relevant.  Thank you." CreationDate="2010-09-10T22:21:10.390" UserId="919" />
  <row Id="3104" PostId="2509" Text="Sure, a biased estimate converging to $p^*$ is going to have a similar central limit theorem, i.e. $p^* - \hat{p}$ must then be assumed to converge. This is simply replacing $p_0$ by $p^*$." CreationDate="2010-09-10T23:37:24.213" UserId="352" />
  <row Id="3105" PostId="2549" Text="Actually I think nowadays the skew in a lot of countries is more towards seniors, not tots." CreationDate="2010-09-11T00:09:34.637" UserId="830" />
  <row Id="3106" PostId="2549" Text="Perhaps, it is skewed the other way but the general point stands. For skewed distributions a median may make more sense than the mean." CreationDate="2010-09-11T00:21:30.863" />
  <row Id="3108" PostId="2550" Score="1" Text="I think you meant &quot;The median is more resistant to such errors than the mean&quot;. I agree with your comments though, and I believe the US census typically reports median's for many categories in official reports (not just the age) for basically all of the same reasons. Income is maybe even a better example than age to illustrate such points." CreationDate="2010-09-11T01:54:37.943" UserId="1036" />
  <row Id="3112" PostId="2539" Text="@srikant: Thanks very much. This is DEFFINITELY going in the right direction. Plus you have clearly outlined your thought process, and explained things as you are going along. Very, very good. Give me a little time to reflect on your answer and make sure that it addresses all the issues I need to address." CreationDate="2010-09-11T07:22:13.680" UserId="1216" />
  <row Id="3113" PostId="2539" Text="@srikant: The more I read your answer, the more I love it - its a shame that I can't vote it up, whilst I'm doing a little bit more reading around the subject area. I have a couple of further questions for you. 1). In the Bayesian model you described above, are there any underlying assumptions of the observed variables regarding: i). normality ii). independence (i.e. zero auto correlation)?  If there are any assumptions what are they and how does that affect the &quot;appropriateness&quot; of bayesian model, if one (or both) of the underlying assumptions are &quot;violated&quot; ?" CreationDate="2010-09-11T07:33:08.507" UserId="1216" />
  <row Id="3114" PostId="2539" Text="Second question: You have (understandably), assigned a nominal scale to the observed variable (which light bulb is switched on), which you called 'Severity'. Will the Bayesian model still be an appropriate one, if instead of using a nominal scale, I use an ordinal one - i.e. there is no 'ranking' between the signals given. For example Level1, .. LevelN merely indicate the level of a building in which the attact event is predicted to occur." CreationDate="2010-09-11T07:38:53.547" UserId="1216" />
  <row Id="3115" PostId="2558" Text="A question though (on other polynomials): my aim is to approximate a multivariate gaussian by polynomial expansions: which of the different approaches (Chebyshev, Hermite,  Laguerre) is considered best for approximation ?" CreationDate="2010-09-11T07:47:15.700" UserId="603" />
  <row Id="3116" PostId="2539" Text="Srikant: Could you please elaborate on this statement: &quot;In order to compute the required probability, P(S=s|W=w) you need to have some sense of P(S=s) which you can estimate or guess based on previous experience.&#xA;&#xA;" CreationDate="2010-09-11T07:52:20.207" UserId="1216" />
  <row Id="3117" PostId="2539" Text="@Srikant: Could you please elaborate on the following statement (i.e. how do I do what you suggested, in practise): &quot;You can then assess the accuracy of the device by looking at the following probabilities: P(S=s|W=s).&quot;" CreationDate="2010-09-11T07:53:49.087" UserId="1216" />
  <row Id="3118" PostId="730" Text="but see a nice discussion around this quote on Gelman's blog, http://j.mp/9SgIBO" CreationDate="2010-09-11T10:21:25.927" UserId="930" />
  <row Id="3119" PostId="2549" Text="I just updated my answer on math.stackexchange to emphasize just that point.  People look for symmetry and can incorrectly impose symmetry when it isn't there.  When you report the median, you give an answer that is symmetric -- the median splits the population in half -- even though the distribution is not symmetric." CreationDate="2010-09-11T14:11:38.210" UserId="319" />
  <row Id="3120" PostId="2542" Score="2" Text="Here's a page on exactly how good the normal approximation of the t distribution is for n=30.  http://www.johndcook.com/normal_approx_to_t.html" CreationDate="2010-09-11T14:25:20.370" UserId="319" />
  <row Id="3121" PostId="2545" Score="1" Text="Your example illustrates the value of robust statistics.  The *sample median* estimates the location parameter of a Cauchy distribution well.  One could argue that the weakest link in using a t-test with 30 samples is the t-test, not the 30 samples." CreationDate="2010-09-11T14:35:20.043" UserId="319" />
  <row Id="3122" PostId="2550" Text="Oops: thanks for catching that typo, Andy." CreationDate="2010-09-11T14:54:38.053" UserId="919" />
  <row Id="3123" PostId="2547" Text="@Shane: But perhaps different sites hold the potential to garner different answers from different points of view?" CreationDate="2010-09-11T14:56:20.507" UserId="919" />
  <row Id="3124" PostId="2561" Text="Edward Tufte is a statistician.  Started his career with BA and MS in statistics from Stanford, taught and wrote books about statistics for political scientists and is a fellow of the ASA.  " CreationDate="2010-09-11T15:13:09.837" UserId="1107" />
  <row Id="3125" PostId="2563" Score="2" Text="I am a bit unclear as to what exactly you are asking and how data analysis enters the picture? Please elaborate." CreationDate="2010-09-11T16:37:35.250" />
  <row Id="3126" PostId="2539" Text="Reg assumptions- Normality is irrelevant as the variables are discrete. Using the empirical proportions to estimate P(W=w|S=s) is equivalent to assuming independent bernoulli trials for the events; an assumption which may or may not be reasonable given your context. Your estimates for this probability will be 'off' if this assumption is violated. And consequently your assessment of the reliability of the device will also be affected. " CreationDate="2010-09-11T16:45:16.483" />
  <row Id="3127" PostId="2539" Text="Reg nominal vs ordinal- you have your terms switched. Ordinal implies ranking and nominal does not imply ranking. Yes, the model will hold for both ordinal and nominal data. Do note that if you have ordinal data we can develop a better model by exploiting the ordering of the data. Currently, for ordinal data the model is not as powerful as it can be as it does not use the ordering information present in the data. " CreationDate="2010-09-11T16:49:20.407" />
  <row Id="3128" PostId="2539" Text="Reg P(S=s): This is application specific but perhaps you could take a random sample of your target population to which you wish to generalize and assess P(S=s)" CreationDate="2010-09-11T16:50:19.267" />
  <row Id="3129" PostId="2539" Text="Reg criteria for P(S=s|W=s): I do not know of any explicit criteria. Obviously, this should be as close to 1 as possible. So, perhaps compute how far the vector {P(S=s|W=s)} is from a vector of 1s by computing the Mean Squared Error. That way you have a scalar measure of the performance of the device which you can then compare across several potential designs of the warning system." CreationDate="2010-09-11T16:52:43.130" />
  <row Id="3130" PostId="1406" Text="Unfortunately it is popular among those that don't understand or trust statistics.  Once had it quoted to me when I gave a scientist an estimate of the number of hours it would take me to design and analyze his experiment.  He didn't seem to appreciate the fact that noise can look like effects, and effects of interest can be hidden from intuition by noise." CreationDate="2010-09-11T17:05:56.130" UserId="1107" />
  <row Id="3131" PostId="1330" Text="Love this one -- a wonderful bonus of being a statistician." CreationDate="2010-09-11T17:13:07.987" UserId="1107" />
  <row Id="3132" PostId="2542" Text="That's good stuff.  Thanks for sharing." CreationDate="2010-09-11T17:53:41.530" UserId="1108" />
  <row Id="3134" PostId="2563" Text="How is ranking not related to data analysis?  You're trying to combine user data (up/down votes) to generate a rank statistic that conveys information.  Admittedly, more work on this has been done in machine learning than statistics, but that's on topic, right?" CreationDate="2010-09-11T18:56:10.023" UserId="251" />
  <row Id="3135" PostId="2574" Text="@Dirk: &quot;That is different from describing your dataset with an estimated  density or histogram.&quot; --- In case of pdf we are also somehow ESTIMATING the density. Are we not? Can you shed more light on your last paragraph. Thanks." CreationDate="2010-09-11T19:44:54.143" UserId="1102" />
  <row Id="3136" PostId="2563" Text="@ars It *seems* on topic but the question was/is a bit unclear to me. An algorithm can mean many different things depending on context." CreationDate="2010-09-11T19:47:49.887" />
  <row Id="3137" PostId="2574" Text="@Harpreet You are not estimating the shape of the PDF since as @Dirk indicated it has closed form, you just specify its parameters (e.g. $\mu$ and $\sigma^2$ for a gaussian). It will not necessarily &quot;fit&quot; the data. Now, there exist several kind of non-parametric density estimates, where you only use the data at hand (plus some kernel specifications or window span, etc.); see e.g., online help for the `density` R function." CreationDate="2010-09-11T19:57:24.090" UserId="930" />
  <row Id="3138" PostId="2575" Text="@ars I must admit I never fully understand why Gelman advocates the use of histogram with small bin width; why not using stripchart plot or raw data with superimposed kernel density estimates, which much better convey the empirical distribution of the observed data?" CreationDate="2010-09-11T20:06:31.440" UserId="930" />
  <row Id="3139" PostId="2575" Score="1" Text="@chl: There are of course other good visualization methods to get a sense of sampling variability.  But on the narrower comparison of histogram v. pdf under discussion here, I think his point is well made.  " CreationDate="2010-09-11T20:18:52.423" UserId="251" />
  <row Id="3140" PostId="2577" Text="It's not absolutely necessary; I was hoping whether there was such a technique so there's more rigour than eye-balling. If nothing exists, well, this is it, but perhaps there is." CreationDate="2010-09-11T21:04:52.813" UserId="840" />
  <row Id="3141" PostId="2561" Text="@Kingsford My fault! I was initially thinking of another citation, not from Tufte and didn't remove my first words... I UPDATED my response. Many thanks!" CreationDate="2010-09-11T21:11:41.153" UserId="930" />
  <row Id="3142" PostId="2574" Text="@chl: Thanks! PS: How do you include these formatting in text - like italics, selection etc - in comments?" CreationDate="2010-09-11T21:27:17.063" UserId="1102" />
  <row Id="3144" PostId="2573" Text="Could you please clarify whether this question concerns data (whose distribution could be represented by a histogram) or theoretical constructs (such as a pdf, which describes a probability distribution)." CreationDate="2010-09-11T21:42:33.967" UserId="919" />
  <row Id="3145" PostId="2573" Text="@whuber: I am asking it from perspective of application towards data analysis. Whether to employ histogram or probability dist. func (pdf), and why?" CreationDate="2010-09-11T21:59:13.897" UserId="1102" />
  <row Id="3146" PostId="2577" Text="Oh sorry, I didn't mean to imply there weren't any techniques, only that you might be better served with collapsing into categories rather than being misled by a distinction between ranks x and x+1.  For example, I think the scoring method you propose is a fine way to start." CreationDate="2010-09-11T22:01:53.480" UserId="251" />
  <row Id="3147" PostId="2573" Score="2" Text="But where does the pdf come from?  By definition, a pdf describes a theoretical probability distribution.  Do you perhaps mean the edf (empirical distribution function)?" CreationDate="2010-09-11T22:38:37.003" UserId="919" />
  <row Id="3148" PostId="2563" Text="@Srikant: there was a comment earlier about *closing* the question which I was responding to." CreationDate="2010-09-11T22:56:18.683" UserId="251" />
  <row Id="3149" PostId="2367" Text="@mbq: No --  the AIC/LOO proof by Stone 1977 does *not* assume linear models.  For this reason, unlike Shao's result, it's widely quoted; see for example the model selection chapters in either EOSL or the Handbook of Computational Statistics, or really any good chapter/paper on model selection.  It's only a bit more than a page long and worth reading because it's somewhat neat for the way he avoids having to compute the Fisher information/Score to derive the result." CreationDate="2010-09-12T02:45:40.563" UserId="251" />
  <row Id="3150" PostId="2367" Text="@Rob: right, it's not k-fold.  In fact, Shao devises the BICV procedure (Balanced Incomplete CV) in the referenced paper because the LKO procedure is so computationally expensive (C(n, k) subsets)." CreationDate="2010-09-12T02:48:17.893" UserId="251" />
  <row Id="3151" PostId="2583" Text="I don't think it is relevant whether I know the true model in the wild. If there is a 'true' model, I would prefer a method which is more likely to find it. " CreationDate="2010-09-12T03:49:05.380" UserId="795" />
  <row Id="3152" PostId="200" Text="+1 for this answer. I'll embellish it by saying that &quot;meaningful&quot; should also be considered in the context of decision making. If a given pattern is reliably extracted from the data generating process, is there a business/research decision that will be made differently by its presence? If so, you can estimate the Value of Information with respect to the effort required to do the data mining and the potential (monetary) gain from confidently making a different decision.  " CreationDate="2010-09-12T03:56:17.867" UserId="1080" />
  <row Id="3153" PostId="2583" Text="@shabbychef: I don't disagree.  But note: &quot;If there is a 'true' model&quot; *and* it's under consideration .. how would you know this a priori?  " CreationDate="2010-09-12T04:08:01.743" UserId="251" />
  <row Id="3154" PostId="430" Text="ditto. I usually put it in terms of 52 cards, and the goal is to find the ace of spades." CreationDate="2010-09-12T04:08:32.313" UserId="795" />
  <row Id="3155" PostId="2583" Text="Note also that my second paragraph actually makes the point in your comment.  This is a nice property, but it's not all clear how applicable it is in the wild; even though it's comforting in some sense, it may be misguided.  " CreationDate="2010-09-12T04:13:58.460" UserId="251" />
  <row Id="3156" PostId="2400" Text="Yes. The entropy for square loss is constant and the entropy for 0-1 loss is min(p,1-p). What's also interesting is that these have strong correspondences to divergences too. The square loss to the Hellinger divergence and 0-1 loss to variational divergence. Since entropies defined like this they are necessarily concave functions and it turns out the f-divergence built using f(p) = -entropy(p). Bob Williamson and I have explored some of this in our paper: http://arxiv.org/abs/0901.0356 . It's fun stuff. " CreationDate="2010-09-12T05:39:17.243" UserId="1201" />
  <row Id="3157" PostId="2580" Text="do you know the probability distribution of the values in **each bin** ? The y axis label makes me think that this could be Poissonian or Multinomial ? (assuming a model gives you the mean in each bin)" CreationDate="2010-09-12T07:47:55.857" UserId="961" />
  <row Id="3158" PostId="2574" Text="@Harpreet This is just Markdown syntax, as for editing a post through the online editor: `*ab*` gives *ab* (italic) `**ab**` gives **ab** (bold) `$\sqrt{2}$`=$\sqrt{2}$" CreationDate="2010-09-12T08:00:35.593" UserId="930" />
  <row Id="3159" PostId="2585" Score="1" Text="@Kevin Should this be community wiki? I know that more than one reasonable answer could be given, but it seems like a question where reputation should be awarded." CreationDate="2010-09-12T08:43:32.080" UserId="183" />
  <row Id="3160" PostId="2531" Text="I don't see it?" CreationDate="2010-09-12T09:54:53.347" UserId="1124" />
  <row Id="3162" PostId="2581" Text="Say, you have a portfolio of assets and you want to estimate the sensitivity of the diversification benefit to a severe change in correlations (therefore, the need to stress correlation matrix). Stressing the eigenvalues seems like a mathematically attractive idea, however, it doesn't necessarily reveal that much - wouldn't be more useful to stress actual correlation because you can express a view about specific variables." CreationDate="2010-09-12T10:59:48.907" UserId="1250" />
  <row Id="3163" PostId="2580" Text="The data is essentially drawn from two Poisson processes, but there are hidden variables that I can't correct for, leading to overdispersion. Thus, a negative binomial is definitely a better model. (see the new image/text I added above). I need to show that my nb model fits better quantitatively." CreationDate="2010-09-12T11:36:40.610" UserId="54" />
  <row Id="3164" PostId="2580" Score="1" Text="How about a metric like Mean Squared Error between actual vs predicted values?" CreationDate="2010-09-12T11:42:44.200" />
  <row Id="3165" PostId="2478" Text="@ebony - thanks for your input. I'll leave the question open for a while longer, to encourage other answers." CreationDate="2010-09-12T15:48:10.660" UserId="1212" />
  <row Id="3166" PostId="2592" Text="What do you call 'Pca matrix' or matrix of eigenvalues? There is only a vector of eigenvalues, and eventually a matrix of rotation." CreationDate="2010-09-12T16:21:45.367" UserId="930" />
  <row Id="3167" PostId="2592" Text="Sorry, my inaccuracy - By 'PCA matrix' I meant matrix of rotation" CreationDate="2010-09-12T16:25:43.383" UserId="1260" />
  <row Id="3168" PostId="2558" Score="1" Text="I'm doesn't matter so much _which_ polynomial class you use (up to numerical issues), but you can get better results by the choice of nodes where you sample the function (in this case the bivariate gaussian density) in order to compute the coefficients in front of the polynomials. For this problem, the Chebyshev nodes are optimal, at least when interpolating on a finite interval. see http://en.wikipedia.org/wiki/Chebyshev_nodes or any decent numerical analysis text (e.g. Cheney &amp; Kincaid) or http://pages.cs.wisc.edu/~amos/412/lecture-notes/lecture09.pdf" CreationDate="2010-09-12T16:33:13.880" UserId="795" />
  <row Id="3169" PostId="2558" Text="should say 'it doesn't matter'..." CreationDate="2010-09-12T17:20:49.867" UserId="795" />
  <row Id="3170" PostId="2358" Text="@Neil I remove the tag &quot;multi&quot; which does not seem to be used in other places, and which has no clear meaning IMO. Feel free to add it again if you think it's a wrong idea." CreationDate="2010-09-12T17:53:57.660" UserId="930" />
  <row Id="3171" PostId="2580" Text="hrmm - I like that idea, Srikant. It's a lot simpler than what I was thinking, but still makes sense. Throw into a an answer below so I can credit it and send some rep your way.  I'm still interested in hearing other methods, but this may work for now." CreationDate="2010-09-12T18:13:42.983" UserId="54" />
  <row Id="3172" PostId="2588" Text="I found that page too, but it does not mention any relationship to the upper tail, so I was unsure." CreationDate="2010-09-12T18:21:26.733" UserId="977" />
  <row Id="3173" PostId="2482" Text="+1, nice and complete answer, with code to boot!" CreationDate="2010-09-12T18:32:39.780" UserId="251" />
  <row Id="3174" PostId="2307" Text="+1, good answer and serendipitous for me -- much thanks for the paper references, especially the review." CreationDate="2010-09-12T18:36:13.703" UserId="251" />
  <row Id="3175" PostId="2367" Text="@ars But AIC must exist, convergence is poor, and so on. My whole crusade about this topic is against using it mindlessly to support some strange claims like that LOOCV is better/worse than 10-fold on 10 object set or that it works in case of machine learning." CreationDate="2010-09-12T18:39:25.957" UserId="88" />
  <row Id="3176" PostId="2367" Text="@mbq: I was only objecting to the strongly worded comment on &quot;*IC&quot; / &quot;*only* for linear models&quot;, not your crusade where I happily cheer you on.  Your latest comment is spot on.  This is the essence of the objection in my answer, i.e. false sense of comfort through stronger claims than supported by theory.  Model selection is far from a settled science -- and for good reason: as the model space becomes more complex, it's quite hard to prove widely applicable results." CreationDate="2010-09-12T19:24:38.847" UserId="251" />
  <row Id="3177" PostId="2599" Text="Thanks! The Wikipedia article on hierarchical clustering does not link to that one." CreationDate="2010-09-12T20:32:07.187" UserId="977" />
  <row Id="3178" PostId="2599" Score="1" Text="Oh right.  Fixed now under &quot;see also&quot; links, thanks for pointing that out!" CreationDate="2010-09-12T20:38:19.667" UserId="251" />
  <row Id="3179" PostId="2588" Score="1" Text="Well upper tail is by definition P(X&gt;x)." CreationDate="2010-09-12T22:18:59.907" />
  <row Id="3180" PostId="2585" Text="I am more or less looking for opinions, so there's no real &quot;solution&quot; to my question. Though I would be happy to award reputation to good responses...if it's possible." CreationDate="2010-09-12T23:19:32.113" UserId="1253" />
  <row Id="3181" PostId="2585" Text="Doesn't look like I can change it back into a regular question - could a moderator help with this?" CreationDate="2010-09-12T23:28:15.407" UserId="1253" />
  <row Id="3182" PostId="2545" Text="John:&gt; &quot;One could argue that the weakest link in using a t-test with 30 samples is the t-test, not the 30 samples&quot;. Very true, and also the assumption that the data is *iid*. Also, the median is MLE for Cauchy distributed random variables (and hence efficient), but in general you could need more than 30 observations.  " CreationDate="2010-09-12T23:54:36.057" UserId="603" />
  <row Id="3183" PostId="2558" Text="Yes, indeed. But do the Chebyshev node generalize to multivariate functions ? I have tried to search the web and the only class of polynomials that is considered in the multivariate case seems to be the Hermite. " CreationDate="2010-09-12T23:58:37.637" UserId="603" />
  <row Id="3184" PostId="2605" Text="@kwak I would be interested in knowing the exact solution as my proposal is more of a hack rather than a principled approach. Could you provide a pointer like a reference/ a term to search for?" CreationDate="2010-09-13T00:15:42.883" />
  <row Id="3185" PostId="2605" Text="@Srikant:&gt; i don't think it's an 'hack' v.s. principled approach difference. Your answer is in term of eigenvalues whereas Edward's question specifically mentions individual correlation coefficients. As a statistician, i understand that eigenvalues  have better statistical properties and are easier to work with than individual correlation coefficients (in this case the e.V.'s   will have the positive side effect of 'pooling' the correlation coefficients). But your simulation will no longer be in terms of individual assets, which may (or may not) be important for Edward." CreationDate="2010-09-13T00:28:03.433" UserId="603" />
  <row Id="3186" PostId="2605" Text="@kwak Can you shed some light on the structure of the objective function? Why is it formulated the way it is?" CreationDate="2010-09-13T01:51:40.237" />
  <row Id="3187" PostId="2560" Text="While I accepted the other answer, I'd like to say thanks for this one too (which is also quite useful)!" CreationDate="2010-09-13T02:08:49.993" UserId="386" />
  <row Id="3189" PostId="2605" Text="IMO, the above is a really helpful answer. +1 from me." CreationDate="2010-09-13T02:23:13.673" />
  <row Id="3190" PostId="2604" Text="Thanks. That clarifies a bunch of things and opens up a whole slew of new questions which I'll have to do some research on. I guess my main question is, does what you're saying mean that something more simple, like just taking root mean squared error, isn't a valid way to approach this problem? I'll grant that it's probably not as robust and won't give me a p-value, but it's something I could do quickly while I try to track down a copy of the book you reference. Any thoughts would be appreciated." CreationDate="2010-09-13T02:58:32.163" UserId="54" />
  <row Id="3191" PostId="520" Text="I'll give you +1 (so that you now have your 2 original upvotes :), just because I agree that with many data points, Python may be a good option; now the minor point, IMO, is that there're not so many clustering algorithms available in SciPy compared to R." CreationDate="2010-09-13T06:16:10.947" UserId="930" />
  <row Id="3192" PostId="2556" Text="Yes there was a typo. Thanks for that Alex. I don't always proofread :-)" CreationDate="2010-09-13T07:26:45.690" UserId="521" />
  <row Id="3194" PostId="2607" Text="@ kwak - that you! The sign of a small correlation can become important in certain optimisation problems. For example, a small negative correlation (which, statistically might have been positive just as likely due to insignificance) can be combined with 2 large volatilities (and as a result large negative covariance), which will lead to misleading optimisation results. Had the sign been positive, the results would have been very different. Zero correlation would at least not give preference to either." CreationDate="2010-09-13T07:34:38.583" UserId="1250" />
  <row Id="3195" PostId="2607" Text="By the way, I agree that the matrix might become indefinite. However, wouldn't it be feasible to simply make the negative eigenvalues = 0 and recalculate and rescale the matrix so that the diagonal values are 1? Example of the procedure can be found here: http://comisef.wikidot.com/tutorial:repairingcorrelation" CreationDate="2010-09-13T07:39:20.513" UserId="1250" />
  <row Id="3196" PostId="2605" Text="@kwak that was very helpful! It is possible to estimate the limits to stress testing numerically by increasing the stress levels until the minimum eigenvalue of the matrix no longer is &gt;= 0. I have also been exploring stress testing the entire matrix by a single parallel shift (ie each correlation +/- x stress level). Again, it is easy to estimate the limits to stress testing numerically. Rule of thumb: max upward stress is roughly equal to the smallest eigenvalue of a pos-def matrix. Max downward stress is more complicated. Is there a convenient way to solve this problem analytically?" CreationDate="2010-09-13T07:50:26.030" UserId="1250" />
  <row Id="3197" PostId="2610" Text="Thanks for this excellent answer! In fact, the hierarchical clustering module you showed is already part of scipy. Also, scipy provides an implementation of k-means, so I could easily use that." CreationDate="2010-09-13T08:36:32.897" UserId="977" />
  <row Id="3198" PostId="2530" Text="Thanks.  I've done some more searching and your formula appears to be more commonly used (and using a ratio of variances seems simpler and more intuitive to me).  I tried both formulas on two data sets; the results were qualitatively similar in one case and quantitatively identical in the other, so I guess the results of the two versions of the test converge at some point.  I'm going to proceed using your formula.  If anyone else has any insight into the two formulae, it would be great to hear it." CreationDate="2010-09-13T09:01:33.390" UserId="266" />
  <row Id="3199" PostId="2610" Text="Ok, I didn't look in details into this. For k-means, you need to pay attention to the fact that we generally need two outer loops for validating cluster solution (one where you vary the # of clusters and another for varying the seed -- the objective being to minimize the RSS); then you can use the Gap statistic to choose the optimal # of clusters." CreationDate="2010-09-13T09:18:19.767" UserId="930" />
  <row Id="3201" PostId="2605" Text="@Srikant:&gt; The solution is a k dimensional ellipse centered at $a_0$ (the actual observed correlation coefficients). This is like asking 'how far i can walk in the direction given by -w until i meet the boundary of the ellipse (which is represented by the first constraint). The actual distance is $\max.\; a_0'w-a'w$, but since the first term is a constant, it drops from the objective function. So this is given by min. a'w" CreationDate="2010-09-13T10:30:55.607" UserId="603" />
  <row Id="3202" PostId="2605" Text="@Edward:&gt; the numerical approach is okay when $p$ is small. What you do there is that you take a bunch of points at random inside the $k$ dimensional $(-1,1)$ hypercube and then you test each of your random points you test (using the value of the smallest eigenvalue of the resulting matrix) whether that point is inside the $k$ dimensional ellipse that i describe which bound the zone of admissibility. When $k$ increases you approach is no longer feasible because the ellipse will occupy an ever smaller part of the hypercube (this is known as the curse of dimensionality) meaning that the odds... " CreationDate="2010-09-13T10:39:07.837" UserId="603" />
  <row Id="3203" PostId="2605" Text="...of having a randomly selected point falling inside this ellipse (your acceptance rate) will become ever smaller. You would have to make exponentially more tries to have a fixed number of PSD matrices. THat's why it's better to have an explicit formulation of the boundary of the ellipse." CreationDate="2010-09-13T10:43:18.390" UserId="603" />
  <row Id="3204" PostId="2605" Text="@Edward:&gt; your second question (analytical solution). If you want to know by how what quantity $\delta$ you can shrink all of your correlation coefficients towards 0 while still having a PSD matrix, this is easy. If you want to know by how what quantity $\delta$ you can shrink all of your correlation coefficients towards -1 while still having a PSD matrix, this is more complicated. Which one you want?" CreationDate="2010-09-13T10:49:54.630" UserId="603" />
  <row Id="3205" PostId="2607" Text="Edward:&gt; your second point: i need to think about this but i have the impression that what you propose amounts to shrinking all your correlation coefficients by a small amount towards 0." CreationDate="2010-09-13T10:57:46.457" UserId="603" />
  <row Id="3206" PostId="2605" Text="I am interested in the 2nd. Allow me to describe the problem in more detail. I have some initial correlation matrix. I want to stress each correlation in the matrix by the same constant simultaneously (except the diagonal; lets call this global parallel stress since it affects the entire matrix by the same constant). I start with adding 0.01% to each correlation and check if the matrix is still PSD. I continue increasing the stress levels by small increments. Eventually I encounter a stress level above which the matrix would no longer be PSD. Let's call this the upper stress boundary." CreationDate="2010-09-13T11:42:06.463" UserId="1250" />
  <row Id="3207" PostId="2605" Text="I repeat the same procedure for negative stresses and find the lower stress boundary. My general observation was that the minimum eigenvalue initial correlation matrix is roughly equal to the upper stress boundary. Do you think that an analytical solution for this problem is too complicated?" CreationDate="2010-09-13T11:46:04.117" UserId="1250" />
  <row Id="3208" PostId="2605" Text="Generally, I am more interested in the upper stress boundary. I stress the entire matrix at the same time because it is a simple way to reduce diversification benefit across all market variables." CreationDate="2010-09-13T11:55:24.447" UserId="1250" />
  <row Id="3209" PostId="2605" Text="Edward:&gt; maybe open a new threat (that links to this one). Since you marked this one as solved, you're not getting a lot of traffic here (and i think other people in this forum have alot of experience with issues similar to yours). I think your problem is now different enough from the initial one and specific enough  that it deserves a threat of its own.  Also you might want to cross post it in http://www.or-exchange.com/" CreationDate="2010-09-13T12:26:13.267" UserId="603" />
  <row Id="3210" PostId="2598" Text="As @Edward pointed out, please add some details." CreationDate="2010-09-13T12:40:02.730" UserId="88" />
  <row Id="3212" PostId="2612" Text="Thanks a lot for your reply! In principle i'm also interested in how to solve a concrete problem like the one i described (thus thanks for the WinBUGS tip). But at the moment i'm trying to do a simulation study for a seminar paper in which i'd examine the performance (coverage rates etc.) of MI under model misspecification. I suppose i'll just forget about the variance components if I can't find a solution and focus on the fixed effects, but it's frustrating to give up." CreationDate="2010-09-13T13:15:38.563" UserId="1266" />
  <row Id="3213" PostId="2612" Text="@Rok Great idea for the simulation! I'll look forward for this particular issue. I suppose you already search on the r-sig-mixed mailing and Gelman's book on multilevel regression..." CreationDate="2010-09-13T13:20:52.210" UserId="930" />
  <row Id="3214" PostId="2605" Text="Good idea! Will do." CreationDate="2010-09-13T13:22:51.997" UserId="1250" />
  <row Id="3215" PostId="2613" Score="4" Text="The McNemar test is designed for paired data, isn't it? So you cannot compare its output with that of Pearson or Fisher which assumes independent samples..." CreationDate="2010-09-13T13:40:15.897" UserId="930" />
  <row Id="3216" PostId="2613" Text="OK, can you put your comment as answer?" CreationDate="2010-09-13T14:08:18.777" UserId="603" />
  <row Id="3217" PostId="2617" Score="1" Text="There are many ways to interpret (or misinterpret) your description, Daniel, so let me know where this attempt at a clarification might be wrong: You have a set of (x,y) coordinates.  You apply a function to them to obtain a set of (x,y,z) values, a &quot;terrain map.&quot;  The z values can in principle lie in [-2.25,2.25] but actually they lie in [-1.75,1.75] with the middle 40% of them in [-0.4,0.4].  You seek a transformation f so that the collection of f(z) is uniformly distributed in [-1,1].  (If this characterization is correct, there's a simple solution involving a linear transform of the rank.)" CreationDate="2010-09-13T14:58:25.503" UserId="919" />
  <row Id="3218" PostId="2612" Text="I looked now, tanks for the references! Unfortunately, there's nothing on MI in the r-sig-mixed archives; and Gelman only gives the basic formula on how to combine inferences from MI when we have variation within and between imputations given (§25.7)." CreationDate="2010-09-13T15:14:20.333" UserId="1266" />
  <row Id="3219" PostId="2607" Score="2" Text="@kwak: Have I misinterpreted that last statement?  The eigenvalues of a 3 x 3 correlation matrix, for instance, are determined by three correlation coefficients (a,b,c).  For (a,b,c) = (-0.9, 0.5, -0.1), the ev's are (2.07415, 0.91532, 0.0105346).  Changing c from -0.1 to -0.01 changes the ev's to (2.03383, 0.99151, -0.0253402).  Thus, a relatively small *decrease* in absolute value of a single coefficient can destroy the positive definiteness of the matrix." CreationDate="2010-09-13T15:16:41.043" UserId="919" />
  <row Id="3220" PostId="2619" Score="1" Text="I would recommend that you to split this across three separate questions." CreationDate="2010-09-13T15:20:50.180" UserId="5" />
  <row Id="3221" PostId="2619" Text="Actually, now that I read this a little closer, I see that the answer for each is very closely related." CreationDate="2010-09-13T15:25:22.933" UserId="5" />
  <row Id="3222" PostId="2615" Text="@edward You may want to explicitly link to the previous question on this issue as @kwak's answer on that question may be relevant here." CreationDate="2010-09-13T15:35:27.260" />
  <row Id="3223" PostId="2615" Text="Good idea! Added." CreationDate="2010-09-13T15:51:14.947" UserId="1250" />
  <row Id="3224" PostId="2615" Text="Also, on re-reading your question it is not clear what you mean by &quot;minimum eigenvalue initial correlation matrix&quot;. Could you clarify what you mean by that term?" CreationDate="2010-09-13T15:57:32.873" />
  <row Id="3225" PostId="2617" Text="@whuber - This sounds like a perfect interpretation." CreationDate="2010-09-13T16:07:00.277" UserId="1267" />
  <row Id="3226" PostId="2619" Text="I felt that the heart of the question is comparing two different distributions, I just happen to list three different ways to do it." CreationDate="2010-09-13T16:14:58.533" UserId="559" />
  <row Id="3227" PostId="2607" Text="+1 counter example: i had assumed that making the ellipse more spherical was a transformation that preserved the convexity of the associated quadratic form. Corrected (i.e. strike in the response). " CreationDate="2010-09-13T16:19:00.680" UserId="603" />
  <row Id="3228" PostId="2622" Text="I actually have the Agresti textbook on my desk right now and I have been using it. The problem is that I didn't know what specific methodology I should be using." CreationDate="2010-09-13T16:32:54.710" UserId="559" />
  <row Id="3229" PostId="2558" Score="1" Text="ah, yes, for some reason I thought the function was separable.  In higher dimensions, I only have experience in spline interpolation, e.g. for finite-element method compuations. (oddly enough, my thesis was on the topic). the big takeaway in planar meshing to minimize interpolation error is to minimize the maximum angle appearing in the triangulation. the margin does not permit me to be more detailed than this ;) you can look up any of the papers by J. Shewchuk (see 'finite element quality' at http://www.cs.cmu.edu/~jrs/jrspapers.html) or N. Walkington &amp; G. Miller, etc. etc. hth." CreationDate="2010-09-13T16:34:25.813" UserId="795" />
  <row Id="3230" PostId="2615" Text="Minimum eigenvalue initial correlation matrix: 1) calculate the eigenvalues of the correlation matrix prior to stress testing (i.e. initial matrix), 2) select the smallest eigenvalue, 3) this eigenvalue will be roughly equal to the upper stress boundary." CreationDate="2010-09-13T17:02:22.853" UserId="1250" />
  <row Id="3231" PostId="2623" Score="1" Text="if your series is non stationnary, the ACF will decline very slowly, to the point of being useless (it essentially a constant). What do you mean by 'have any meaning' ? " CreationDate="2010-09-13T17:02:54.663" UserId="603" />
  <row Id="3232" PostId="2603" Text="The location changes after each attempt" CreationDate="2010-09-13T17:47:26.360" UserId="64" />
  <row Id="3233" PostId="2607" Text="@kwak: Thank you.  Issues involving positive definiteness can be subtle." CreationDate="2010-09-13T17:48:35.510" UserId="919" />
  <row Id="3234" PostId="2613" Score="2" Text="Don't know if your udpate calls for a new answer, anyway I'll continue to comment on. For a given # of cases, power for detecting an effect can be increased by selecting more controls than cases (which also come into play in CIs width). Power increases as one moves from a 1:1 to a 1:2 matching but the benefits are less interesting above that ratio. However, it is still useful to have more controls in case of exclusion during sensitivity analysis or things like that. If you're interested in CC design, look at this Lancet series, http://j.mp/aZWRrg" CreationDate="2010-09-13T18:29:50.643" UserId="930" />
  <row Id="3235" PostId="2625" Text="The problem with this method is that I never actually have a complete map. The my noise function generates [x,y] independently from [x+n,y+m], and should have, on its own, a uniform distribution. I'm trying to make the function uniform. A uniform distribution over the map is simply a byproduct. (Granted, I didn't really make this clear in the question; +1 for the idea.)" CreationDate="2010-09-13T19:07:22.503" UserId="1267" />
  <row Id="3236" PostId="2622" Score="2" Text="@Elpezmuerto Very briefly, to complement @ars answer, question 1 can be answered with a conditional or trellis plot, e.g. sth like `dist ~ occ | isLand` using Lattice, or see the `coplot()` function in the `vcd` package -- this is for exploratory purpose; question 2 calls for a prediction model; depending on the variable you consider as your outcome, it may be logistic regression (e.g. if Y=isLand), a linear regression (e.g. if Y=distance), or directly a log-linear model providing you categorize your continuous measurement; question 3 is clearly a log-linear model as suggested by @ars." CreationDate="2010-09-13T19:10:03.573" UserId="930" />
  <row Id="3237" PostId="2622" Score="1" Text="@Elpezmuerto @ars Thanks to the work of Laura Thompson, Agresti's book is available in R too, http://j.mp/9fXheu :-)" CreationDate="2010-09-13T19:12:25.597" UserId="930" />
  <row Id="3238" PostId="2620" Text="This is looking good, but I'm having trouble interpreting my results (which I edited in above.) I think it means my starting distribution isn't exactly normal, but I can't seem to tweak it to work. I'm using `x = (1 - erf(x))/2`. Am I missing a constant scalar of some sort?" CreationDate="2010-09-13T19:17:12.150" UserId="1267" />
  <row Id="3239" PostId="2622" Score="2" Text="@chl: that's a great find! Thank you.  @Elpezmuerto:  There's a series of examples in Agresti concerning crabs -- I'm pretty sure there's a continuous variable (size of crab?) along with a color (range) and a boolean (can't recall).  So fairly close to your case -- it's probably instructive to read through those examples which span at least 2 chapters (one chapter is logistic regression I believe).  " CreationDate="2010-09-13T19:33:44.110" UserId="251" />
  <row Id="3240" PostId="2613" Text="good link, thanks." CreationDate="2010-09-13T19:35:19.127" UserId="603" />
  <row Id="3241" PostId="2603" Text="as @Edward said. I could ask such a thing in a homework actually... Now Martin, can you figure out what's the chance you never find that treasure?" CreationDate="2010-09-13T19:47:02.617" UserId="1124" />
  <row Id="3242" PostId="2622" Text="@ars These are esp. chapters 4 and 5, with carapace width and weight as continuous variables and spine condition as another categorical (ordinal) variable, used in Poisson and Logistic regression :)" CreationDate="2010-09-13T19:51:01.130" UserId="930" />
  <row Id="3243" PostId="2620" Score="1" Text="@Daniel: Don't think you're missing a constant, but scaling might help.  Think your intuition about normality is correct though.  I updated my answer with some suggestions." CreationDate="2010-09-13T19:56:09.467" UserId="251" />
  <row Id="3244" PostId="2622" Text="@Ars, I actually have been using the crab examples before you mentioned it. So atleast I know I am on the right track" CreationDate="2010-09-13T20:18:52.840" UserId="559" />
  <row Id="3245" PostId="2625" Text="@Daniel: Hmm, your actual situation is a little unclear.  Maybe you could edit the question to say more about what your doing.  Anyway, I'm operating on the assumption that the references to normality may be a bit of a red herring." CreationDate="2010-09-13T20:23:28.970" UserId="919" />
  <row Id="3246" PostId="2617" Text="@Daniel: Which probability integral are you using?  If you used the one appropriate for the upper (green) curve, then the lower (red) curve should be perfectly horizontal." CreationDate="2010-09-13T20:24:27.740" UserId="919" />
  <row Id="3247" PostId="2617" Text="@whuber - I'm using cubic interpolation between integral points generated from a hash function." CreationDate="2010-09-13T20:32:53.783" UserId="1267" />
  <row Id="3248" PostId="2496" Text="Thank you, I will look into this, I already know how to do a regression analysis of this data, but maybe the chapter also has a solution to my problem. " CreationDate="2010-09-13T20:43:56.600" UserId="1084" />
  <row Id="3249" PostId="2617" Text="@Daniel: Although I'm quite familiar with all those terms, without more details it's hard to say whether the problems with the lower red curve are to be expected (e.g., if you're using a spline with a small number of knots) or are actually serious.  I'm startled by the amount of error in the tails, though." CreationDate="2010-09-13T20:46:57.097" UserId="919" />
  <row Id="3250" PostId="2617" Text="@whuber - I'm taking the X,Y coordinates of the four corner points, running them and a seed value through MurmurHash2, and using cubic interpolation calculate get the points between the corners. Since all this can be done independently for each point, I want the *function* to be uniform, not only the map." CreationDate="2010-09-13T20:57:52.330" UserId="1267" />
  <row Id="3251" PostId="2632" Text="I think this may be the way I'm going to go. I fear you're right about the hash function not being completely normal." CreationDate="2010-09-13T20:59:45.713" UserId="1267" />
  <row Id="3252" PostId="2632" Text="@Daniel: I suspect it doesn't matter if it's normal or not. If you want to know how normal it is, generate N numbers from a normal distribution, sort it, and plot against the numbers in your table. If your table is normal you'll get a straight line (QQ plot). If not, it will not be straight. Useful thing." CreationDate="2010-09-13T21:14:12.867" UserId="1270" />
  <row Id="3253" PostId="2632" Text="@Daniel: There's a really brain-dead simple way to generate a pretty good normal random number. Just add together 12 uniform numbers from the range -0.5 to 0.5." CreationDate="2010-09-13T21:16:56.877" UserId="1270" />
  <row Id="3254" PostId="2496" Text="It'll still be a regression analysis, just one that takes unto account (i.e. corrects) the fact that 4 measurements on the same observations are correlated with one another." CreationDate="2010-09-13T21:18:45.787" UserId="603" />
  <row Id="3255" PostId="2621" Text="Nice explanation. Thanks!" CreationDate="2010-09-13T21:23:32.863" UserId="1102" />
  <row Id="3256" PostId="2574" Text="Thanks chl! Is it true in just stack exchange or its an html formatting?" CreationDate="2010-09-13T21:25:55.683" UserId="1102" />
  <row Id="3257" PostId="2617" Text="@Daniel: Aha!  Do you want the function to have a uniform distribution on each rectangle, or does it have to be uniform *globally*?  (Until now the latter was my interpretation, but your comment suggests it's the former.)  The first interpretation has a definite mathematical answer." CreationDate="2010-09-13T21:34:56.557" UserId="919" />
  <row Id="3258" PostId="750" Score="2" Text="There are three kinds of lies: Lies, damned lies and fake statistics." CreationDate="2010-09-13T22:01:51.430" UserId="1124" />
  <row Id="3259" PostId="2617" Text="@whuber - The *limit* of the distribution (if that makes any sense) should be uniform. But any single point should be also. So if I pick, say (45.32,-12.1), it should have equal probability of being any number [-1,1]. (45.32,-12.2) should also have equal probability of being any number [-1,1], though in reality it will be very very close to the number selected before. Any integral point [(0,0),(1,4),etc] already has uniform distribution. But interpolation between points skews the distribution." CreationDate="2010-09-13T22:15:47.930" UserId="1267" />
  <row Id="3260" PostId="2558" Text="*the big takeaway in planar meshing to minimize interpolation error is to minimize the maximum angle appearing in the triangulation.* meaning Delaunay triangularization ? this is nice because its quiet simple to do. Thanks for the papers, i will have a look." CreationDate="2010-09-13T22:25:20.970" UserId="603" />
  <row Id="3261" PostId="2617" Text="Interestingly enough, the distribution of points generated from the hash and 1 dimensional cubic interpolation seems to not be normal at all. It looks almost uniform, but after generating a billion values, there is still noticeable noise in the PDF." CreationDate="2010-09-13T22:25:57.757" UserId="1267" />
  <row Id="3262" PostId="2617" Text="@Daniel: I'm still lost, but I'm getting there.  Let's see: you are generating hash values (which, if it's a good hash, should act like independent uniformly distributed random numbers) at points with *integral* coordinates.  Then you're interpolating those values to all other coordinates.  You find that the distribution of all values is not uniform.  You would like to make it so by transforming the interpolated values.  Am I close now?  If so, should the transformation be predetermined or can it depend on the hashes at the integral coordinates?" CreationDate="2010-09-13T22:48:34.983" UserId="919" />
  <row Id="3263" PostId="2558" Text="A small maximum angle is implied by a large lower bound on minimum angle ($\pi - 2\alpha$ where $\alpha$ is the lower bound on angle); Delaunay Triangulations maximize this lower bound over all triangulations. Shewchuk explains this fairly well, IIRC. My thesis less so ;) You are more free however, because you are not constrained to include certain edges (meshing algorithms are usually applied to weird shapes), and can put your mesh points anywhere. Practically speaking, you want to use Strang &amp; Persson's meshing code, in Matlab, or use Shewchuk's Triangle, which is a high quality standalone." CreationDate="2010-09-13T23:12:24.730" UserId="795" />
  <row Id="3264" PostId="2619" Text="For `occupants` what you've got is an ordinal variable, so I wouldn't think of it as categorical. Especially with 8 values, it's almost continuous." CreationDate="2010-09-14T00:07:09.600" UserId="1270" />
  <row Id="3265" PostId="2617" Text="@whuber - You seem to be much less lost than you make yourself out to be. If I understand your question, I believe I want a predetermined transformation. I know how I could transform any given map so that it's uniform. But I need a transformation that generally works all the time. So the I can add on to my map indefinitely, and it will retain uniformity (and probably get more and more uniform, too.)" CreationDate="2010-09-14T00:37:28.267" UserId="1267" />
  <row Id="3266" PostId="2636" Score="3" Text="The distribution of age is certainly not log normal." CreationDate="2010-09-14T00:51:17.043" UserId="159" />
  <row Id="3267" PostId="2622" Text="@chl: totally impressed by your consistent thoroughness in all responses! You're setting a high bar though.  :)  @Elpezmuerto: totally on the right track.  I'm guessing you're more or less aware of the methodology and just need to start applying it to your data.  Then ask specific questions if you hit a wall or are unsure.  It's the only way to learn modeling, I think." CreationDate="2010-09-14T01:25:27.807" UserId="251" />
  <row Id="3268" PostId="2633" Text="Oops, you're using agglomerative methods, so the &quot;1 or 2 clusters&quot; part doesn't make sense -- the &quot;inverse&quot; applies, you don't want n-1 or n-2 singletons, etc, i.e. letting clustering work for a bit before applying validity criteria shouldn't be problematic." CreationDate="2010-09-14T01:29:13.960" UserId="251" />
  <row Id="3269" PostId="2036" Score="1" Text="@Gong-Yi: nice, thanks for updating with the answer!  " CreationDate="2010-09-14T01:32:35.497" UserId="251" />
  <row Id="3270" PostId="2639" Text="No one to explain?" CreationDate="2010-09-14T02:18:56.707" UserId="1102" />
  <row Id="3271" PostId="2639" Text="Do you really mean you have 1 observation from A? It seems like you are likely to make an error with so little data." CreationDate="2010-09-14T03:17:16.217" UserId="795" />
  <row Id="3272" PostId="2497" Text="+1, Very informative, thank you." CreationDate="2010-09-14T03:27:28.800" UserId="251" />
  <row Id="3273" PostId="2639" Text="Its not one observation, it is one distribution or realization with 40000 values." CreationDate="2010-09-14T03:38:14.150" UserId="1102" />
  <row Id="3274" PostId="2639" Text="So say a sample has size n = 40000.  Then you have one such sample of random variates from A, and 100 replicates each for B, C and D?" CreationDate="2010-09-14T03:55:52.653" UserId="251" />
  <row Id="3275" PostId="2641" Text="Great question. I would add &quot;odds&quot; and &quot;chance&quot; in there too :)" CreationDate="2010-09-14T05:28:41.807" UserId="74" />
  <row Id="3276" PostId="2641" Score="2" Text="I think you should take a look at this question http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675#675 because Likelihood is for statistic purpose and probability for probability. " CreationDate="2010-09-14T06:04:20.573" UserId="223" />
  <row Id="3278" PostId="2649" Text="I'm having trouble doing hyperlinks. Can someone offer some pointers please?" CreationDate="2010-09-14T08:47:05.083" UserId="521" />
  <row Id="3279" PostId="2650" Score="1" Text="If you need a command to do that, you better provide us at least with the language/package you use. Are we talking SAS, SPSS, Stata, SPlus, Minitab, R, ... ?" CreationDate="2010-09-14T09:36:46.557" UserId="1124" />
  <row Id="3280" PostId="2604" Score="1" Text="imagine that you had a set of points (x,y) and you were considering whether you might fit a straight line or a quadratic. If you compared the RMSE, the quadratic *would always beat the straight line*, because the line is a quadratic with one parameter set to zero: if the least squares estimate of the parameter is exactly zero (which has zero probability for continuous response), it's a tie, and in every other case the line loses. It's the same with the Poisson vs the negative binomial - a free Negative Binomial can always fit at least as well as a free Poisson." CreationDate="2010-09-14T10:28:42.733" UserId="805" />
  <row Id="3281" PostId="2651" Text="Thanks a million for your valuable answer, it really does help, for your previous remark, i'm a stata user." CreationDate="2010-09-14T10:53:58.400" UserId="1251" />
  <row Id="3282" PostId="1315" Text="You're getting lots of useful ideas for different distributions. To answer your question &quot;I want to know what statistical distribution this is&quot;, the usual method is a QQ plot, easy to generate in R." CreationDate="2010-09-14T11:25:03.753" UserId="1270" />
  <row Id="3283" PostId="1315" Text="Of course, if all you want is a smooth curve, take all your data, sort it to make an empirical CDF, smooth it, and take the first derivative numerically. That's your curve." CreationDate="2010-09-14T11:29:03.463" UserId="1270" />
  <row Id="3284" PostId="2651" Score="1" Text="In Stata, I wouldn't know a command to check that easily. When using implemented goodness of fit tests, you have to be aware you cannot compare the values across non-nested models in many cases, and definitely not across different frameworks." CreationDate="2010-09-14T11:54:18.837" UserId="1124" />
  <row Id="3285" PostId="2651" Text="@Ama Maybe worth looking at the UCLA Resources Center for Stata, http://j.mp/9xBk6U" CreationDate="2010-09-14T12:14:36.697" UserId="930" />
  <row Id="3286" PostId="2654" Text="Thanks for the suggestion. Perhaps, my description makes it sound like a sigmoidal process, but the data is typically not as smooth as a sigmoidal process might suggest. The theory and the data suggest that some form of relatively abrupt change occurs, which leads to the commencement of the rise from values around 0 to values around 1. Thus, it's something like a segmented nonlinear regression with unknown change points." CreationDate="2010-09-14T13:26:14.757" UserId="183" />
  <row Id="3287" PostId="1234" Text="In quiet periods price volatility tends to be lower, so prices closer to the mean can be considered outliers. However, because you use MAD for (presumably) an entire trading day (or even longer) these outliers are less than 3 MAD away from the median and will not be filtered. The reverse is true for busy periods with higher price movements (acceptable price movements will be filtered). Thus the problem reduces to properly estimating the MAD at all times, which is the issue to begin with. " CreationDate="2010-09-14T13:39:17.310" UserId="127" />
  <row Id="3288" PostId="2604" Text="Nice explanation - I get what you're saying now. I think my case is a little different, because I'm not doing regression to get a fit, but rather, I'm basing the extra NB parameter on outside information (I expect the var/mean ratio to be N).  Since Poisson is the special case where N=1, what I'm really comparing is the choice of N. I agree that if I was doing regression, the NB would always be able to find a better fit, because it's less constrained. In my case, where I'm choosing a value for N up front, it would certainly be possible to choose some crazy value of N that makes the fit worse." CreationDate="2010-09-14T13:40:22.313" UserId="54" />
  <row Id="3289" PostId="2654" Score="1" Text="The sigmoidal functions become step functions with appropriate prarameters, but are you meaning you are wanting something like a level shift in time series analysis?" CreationDate="2010-09-14T13:47:30.800" UserId="229" />
  <row Id="3290" PostId="2622" Text="@Ars, I felt I hit an information wall but you guys are getting me on the right track. I am also considering using a generalized linear model because max occupants is a discrete response variable that is a count as a possible outcome. Agresti (page 74) suggests generalized linear model then." CreationDate="2010-09-14T13:51:06.417" UserId="559" />
  <row Id="3291" PostId="2496" Text="Then I think think I have explained my question properly. I already understand how to do a regression analysis in a within subject design. What I don't understand is how to adjust df when comparing means (and correlating) slope values acquired in a repeated design." CreationDate="2010-09-14T14:06:42.027" UserId="1084" />
  <row Id="3292" PostId="2496" Text="In fact a simple regression line, that minimizes distance the vertical distance, is not appropriate for finding the relation between two dependent variables, one has to calculate the minimized euclidean distance." CreationDate="2010-09-14T14:08:04.477" UserId="1084" />
  <row Id="3293" PostId="2649" Score="2" Text="@thylacoleo In the editor you will see icons for bold, italic and right next to that you would see an icon that looks like a 'linked-chain'. Click that icon and paste the hyperlink in the dialog box. In the main text of the question you will something along the following lines: &quot;[link text][1]&quot; Replace link text with what you want the link to say. You can do links in comments also like so: [link text] (url to link). Remove the space between the closing ']' and the opening '(' when you actually use this methid in the comments." CreationDate="2010-09-14T14:10:57.143" />
  <row Id="3294" PostId="1234" Text="Ok: Croux had a paper on robust GARCH estimation. It basically acts as a time varying MAD estimate. here's the link, but i don't think it has been implemented. http://ideas.repec.org/a/jof/jforec/v21y2002i5p381-93.html" CreationDate="2010-09-14T14:13:03.443" UserId="603" />
  <row Id="3295" PostId="2496" Text="Adam1:&gt; once you control (by fixed or random effect) for the correlation between measurement, you will also increase the number of parameters of your model, hence decrease the number of df." CreationDate="2010-09-14T14:17:13.840" UserId="603" />
  <row Id="3297" PostId="2648" Text="Does $Y$ increases monotoneously ? (it's not clear from your description whether the first derivative of $Y$ is always $\geq0$)" CreationDate="2010-09-14T14:31:52.953" UserId="603" />
  <row Id="3298" PostId="2496" Text="Kwak:&gt; not sure how to explain myself better. I think you are trying to help me do a regression, but that is not my question - I already understand that, what I am trying to do is to apply a t-test or a correlation to individual slope values. I am asking if I am loosing df treating each slope value as 1 df even though it was calculated over repeated measurements." CreationDate="2010-09-14T14:39:53.830" UserId="1084" />
  <row Id="3299" PostId="2651" Text="Joris, Chl, many many thinks to both of you, your advices are really so valuable!!!!" CreationDate="2010-09-14T14:40:53.923" UserId="1251" />
  <row Id="3300" PostId="2615" Score="1" Text="Apparently a solution had been founded on the OR exchange forum (http://www.or-exchange.com/questions/695/analytical-solutions-to-limits-of-correlation-stress-testing). Edward:&gt; Maybe you should repost mark this question as solved" CreationDate="2010-09-14T14:53:02.267" UserId="603" />
  <row Id="3301" PostId="2617" Text="@Daniel: I don't think your hope is going to play out exactly. A fixed transformation can be devised to uniformize the data within any given cell.  That distribution will, of course, depend on the hash values at the four corners.  The whole grid thereby becomes an equally-weighted mixture of uniforms which have varying ranges (and those ranges are correlated among neighboring cells).  There will be tendency for fewer values to be generated near the extreme end of the z-interval.  However, for any given set of (x,y) coordinates, you can generate several independent sets of z's... (continued)." CreationDate="2010-09-14T15:05:16.690" UserId="919" />
  <row Id="3302" PostId="2617" Text="... With this set, you can either create a lookup table (as suggested by Mike Dunlavey) or, more simply, fit a spline to the edf for an efficient approximation to the rank transform I suggested.  That empirically-determined, simulation-based transform should work well for all future realizations of z's using the same grid (or, I believe, grids of similar size and shape)." CreationDate="2010-09-14T15:07:41.647" UserId="919" />
  <row Id="3303" PostId="2639" Text="@ars: Yes. However A,B,C,D are not necessarily related." CreationDate="2010-09-14T15:15:06.443" UserId="1102" />
  <row Id="3304" PostId="2604" Text="I'm certainly going to read up on the smooth tests of goodness of fit that you suggested though.  Thanks for the informative answers." CreationDate="2010-09-14T15:31:55.557" UserId="54" />
  <row Id="3305" PostId="2647" Text="The distinction between discrete and continuous variables disappears from the point of view of measure theory." CreationDate="2010-09-14T15:48:30.070" UserId="919" />
  <row Id="3306" PostId="2639" Text="ok, I'm going to delete my 'answer', because I have no idea what your problem is and what you are trying to solve." CreationDate="2010-09-14T16:31:54.597" UserId="795" />
  <row Id="3307" PostId="2660" Text="Just in case it helps for further reading, link to the wikipedia article for KS test: http://en.wikipedia.org/wiki/K-S_Test" CreationDate="2010-09-14T16:59:56.223" UserId="251" />
  <row Id="3308" PostId="2653" Text="Thank you very much Chl :)" CreationDate="2010-09-14T18:24:45.127" UserId="1251" />
  <row Id="3309" PostId="2661" Text="Is your question related to some extent to assessing model goodness of fit or the comparison of two nested models?" CreationDate="2010-09-14T19:28:38.110" UserId="930" />
  <row Id="3310" PostId="2628" Text="thanks for change the title Jeromy. better this way." CreationDate="2010-09-14T19:29:12.630" UserId="1154" />
  <row Id="3311" PostId="2661" Text="@chl, both, i have data and need to determine the proper fit and how well it fits" CreationDate="2010-09-14T19:43:01.677" UserId="559" />
  <row Id="3312" PostId="2657" Text="@James I don't really understand how the conclusion you reached about the issue of tossing a coin is related to the question asked. Maybe you could expand a little bit on that point? Statistical tests seem to be relevant to the extent they help to infer observed results to a larger population, whether it be a reference or general population. The question here seems to be: Given that the sample is close to the population of test takers for a fixed period of time (here, one year), is classical inference the right way to reach a decision about possible differences at the individual level?" CreationDate="2010-09-14T19:43:49.150" UserId="930" />
  <row Id="3313" PostId="2660" Text="Thank you, ars.  I figure anyone can search Wikipedia themselves, but you have made it more convenient by providing the link." CreationDate="2010-09-14T20:07:26.530" UserId="919" />
  <row Id="3314" PostId="2647" Text="@whuber yes but an answer using measure theory is not that accessible to everyone." CreationDate="2010-09-14T20:09:06.713" />
  <row Id="3315" PostId="2647" Text="@Srikant: Agreed.  The comment was for the benefit of the OP, who is a mathematician (but perhaps not a statistician) to avoid being misled into thinking there is something fundamental about the distinction." CreationDate="2010-09-14T20:36:32.933" UserId="919" />
  <row Id="3316" PostId="2663" Score="1" Text="+1 for the sensible discussion; a few points though.  Inferential machinery is unavailable for population analysis, but in many modeling cases, I'd question whether one ever has *the* population data to begin with -- often, it's not very hard to poke holes.  So it's not *always* an appeal to a super population as the means to deploy inference.  Rather than &quot;super population&quot;, the better way is to assume a data generating process yielding, for example, the year to year test taking cohorts in question.  That's where the stochastic component arises." CreationDate="2010-09-14T20:41:59.020" UserId="251" />
  <row Id="3317" PostId="2664" Text="This was an OR question to begin with :), i'm glad an elegant solution was found." CreationDate="2010-09-14T21:10:16.017" UserId="603" />
  <row Id="3318" PostId="2539" Text="Thanks for the response. Apologies for the delay in accepting your answer, I had not realized you had answered my questions." CreationDate="2010-09-14T21:15:23.703" UserId="1216" />
  <row Id="3319" PostId="2668" Text="Here I am thinking of all of frequentist/bayesian probability and all of descriptive/exploratory/inferential statistics." CreationDate="2010-09-14T21:57:29.740" UserId="1108" />
  <row Id="3321" PostId="2663" Score="1" Text="I don't think there as any disagreement here, except for the lack of inferential machinery for population analysis.  Randomization tests are applicable to populations and can reasonably test whether the data generating process is likely due to a random generating process versus a systematic generating process.  They do not assume random sampling and are a rather direct test of chance versus systematic variation.  Our traditional tests happen to stand in pretty well for them." CreationDate="2010-09-14T22:45:07.637" UserId="485" />
  <row Id="3322" PostId="2663" Text="That's true re: &quot;lack of inferential machinery&quot;.  Careless wording on my part, especially since I liked the point you made about randomization tests in your answer.  " CreationDate="2010-09-14T23:02:42.550" UserId="251" />
  <row Id="3324" PostId="2667" Text="+1 really nice question." CreationDate="2010-09-15T00:09:05.813" UserId="352" />
  <row Id="3325" PostId="2667" Text="I'm not immediately sure how to solve this, but it seems reasonable that an estimate of the cdf can still be obtained. It might not be continuous, but I don't see that as a huge problem." CreationDate="2010-09-15T00:11:30.037" UserId="352" />
  <row Id="3326" PostId="2649" Text="Thanks Srikant, that was great!" CreationDate="2010-09-15T00:22:54.683" UserId="521" />
  <row Id="3327" PostId="2641" Text="Wow, these are some _really_ good answers.  So a big thanks for that!  Some point soon, I'll pick one I particularly like as the &quot;accepted&quot; answer (although there are several that I think are equally deserved)." CreationDate="2010-09-15T01:13:24.733" UserId="386" />
  <row Id="3328" PostId="2671" Score="4" Text="For the formulas, you can get a jump start at http://meta.math.stackexchange.com/questions/480/math-markup-diagrams-etc-pointers-please ." CreationDate="2010-09-15T02:34:19.617" UserId="919" />
  <row Id="3329" PostId="2664" Text="@edward when you get a chance please update your answer with the proof for the benefit of future readers." CreationDate="2010-09-15T03:23:10.553" />
  <row Id="3330" PostId="2670" Text="You're assuming independence and a probability of 1/12 of being born in January. Neither are quite correct (e.g twins, triplets etc exist and have a tendency to be found together after birth, and the proportion of births that are in January isn't actually 1/12, though it's pretty close)." CreationDate="2010-09-15T03:24:38.067" UserId="805" />
  <row Id="3331" PostId="2635" Score="1" Text="@alone Your question is not clear. Could you please edit the text and add a bit more context?" CreationDate="2010-09-15T03:25:47.133" />
  <row Id="3332" PostId="2604" Text="Sorry about not realizing that the data didn't come into the choice of overdispersion parameter. There may be some argument for doing it your way, but if the external estimate is likely to reflect what you actually observe, the NB still may have some advantage depending on circumstances." CreationDate="2010-09-15T03:28:14.007" UserId="805" />
  <row Id="3333" PostId="2671" Text="Thanks whuber :-)" CreationDate="2010-09-15T03:47:32.543" UserId="521" />
  <row Id="3334" PostId="2606" Text="This was the right answer for my particular situation, even though Glen_b's response helped me learn more.  So more upvotes for him, accepted answer for Srikant. Everybody wins - thanks all." CreationDate="2010-09-15T04:35:43.363" UserId="54" />
  <row Id="3335" PostId="162" Score="2" Text="Spring force is proportional to the deformation, so this is not a least squares regression! " CreationDate="2010-09-15T05:31:36.913" UserId="795" />
  <row Id="3336" PostId="162" Text="Nice try! Depends on the spring. For example, if the spring constant is 1/sigma, works great ;)" CreationDate="2010-09-15T05:54:10.083" UserId="74" />
  <row Id="3337" PostId="2648" Text="@kwak The data itself varies from time point to time point. The line of best fit is generally either stable or monotonically increasing." CreationDate="2010-09-15T06:52:37.680" UserId="183" />
  <row Id="3338" PostId="2670" Text="@stat-teacher LaTeX works here. Just enclose formulas in dollar signs $...$" CreationDate="2010-09-15T07:08:27.080" UserId="183" />
  <row Id="3339" PostId="2657" Text="@chl Yes, but it seems to OP is trying to infer an underlying probability of success. The tests compare the observed proportions to the theoretical distribution to determine if there is a difference for a given level of confidence. You are testing for any form of randomness, not just sampling error randomness." CreationDate="2010-09-15T09:02:35.930" UserId="229" />
  <row Id="3340" PostId="2675" Score="2" Text="A little bit more detail would be ideal to get the question. At least, how does your data look like, who are the subscribers, what do they have to do with average time, and how is 'average time in service' defined?" CreationDate="2010-09-15T11:05:08.787" UserId="442" />
  <row Id="3342" PostId="2675" Text="I agree with Henrik. The more precise your question is, the better the answer will be." CreationDate="2010-09-15T12:12:02.503" UserId="8" />
  <row Id="3343" PostId="2662" Text="So for clarification its ok to upvote answers (you want to give rep to people for participating), you just don't think it is fair (or appropriate) to give rep for asking the question to begin with?" CreationDate="2010-09-15T12:33:17.610" UserId="1036" />
  <row Id="3344" PostId="2662" Text="@andy yes that is the intent. As the project evolves it will be necessary to reward effort as when someone provides a good answer. However, the questions itself are more about moving the project from one phase to the next. So, I do not think the qn 'asker' should get rep. " CreationDate="2010-09-15T13:03:22.257" />
  <row Id="3345" PostId="2654" Text="@James My assumption is that a latent change occurs that triggers the initial rising process. I'm also interested in identifying the exact time point when this occurs. Thus, I see this as a true step function, whereas I would have thought a sigmoidal function could be parameterised to approximate this, but actually, it would still be continuous in the sense that it has a second derivative. " CreationDate="2010-09-15T14:10:04.080" UserId="183" />
  <row Id="3346" PostId="2652" Text="@Thylacoleo Thanks. I'll explore some of these models." CreationDate="2010-09-15T14:17:19.590" UserId="183" />
  <row Id="3347" PostId="2680" Text="@Srikant: Would you mind explaining which two independent variables are being summed?" CreationDate="2010-09-15T14:39:47.490" UserId="919" />
  <row Id="3348" PostId="2681" Score="1" Text="@G. Jay: Could you elaborate a little on what &quot;standard and important result&quot; this is and how it applies in this situation?" CreationDate="2010-09-15T14:40:37.410" UserId="919" />
  <row Id="3349" PostId="2673" Text="Great point!  Note, however, that this report would only be directly relevant for a room full of six and seven year olds ;-).  Seriously, if you are concerned about the statistics of the question (which I think is great), then it gets interesting once you realize people control birth month and even day of the week (doctors induce labor, etc.) and that over time the degree of control as well as what is favored can change.  But please don't take these remarks as quibbles: I'm only pointing out some intriguing consequences of the path you have indicated here." CreationDate="2010-09-15T14:47:47.887" UserId="919" />
  <row Id="3350" PostId="2648" Text="Jeromy:&gt; then, i don't think a Sigmoid function is what you want (because of symmetry assumption built unto it)." CreationDate="2010-09-15T15:02:56.853" UserId="603" />
  <row Id="3351" PostId="2680" Text="I'd assume red and non-red cars. " CreationDate="2010-09-15T15:23:24.290" UserId="1287" />
  <row Id="3352" PostId="2680" Text="@whuber @msalters yes that is correct. My answer addresses the OP's first qn. No of cars per hour is poisson. No of cars per hr = No of red cars per hr + No of non-red cars per hr. Hence, No of red cars per hr is poisson." CreationDate="2010-09-15T15:26:18.467" />
  <row Id="3353" PostId="2682" Text="Sorry, I had thought it was clear that the color here should be taken as red or non-red. The chance of a car being red obviously doesn't depend on a previous car being blue or green, but does depend on the previous car being red." CreationDate="2010-09-15T15:29:43.007" UserId="1287" />
  <row Id="3354" PostId="2680" Text="Hadn't realized it this explicitly, but I can indeed get at the results I need by treating my &quot;all cars&quot; not as an independent variable, but as the sum of two variables with related distributions. Then σ(cars) = σ(red cars) + σ(non-red cars) = σ(red cars) * (1/p(red))" CreationDate="2010-09-15T15:37:19.087" UserId="1287" />
  <row Id="3355" PostId="1924" Text="The difference between $I$ and $H$ is that $H$ is expected second derivative, whereas I is expected product of first derivatives. $I$ is not always the same as $-H$, for instance, for uniform distribution" CreationDate="2010-09-15T15:39:12.473" UserId="511" />
  <row Id="3356" PostId="2681" Text="Sure.  I said &quot;standard&quot; because I have seen it usually covered in the first few chapters of books on stochastic processes. (Didn't mean to seem snooty;  hope it didn't come off that way).  I said &quot;important&quot; because a) Poisson processes form the foundation for renewal processes in general (where interarrival times are not necessarily exponentially dist'd but are independent with dist'n F), and b) decomposition of Poisson processes in particular gives, for instance, a handy way to  simulate continuous time Markov chains with a computer.  Durrett has a nice discussion of this point." CreationDate="2010-09-15T15:45:31.417" UserId="1108" />
  <row Id="3357" PostId="2681" Text="By the way, as whuber noticed, the process determining which cars are &quot;red&quot; and which ones &quot;not-red&quot; should be independent of the Poisson process;  otherwise, all bets are off.  :-)  The OP did mention independence, but it can be tricky to make things perfectly clear." CreationDate="2010-09-15T15:48:58.180" UserId="1108" />
  <row Id="3358" PostId="2680" Text="@Srikant: Cool!  Thanks for the reference to Raikov's Theorem.  (I might have spent a lot of time trying to think of a counterexample otherwise...)" CreationDate="2010-09-15T15:50:26.520" UserId="919" />
  <row Id="3359" PostId="2673" Text="@whuber I agree with you, my post was rather simplistic (I just found it interesting!) Two other points of interest. First, in the report it gives different birth rates classified by race. Secondly, I'm a bit confused why the Monday birth rate isn't much higher (since the rate is low over the weekend). Instead, it seems Tuesday is the day of choice." CreationDate="2010-09-15T15:54:05.950" UserId="8" />
  <row Id="3360" PostId="2681" Text="@G. Jay: Thank you!" CreationDate="2010-09-15T15:56:33.583" UserId="919" />
  <row Id="3361" PostId="162" Score="2" Text="no, no, the point is that in static equilibrium, the sum of forces would be zero; assuming equal spring constants, you would be minimizing the sum of absolute deviations, i.e. $L_1$ regression, not least squares. This ignores the fact that the springs would have to be freely floating on the stick, so they would shift so that deformation would not be entirely in the $y$ direction, resulting in something like a Principal Components fit, but with absolute errors. " CreationDate="2010-09-15T16:19:10.160" UserId="795" />
  <row Id="3363" PostId="2667" Text="Like Robin says, definition of entropy depends on choice of measure. For discrete distributions it's clear-cut, but for continuous there's no agreement on the right measure to use, so you have a spectrum of different entropies. Perhaps you should ask why you need entropy in a first place, ie, what exactly do you expect to learn from it?" CreationDate="2010-09-15T17:06:25.230" UserId="511" />
  <row Id="3364" PostId="1693" Text="thanks, useful links" CreationDate="2010-09-15T17:11:21.007" UserId="511" />
  <row Id="3365" PostId="2662" Text="@Srikant: You should post your proposal (from the question) as an answer so that people can vote on that too..." CreationDate="2010-09-15T17:11:38.743" UserId="5" />
  <row Id="3366" PostId="2228" Text="Its not only lme4. this is the instalation output now: Attaching package: 'Matrix'&#xA;&#xA;The following object(s) are masked from 'package:reshape':&#xA;&#xA;    expand&#xA;&#xA;The following object(s) are masked from 'package:base':&#xA;&#xA;    det&#xA;&#xA;&#xA;Attaching package: 'lme4'&#xA;&#xA;The following object(s) are masked from 'package:stats':&#xA;&#xA;    AIC&#xA;" CreationDate="2010-09-15T17:26:59.913" UserId="1084" />
  <row Id="3369" PostId="2673" Text="@csgillespie, would *you* (as a hypothetical obstetrician) like to come in on Monday mornings to deliver babies, lol?" CreationDate="2010-09-15T18:14:19.843" UserId="919" />
  <row Id="3371" PostId="2662" Text="@shane What you say makes sense. Perhaps, in the future, the project owner (me for now or someone else) can trim the question text to make it a bit more precise and post their own ideas as an answer. I will do it shortly. " CreationDate="2010-09-15T18:35:00.607" />
  <row Id="3375" PostId="2106" Text="updated ez, but still doesn't work 2: Anova() failed for some reason (maybe there are too few Ss?). should having too few Ss matter?" CreationDate="2010-09-15T19:17:40.950" UserId="1084" />
  <row Id="3376" PostId="2228" Text="now it works, not sure what fixed it." CreationDate="2010-09-15T19:18:06.537" UserId="1084" />
  <row Id="3378" PostId="2644" Text="I'm using a hash function to determine the integral points, which is why they have uniform distribution already. Continuity is important, however, between the points. I don't want completely random noise." CreationDate="2010-09-15T19:26:33.490" UserId="1267" />
  <row Id="3379" PostId="2688" Text="What R package did you use for HC3 estimation? `sandwich`, `contrast`?" CreationDate="2010-09-15T19:46:25.407" UserId="930" />
  <row Id="3380" PostId="2617" Text="I generated 10 billion numbers, found the best fit Normal distribution, and applied the inverse probability transform. It seems to work, but my data's not quite normal." CreationDate="2010-09-15T19:46:28.213" UserId="1267" />
  <row Id="3381" PostId="2688" Text="Another question while we are in: What is the design you are considering, I mean is there any clustering or multiple predictors, or is it a simple linear regression? This may help the reader to better understand the context of your study." CreationDate="2010-09-15T20:47:17.963" UserId="930" />
  <row Id="3382" PostId="2695" Score="3" Text="That's very layman ;-)" CreationDate="2010-09-15T21:24:57.753" UserId="88" />
  <row Id="3383" PostId="2688" Text="Have you tried re-expressing the dependent variable to stabilize variance?" CreationDate="2010-09-15T21:34:23.450" UserId="919" />
  <row Id="3384" PostId="2682" Text="@MSalters: thank you for the clarification.  I'll leave my answer up for amusement value ;-)." CreationDate="2010-09-15T21:35:11.593" UserId="919" />
  <row Id="3385" PostId="2691" Score="6" Text="Good question. I agree with the quote as well. I believe there are many people in statistics and mathematics who are highly intelligent, and can get very deep into their work, but don't deeply understand what they are working on. Or they do, but are incapable of explaining it to others.I go out of my way to provide answers here in plain English, and ask questions demanding plan English answers. " CreationDate="2010-09-15T21:43:29.863" UserId="74" />
  <row Id="3387" PostId="333" Text="Looks like Louis Bachelier started it all in 1900 http://en.wikipedia.org/wiki/Louis_Bachelier" CreationDate="2010-09-15T21:58:06.177" UserId="74" />
  <row Id="3388" PostId="2699" Score="1" Text="They have ratings break downs by gender and age groups as well.  I suspect that they may average within those groups and then taking the average of the group means for their reported score." CreationDate="2010-09-15T22:04:57.613" UserId="196" />
  <row Id="3389" PostId="2699" Text="@drnexus Yes, good point about age, but I'm not sure about gender..." CreationDate="2010-09-15T22:07:21.353" UserId="8" />
  <row Id="3390" PostId="2686" Score="1" Text="Hint: being more concise will probably help you get an answer. " CreationDate="2010-09-15T22:09:49.073" UserId="88" />
  <row Id="3394" PostId="2548" Text="@whuber (+1) ...about the first remark." CreationDate="2010-09-15T22:41:15.723" UserId="88" />
  <row Id="3395" PostId="2636" Text="I do not think you can infer age is log-normally distributed just from the fact that it is always positive. The gamma and the Weibull distributions are also always positive, so why not choosing those ones? " CreationDate="2010-09-15T23:08:27.347" UserId="582" />
  <row Id="3396" PostId="2695" Text="It's a little mathy, but the best way to understand something is to derive it." CreationDate="2010-09-16T01:08:28.433" UserId="7" />
  <row Id="3397" PostId="2695" Score="4" Text="@Vince: You have an exceptionally well-educated grandmother :-)." CreationDate="2010-09-16T01:44:58.330" UserId="919" />
  <row Id="3398" PostId="2703" Score="1" Text="Welcome to the stats site @Joel!  If you get a chance, you might also contribute to the discussion on our proposed distributed StackExchange data analysis project: http://stats.stackexchange.com/questions/2512/what-should-be-our-first-polystats-project." CreationDate="2010-09-16T02:11:33.470" UserId="5" />
  <row Id="3399" PostId="1851" Text="I'm still confused by the question, because it uses some words in ways that are inconsistent with the definitions I know.  A &quot;parametric representation&quot; of a (hyper)surface in $N$ dimensions is a function of $N-1$ variables, for instance.  A &quot;function of $L$&quot; couldn't do anything for you in this regard.  I'm thinking that perhaps you're looking for $L$ itself to be a function of $N-1$ angular parameters.  Could you perhaps clarify this or give an example of what you're looking for as a solution?" CreationDate="2010-09-16T03:12:21.817" UserId="919" />
  <row Id="3400" PostId="2687" Score="1" Text="The mean at different levels of the repeated measures factor could change, even when r = 1.0 between scores at different time points." CreationDate="2010-09-16T03:14:23.857" UserId="183" />
  <row Id="3402" PostId="2674" Text="very good. I'll sum the two and see how that works!" CreationDate="2010-09-16T04:45:18.490" UserId="795" />
  <row Id="3403" PostId="2691" Score="1" Text="This was asked on the Mathematics site in July, but not as well and it didn't get many answers (not surprising, given the different focus there).  http://math.stackexchange.com/questions/1146/intuitive-way-to-understand-principal-component-analysis " CreationDate="2010-09-16T05:03:44.287" UserId="919" />
  <row Id="3404" PostId="2688" Text="I´m using the sandwich package for the HC3 estimator. I´m using just simple linear regression. Intuitively I feel more comfortable with the bootstrap version and I guess I´ll stick with that. //thx for the input" CreationDate="2010-09-16T05:19:58.830" UserId="1291" />
  <row Id="3405" PostId="2701" Text="Note that it's often not possible to calculate the mean (without making parametric assumptions about the distribution) as the longest observed time is usually too short for the survival curve to reach zero, often it's not even close. You can estimate some of the quantiles, such as the median time, however, though which ones depends on the proportion who experience the event." CreationDate="2010-09-16T06:09:56.287" UserId="449" />
  <row Id="3407" PostId="2704" Score="1" Text="Similar to explanation by Zuur et al in Analyzing ecological data where they talk about projecting your hand on an overhead projector. You keep rotating your hand so that the projection on the wall looks pretty similar to what you think a hand should look like." CreationDate="2010-09-16T09:00:49.823" UserId="144" />
  <row Id="3408" PostId="2654" Text="@Jeromy I'm still not sure why a sigmoidal function isn't appropriate without seeing the shape of the data, but you might find this paper useful: http://www.unc.edu/~jbhill/tsay.pdf" CreationDate="2010-09-16T09:23:34.137" UserId="229" />
  <row Id="3409" PostId="2698" Text="To add to this, when you have (near-)equal semiaxes (i.e. the ellipsoid has a (near-)circular slice), it indicates that the two pieces of data corresponding to those axes have (near-)dependency; one can talk about principal axes for an ellipse, but circles only have one radius. :)" CreationDate="2010-09-16T09:43:26.663" UserId="830" />
  <row Id="3410" PostId="2712" Text="Thanks, but I'm trying to avoid 'confidence interval'. I seek for some formal distance metric to interpret error. I know what I'm gone pick, I just need metric 'how sure am I' (in %)." CreationDate="2010-09-16T09:50:22.803" UserId="1313" />
  <row Id="3411" PostId="2715" Text="If someone can help me to better tag this question..." CreationDate="2010-09-16T10:23:30.607" UserId="930" />
  <row Id="3412" PostId="2712" Score="1" Text="It seems you need some since I'm not familiar with, so I hope someone else will give you a more satisfying answer; nevertheless getting/assuming distribution would be a inevitable starting point for such an analysis." CreationDate="2010-09-16T11:46:28.047" UserId="88" />
  <row Id="3413" PostId="2717" Score="1" Text="You should consider adding the information that you have already tried UPGMA (and others that you may have tried) :)" CreationDate="2010-09-16T11:49:07.717" UserId="977" />
  <row Id="3414" PostId="2715" Text="@chl I recently made [rule-of-thumb] tag, should work here. Also, if you don't find good tags, use [for-retag]." CreationDate="2010-09-16T11:51:10.377" UserId="88" />
  <row Id="3415" PostId="2715" Text="@mbq Thanks for the retag info." CreationDate="2010-09-16T11:52:54.497" UserId="930" />
  <row Id="3416" PostId="2717" Text="done..........." CreationDate="2010-09-16T12:15:38.187" UserId="1316" />
  <row Id="3417" PostId="2715" Text="@chi Perhaps, we should have one rule per answer?" CreationDate="2010-09-16T12:20:29.343" />
  <row Id="3418" PostId="854" Score="1" Text="@Michel I agree that computing resources allow to easily perform permutation and exact tests, or long runs of Monte Carlo simulations, but here is a paper (from *Statistics in Medicine*) where the authors argued against the systematic use of Fisher's exact test and rather recommended choosing the test according to the question that is posed: http://bit.ly/9qflht." CreationDate="2010-09-16T12:20:34.290" UserId="930" />
  <row Id="3419" PostId="2715" Text="@Srikant Well, if you feel it would be more readable by others, why not. Otherwise, I can manage to summarize the poll at the end of my question. In fact, I was expecting that people would like to enumerate 3 or more rules they always keep in mind, for every kind of data analysis project." CreationDate="2010-09-16T12:26:56.053" UserId="930" />
  <row Id="3420" PostId="2680" Text="Pardoxically the Central Limit Theorem states that if you keep adding independent identically distributed Poisson variables the distribution will converge to a normal distribution. Go figure... http://en.wikipedia.org/wiki/Central_limit_theorem" CreationDate="2010-09-16T13:11:34.167" UserId="521" />
  <row Id="3421" PostId="2712" Text="I have already used mine metric.&#xA;Thanks in any case. " CreationDate="2010-09-16T13:41:34.053" UserId="1313" />
  <row Id="3422" PostId="2680" Text="@thylacoleo I do not think there is a paradox because of two reasons. CLT applies to a suitable normalization of the sum and not to the sum itself. You need to control the growth in the numerator by a suitable scale factor as otherwise there is no convergence." CreationDate="2010-09-16T13:46:44.937" />
  <row Id="3423" PostId="2715" Text="@chl It is not as much about readability but about voting. Having one rule per answer would ensure that the best rules float up. Right now, if an answer has three rules there is no  way to vote differentially for the three rules." CreationDate="2010-09-16T13:49:10.483" />
  <row Id="3424" PostId="2715" Text="@Srikant Ok. I changed my question accordingly. Sorry for @mbq and @suncoolsu who already gave multiple answers." CreationDate="2010-09-16T14:02:11.110" UserId="930" />
  <row Id="3425" PostId="2715" Text="@chl -  @mbq and @suncoolsu can always split their answers into multiple answers if they want to. This is a CW question so there are no rep issues." CreationDate="2010-09-16T14:03:32.727" />
  <row Id="3426" PostId="2675" Text="Average time in service is simply the number of days from when they first appear in our system, to when they choose to leave.  The data available is the subscriber identifier, the current state (i.e. whether or not they've left yet) and the date they joined (and also the date they left if applicable) " CreationDate="2010-09-16T14:08:10.393" UserId="1284" />
  <row Id="3427" PostId="2678" Text="Excellent thanks, i'll read up on that shortly." CreationDate="2010-09-16T14:09:54.263" UserId="1284" />
  <row Id="3428" PostId="2698" Score="1" Text="I would be more cautious here, J.M.  First, just to clarify, by &quot;near-dependency&quot; you must mean &quot;nearly independent.&quot;  This would be true for a multinormal variate, but in many cases PCA is performed with data that are markedly non-normal.  Indeed, the clustering analyses that follow some PCA calculations can be viewed as one way to assess a strong form of non-normality.  Mathematically, circles *do* have principal axes, but they are just not uniquely determined: you can choose any orthogonal pair of radii as their principal axes." CreationDate="2010-09-16T14:11:11.427" UserId="919" />
  <row Id="3429" PostId="2712" Text="Consider adding your solution as an answer; it is fair to do so, users that will read this in future will get the solution and you could earn some reputation." CreationDate="2010-09-16T14:18:51.097" UserId="88" />
  <row Id="3430" PostId="2711" Score="3" Text="Requests to &quot;avoid explanation&quot; and a reluctance to use probability statements suggest an intention to use statistics to sanctify a result rather than understand or improve it.  That can make for a difficult conversation, because statisticians do not share this point of view." CreationDate="2010-09-16T14:20:47.210" UserId="919" />
  <row Id="3431" PostId="2678" Text="Is there any software around to assist in the analysis?" CreationDate="2010-09-16T14:26:11.403" UserId="1284" />
  <row Id="3432" PostId="2729" Text="pam actually gives that out of the box (when using plot on the pam).  I'm still open for other solutions :)" CreationDate="2010-09-16T15:53:27.270" UserId="253" />
  <row Id="3433" PostId="2729" Text="@Tal I know, but you haven't mentioned it." CreationDate="2010-09-16T16:07:23.477" UserId="88" />
  <row Id="3434" PostId="2698" Text="Yes, sorry, I suppose &quot;the principal axes of a circle are indeterminate&quot; would have been a better way of putting it." CreationDate="2010-09-16T16:13:22.370" UserId="830" />
  <row Id="3435" PostId="2701" Text="@onestop: thanks, I fixed my answer to note the mean/censored issue." CreationDate="2010-09-16T16:23:59.730" UserId="251" />
  <row Id="3436" PostId="1653" Score="3" Text="@AndyF Not `lm()`, unless you move to mixed-models with the `nlme` or `lme4` package, but there's a handy way to handle repeated-measurements through appropriate specification of the `Error` term in `aov()`, see more details on Baron &amp; Li tutorial, §6.9, http://j.mp/c5ME4u" CreationDate="2010-09-16T17:25:46.703" UserId="930" />
  <row Id="3437" PostId="1723" Text="@aL3xa I think the randomization approach is better appropriate given your research field; notwithstanding the fact that usual parametric tests provide good approximation to exact permutation tests, non-parametric tests also imply some kind of assumption (e.g. on shape of the distribution). I even wonder how we might really define what is a deviation from normality in small-sample study. I think you should ask for further discussion about this point in a separate question." CreationDate="2010-09-16T17:35:55.230" UserId="930" />
  <row Id="3438" PostId="2663" Text="Thanks a lot for your answers. " CreationDate="2010-09-16T18:52:03.680" UserId="1154" />
  <row Id="3439" PostId="2663" Text="sorry. I have difficulties to understand how i would compute permutations and what conclusions I'll be able to made for them." CreationDate="2010-09-16T18:53:16.250" UserId="1154" />
  <row Id="3440" PostId="2700" Score="1" Text="+1 for really caring to answer." CreationDate="2010-09-16T18:55:35.137" UserId="851" />
  <row Id="3441" PostId="2686" Text="I agree with you entirely. My question is long and rambling but that is my problem. I do not really know what my question is. Can you help me?" CreationDate="2010-09-16T19:11:24.753" UserId="104" />
  <row Id="3442" PostId="2738" Text="Thanks gd047 a good answer indeed." CreationDate="2010-09-16T19:19:45.627" UserId="253" />
  <row Id="3443" PostId="2731" Text="Thanks ars, I didn't know of this one: +1" CreationDate="2010-09-16T19:20:11.397" UserId="253" />
  <row Id="3444" PostId="2729" Text="True, I simply forgot:  +1  :)" CreationDate="2010-09-16T19:20:34.237" UserId="253" />
  <row Id="3445" PostId="2741" Text="So a set of &quot;If-Then&quot; rules is basically a forest of single-level decision trees (i.e., single binary splits on a single predictor)?" CreationDate="2010-09-16T19:30:28.643" UserId="1121" />
  <row Id="3446" PostId="2741" Score="1" Text="@user1121 Yes, but only in case of binary classification.  " CreationDate="2010-09-16T19:35:45.173" UserId="88" />
  <row Id="3447" PostId="2729" Score="2" Text="Also I still think that this is a very good method." CreationDate="2010-09-16T19:39:16.683" UserId="88" />
  <row Id="3448" PostId="2716" Text="Hey Peter! Good to see you here. This is a really good, simple, no math answer. " CreationDate="2010-09-16T19:40:02.957" UserId="29" />
  <row Id="3449" PostId="749" Text="Indeed, this is too much!" CreationDate="2010-09-16T20:49:35.187" UserId="930" />
  <row Id="3450" PostId="2748" Score="1" Text="It might help to explain more about your study design: how are cases and controls sampled?" CreationDate="2010-09-16T20:58:19.957" UserId="449" />
  <row Id="3451" PostId="2746" Text="Perhaps you could indicate what PSD acronym stands for?" CreationDate="2010-09-16T21:03:21.310" UserId="930" />
  <row Id="3452" PostId="2746" Text="positive semi definite?" CreationDate="2010-09-16T21:05:42.587" UserId="795" />
  <row Id="3453" PostId="2738" Score="2" Text="@Tal Galili You are welcome Tal! If you find out some other method for choosing the number of clusters, share it with us." CreationDate="2010-09-16T21:12:05.827" UserId="339" />
  <row Id="3454" PostId="2751" Text="Well, we answer quite at the same time. Nice reference for `metan`!" CreationDate="2010-09-16T21:12:52.347" UserId="930" />
  <row Id="3455" PostId="2746" Text="@shabbychef Well, let's go this way; I was thinking of this too but others might not grasp the title at first sight..." CreationDate="2010-09-16T21:14:42.313" UserId="930" />
  <row Id="3456" PostId="2729" Text="I'll look into it more then - thanks :)" CreationDate="2010-09-16T21:19:10.247" UserId="253" />
  <row Id="3457" PostId="2746" Text="Correction added!" CreationDate="2010-09-16T21:22:52.917" UserId="1250" />
  <row Id="3458" PostId="796" Score="1" Text="Meta-response: Good point.  But your data might not behave like mine! :-)" CreationDate="2010-09-16T23:56:56.533" UserId="919" />
  <row Id="3459" PostId="2751" Text="Thansk everyone - I only have one group - I'm trying to calculate Orwin's (1983) modification of Rosenthal's Failsafe N and it seems like this is only possible using Cohen's d. Any suggestions?" CreationDate="2010-09-17T00:07:51.743" />
  <row Id="3460" PostId="2654" Text="@James Thanks the paper looks good" CreationDate="2010-09-17T02:13:13.983" UserId="183" />
  <row Id="3461" PostId="2761" Text="I'm sorry, what distinction are you making between the large sample formula and the small sample formula?  Are you calculating the variances using a population formula in large samples rather than using a sample estimate of the population variance?" CreationDate="2010-09-17T05:53:05.863" UserId="196" />
  <row Id="3462" PostId="2772" Text="@gappy do you have a link to the articles (free pdf or publisher)?" CreationDate="2010-09-17T07:16:12.750" UserId="183" />
  <row Id="3463" PostId="2739" Score="1" Text="Do this question and the answer refer solely to machine-learning?" CreationDate="2010-09-17T08:12:33.733" UserId="442" />
  <row Id="3464" PostId="2752" Text="+1 nice links. Thanks." CreationDate="2010-09-17T08:15:31.207" UserId="442" />
  <row Id="3465" PostId="2765" Text="Howell is very good, but not very mathematical." CreationDate="2010-09-17T08:30:35.273" UserId="442" />
  <row Id="3467" PostId="2712" Text="OK, I will do that. Thanks!" CreationDate="2010-09-17T08:45:05.710" UserId="1313" />
  <row Id="3468" PostId="2780" Text="+1 Great link, I missed this reference -- so I guess now it can be handled within R without any problem :) I put the PDF at this address in case someone want to read it: http://j.mp/bnnwsu" CreationDate="2010-09-17T09:08:23.827" UserId="930" />
  <row Id="3469" PostId="2781" Score="1" Text="Thanks. The idea of using a clustering procedure had occurred to me. I imagine the challenge would be to adequately capture and weight the possible individual-level curve features in a theoretically meaningful way. I'll have a look at see how it works in kml." CreationDate="2010-09-17T09:08:49.560" UserId="183" />
  <row Id="3470" PostId="2781" Text="Well, it works pretty well although the interface is awful (and I know the guy who build it :) -- I used it two months ago for separating clinical groups based on individual profiles on developmental measurements (Brunet-Lézine)." CreationDate="2010-09-17T09:12:09.407" UserId="930" />
  <row Id="3471" PostId="2772" Text="@gappy,@Jeromy. Links added." CreationDate="2010-09-17T10:59:02.450" UserId="159" />
  <row Id="3473" PostId="2781" Text="Thank you for the functional data analysis link." CreationDate="2010-09-17T11:16:18.850" UserId="919" />
  <row Id="3474" PostId="2775" Text="If the entries are generated from a Normal distribution rather than a uniform, the decomposition you mention ought to be SO(n) invariant (and therefore equidistributed relative to the Haar measure)." CreationDate="2010-09-17T11:47:03.640" UserId="919" />
  <row Id="3475" PostId="2781" Score="1" Text="Here's another primary reference for FDA: http://www.psych.mcgill.ca/misc/fda/  " CreationDate="2010-09-17T12:02:37.287" UserId="364" />
  <row Id="3476" PostId="2787" Text="What would be the goals of qualitative and quantitative testing? " CreationDate="2010-09-17T13:25:28.547" />
  <row Id="3477" PostId="2789" Score="1" Text="I have never heard of this approach, but it seems somehow reasonable. But, &quot;experiment&quot; should rather be a random factor, than a fixed one. These two experiments are just two randomly drawn from the population of possible experiments with this design. They are in no way fixed." CreationDate="2010-09-17T13:32:51.167" UserId="442" />
  <row Id="3478" PostId="2789" Text="Ah, you're indeed correct that experiment should be a random factor. I'll update the answer to reflect this. Thanks!" CreationDate="2010-09-17T13:35:03.570" UserId="364" />
  <row Id="3479" PostId="2792" Score="3" Text="@Henrik Baayen's textbook, *Practical Data Analysis for the Language Sciences with R*, may still be found at http://bit.ly/aQ1KGb" CreationDate="2010-09-17T13:59:15.447" UserId="930" />
  <row Id="3480" PostId="2775" Text="Interesting. Can you provide a reference for this?" CreationDate="2010-09-17T14:39:46.903" UserId="30" />
  <row Id="3481" PostId="2786" Text="@J.M. : Nice reference: This appears to be the most efficient solution (asymptotically)." CreationDate="2010-09-17T14:46:17.283" UserId="919" />
  <row Id="3483" PostId="2792" Text="Thanks. I just had it in a kind of unusable Version with &quot;Draft&quot; in the background." CreationDate="2010-09-17T17:34:43.930" UserId="442" />
  <row Id="3484" PostId="2765" Text="It's plenty mathematical. More computational." CreationDate="2010-09-17T18:30:26.650" UserId="1334" />
  <row Id="3487" PostId="2786" Text="@whuber: Heh, I picked it up from Golub and Van Loan (of course); I use this all the time to help in generating test matrices for stress-testing eigenvalue/singular value routines. As can be seen from the paper, it's essentially equivalent to QR-decomposing a random matrix like what kwak suggested, except that it is done more efficiently. There's a MATLAB implementation of this in Higham's Text Matrix Toolbox, BTW." CreationDate="2010-09-17T19:14:47.173" UserId="830" />
  <row Id="3488" PostId="2772" Score="2" Text=" 0  down vote&#xA;	&#xA;&#xA;Gappy:&gt; this is not an answer to your question, but an alternative, more recent, and often more potent in out of sample forecasting, approach to this problem:&#xA;&#xA;Large Bayesian VARs, see this recent paper http://ideas.repec.org/p/cpr/ceprdp/6326.html&#xA;" CreationDate="2010-09-17T19:56:02.003" UserId="603" />
  <row Id="3489" PostId="2786" Text="@J.M.:&gt; Thanks for the matlab implementation. Would you by any chance know of a Haar pseudo-random matrix generator in R?" CreationDate="2010-09-17T19:58:44.687" UserId="603" />
  <row Id="3490" PostId="2786" Text="@kwak: No idea, but if there's no implementation yet, it shouldn't be too hard to translate the MATLAB code to R (I can try to whip one up if there really isn't any); the only prerequisite is a decent generator for pseudorandom normal variates, which I'm sure R has." CreationDate="2010-09-17T20:06:23.177" UserId="830" />
  <row Id="3491" PostId="2786" Text="@J.M.:&gt; yes i'll probably translate it my self. Thanks for the links, Best." CreationDate="2010-09-17T20:13:19.893" UserId="603" />
  <row Id="3493" PostId="2740" Text="OK - I think this is the right conceptual answer - I hadn't appreciated the difference between the effect of the correlation across repeated measures on the within group power (improved) and between groups power (degraded).  Of course, given that this is a clinical trial, what I really care about is the power to detect a group x time interaction.  I believe I should be using in g*power the &quot;repeated measures, within-between interaction&quot; model, which does show increasing power with increasing correlation over the repeated measure." CreationDate="2010-09-17T23:11:20.400" UserId="1290" />
  <row Id="3495" PostId="527" Score="1" Text="@robin: &quot;matrix&quot; in this context is standard analytical chemistry terminology; it refers to the medium where the entities to be analyzed for (the &quot;analytes&quot;) can be found. For instance, if you're analyzing for the concentration of lead in tap water, lead is the analyte, and water is the matrix." CreationDate="2010-09-18T00:49:04.920" UserId="830" />
  <row Id="3496" PostId="2806" Score="1" Text="There are loads of public SVD algorithms. See http://en.wikipedia.org/wiki/Singular_value_decomposition#Implementations. Can't you use or adapt one of them? Also, R is open-source, and under a GPL licence, so why not borrow its algorithm if it does the job?" CreationDate="2010-09-18T03:16:10.637" UserId="159" />
  <row Id="3497" PostId="2806" Text="@Rob:  I'd like to avoid practically writing a linear algebra library, and I also want to avoid the GPL's copyleft.  Also, I've looked at bits and pieces of the R source code before and it's generally not very readable." CreationDate="2010-09-18T03:25:47.253" UserId="1347" />
  <row Id="3498" PostId="2810" Text="The Wikipedia algorithm cites this and is equivalent to this for the case of finding one principal component at a time." CreationDate="2010-09-18T03:26:12.870" UserId="1347" />
  <row Id="3499" PostId="2806" Score="1" Text="Am I missing something? You have &gt;10K features but &lt;1K samples? This means the last 9K components are arbitrary. Do you want  all 1K of the first components?" CreationDate="2010-09-18T03:59:16.640" UserId="795" />
  <row Id="3500" PostId="2806" Score="1" Text="In any event, you can't escape having to implement SVD, though thanks to much numerical linear algebra research, there are now a lot of methods to choose from, depending on how big/small, sparse/dense your matrix is, or if you want just the singular values, or the complete set of singular values and left/right singular vectors. The algorithms are not terribly hard to understand IMHO." CreationDate="2010-09-18T05:39:04.433" UserId="830" />
  <row Id="3501" PostId="1921" Text="Nice reference! In the same vein, Categorical data analysis: Away from ANOVAs. (transformation or not) and towards logit mixed models. T. Florian Jaeger (JML 2008 59), http://j.mp/a8h763." CreationDate="2010-09-18T07:42:54.967" UserId="930" />
  <row Id="3502" PostId="2799" Text="+1 the classic." CreationDate="2010-09-18T08:21:23.937" UserId="442" />
  <row Id="3503" PostId="2680" Text="@Srikant I'm pretty sure the CLT applies to sums. http://cnx.org/content/m16948/latest/" CreationDate="2010-09-18T08:28:43.107" UserId="521" />
  <row Id="3504" PostId="2680" Text="@ Srikant I'm pretty sure the CLT applies to sums. http://cnx.org/content/m16948/latest/  I think the paradox is resolved because the mean of the sum of n iid Poisson distributions is n*λ, whilst the SD  is sqrt(n*λ). See &quot;Sums of Poisson-distributed random variables&quot; http://en.wikipedia.org/wiki/Poisson_distribution#Properties  And for large  λ (say &gt;1000) the Poisson can be approximated as the normal distribution with mean λ and variance λ. http://en.wikipedia.org/wiki/Poisson_distribution#Related_distributions" CreationDate="2010-09-18T08:36:02.370" UserId="521" />
  <row Id="3506" PostId="2782" Score="3" Text="@Jeromy Is it a quotation? I'm just wondering how trying alternative testing procedures (not analysis strategies!) may not somewhat break control of Type I error or initial Power calculation. I know SAS systematically returns results from parametric and non-parametric tests (at least in two-sample comparison of means and ANOVA), but I always find this intriguing: Shouldn't we decide before seeing the results what test ought to be applied?" CreationDate="2010-09-18T09:57:47.090" UserId="930" />
  <row Id="3507" PostId="1701" Text="I used the kmeans R function with a random initialization so the results were different for each clustering attemp. Now I am reading the paper suggested above by Suresh." CreationDate="2010-09-18T10:19:05.840" UserId="221" />
  <row Id="3509" PostId="2636" Text="@Rob: @nico: I'm sure you're right. It was a poor choice of example. Typically we model pharmacometric parameters like volume and clearance." CreationDate="2010-09-18T12:53:10.543" UserId="1270" />
  <row Id="3510" PostId="2820" Score="1" Text="Hi Tal, thanks! Bonferroni does not seem appropriate to me - if one of my SNPs *is* causal and others are associated with it, there should be a signal, and Bonferroni has always looked too conservative to me (I usually prefer Holm's stepwise correction). The FDR you link to and p.adjust do not consider *combined* evidence (and the FDR still requires me to understand the correlation of my tests, the original question). multcomp may help, though at first glance it seems like it deals more with multiple tests within a *single* model, whereas I have *multiple* models. I'll dig deeper..." CreationDate="2010-09-18T14:12:53.830" UserId="1352" />
  <row Id="3511" PostId="2796" Text="Lot's of things are known about Fisher Information, it's integrals of fisher information that I'm not sure about. I'm not familiar with what you say about Fisher Information turning into some known divergence on intergration" CreationDate="2010-09-18T14:43:09.403" UserId="511" />
  <row Id="3512" PostId="2822" Text="Wow, thanks for going to all this trouble! I understand your qualms about bootstrapping, and I'm almost convinced. I think my main complication is the numerical covariate I have that will certainly be necessary (either by itself or in interaction with genotype), and that seems to rule out mt.maxT and plink, although I may need to look into plink again. But I will certainly dig through the references you provided!" CreationDate="2010-09-18T15:56:52.983" UserId="1352" />
  <row Id="3513" PostId="2822" Text="You can always work with the residuals of your GLM to get ride of your covariates, though you lost some Df that will be difficult to account or reintroduce afterwards (e.g. for computing p-value)." CreationDate="2010-09-18T16:08:45.203" UserId="930" />
  <row Id="3514" PostId="604" Text="I also don't get how you get perfect fit with a line -- what if your points don't lie on any line?" CreationDate="2010-09-18T18:06:19.457" UserId="511" />
  <row Id="3515" PostId="2822" Text="Hm, residuals from my logistic regression? Would that be legitimate?" CreationDate="2010-09-18T18:09:35.097" UserId="1352" />
  <row Id="3516" PostId="2822" Text="Yes, why not? It is not uncommon to remove the variance accounted for by other covariates and then move on 2nd-level analysis with your residualized data. It is often faster (for instance, plink is pretty slow with categorical covariates, while it's ok with continuous ones; `snpMatrix` or simply `glm()` performs quite better on this point but you cannot embed a lot of SNPs within `glm()`...); the problem is that getting the corrected p-value at the end of your 2nd analysis is rather tricky (because you have to account for the parameters already estimated). " CreationDate="2010-09-18T18:27:33.777" UserId="930" />
  <row Id="3517" PostId="2822" Text="For an illustration of how people are working with residuals, see for example p. 466 of Heck et al. Investigation of 17 candidate genes for personality traits confirms effects of the HTR2A gene on novelty seeking. *Genes, brain, and behavior* (2009) vol. 8 (4) pp. 464-72" CreationDate="2010-09-18T18:31:55.277" UserId="930" />
  <row Id="3518" PostId="2809" Text="@ars Thanks for the links; MASAL approach looks interesting." CreationDate="2010-09-18T18:35:19.887" UserId="930" />
  <row Id="3519" PostId="2811" Text="@Andy Thanks for the article of Muthén!" CreationDate="2010-09-18T18:36:32.583" UserId="930" />
  <row Id="3520" PostId="2822" Text="I think the Nyholt's paper is worth looking at. I'm just re-reading it and it seems to provide good reference and a well-founded framework. Maybe you'll have to check if there isn't more recent publications on this topic, e.g. http://j.mp/9jHTpI but I just look at the abstract." CreationDate="2010-09-18T18:39:35.820" UserId="930" />
  <row Id="3521" PostId="2822" Text="Last but not least, Nicodemus has been very active for promoting the use of Random Forests when working with SNPs, and RFs are great because you get measure of variable importance and % of classification error on resampled subjects, see e.g. http://j.mp/cstgeO. There is also a paper on how to use haplotype block instead of PCA like suggested in the aforementioned paper, http://j.mp/c4V45i" CreationDate="2010-09-18T18:46:08.160" UserId="930" />
  <row Id="3522" PostId="2826" Text="Thanks. Unfortunately, few of my data are numeric, but this is a useful phenomenon to know about. Incidentally, there was a nice programme about this on BBC Radio 4 a few years ago - looks like it's still available in RealAudio: http://www.bbc.co.uk/radio4/science/further5.shtml" CreationDate="2010-09-18T19:24:44.963" UserId="1343" />
  <row Id="3523" PostId="2778" Text="I have found the book by Maxwell and Delaney, and having read already 20-30 pages, I must say it is very nice... I'll keep on reading and I think I'll find the answers I'm looking for, thanks!" CreationDate="2010-09-18T19:58:22.763" UserId="1320" />
  <row Id="3524" PostId="2820" Text="Hello Stephan.  I understand, sorry for not helping more.  Good luck!  Tal" CreationDate="2010-09-18T20:14:01.537" UserId="253" />
  <row Id="3525" PostId="2810" Text="OK, I see the link now.  This is a fairly simple approach, and like Wikipedia mentions, there are advances upon this basic idea.  On reflection though, you're going to have to deal with some kind of trade-offs (convergence in this case).  I wonder if you're asking the right question here.   Are there really no good bindings to linalg libraries for D?" CreationDate="2010-09-18T20:26:54.510" UserId="251" />
  <row Id="3526" PostId="2828" Text="can you give more information about what attributes are available about the models? " CreationDate="2010-09-18T20:42:21.607" UserId="795" />
  <row Id="3527" PostId="1815" Score="1" Text="See also this question  (asked with specific reference to designining clinical trials): http://stats.stackexchange.com/questions/2770/good-text-on-clinical-trials/" CreationDate="2010-09-18T21:02:27.720" UserId="266" />
  <row Id="3528" PostId="2828" Text="This is kind of an open question, so my question would be -- what kind of attributes do I need to be able to measure complexity? At the most basic level, a probability model is a set of probability distributions, and I fit the model to data by picking the best fitting member" CreationDate="2010-09-18T21:32:21.587" UserId="511" />
  <row Id="3529" PostId="2835" Text="you don't need to tweak it. Quantreg will happily digest irregularly spiced time series." CreationDate="2010-09-18T21:57:54.807" UserId="603" />
  <row Id="3530" PostId="2214" Text="We need a finite mean (first moment) to compute an L-estimator (don't we?). Levy distributed r.v. do not come with such niceties. Correct me if i'm wrong. " CreationDate="2010-09-18T22:02:30.867" UserId="603" />
  <row Id="3531" PostId="2835" Text="No doubt about that. But does it do *local* quantile regression? - If not, one can play around with the weights and just calculate lots of models" CreationDate="2010-09-18T22:06:46.603" UserId="1352" />
  <row Id="3532" PostId="2830" Text="Hm...the paper is all about regression, I wonder if this can be used for discrete probability estimation. Also, I don't really understand the motivation he gives for it -- gdf is a degree of sensitivity of parameters to small changes in data, but why is it important? I could choose a different parameterization where small changes in parameters in original parameterization correspond to large changes in the new parameterization, so it'll seem more sensitive to data, but it's the same model" CreationDate="2010-09-18T22:07:39.080" UserId="511" />
  <row Id="3533" PostId="2830" Text="Yaroslav:&gt; *I could choose a different parameterization where small changes in parameters in original parameterization correspond to large changes in the new parameterization, so it'll seem more sensitive to data * can you give an example (involving an affine equivariant estimator)? Thanks," CreationDate="2010-09-18T22:24:39.893" UserId="603" />
  <row Id="3534" PostId="2830" Score="1" Text="DoF in linear regression work out to the trace of the hat matrix or the sum of sensitivities -- so the motivation/concept aren't all that far out.  Tibshirani &amp; Knight proposed Covariance Inflation Criterion which looks at covariances of model estimates instead of sensitivities.  GDF seems to have been applied in a number of model procedures like cart and wavelet thresholding (Ye's paper on adaptive model selection has more details), and in ensemble methods to control for complexity, but I don't know of any discrete estimation cases.  Might be worth trying ..." CreationDate="2010-09-18T23:11:29.817" UserId="251" />
  <row Id="3535" PostId="2830" Text="Don't know about &quot;affine equivariant estimators&quot;, but suppose we rely on maximum likelihood estimator instead. Let q=f(p) where f is some bijection. Let p0,q0 represent MLE estimate in corresponding parameterization. p0,q0 are going to have different asymptotic variances, but in terms of modeling data, they are equivalent. So the question comes down to -- in which parameterization is the sensitivity of parameters representative of expected risk?" CreationDate="2010-09-18T23:15:55.140" UserId="511" />
  <row Id="3536" PostId="2833" Text="Good advice most of the time, but what about the case of large datasets, where you can't feasibly look through all the data manually? " CreationDate="2010-09-19T00:51:01.033" UserId="1347" />
  <row Id="3537" PostId="2811" Text="@chl, No problem, Thank you for all the resources you listed here." CreationDate="2010-09-19T03:25:56.087" UserId="1036" />
  <row Id="3538" PostId="2817" Text="Good grief... constructing the covariance matrix is never the best way for SVD. I displayed an example of why explicitly forming the covariance matrix is not a good idea at math.SE: http://math.stackexchange.com/questions/3869/3871#3871 ." CreationDate="2010-09-19T03:30:04.673" UserId="830" />
  <row Id="3539" PostId="2782" Score="2" Text="@chl good point. I agree that the above rule of thumb can be used for the wrong reasons. I.e., trying things multiple ways and only reporting the result that gives the more pleasing answer. I see the rule of thumb as useful as a data analyst training tool in order to learn the effect of analysis decisions on substantive conclusions. I've seen many students get lost with decisions particularly where there is competing advice in the literature (e.g., to transform or not to transform) that often have minimal influence on the substantive conclusions." CreationDate="2010-09-19T05:45:50.497" UserId="183" />
  <row Id="3540" PostId="2106" Text="What's the error message?" CreationDate="2010-09-19T07:06:42.420" UserId="966" />
  <row Id="3541" PostId="2811" Text="@Andy Good point about latent groups. I've seen several applications of latent class analysis &amp; cluster analysis where it seems to be just carving up a continuous variable int categories such low &amp; high (http://jeromyanglim.blogspot.com/2009/09/cluster-analysis-and-single-dominant.html). However, I do have some individual-level longitudinal data which visually look like they are coming from  categorically distinct data generating processes (e.g., always high, always low, gradual increasing, low-then-abrupt-increase, etc.) and within categories there is more continuous variation of parameters." CreationDate="2010-09-19T09:22:23.023" UserId="183" />
  <row Id="3542" PostId="2844" Text="For the 2nd question, you mean the eigenvectors or the component loadings summary?" CreationDate="2010-09-19T09:40:04.747" UserId="930" />
  <row Id="3543" PostId="2844" Text="Hi chl - I mean the output from &quot;summary(pc.cr)&quot; - for some reason, I can't find it. (doing something like summary(pc.cr)[[1]] will get me only part of the table)" CreationDate="2010-09-19T09:44:07.483" UserId="253" />
  <row Id="3544" PostId="2782" Text="@chl no it's not a quotation. But I thought it was good to demarcate the rule of thumb from its rationale and caveats. I changed it to bold to make it clear." CreationDate="2010-09-19T10:11:22.950" UserId="183" />
  <row Id="3545" PostId="2781" Score="1" Text="I found this introduction to FDA link by Ramsay (2008), particularly accessible http://gbi.agrsci.dk/~shd/public/FDA2008/FDA_Sage.pdf" CreationDate="2010-09-19T10:27:29.060" UserId="183" />
  <row Id="3546" PostId="2782" Text="@Jeromy Ok, it makes sense to me to try different transformations and look if it provides a better way to account for the studied relationships; what I don't understand is to try different analysis strategies, although it is current practice (but not reported in published articles :-), esp. when they rely on different assumptions (in EFA vs. PCA, you assume an extra error term; in non-parametric vs. parametric testing, you throw away part of the assumptions, etc.). But, I agree the demarcation between exploratory and confirmatory analysis is not so clear..." CreationDate="2010-09-19T10:29:29.263" UserId="930" />
  <row Id="3547" PostId="2735" Text="Thanks for the link!" CreationDate="2010-09-19T10:36:25.740" UserId="1119" />
  <row Id="3548" PostId="2814" Text="Indeed this is the same proof as the one provided by Paul Rubin on or-exchange.com. Multiplying each correlation by a constant has a very significant drawback though - it is a very very small stress test. It is possible to apply a much larger stress test by adding a constant to each matrix, not multiplying. However, it seems the analytical solutions for that are a little less obvious. " CreationDate="2010-09-19T12:15:13.890" UserId="1250" />
  <row Id="3549" PostId="2809" Text="@ars Thanks for the article and R package links" CreationDate="2010-09-19T12:55:56.783" UserId="183" />
  <row Id="3550" PostId="2807" Text="Electrical potential difference is an example of a real-world quantity that can be negative." CreationDate="2010-09-19T13:03:21.280" UserId="582" />
  <row Id="3551" PostId="2833" Score="1" Text="@dsimcha It also depends of sample size per group. It is known, for example, that when the samples are of equal size the t-test  is robust against departure from the homoscedasticity assumption; if $n_1\neq n_2$, then the probability of a type I error will be $&lt;\alpha$ if the larger $\sigma^2$ is associated with the larger sample, and *vice versa*. See Zar, JH *Biostatistical Analysis* (4th Ed., Prentice Hall, 1998) for further references." CreationDate="2010-09-19T13:17:37.500" UserId="930" />
  <row Id="3552" PostId="2828" Score="1" Text="What, precisely, is &quot;complexity&quot;?  (This is not a flippant question!)  In the absence of a formal definition, we cannot hope to make valid comparisons of something." CreationDate="2010-09-19T13:30:33.050" UserId="919" />
  <row Id="3553" PostId="2415" Score="1" Text="Concerning the first bullet (drug trials): even people who otherwise might not be interested in medical experimentation should read the NYTimes article *New Drugs Stir Debate on Basic Rules of Clinical Trials* (http://www.nytimes.com/2010/09/19/health/research/19trial.html?pagewanted=1&amp;_r=1&amp;th&amp;emc=th ).  The statistically literate reader will immediately see the unstated implications concerning experimental design and using p-values for decision making.  There is a statistical resolution, somewhere, to the life-and-death conundrum described in this article." CreationDate="2010-09-19T13:35:16.840" UserId="919" />
  <row Id="3554" PostId="2852" Text="Could you explain how data were collected and what they intend to measure? Are the different columns of your data matrix independent measurements?" CreationDate="2010-09-19T14:23:14.670" UserId="930" />
  <row Id="3555" PostId="2849" Text="what's the other dimension of the panel (i.e. N)?" CreationDate="2010-09-19T14:41:27.093" UserId="603" />
  <row Id="3556" PostId="2852" Text="Thank you. These are biological data and each column are indepent data under the same measure method. I only want to know the difference between the first column band each one of others." CreationDate="2010-09-19T15:52:51.437" />
  <row Id="3557" PostId="2849" Text="thanks Jeromy for your reply,my second dimension is N=16." CreationDate="2010-09-19T16:27:19.520" UserId="1251" />
  <row Id="3558" PostId="1653" Text="@chl Thanks - I suspected lm couldn't do it." CreationDate="2010-09-19T16:31:31.680" UserId="702" />
  <row Id="3559" PostId="2849" Text="Sorry i mean Kwak" CreationDate="2010-09-19T16:36:57.930" UserId="1251" />
  <row Id="3560" PostId="2849" Text="Ama&gt;: imho that's much too little. Even if the observations were drawn from a single process (i.e. a single run of 4*16=64 *differentiated* observations), the granger test for causality would probably require more data points. Now in your case you would also have to account for the loss of degrees of freedom to control for the fact that your observations are coming from different process. " CreationDate="2010-09-19T16:52:24.340" UserId="603" />
  <row Id="3561" PostId="2811" Text="@Jeromy, I don't think the work I cited would discourage people from using such methods to identify latent groups. I would say the point of the work is that you can't use such methods to solely infer the existence of groups, because you will always find groups, even in random data. It is up to more subjective interpretation whether those groups you find are real or are simply artifacts of the method. You could identify some logical theories that generate such processes and then see if the groups identified fit within those theories." CreationDate="2010-09-19T16:59:23.633" UserId="1036" />
  <row Id="3562" PostId="2849" Text="sure, you are right, but is there any possibility to bypasse this issue??" CreationDate="2010-09-19T16:59:44.570" UserId="1251" />
  <row Id="3563" PostId="2846" Text="Great topic - You'll get many +1's for this question..." CreationDate="2010-09-19T17:07:02.247" UserId="253" />
  <row Id="3564" PostId="1653" Text="@AndyF `aov()` is built on top of the `lm()` function but include additional argument, called *Special* terms, like `Error`." CreationDate="2010-09-19T17:44:29.677" UserId="930" />
  <row Id="3565" PostId="2856" Text="@Andy Thanks, nice reference!" CreationDate="2010-09-19T17:45:57.227" UserId="930" />
  <row Id="3566" PostId="2828" Text="That's what I'm asking essentially" CreationDate="2010-09-19T17:49:14.097" UserId="511" />
  <row Id="3567" PostId="2856" Text="@Andy Thanks, I like the reference too!" CreationDate="2010-09-19T18:03:19.743" UserId="1307" />
  <row Id="3568" PostId="2847" Text="+1 I also use FactoMineR but I remember that when I tried it's PCA method on a really large dataset, I never got results." CreationDate="2010-09-19T18:11:01.953" UserId="339" />
  <row Id="3569" PostId="2828" Text="But can't you at least give us a hint as to what aspect of a model you're trying to capture in the word &quot;complexity&quot;?  Without that, this question is just to ambiguous to admit one reasonable answer." CreationDate="2010-09-19T18:14:42.803" UserId="919" />
  <row Id="3570" PostId="2859" Text="@whuber Nice remark! Apart from looking for a plausible interpretation of the data, do you think the use of an exact permutation test is not an option there?" CreationDate="2010-09-19T18:22:24.360" UserId="930" />
  <row Id="3571" PostId="2847" Text="@gd047 It failed for me too, though it is based on an SVD (might be optimized to handle large data set :)" CreationDate="2010-09-19T18:26:52.620" UserId="930" />
  <row Id="3572" PostId="2834" Text="@chl, would you mind clarifying what you mean by &quot;(a) our conclusions should not depend on the particular sample used for the comparison&quot;?  I'm having trouble due to the ambiguity of &quot;sample&quot; in this context: does it mean &quot;statistical sample&quot; (a set of data presumed to represent a process or population) or &quot;environmental sample&quot; (a bit of water, soil, air, or tissue, typically).  With either meaning I can't quite draw the logical line to your conclusion that this &quot;precludes any method based on correlations.&quot;" CreationDate="2010-09-19T18:36:54.130" UserId="919" />
  <row Id="3573" PostId="2859" Score="2" Text="@chl: I think an exact permutation test would work fine; I implicitly applied two of them in my response.  The reason why it should work is that the *actual* test statistic itself will not be affected by ties, because none of the data in the first column is tied within anything, so that their ranks are all uniquely determined.  For the same reason, a Wilcoxon test that does *not* adjust for ties should give correct results!" CreationDate="2010-09-19T18:40:38.190" UserId="919" />
  <row Id="3574" PostId="1653" Text="aov() is simply a wrapper to lm().  It does some contrast coding behind the scenes and packages the result in the ANOVA style.  All of it is modeled by lm().  In the article I referenced above, it tells you how to set up coding to do repeated contrasts in regression models, including lm()." CreationDate="2010-09-19T18:51:29.937" UserId="485" />
  <row Id="3575" PostId="2859" Text="@whuber Thanks for this." CreationDate="2010-09-19T18:55:54.890" UserId="930" />
  <row Id="3576" PostId="2833" Score="2" Text="@dsimcha re large datasets: depends on what you mean by &quot;large&quot;. Many observations? Use good graphics (boxplot, jittered dotplots, sunflowerplots). Many independent variables? Yes, you have a point there... But if you have so many IVs that you can't plot the DV against each IV, I'd question using an ANOVA at all - it looks like it may be hard to interpret in any case. Some smart machine learning approaches may be better (Brian D. Ripley: &quot;To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'.&quot;)" CreationDate="2010-09-19T19:13:43.703" UserId="1352" />
  <row Id="3577" PostId="2834" Text="@whuber Well, I mean the collection of observed data (e.g. glucose concentration) which, ideally, should be representative of the likely range of what is being measured. Relying on correlation may be misleading because it depends on the sampled units (e.g. patients in an hospital): we can get a higher correlation just by getting one or more extreme measurement on either scale, although the relation between the two methods is still the same. Hence, the idea is that the distribution of the measure of interest should not influence our conclusion about methods comparability. (...)" CreationDate="2010-09-19T19:16:55.103" UserId="930" />
  <row Id="3578" PostId="2834" Text="@whuber (...) What we want to assess is the *agreement beyond the data*, not the relationship in the data (I'm quoting Carstensen 2010 p. 8-9)." CreationDate="2010-09-19T19:17:16.197" UserId="930" />
  <row Id="3579" PostId="2814" Text="what exactly is your question, then?" CreationDate="2010-09-19T19:55:00.483" UserId="795" />
  <row Id="3580" PostId="854" Text="@chl: I don't have (free) access to the article. I'm very interested in the arguments. What can be more exact than Exact?" CreationDate="2010-09-19T20:11:10.880" UserId="506" />
  <row Id="3581" PostId="854" Score="1" Text="@Michel Here is the electronic copy in case you want to read it (it's very instructive and includes a lot of references): http://j.mp/b2Wbpn." CreationDate="2010-09-19T20:20:25.157" UserId="930" />
  <row Id="3582" PostId="2828" Text="amount of data needed to identify a good member of the model" CreationDate="2010-09-19T21:38:19.577" UserId="511" />
  <row Id="3583" PostId="2814" Text="Same as before - I am looking for an analytical solution for correlation matrix stress testing. Solutions have been suggested for stress testing utilising multiplication but not addition of stress levels (addition was my original question above). Either way, thank you everyone." CreationDate="2010-09-19T22:02:14.270" UserId="1250" />
  <row Id="3584" PostId="2862" Text="You really should ask your advisor. IMO, asking random strangers on the internet is not a good idea. In any case, this should be CW." CreationDate="2010-09-19T22:43:26.373" />
  <row Id="3585" PostId="2862" Score="1" Text="I wouldn't necessarily assume that this should be CW since the OP is seeking a specific answer..." CreationDate="2010-09-19T22:44:37.383" UserId="5" />
  <row Id="3586" PostId="2862" Text="Can you give us some sense of what kind of specialty your looking for?  This is a very broad field, covering anything from market microstructure effects all the way up to long-term econometrics.  I would also agree with Srikant that discussing with an advisor will be your best bet in the long run, although seeking advice here won't hurt as a start." CreationDate="2010-09-19T22:46:52.590" UserId="5" />
  <row Id="3587" PostId="1164" Score="3" Text="Tests relying on many assumptions are more powerful when those assumptions are satisfied. We can test for significance of deviation assuming that observations are IID Gaussian, which gives mean as statistic. A less restrictive set of assumptions tells us to use median. We can go further and assume that observations are correlated to get even more robustness. But each step reduces the power of our test, and if we make no assumptions at all, our test is useless. Robust tests implicitly make assumptions about data and are better than classical only when those assumptions match reality better" CreationDate="2010-09-19T23:20:07.890" UserId="511" />
  <row Id="3588" PostId="2846" Score="1" Text="@Tal .. thanks :-) But I wanted to know more and possibly expand the Wiki article, which is not too informative." CreationDate="2010-09-19T23:45:37.980" UserId="1307" />
  <row Id="3589" PostId="2814" Text="ah, I think I see: you want to add a fixed amount to every off-diagonal element? I think my other answer can be adapted to that case." CreationDate="2010-09-20T04:14:54.913" UserId="795" />
  <row Id="3590" PostId="1884" Text="See also http://en.wikipedia.org/wiki/Wald–Wolfowitz_runs_test , where the mean number of runs matches that given here." CreationDate="2010-09-20T04:51:32.113" UserId="795" />
  <row Id="3591" PostId="2221" Text="yes, it is correct. Both formulations are a little bit fishy, though. Imagine if the dart board were 30 feet in diameter. I don't think either of these models would be appropriate in that case" CreationDate="2010-09-20T05:27:01.637" UserId="795" />
  <row Id="3592" PostId="2852" Text="When you say you &quot;want to compare the difference between the first column [and the] other columns&quot; are you implying that row 1 represents 6 different measurements on the same object, subject, unit (whatever), row 2 likewise, etc? So we have only 6 things being measured. Or are we looking at 36 different units being measured? And you have not explained why you want to &quot;compare the difference...&quot;. What are you trying to test?" CreationDate="2010-09-20T06:35:54.450" UserId="521" />
  <row Id="3593" PostId="2885" Score="1" Text="I've read that paper and it seems like Stochastic Complexity fixes the problem of not being able to distinguish between models of same dimensions, but introduces a problem of sometimes not being able to distinguish between models of different dimensions.  Geometric distribution is assigned infinite complexity, surely not what we'd expect for such a simple model!" CreationDate="2010-09-20T06:45:50.200" UserId="511" />
  <row Id="3594" PostId="1857" Text="That rule of thumb mainly applies to classical methods like l2 regularized maximum likelihood, L1 regularized methods can learn effectively when number of adjustable parameters is exponential in the number of observations (ie, Miroslav Dudik, 2004 COLT paper)" CreationDate="2010-09-20T07:03:45.523" UserId="511" />
  <row Id="3595" PostId="2885" Text="Very good point about infinite stochastic complexity (SC). Solutions to the problem of infinite SC exist, but are not very elegant; Rissanen's renormalization works well in linear models, but is not easy to do for the Poisson/Geometric problem. The MML (or SMML) encoding of Poisson/Geometric data is fine though." CreationDate="2010-09-20T07:07:59.100" UserId="530" />
  <row Id="3596" PostId="2846" Text="Great incentive :)" CreationDate="2010-09-20T07:19:51.593" UserId="253" />
  <row Id="3597" PostId="2853" Text="+1 for the great answer that wraps everything up. How to apply the mentioned numerical procedures is nicely and applicably described in Tabachnik and Fidell's Using Multivariate Statistics (for SPSS and SAS): http://www.amazon.com/Using-Multivariate-Statistics-Barbara-Tabachnick/dp/0205459382/ref=sr_1_1?ie=UTF8&amp;s=books&amp;qid=1284969084&amp;sr=8-1 (But see the Erratas on the accompanied web page)" CreationDate="2010-09-20T07:53:29.283" UserId="442" />
  <row Id="3598" PostId="2888" Text="@Yaroslav Yes, the Wu et al. 2009 is a nice paper. Incidentally, I've been working on GWAS and ML during the last two years; now I'm trying to go back to clinical studies where most of the time we have to deal with imperfect measurements, missing data, and of course... a lot of interesting variables from the point of view of the physicist!" CreationDate="2010-09-20T08:37:51.073" UserId="930" />
  <row Id="3599" PostId="2836" Score="1" Text="Just a quick note: minimum description length is very powerful and useful, but it can take ages to obtain results, especially when using normalized maximum likelihood withslighltty larger datasets. I once took 10 days running FORTRAN code to get it for just one model" CreationDate="2010-09-20T09:02:18.237" UserId="447" />
  <row Id="3600" PostId="2862" Score="1" Text="And either way - I think this should be a wiki..." CreationDate="2010-09-20T09:22:51.413" UserId="253" />
  <row Id="3601" PostId="2892" Text="Do you have in mind any particular application, that is do you seek for general advices about how many EVs we need to consider apart from any application (i.e. on a pure mathematical side) or should it apply to a specific context (e.g. factor analysis, PCA, etc.)?" CreationDate="2010-09-20T10:32:13.380" UserId="930" />
  <row Id="3602" PostId="2892" Text="I'm interested more in the mathematical side, ie eigenvalues as a property of the data  underlying a correlation matrix. If it makes sense to discuss this in terms of specific context, feel free to do so too." CreationDate="2010-09-20T11:00:01.263" UserId="1250" />
  <row Id="3603" PostId="2894" Text="Any reason why you marked this as CW? The question seems specific enough to allow for *one* right answer and thus should not be CW." CreationDate="2010-09-20T14:24:05.950" />
  <row Id="3604" PostId="2834" Text="@chl: Thank you; that clarifies your position well.  This is essentially an exercise in *calibration* except that we do not appear to have a reference standard for comparison; we merely assume that the physical samples chosen by the experimenter cover some range of true concentrations.  Thus, as you write, correlation *per se* is not necessarily a useful measure of agreement among the two methods.  Typically though, especially for chemical analyses, the true concentration is known (because the experimenter introduced a known amount of a substance into the matrix)." CreationDate="2010-09-20T14:25:48.863" UserId="919" />
  <row Id="3606" PostId="2884" Score="1" Text="At this point you need to consider how the data were generated and you need to more formally specify your study objectives.  The Kruskal-Wallis test, as I recall, looks for a shift of location but assumes homogeneous variances, which obviously is not the case here (compare V4 to V6).  The data-generation process may suggest useful analyses (e.g., these data might be rescaled counts or they could be autocorrelated or the low values might represent measurement noise only)." CreationDate="2010-09-20T14:38:19.013" UserId="919" />
  <row Id="3607" PostId="2834" Text="@whuber That's right. In the absence of a gold standard, we are merely interested in the extent to which the two methods yield &quot;comparable&quot; results, hence the idea of relying on so-called limits of agreement. Although the true measure may be known in advance, each measurement instrument carry its own measurement error -- at least for those I used to deal with in the biomedical (e.g. blood glucose concentration) and neuropsychological (e.g. depression level) domain." CreationDate="2010-09-20T14:42:39.633" UserId="930" />
  <row Id="3608" PostId="2894" Text="Although it doesn't answer your question, you may find this answer to be helpful in your quest http://stats.stackexchange.com/questions/1807/how-to-perform-students-t-test-having-only-sample-size-sample-average-and-popul/1836#1836" CreationDate="2010-09-20T14:44:25.233" UserId="1036" />
  <row Id="3610" PostId="1929" Text="Srikant:&gt; No. The definition has to have two important feature of the univariate median. a) Invariant to monotone transformation of the data, b) robust to contamination by outliers. None of the extentions you propose have these. The Tukey depth has these qualities. " CreationDate="2010-09-20T16:57:58.290" UserId="603" />
  <row Id="3611" PostId="2894" Text="@Srikant Because I didn't understand the nature of CW posts. (I went just now and looked it up on meta -- oops.) I thought I was being helpful. Alas, it looks like there's no way to undo it." CreationDate="2010-09-20T17:01:14.617" UserId="1376" />
  <row Id="3612" PostId="2891" Text="Is this compositionnal data (i.e. do the lines sum to a constant)?   http://en.wikipedia.org/wiki/Compositional_data" CreationDate="2010-09-20T17:01:58.563" UserId="603" />
  <row Id="3613" PostId="2890" Text="I don't really understand &quot;model mimicry&quot;, but cross-validation seems to just postpone the task of assessing complexity. If you use data to pick your parameters *and* your model as in cross-validation, the relevant question becomes how estimate the amount of data needed for this &quot;meta&quot;-fitter to perform well" CreationDate="2010-09-20T17:09:21.123" UserId="511" />
  <row Id="3614" PostId="2902" Text="Thank you! That's a good idea. However I'm not very clear what exactly do you mean by &quot;finding some banks&quot;?" CreationDate="2010-09-20T18:01:53.107" UserId="1075" />
  <row Id="3615" PostId="1929" Text="@kwak What you say makes sense. " CreationDate="2010-09-20T18:25:06.690" />
  <row Id="3616" PostId="2902" Text="the eurozone banking market is still highly fragmented -and even more so before the crisis. You'll find plenty of mid-sized banks in various countries quoted on the local stock exchange that are very similar to one another (same size, assets, historical local community supports). It should be easy to find a large sample of such banks that are very similar to one another. You can start here; http://www.stoxx.com/indices/index_information.html?symbol=SX7P " CreationDate="2010-09-20T18:25:23.497" UserId="603" />
  <row Id="3617" PostId="2894" Text="@josh that's fine. I was just curious about your choice." CreationDate="2010-09-20T18:25:54.347" />
  <row Id="3618" PostId="2904" Text="are BetaH, BetaM, BetaL also parameters (i.e. things to estimate)?" CreationDate="2010-09-20T18:31:58.943" UserId="603" />
  <row Id="3619" PostId="1929" Text="@Srikant:&gt; Check the R&amp;S paper cited by Gary Campbell above ;). Best," CreationDate="2010-09-20T18:38:45.233" UserId="603" />
  <row Id="3620" PostId="1929" Text="@kwak On thinking some more, the taxicab metric does have the features you mentioned as it basically reduces to univariate medians. no?" CreationDate="2010-09-20T18:40:07.397" />
  <row Id="3621" PostId="2904" Text="Kwak -- yes; I'll clarify that in the question statement." CreationDate="2010-09-20T18:40:28.190" UserId="53" />
  <row Id="3622" PostId="1929" Text="@Sri:&gt; no. For instance: if you pre-multiply $x,y$ by a non singular,symmetric 2 \times 2$ matrix $A$ the ranking of the $|(x_i,y_i)-(m_x,m_y)|$ will change. The taxicab metrix is not affine invariant. The median is invariant to a even larger group of transformation (it is monotone invariant). Same for robustness, an arbitrary small contamination of your dataset by an observation $x_i$ located at $+\infty$ will cause the location of $m_x$ to shift without bounds, hence changing, again, all the rankings of the $|(x_i,y_i)-(m_x,m_y)|$. The taxicab metrix has a breakdown point of $0%$. " CreationDate="2010-09-20T18:57:42.323" UserId="603" />
  <row Id="3623" PostId="1929" Text="The median, again, has a break down point of 50 percent (the bivariate tukey depth has a break down point of 33 percent (1/(1+p)) and is monotone invariant." CreationDate="2010-09-20T18:58:37.627" UserId="603" />
  <row Id="3624" PostId="2686" Text="very good question (somehow i just saw it). +1. " CreationDate="2010-09-20T19:05:01.293" UserId="603" />
  <row Id="3625" PostId="2737" Text="Shabbychef:&gt; i think Farrel specifically mentioned an argument against this approach in his question." CreationDate="2010-09-20T19:06:00.403" UserId="603" />
  <row Id="3626" PostId="578" Text="In chemometrics the instrument responses are often nonlinear and heteroscedastic.  At a minimum that imposes a certain amount of caution when conducting and interpreting the regression." CreationDate="2010-09-20T19:06:19.173" UserId="919" />
  <row Id="3627" PostId="2896" Score="3" Text="Why would one need to invoke the CLT when sampling from a known (or assumed) Gaussian distribution?  The mean of even a sample of one will be Normally distributed!" CreationDate="2010-09-20T19:09:50.000" UserId="919" />
  <row Id="3628" PostId="2894" Text="Google &quot;adaptive sampling&quot; and &quot;sequential sampling&quot;.  If you're still stuck, include &quot;Wald&quot; as a keyword and then work forward historically (i.e, look at papers that reference Wald's work on sequential sampling, then look at papers that reference them, etc.)." CreationDate="2010-09-20T19:10:59.647" UserId="919" />
  <row Id="3629" PostId="1929" Text="@kwak I see the point. I will not delete this incorrect answer for the benefit of future readers." CreationDate="2010-09-20T19:13:21.837" />
  <row Id="3631" PostId="2908" Text="Yes, I've read his remarks in (I believe) JASA, and he seems to really try to be respectful to statisticians. Very unlike the way he treats economists." CreationDate="2010-09-20T19:18:21.143" UserId="666" />
  <row Id="3632" PostId="2908" Text="I agree that his books are full of bluster, but the ideas are IMHO valid, and in many cases, scary." CreationDate="2010-09-20T19:41:40.913" UserId="247" />
  <row Id="3633" PostId="1931" Text="phv:&gt; one can ask for 'the' generalization to preserve (in higher dimensions) some of the interesting properties of the median. This severly limits the number of candidates (see the commenting after Srikant's answer below)" CreationDate="2010-09-20T20:10:15.770" UserId="603" />
  <row Id="3634" PostId="1931" Text="@Whuber:&gt; then notion of ordering can be generalized to R^n for unimodal distributions (see my answer below)." CreationDate="2010-09-20T20:11:05.130" UserId="603" />
  <row Id="3635" PostId="1929" Text="@Srikant:&gt; there are no incorrect answer to phv's questions because there are no 'good answers' either; this area of research is still under development. I simply wanted to point out why it is still an open problem. " CreationDate="2010-09-20T20:13:22.633" UserId="603" />
  <row Id="3636" PostId="2908" Text="@John +1 Loved your blog post. Explains Taleb's ideas very well." CreationDate="2010-09-20T20:27:37.163" UserId="666" />
  <row Id="3637" PostId="2911" Text="@Shane Well, does it mean it has nothing to do on stats.stackexchange (and hence vote to close) or so?" CreationDate="2010-09-20T20:45:45.883" UserId="930" />
  <row Id="3638" PostId="2911" Text="@chl: no, not at all.  Just providing some additional links." CreationDate="2010-09-20T20:47:14.027" UserId="5" />
  <row Id="3639" PostId="2894" Text="Is there anything wrong with computing the sample mean, the sample variance and a confidence interval?" CreationDate="2010-09-20T20:48:04.647" UserId="352" />
  <row Id="3640" PostId="2911" Text="@Shane Thanks for the links! The question is open to any statistical software -- I use Python and Stata from time to time, so I wonder if confirmed users may bring interesting recommendations there." CreationDate="2010-09-20T20:51:34.757" UserId="930" />
  <row Id="3641" PostId="2904" Text="I looks to me like you should only need 3 guesses, one for when each of $H'$, $M'$ and $L'$ is largest.  But, I may simply misunderstand the question." CreationDate="2010-09-20T20:55:39.107" UserId="352" />
  <row Id="3642" PostId="2911" Text="Absolutely; although I would add that the recommendations in the above links could really apply to any statistical project (regardless of the language)." CreationDate="2010-09-20T20:58:44.827" UserId="5" />
  <row Id="3643" PostId="2911" Text="@Shane Definitely, yes! I updated my question at the same time." CreationDate="2010-09-20T21:03:00.437" UserId="930" />
  <row Id="3644" PostId="1931" Text="@kwak: could you elaborate a little?  The usual mathematical definition of an ordering of a space is independent of any kind of probability distribution, so you must implicitly have some additional assumptions in mind." CreationDate="2010-09-20T21:14:45.240" UserId="919" />
  <row Id="3645" PostId="2901" Score="1" Text="But doesn't the third (central) moment do this in a more stable and practical way?" CreationDate="2010-09-20T21:16:58.407" UserId="919" />
  <row Id="3646" PostId="2905" Score="1" Text="I'm curious: what is the trick?" CreationDate="2010-09-20T21:20:06.720" UserId="795" />
  <row Id="3647" PostId="2894" Text="@Robby McKilliam: But what data do you use?  This question arises before any data have been collected.  If you collect values one at a time and compute a CI after each new one is added to the dataset, you cannot use standard formulas for the intervals due to the correlated multiple comparisons you are making.  Thus, you need a *stopping rule* that optimizes the sum of the statistical risk of your estimator and the cost of collecting each additional sample." CreationDate="2010-09-20T21:22:44.220" UserId="919" />
  <row Id="3648" PostId="1865" Text="why has this question/answer pair been imported _in toto_ from the math SE?" CreationDate="2010-09-20T21:28:39.703" UserId="795" />
  <row Id="3649" PostId="2913" Text="i.e. the data matrix is m-by-n, where m is (usually) greater than n. Forming the covariance matrix from that will always give you an n-by-n matrix no matter how you increase or decrease m. (That's part of why using the normal equations for least-squares seduces a lot of people...)" CreationDate="2010-09-20T21:38:24.010" UserId="830" />
  <row Id="3650" PostId="2894" Text="@whuber thanks! I'm still digesting the material, but I think that this is exactly what I'm looking for. If this were an answer, I'd accept it..." CreationDate="2010-09-20T21:41:32.760" UserId="1376" />
  <row Id="3651" PostId="2913" Text="@whuber: Well, computing the sample covariance matrix does depend on the number of data points as well and this computation costs something like O(N*P^2) for N data points having P features each. Besides, many of the fast iterative algorithms for PCA (e.g., the Expectation Maximization algorithm for PCA) have a time complexity dependent on N. It can really make a difference if N is large. So one can't say in general that the computational complexity will not at all depend on the number of data points." CreationDate="2010-09-20T22:15:08.057" UserId="881" />
  <row Id="3652" PostId="2913" Text="@ebony1: But O(N*P^2) is O(N), which is exactly what I was saying.  If your N really is around 1000, you can't improve computational performance (which is already going to be fast unless P is huge) by more than a factor of 2.  In most cases that's hardly worth sweating over and in this case, for reasons ably pointed out in your problem statement, it seems like subsampling may not be a good idea." CreationDate="2010-09-20T22:18:15.610" UserId="919" />
  <row Id="3653" PostId="2901" Text="@Whuber:&gt; the third is measuring overall asymmetry, which is not the same thing as tail asymmetry. Because of the higher exponent, the value of the fifth is nearly entirely determined by the tails." CreationDate="2010-09-20T22:31:26.543" UserId="603" />
  <row Id="3654" PostId="2914" Text="Just to throw a couple of ideas on the subject, if the study discloses standard regression statistics you could focus on the t stats and p values of the coefficients.  If the RSquare of the model is high; but, one or more of the variables have a t stat &lt; 2.0; this could be a red flag.  Also, if the sign of the coefficients on some of the variables defy logic that is probably another red flag.  If the study does not disclose a hold out period for the model, that may be another red flag.  Hopefully, you will have other and better ideas. " CreationDate="2010-09-20T23:12:15.183" UserId="1329" />
  <row Id="3655" PostId="2909" Text="How accurate do you need it? Could you just simulate the walk and avoid fully-functional solutions?" CreationDate="2010-09-20T23:20:43.030" UserId="53" />
  <row Id="3658" PostId="604" Text="I understand your question, and I share your conundrum.  I would think that any method to estimate something would be able to replicate its exact estimates if the latter are used as the dependent variable.  The method you use has its own constraints (linear relationships, normal distribution, or whatever).  But, the estimates by definition reflect the constraints of the method you use.  So, you should replicate the estimates exactly.  " CreationDate="2010-09-20T23:35:40.460" UserId="1329" />
  <row Id="3659" PostId="2914" Text="One way is to see how the model performs on other (but similar) data." CreationDate="2010-09-20T23:36:49.820" UserId="5" />
  <row Id="3660" PostId="2919" Text="Is Doug Bate's mixed-effects model the same as a hierarchical model?" CreationDate="2010-09-20T23:45:52.673" UserId="1329" />
  <row Id="3661" PostId="2919" Score="1" Text="yes: http://en.wikipedia.org/wiki/Multilevel_model" CreationDate="2010-09-20T23:56:47.153" UserId="603" />
  <row Id="3662" PostId="2909" Text="good point. I don't need atomic-physics level accuracy. in fact, 3 sigfigs is probably just fine...." CreationDate="2010-09-21T00:11:39.167" UserId="795" />
  <row Id="3663" PostId="2737" Text="The question is somewhat ambiguous; my takeaway was that the OP was looking for 'buzzwords'." CreationDate="2010-09-21T00:13:21.707" UserId="795" />
  <row Id="3664" PostId="2905" Text="Do you mean the _eigenvectors_ of $\tilde{C}$ in 3? not _values_? " CreationDate="2010-09-21T00:16:51.123" UserId="795" />
  <row Id="3665" PostId="2905" Text="no. $\lambda_1$ is a scalar." CreationDate="2010-09-21T00:27:52.160" UserId="603" />
  <row Id="3667" PostId="2833" Text="@Stephan:  Good comment, +1.  Even though this specific question is about ANOVA, I was thinking on a more general level about the question of plots vs. tests when I wrote my response." CreationDate="2010-09-21T01:28:46.920" UserId="1347" />
  <row Id="3668" PostId="1931" Text="@Whuber:&gt; You state: &quot;R1 can be ordered but R2, R3, ... cannot be&quot;. R2,..,R3 can be ordered in many ways by mapping from Rn to R . One such way is the tukey depth. It has many important properties (robustness to some extend, non parametric, invariance,...) but these only hold for the case of unimodal distributions. Let me know if you want more details." CreationDate="2010-09-21T01:47:08.540" UserId="603" />
  <row Id="3669" PostId="2925" Text="what do you mean by 'verify'? does this mean the third party controls the external event? this would contradict the pmf here." CreationDate="2010-09-21T02:39:30.683" UserId="795" />
  <row Id="3670" PostId="2905" Text="This is a very odd procedure; has it been published somewhere?" CreationDate="2010-09-21T02:42:57.420" UserId="795" />
  <row Id="3671" PostId="2913" Text="@whuber: My question wasn't as much about a specific case of 500/500 split. It was just an example. Maybe something like 10% in-sample/90% out-of-sample would have been a better example. My question in general was about the reliability of the embeddings of the out-of-sample data points if you do PCA using a small sample of the data and use the learned projection to compute the embeddings of the out-of-sample data points." CreationDate="2010-09-21T02:51:28.380" UserId="881" />
  <row Id="3672" PostId="2925" Text="Obviously the third party should not control the external event for the question to make sense. As an example of what I mean, suppose we agree that the protocol is this: If it rains tomorrow in New York X=0 otherwise X=1. The draw is based on an external event and is verifiable in the sense that someone can check if it really rained post-facto." CreationDate="2010-09-21T02:58:55.517" />
  <row Id="3674" PostId="2929" Text="So, this would mean when having two distributions to compare, I could plot two Lorenz curves and calculate two Gini coefficients for them. This should then be comparable with creating one QQ plot with the advantage that the Lorenz/Gini solution would allow me to compare the mentioned distributions numerically, whereas the QQ-plot only allows me to compare visually. Correct?" CreationDate="2010-09-21T03:56:29.570" UserId="608" />
  <row Id="3675" PostId="2931" Text="Wow, thanks Andrew Moore's tutorial on cross-validation is world class." CreationDate="2010-09-21T04:08:38.987" UserId="1329" />
  <row Id="3676" PostId="2927" Text="Thanks for the links to the AIC and BIC tests.  Do they add much value vs Adjusted R Square that does a similar thing by penalizing models for adding variables?" CreationDate="2010-09-21T04:10:47.860" UserId="1329" />
  <row Id="3677" PostId="2929" Text="What question are you trying to answer? If you are asking, &quot;are these two distributions the same?&quot;, then you should use the QQ plot for a visual and the Kolmogorov-Smirnov test for the similarity of the distributions. This test basically uses the absolute maximum distance from the 45-degree line in the QQ plot as its statistic. The KS test can also be thought of as the maximum distance between two cumulative distribution functions plotted in the same figure. A CDF is a general concept; a Lorenz curve is a special case of the CDF when the domain of the random variable is from 0 to 1." CreationDate="2010-09-21T05:28:26.187" UserId="401" />
  <row Id="3678" PostId="2937" Text="Aren't these tests not restricted to hypotheses testing (testing the null hypothesis that they are the same)? I would like to have a similarity measure that quantifies their (dis)similarity in addition to simply testing IF they are (dis)similar and preferably in some comparable form (e.g. 20% dissimilar or 0.2 (i.e. a normalized value between 0 and 1 as a measure for similarity) etc.). Would you recommend to use the p value for that? Thanks in advance!" CreationDate="2010-09-21T05:40:39.733" UserId="608" />
  <row Id="3679" PostId="2929" Text="I have a distribution ranging from 1 to 1024 (screen coordinates) with frequencies of user activities at these coordinates. I produce a density plot of this distribution and, since I have a number of activities I would like to compare, I have several of such distributions. So, to answer your question, my rage is naturally not [0,1]. However, if I normalize the screen between 0% and 100% I can fit it that way (just like in economic problems where the income level is scaled between 0% and 100% in 20% steps). Does that sound reasonable? " CreationDate="2010-09-21T05:47:51.763" UserId="608" />
  <row Id="3680" PostId="2929" Text="I'm sorry, but I'm still not understanding the hypothesis that you want to test. Is it, &quot;clicks are uniformly distributed across the screen&quot;? If so, you can use the KS test to answer whether your distribution of clicks is statistically different from a normal distribution. You wouldn't need to normalize or think of this as a Lorenz curve; just think of it as a CDF and the KS test statistic is the maximum absolute distance between your CDF (with a domain from 0 to 1024) to the CDF of the uniform &quot;reference&quot; distribution (the 45-degree line)." CreationDate="2010-09-21T06:01:17.367" UserId="401" />
  <row Id="3681" PostId="2929" Text="The KS test can compare one empirical distribution to a known theoretical distribution or one empirical distribution to another empirical distribution. It is a nonparametric test for equality of distributions. See the Wikipedia page: http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test" CreationDate="2010-09-21T06:02:40.580" UserId="401" />
  <row Id="3682" PostId="2905" Text="@Shabbychev:&gt; no, but i had the opportunity to work on a related problem (just not one involving time series) a while ago (same problem as this one http://stats.stackexchange.com/questions/2572/correlation-stress-testing )" CreationDate="2010-09-21T06:45:22.317" UserId="603" />
  <row Id="3683" PostId="2935" Score="1" Text="Gaetan:&gt; Elane seems to suggest that the slope in the summer are different from the slopes in the winter (not just the intercept)." CreationDate="2010-09-21T06:50:29.373" UserId="603" />
  <row Id="3684" PostId="1735" Text="Elaine:&gt; what do you want to do? From your question i gather that you want a relative ranking of the coefficient for winter and one for summer. Is this correct, or just an ersatz for something else you have not been able to do. " CreationDate="2010-09-21T06:52:10.103" UserId="603" />
  <row Id="3685" PostId="2927" Score="1" Text="@Gaeten, Adjusted R-squared will increase when an F-test of the before vs after model is significant, so they are equivalent, except normally calculating an adjusted R-squared doesn't return a p-value." CreationDate="2010-09-21T07:33:42.250" UserId="521" />
  <row Id="3686" PostId="2927" Score="1" Text="@Gaeten - AIC &amp; BIC are more general than F-tests and adjusted R-squared which are usually limited to models fit by least squares. AIC &amp; BIC can be used for any ,model where the likelihood can be calculated and degrees of freedom can be known (or estimated)." CreationDate="2010-09-21T07:44:14.057" UserId="521" />
  <row Id="3687" PostId="2929" Text="I simply want to know how similar they are. Lets say I have clicks across the screen for one distribution and eye fixations (measured with an eye tracker) for the other. Now I want to know what is the % of similarity between them using some normalized qualifier/measure. This could be accompanied by a test of course, and I take your advice on that, but in addition to such a test, I still would like to quantize the similarity somehow. " CreationDate="2010-09-21T07:55:52.923" UserId="608" />
  <row Id="3688" PostId="2937" Text="Ampleforth:&gt; yes, you can use the p-value as a continuous (not just 0-1) and normalized measure of the differences between the two distributions. It'll be more accurate (in the sense of precise,  or powerfull) than Gini's I." CreationDate="2010-09-21T08:29:00.083" UserId="603" />
  <row Id="3689" PostId="1071" Text="Also (in addition to SV), can you please clarify why do you want to use beta-binomial analysis? The way I understand your question, you probably want to compare 1 .. 5 categories (right?)" CreationDate="2010-09-21T08:32:04.717" UserId="1307" />
  <row Id="3690" PostId="2913" Text="@Ebony:&gt; Whuber's point is still valid, in the sense that even with 1%-99% split, your proposal does not affect the (very low) O(n) complexity of the overall procedure so from a theoretical point of view your concerns are void. Shifting to a practical point of view, you can perform PCA on 100K's of observations and 10K's features in seconds on a modern computer. It's hard to imagine a setting in which splitting the sample would be worthwhile." CreationDate="2010-09-21T08:45:49.390" UserId="603" />
  <row Id="3691" PostId="2890" Text="@Yaroslaw: I don't really understand your issue with cross-validation, but to be honest I am no expert there. However, I would really like to make a point for measuring model mimicry. Therefore, see my updated answer." CreationDate="2010-09-21T08:46:20.653" UserId="442" />
  <row Id="3693" PostId="2828" Text="Yaroslav:&gt; Shouldn't that be *Amount of data needed to identify when a model is **not** a good description of the data*?" CreationDate="2010-09-21T09:01:03.227" UserId="603" />
  <row Id="3694" PostId="2896" Text="Good point! Didn't RTQ properly." CreationDate="2010-09-21T09:22:29.153" UserId="229" />
  <row Id="3695" PostId="2929" Text="@Ampleforth But it seems to me that in this case you should rather focus on some kind of distance (e.g. euclidean) between eye coordinates on the screen and mouse position for each event (I assume that you have 2 (x,y) pairs -- eye + mouse -- for each mouse click). Then you can derive a measure of overall error (e.g. RMSE)." CreationDate="2010-09-21T09:38:18.470" UserId="930" />
  <row Id="3696" PostId="2700" Score="1" Text="The tutorial was really great. Could you suggest any further tutorials as a follow-up?" CreationDate="2010-09-21T09:54:57.307" UserId="1250" />
  <row Id="3698" PostId="2913" Score="1" Text="@kwak: Okay, consider this -- if P &gt;&gt; N, then it's suggested that you do PCA using the eigen-decomposition of X'*X (which is NxN) rather of the covariance matrix X*X' (which is PxP). In this setting, the computational complexity is going to be O(N^3). If N is reasonably large (even though smaller than P) I don't think you can ignore the cubic complexity." CreationDate="2010-09-21T11:21:13.590" UserId="881" />
  <row Id="3700" PostId="2737" Text="I google searched time-based boxcar and came up with very little. Can you point us to a worked example?" CreationDate="2010-09-21T13:03:43.460" UserId="104" />
  <row Id="3701" PostId="2907" Text="@kwak do you have a url that you can point me at. I have never done a time series analysis in my life. But it looks as if this is what I need to teach myself. Is there a worked example that I can look at that uses time series filters?" CreationDate="2010-09-21T13:06:25.183" UserId="104" />
  <row Id="3702" PostId="2884" Text="Thank you,whuber!" CreationDate="2010-09-21T13:14:27.787" />
  <row Id="3703" PostId="2940" Text="Does this meet the criteria of verifiability? Potentially, someone can argue that I fudged the recording to get the draw I wanted. BTW, I believe that the norms on SE sites *do not require* you to answer OP by name. However, feel free to address the OP by name if you feel that makes you comfortable. " CreationDate="2010-09-21T13:57:40.130" />
  <row Id="3704" PostId="2936" Text="What would be a specific example for an external event that I could use for $Y$?" CreationDate="2010-09-21T13:58:31.033" />
  <row Id="3705" PostId="2913" Score="3" Text="@ebony1: It is true that the situation changes when P &gt;&gt; N.  That's an interesting refinement of your original problem, because in effect any model is going to be a gross overfitting: more explanatory variables than data.  That does not augur well for the prospects of obtaining stable results via subsampling.  At a minimum, it points to the importance of retaining as much data as possible." CreationDate="2010-09-21T14:08:24.857" UserId="919" />
  <row Id="3706" PostId="2944" Score="1" Text="@2 is clever.  It might even be better to draw correlated random subsamples: e.g., randomly partition the data into $k$ parts of approximately $n/k$ values each and run $k$ PCAs for a computational cost of only $1 / k^2$ of a full PCA." CreationDate="2010-09-21T14:11:01.777" UserId="919" />
  <row Id="3707" PostId="2937" Score="2" Text="@shabbychef: +1 for the second suggestion.  The Q-Q plot is one of the strongest and most insightful ways to compare two empirical distributions, because it graphically displays variations between the two throughout their range.  It is also a natural graphical adjunct to the KS test, whose statistic is the maximum deviation between the Q-Q plot and its reference line (Y==X)." CreationDate="2010-09-21T14:15:28.080" UserId="919" />
  <row Id="3709" PostId="2851" Text="Thanks! I will try to get hold of the book." CreationDate="2010-09-21T14:33:33.757" UserId="1084" />
  <row Id="3710" PostId="2940" Text="You are right, it is better to answer the question, not the person asking it." CreationDate="2010-09-21T14:53:13.880" UserId="603" />
  <row Id="3711" PostId="2907" Text="@farel:&gt; if you click on the blue words, you will be directed to actual websites. The last document in particular contains working out examples (repeated in the edted version of my post)." CreationDate="2010-09-21T14:57:17.853" UserId="603" />
  <row Id="3712" PostId="2939" Text="Note that (as I understand it), the K-S test is for testing empirical data against an a priori distribution. It's not appropriate for comparing two empirical distributions, nor is it appropriate to compare empirical data against an a priori distribution whose parameter values were estimated from the empirical data." CreationDate="2010-09-21T15:07:19.677" UserId="364" />
  <row Id="3713" PostId="2947" Text="@Mike Your paper includes very nice figures :)" CreationDate="2010-09-21T15:19:47.427" UserId="930" />
  <row Id="3714" PostId="2936" Score="1" Text="As an example I first thought of old numbers games where they used some decimal point of specific stock reports in the news paper. Of course you would worry about the true randomness of this metric though, so I imagine there are better examples." CreationDate="2010-09-21T15:31:31.480" UserId="1036" />
  <row Id="3715" PostId="2915" Text="Should this be CW as it is asking for a collection of resources? What would constitute a 'correct' answer for such a question?" CreationDate="2010-09-21T15:37:31.407" />
  <row Id="3716" PostId="2939" Score="1" Text="@Mike, you can use the K-S test to compare two empirically derived distributions, see Charlie's prior answer and comments http://stats.stackexchange.com/questions/2918/lorenz-curve-qq-plot/2929#2929" CreationDate="2010-09-21T15:39:48.407" UserId="1036" />
  <row Id="3717" PostId="2944" Text="@Whuber:&gt; thanks. However i have two questions: a) PCA on the full dataset is not the same as the mean of PCA on the parts (do you know of a way to recover the 'total-sample' pca from the 'partial-sample' ones?). b) wouldn't we (under your proposal) lose the interpretation of the 'partial-sample' PCA in terms of sampling variation ? In other words, do the distribution of the 'partial-sample' pca constructed from correlated sample have a statistical interpretation ?" CreationDate="2010-09-21T15:41:56.580" UserId="603" />
  <row Id="3718" PostId="2947" Text="@chi: They were all created in R using ggplot2. It's a fantastic graphics production system!" CreationDate="2010-09-21T15:42:11.167" UserId="364" />
  <row Id="3719" PostId="2915" Text="I am new here, so I don't know what CW is?  Does it mean Community Wiki or something?  And, what does this mean?  Should I have posted this question somewhere else within this site?  And, where? and how? " CreationDate="2010-09-21T15:50:30.073" UserId="1329" />
  <row Id="3720" PostId="2935" Text="kwak, you raise an excellent point.  I have been told that linear regression can handle that whereby you have dummy variables for the intercepts of the seasons and another one for their respective slope coefficients.  Unfortunately, I have not used such a method.  So, I can't explain it further. " CreationDate="2010-09-21T15:55:53.830" UserId="1329" />
  <row Id="3721" PostId="2939" Text="@Andy, Ah, I took point 3 from http://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm as having the corollary that you can't compare two empirical CDFs, but I see that my assumption wasn't appropriate. Good to know, thanks! " CreationDate="2010-09-21T16:02:47.997" UserId="364" />
  <row Id="3722" PostId="2942" Text="Thank you, the paper is quite helpful." CreationDate="2010-09-21T16:05:14.467" UserId="1381" />
  <row Id="3724" PostId="2915" Score="1" Text="@gaetan When you ask a question or edit a question there is a check box that appears at the bottom of the question box towards the right hand side that says 'Community wiki'. You need to check that box *if* you wish to make a question or answer CW. The idea behind CW is enable the community to edit the questions/answers collaboratively and to prevent users from gaining rep for certain type of questions. See this link for more details: http://meta.stats.stackexchange.com/questions/6/what-should-our-faq-contain/198#198  " CreationDate="2010-09-21T16:18:24.963" />
  <row Id="3725" PostId="2937" Text="@whuber: good point about KS statistic being the maximum deviation. " CreationDate="2010-09-21T16:26:45.117" UserId="795" />
  <row Id="3726" PostId="2937" Text="You can use the K-S test statistic as a measure of dissimilarity. For a two-sided test, the K-S test statistic is the largest absolute difference between the two distribution functions. As the distribution functions lie in [0,1], the difference also lies in [0,1]. If the two distributions differ slightly the p-value will tend to 0 as the sample sizes increase (since the power of the test increases) but the K-S test statistic will tend to a limit." CreationDate="2010-09-21T16:33:21.053" UserId="449" />
  <row Id="3728" PostId="2920" Text="For MSD, if MSD = t_{alpha,2n-2}*sd*sqrt(2/n), is SE = MSD*n/(t_{alpha,n}*sqrt(2)) correct?" CreationDate="2010-09-21T16:35:22.887" UserId="1381" />
  <row Id="3729" PostId="2950" Score="2" Text="kmeans() (you can find it by typping ?? folowed by the name of what you want as in ??kmean)" CreationDate="2010-09-21T17:36:02.947" UserId="603" />
  <row Id="3730" PostId="2950" Text="Another useful way to find functions is the `apropos` function.  " CreationDate="2010-09-21T17:38:11.993" UserId="5" />
  <row Id="3731" PostId="2950" Score="3" Text="@kwak: why not post your comment as an answer?" CreationDate="2010-09-21T17:38:45.440" UserId="5" />
  <row Id="3733" PostId="2945" Score="1" Text="I would agree that Cross Validation solves the problem of measuring model complexity. Maybe I'm asking the wrong question, because a practical question is the sample complexity of the fitting procedure. Cross-validated learner would try different models and pick the one with lowest cross validation error. Now the question is -- is this learner more likely to overfit than one that fits a single model by maximum likelihood? " CreationDate="2010-09-21T18:30:22.880" UserId="511" />
  <row Id="3735" PostId="2945" Text="Yaroslav Bulatov:&gt; yes but you can use ML only to compare nested models. Insofar as you specified (in your question) mentioned models with the same number of parameters, then they cannot be nested. " CreationDate="2010-09-21T18:39:43.600" UserId="603" />
  <row Id="3737" PostId="2851" Score="1" Text="Hurlbert, S.H. (1984) Pseudoreplication and the design of ecological field experiments. Ecological Monographs, 54, 187–211." CreationDate="2010-09-21T20:34:04.193" UserId="603" />
  <row Id="3738" PostId="2953" Score="4" Text="You can also do `RSiteSearch(&quot;k-means&quot;)`" CreationDate="2010-09-21T20:42:23.407" UserId="582" />
  <row Id="3739" PostId="2953" Text="Thanks: i did know of this feature ;)" CreationDate="2010-09-21T21:09:33.237" UserId="603" />
  <row Id="3740" PostId="2950" Text="@all:&gt; i propose to change the title of the post, so as to make it easier to find for someone looking for the R implementation of other procedures (beside k-means). " CreationDate="2010-09-21T21:11:13.487" UserId="603" />
  <row Id="3741" PostId="2939" Text="However, point 3 does imply that you can't use K-S to test whether your data come from a normal distribution with mean and sd *estimated from the data*. This is a popular error among the psychology students I meet." CreationDate="2010-09-21T21:28:00.580" UserId="1352" />
  <row Id="3742" PostId="2945" Text="Another issue is that cross-validation doesn't add to our understanding of model complexity. Measures like AIC/BIC make it clear that lots of parameters encourage overfitting. Now the question becomes -- what aspects of the model besides dimension increase capacity to overfit?" CreationDate="2010-09-21T21:56:49.153" UserId="511" />
  <row Id="3743" PostId="1858" Text="@mbq Did you find any paper finally?" CreationDate="2010-09-21T22:00:31.033" UserId="930" />
  <row Id="3744" PostId="2950" Text="@kwak Yes, I think it's better to change the title because k-means vs. kNN seems ambiguous." CreationDate="2010-09-21T22:07:11.987" UserId="930" />
  <row Id="3745" PostId="1858" Text="@chl Not yet; but thanks for reminder." CreationDate="2010-09-21T22:11:37.700" UserId="88" />
  <row Id="3747" PostId="2945" Text="Yaroslav:&gt; Again, very good point. " CreationDate="2010-09-21T22:28:07.310" UserId="603" />
  <row Id="3748" PostId="1858" Text="@mbq There's no hurry :) Didn't find anything interesting myself; maybe Pubmed isn't the right search engine for this particular case..." CreationDate="2010-09-21T22:34:03.017" UserId="930" />
  <row Id="3749" PostId="1858" Text="@chl That's also my problem here. It really seems n&lt;&lt;p has became a synonym for biomed data." CreationDate="2010-09-21T22:46:27.957" UserId="88" />
  <row Id="3750" PostId="2888" Text="BTW, I just came across a paper that made me think of this question...it's very rare for Machine Learning papers to talk about confidence intervals, but here's a notable exception http://www.ncbi.nlm.nih.gov/pubmed/19519325" CreationDate="2010-09-21T22:50:39.667" UserId="511" />
  <row Id="3752" PostId="2954" Text="I believe the k nearest-neighbors test is a multivariate analysis. Perhaps I have confused cluster analysis with classification, but I don't think either apply in this situation. I already have categorized the variables into groups, and now want to see if these groups statistically differ from one another. This test is different that an ANOVA or linear model. Any suggestions? Thanks." CreationDate="2010-09-21T23:03:10.253" />
  <row Id="3754" PostId="2820" Text="Hello Stephan,&#xA;I still think you can still use the method=BY (for Benjamini Hochberg Yekuteli Procedure) in p.adjust in R, as pointed by Tal.&#xA;&#xA;Definitely, using Bonferroni can be conservative. " CreationDate="2010-09-22T03:44:28.350" UserId="1307" />
  <row Id="3755" PostId="2960" Text="But the question is: how would you verify that to someone on the other side of the world?  See item b) in the question." CreationDate="2010-09-22T05:16:54.667" UserId="71" />
  <row Id="3756" PostId="2963" Text="Tangent: Dirk Eddelbuettel wrote an R package (http://cran.r-project.org/web/packages/random/index.html) to access random numbers.  Not verifiable, however." CreationDate="2010-09-22T05:27:53.243" UserId="71" />
  <row Id="3757" PostId="2962" Score="2" Text="I'm not aware of such a function, but perhaps someone could look at the formulas in Olejnik and Algina  (2003) http://www.cps.nova.edu/marker/olejnik2003.pdf and write a function" CreationDate="2010-09-22T05:34:03.457" UserId="183" />
  <row Id="3758" PostId="2962" Score="2" Text="@Jeromy Nice reference! This one is worth looking too: Recommended effect size statistics for repeated measures designs (BRM 2005 37(3)), http://j.mp/cT9uEQ " CreationDate="2010-09-22T06:56:03.430" UserId="930" />
  <row Id="3759" PostId="2954" Text="@Sharon I'm a little bit puzzled about the term 'randomization test' (the only paper I found using this keyword, http://j.mp/9OCB9I, seems rather to focus on power considerations) -- I rest on my idea that you could use cross-validation to assess the classification accuracy in your data, e.g. leave-one-out or k-fold CV for this purpose (available in `kknn` and `class` packages). Now, I don't know on which basis your classes were defined, but it is generally meaningless to use ANOVA to test if the groups differ on the variables that were used to define class membership." CreationDate="2010-09-22T07:30:34.920" UserId="930" />
  <row Id="3760" PostId="2962" Score="2" Text="@chl Thanks. Apparently, ezANOVA() in the ez package in R reports generalised eta squared." CreationDate="2010-09-22T07:49:19.490" UserId="183" />
  <row Id="3761" PostId="2958" Text="Can you provide more detail Justin?" CreationDate="2010-09-22T07:54:06.930" UserId="521" />
  <row Id="3763" PostId="2954" Text="@Sharon; can you explain your data a bit better (I'm a (sort of) ecologist and use isotopes all the time so you can be quite technical if it helps)?" CreationDate="2010-09-22T11:54:48.233" UserId="1390" />
  <row Id="3765" PostId="2958" Text="What else do you know about the distribution apart from its 16th percentile (which is what i assume you mean by &quot;16% quartile&quot;)?" CreationDate="2010-09-22T12:09:02.947" UserId="449" />
  <row Id="3766" PostId="2963" Text="@Matt This works perfectly." CreationDate="2010-09-22T12:56:36.803" />
  <row Id="3768" PostId="2967" Text="Thanks for the great suggestion. I did a quick analysis using isoreg in R. It was certainly robust to the outliers (i.e., the very slow times). It was a little bit more stepped than would seem appropriate if it was updating its estimate based on every observation. I also assume that it also wouldn't deal well with the situation where the runner experiences an injury or perhaps changes their running style and thus the runner's latent performance actually declines." CreationDate="2010-09-22T13:22:25.880" UserId="183" />
  <row Id="3769" PostId="2970" Text="Thanks. I have a concern with the nonlinear regression approach (or the linear regression of transformed x and y). 1. It assumes the functional form is known; 2. least squares is influenced by outliers, which I would argue are less relevant to the quantity of interest; 3. The quantity of interest is more like median of a positively skewed running time distribution with outliers, whereas least squares would estimate the mean." CreationDate="2010-09-22T13:28:48.333" UserId="183" />
  <row Id="3770" PostId="2967" Text="Jeromy:&gt; first point/ have you tried fudging with the bandwith selector ? Second point/ Injury is a valid example (running style is not since it does not impact latent [potential] speed): see edited answer." CreationDate="2010-09-22T13:30:09.703" UserId="603" />
  <row Id="3771" PostId="2967" Text="@kwak Thanks. I'll play with the bandwidth selector. As for running style, I guess your argument would be that the runner could go back to their original running style and return to their previous speed. Sometimes I'd be interested in seeing the person's new potential in terms of perhaps a new running style that they have committed to." CreationDate="2010-09-22T13:35:20.377" UserId="183" />
  <row Id="3772" PostId="2967" Text="@Jeromy:&gt; see updated answer." CreationDate="2010-09-22T13:40:04.550" UserId="603" />
  <row Id="3773" PostId="2968" Text="+1 Because, it is a great idea." CreationDate="2010-09-22T13:47:02.663" UserId="442" />
  <row Id="3774" PostId="2820" Text="suncoolsu, I think that this method only works when the correlation is positive (not negative) between the variables.  Cheers." CreationDate="2010-09-22T13:57:04.110" UserId="253" />
  <row Id="3775" PostId="2970" Text="@Jeromy: I'm sure you're right. Just sharing my ignorance." CreationDate="2010-09-22T14:24:23.967" UserId="1270" />
  <row Id="3776" PostId="2884" Text="Whuber,but I foud &quot;The Kruskal-Wallis test  does  not  assume  population  normality  nor  homogeneity  of  variance,  as  does  the parametric ANOVA,  and  only  requires  ordinal  scaling  of  the  dependent  variable.&quot; that you can see from http://www.jstor.org/pss/1165320 or http://oak.ucc.nau.edu/rh232/courses/EPS525/Handouts/Understanding%20the%20One-way%20ANOVA.pdf." CreationDate="2010-09-22T14:52:44.943" />
  <row Id="3777" PostId="2701" Text="That is great thanks, the graphpad page was perfect - Fig 6.4 gave me what I needed to know.  We havent yet got 50% dead, so technically it's not possible to work out, but i did some forecasting to get to that point." CreationDate="2010-09-22T15:13:36.870" UserId="1284" />
  <row Id="3778" PostId="2947" Text="What do you mean with fitted CDF?" CreationDate="2010-09-22T15:44:54.097" UserId="608" />
  <row Id="3779" PostId="2929" Text="I believe that would be too noisy. My wish to focus on distributions rather than sequences of data points is based on the fact that the data is rather noisy. I think distributions provide the general tendencies rather than individual pattern. Also, I am summarizing over many users which I can combine in that way to detect the main trends. So, I basically have two PDF (or CDFs) and would like to know how similar they are. I like the idea of the Gini coefficient that uses the relative area under a Lorenz curve but might need something slightly different." CreationDate="2010-09-22T15:51:45.943" UserId="608" />
  <row Id="3780" PostId="2929" Text="By the way, thank you all for continuing this discussion. I just realized that they became rather lengthly :)" CreationDate="2010-09-22T15:53:02.740" UserId="608" />
  <row Id="3781" PostId="2947" Text="@Ampleforth, in that paper, I fit a distribution to empirical data, so by &quot;fitted CDF&quot; I meant the theoretical CDF of the fitted distribution. Sorry, I see how I could have been more clear!" CreationDate="2010-09-22T17:06:35.607" UserId="364" />
  <row Id="3782" PostId="2976" Text="Would factor analysis not do the job for qn 1? Question 2 is a bit vague. 'Relationship' seems a synonym for 'correlation' or at least one form of relationship is linear relationship and correlation captures that. Perhaps, you need to clarify qn 2." CreationDate="2010-09-22T17:08:32.457" />
  <row Id="3783" PostId="2951" Text="@Petr: many thanks for pointing this out to me, I had completely missed the reference to the weights in the documentation." CreationDate="2010-09-22T17:42:54.440" UserId="1007" />
  <row Id="3784" PostId="2977" Text="@Srikant I know all this. CV is a means for finding &quot;best&quot;. What is the definition of &quot;best&quot;?" CreationDate="2010-09-22T18:13:26.627" UserId="1134" />
  <row Id="3785" PostId="2977" Text="@bart 'best model' = a model that 'best' captures global patterns while avoiding local features of a data. That is the best I can do for a non-math description. Perhaps, someone else can elaborate a bit more or be more specific." CreationDate="2010-09-22T18:15:38.403" />
  <row Id="3786" PostId="2977" Text="@bart: &quot;best&quot; means the the function that fits the training data the best, and that &quot;generalizes&quot; well to the validation/unseen-test set data. I think this is quite clear from Srikant's answer. There are many ways to formally define a good generalization behavior. In a non-formal sense, you can think of it as finding a function that is &quot;smooth&quot; and not much wiggly. Trying to fit solely on the training data may lead to the wiggly looking function whereas smoothness usually ensures that the function will do reasonably well on both training and validation/test data." CreationDate="2010-09-22T18:27:35.370" UserId="881" />
  <row Id="3787" PostId="2977" Text="@ebony: You are missing the point. I've rephrased the question to hopefully make it clearer" CreationDate="2010-09-22T18:34:28.597" UserId="1134" />
  <row Id="3788" PostId="2980" Text="@Keith I thought the meaning of 'external event' was clear. To clarify, because of the constraints imposed on the protocol, the external event should be the source of the random generation and the draw must be verifiable. Your answer addresses verifiability but not the external event part. Nevertheless, +1 for an interesting answer. Out of curiosity, doesn't the algorithm breakdown if both Bob and Alice cheat and report a value (0 or 1 as they choose) *without* flipping a coin?" CreationDate="2010-09-22T19:09:32.683" />
  <row Id="3789" PostId="2807" Text="@nico:  Sure it can be negative, but there's some finite limit to it because there are only so many protons and electrons in the Universe.  Of course this is irrelevant in practice, but that's my point.  Nothing is **exactly** normally distributed (the model is wrong), but there are lots of things that are close enough (the model is useful).  Basically, you already knew the model was wrong, and rejecting or not rejecting the null gives essentially no information about whether it's nonetheless useful." CreationDate="2010-09-22T19:39:17.780" UserId="1347" />
  <row Id="3790" PostId="2947" Text="Oh, please don't apologize. My lack of statistics is rather large and that is the only problem here ;) Also I did not read your paper, but only glanced through your graphs which I really liked. " CreationDate="2010-09-22T19:57:07.493" UserId="608" />
  <row Id="3791" PostId="2984" Score="1" Text="Is $\lambda$ a parameter that is free to be chosen?" CreationDate="2010-09-22T21:49:03.633" UserId="352" />
  <row Id="3792" PostId="2984" Text="@Robby:&gt; thanks. I slightly appended the text to make the distinction between parameters and hyperparameters clear." CreationDate="2010-09-22T22:12:04.800" UserId="603" />
  <row Id="3793" PostId="2979" Text="@paul:&gt; this is not the same model (i.e. Kelly's model is non linear in the parameters)." CreationDate="2010-09-22T22:35:14.190" UserId="603" />
  <row Id="3794" PostId="2835" Text="@Stephan:&gt; yes it does local quantile regression as well (see the vignette (http://cran.r-project.org/package=quantreg)." CreationDate="2010-09-22T22:40:08.920" UserId="603" />
  <row Id="3795" PostId="2984" Text="@kwak: I'm sorry to say I haven't a clue what this means. What do the symbols p, q, lambda, x, y, m and beta signify?" CreationDate="2010-09-22T23:05:43.507" UserId="1134" />
  <row Id="3796" PostId="2985" Text="@Robby: You understand the question. I'll follow the links." CreationDate="2010-09-22T23:09:13.920" UserId="1134" />
  <row Id="3797" PostId="2984" Text="@bart:&gt; My answer is essentially the same as Srikant's. Where he provides an intuitive explication, I wanted to add a more rigorous one for the benefits of future visitors that may have the same question as you, but are more familliar with math than non-formal language. All the symbols you mention are defined in my answer (altough, again, this is done formally). " CreationDate="2010-09-22T23:35:03.450" UserId="603" />
  <row Id="3798" PostId="2985" Text="You should know that these links are unlikely to take you anywhere `practical'. If you are trying to *build* something using cross validation (or some other sort of model selection) then in practice it is likely to always come down to something heuristic and a bit ad-hoc (although I agree this is unsatisfying). " CreationDate="2010-09-22T23:36:43.837" UserId="352" />
  <row Id="3799" PostId="2985" Text="@Robby: Now we are getting somewhere. http://en.wikipedia.org/wiki/Minimum_message_length seems to be what I was thinking. Thanks!" CreationDate="2010-09-22T23:48:45.697" UserId="1134" />
  <row Id="3800" PostId="2985" Text="@Robby: No worries. This is just reflection, not practical." CreationDate="2010-09-22T23:50:31.153" UserId="1134" />
  <row Id="3801" PostId="2984" Text="@kwak: Where, for example, is p defined?" CreationDate="2010-09-22T23:54:18.337" UserId="1134" />
  <row Id="3802" PostId="2986" Text="@user1396: many thanks! I was thinking I had to do this 'by hand'. : ) I ran `community.to.membership(ag, z$merges, steps=12)` on the output of running `fastgreedy.community()` with the graph provided as an example in the docs and the output was `$membership&#xA; [1] 2 2 2 2 2 0 0 0 0 0 1 1 1 1 1&#xA;&#xA;$csize&#xA;[1] 5 5 5`&#xA;I am confused by the repeated values in the $membership vector. How should I read them? I thought this would contain the vertice names. many thanks!" CreationDate="2010-09-23T00:41:25.187" UserId="1007" />
  <row Id="3803" PostId="2984" Text="@bart:&gt; line 1 (p,q)&gt;= 1 means p and q are taken to be numbers larger than 1. Latter on (line 1 of last §), it says that for that particular property to hold p should be equal to either 1 or 2. " CreationDate="2010-09-23T00:45:51.087" UserId="603" />
  <row Id="3804" PostId="2986" Text="@user1396: I think I got it; a '2' in the i-th element in the membership vector is telling me the i-th vertex belongs to community 2. correct? :)" CreationDate="2010-09-23T00:47:25.193" UserId="1007" />
  <row Id="3805" PostId="2979" Text="Paul-- believe it or not, we tried this as a starting point for the  optimization (taking into account the concerns you raised), but the estimator was way off from where we'd expect it to converge to. No good explanation why. We do have a reasonable workaround, which I'll be sharing with the question shortly." CreationDate="2010-09-23T01:15:09.080" UserId="53" />
  <row Id="3806" PostId="2979" Text="@kwak:&gt; what do you mean by &quot;this&quot;? Not the same model as 2sls or not the same model as what the GHK simulator is usually applied to, ie not a multinomial probit? I know it's not a multinomial probit, that's why I suggested a slight variant of the GHK simulator. As to whether 2sls applies, doesn't the equation for W have some error term, say, e, and doesn't the full model imply E[e|Z,X] = 0 and E[(H,M,L)|Z] not 0? " CreationDate="2010-09-23T01:17:15.533" UserId="1229" />
  <row Id="3807" PostId="2976" Text="You have stated what you want to do. What is your question? Is it about implementation or whether your analysis approach is appropriate? or something else?" CreationDate="2010-09-23T01:21:15.650" UserId="183" />
  <row Id="3808" PostId="2980" Text="Hi Srikant, yes, the algorithm requires at least one honest party. You have no internal source of randomness at all? Then you could take the SHA-256 of the PDF version of The New York Times every day, or the SHA-256 of the volume and closing price of all the stocks in the Dow Jones Industrial Average, or the really the secure hash of anything that can be mutually observed and that you can't influence. If you want one bit, take one bit of the SHA-256. If you want a normal distribution, take the whole thing (256 bits) as a uniform deviate and use the Box-Muller transform to get a normal deviate." CreationDate="2010-09-23T02:47:10.723" UserId="1122" />
  <row Id="3809" PostId="2980" Text="@Keith Yes, your modification using the hash of NYT/stock prices should also work although it has the one weakness of requiring at least 1 honest party. Interesting way to approach the issue." CreationDate="2010-09-23T03:06:53.443" />
  <row Id="3810" PostId="3000" Text="Criteria will be 10% difference in probability of good vs bad outcome. Or lets say since it will be logistic regression, odds ratio = 2. alpha= 0.05, power=80%, I do not yet know what the pooled variance on the continuous variable is but let us assume that the standard deviation is 7mmHg.   Sequential analysis would be good but the final outcome is two  years after the measurement is taken." CreationDate="2010-09-23T03:36:42.700" UserId="104" />
  <row Id="3811" PostId="2988" Text="Do you expect some dropouts during follow-up? Are there any other covariates to be included in your model?" CreationDate="2010-09-23T06:34:13.363" UserId="930" />
  <row Id="3812" PostId="2979" Text="@paul:&gt; you state *You could avoid the problem altogether by simply estimating* which is certainly correct. Nonetheless, the model you propose is a different one than the one Kyle wanted to fit (it is not a reformulation of the same model)." CreationDate="2010-09-23T07:37:05.127" UserId="603" />
  <row Id="3813" PostId="2989" Text="Thanks alot (both for the clever initial suggestion and for the folow thru). " CreationDate="2010-09-23T07:39:16.557" UserId="603" />
  <row Id="3814" PostId="2835" Text="Nice, I didn't know that - thanks!" CreationDate="2010-09-23T08:18:55.463" UserId="1352" />
  <row Id="3817" PostId="3009" Text="@bernd Thanks for the references." CreationDate="2010-09-23T09:21:59.713" UserId="183" />
  <row Id="3820" PostId="3009" Text="@bernd I pointed to similar books in my own response, sorry didn't see your response while I was writing... Card's book is very great!" CreationDate="2010-09-23T09:36:32.393" UserId="930" />
  <row Id="3822" PostId="3005" Text="@kwak Ok, the LARS algorithm seems largely more sophisticated than simple thresholding on variable importance, but the point is that I don't see a clear relation between the penalty parameter and the # of variables that are asked to be kept in the model; it seems to me we cannot necessarily find a penalty parameter that would yield exactly a fixed # of variables." CreationDate="2010-09-23T09:51:08.970" UserId="930" />
  <row Id="3823" PostId="3010" Text="Wow. That's a great set of references. I'm assuming that husbands and wives would not be interchangeable. I wonder if that matters." CreationDate="2010-09-23T09:51:37.143" UserId="183" />
  <row Id="3825" PostId="3010" Text="It depends on the questions asked in the questionnaire. I came across a clinical study dealing with erectile function and sexual life (wife + husband), in this case the role of the respondents are not symmetric." CreationDate="2010-09-23T09:55:19.353" UserId="930" />
  <row Id="3826" PostId="3009" Text="@chl Great, that you mentioned the Thompson/Walker-paper about &quot;The Dyad as the Unit of Analysis&quot;. " CreationDate="2010-09-23T10:00:39.687" UserId="307" />
  <row Id="3827" PostId="3005" Text="@chl:&gt; S-PLS you mean ?(you wrote LARS which is a different thing from either algorithm you discuss). Indeed, there is a monotoneous relationship between the penalty parameter and the # of component, but it is not a linear relationsip and this relationship varies on a case per case basis (is dataset/problem dependant)." CreationDate="2010-09-23T10:04:11.630" UserId="603" />
  <row Id="3829" PostId="2963" Text="+1 I use it often to make seeds." CreationDate="2010-09-23T10:06:12.360" UserId="88" />
  <row Id="3830" PostId="3005" Text="@kwak L1-penalty may be achieved using LARS, unless I am misleading. Your second point is what I have in mind in fact; have you any reference about that point?" CreationDate="2010-09-23T10:21:59.497" UserId="930" />
  <row Id="3832" PostId="3005" Text="@chl:&gt;* L1-penalty may be achieved using LARS, unless I am misleading* i didn't know that (and sort of doubt it). Can you provide a reference ? Thanks. for your second question: look On the “degrees of freedom” of the lasso&#xA;&#xA;Hui Zou, Trevor Hastie, and Robert Tibshirani&#xA;Source: Ann. Statist. Volume 35, Number 5 (2007), 2173-2192. (there are many ungated versions)." CreationDate="2010-09-23T10:47:08.533" UserId="603" />
  <row Id="3833" PostId="3005" Score="1" Text="@kwak Check out Tibshirani's webpage, http://www-stat.stanford.edu/~tibs/lasso.html and the `lars` R package; other methods include coordinate descent (see JSS 2010 33(1), http://bit.ly/bDNUFo), and the Python `scikit.learn` package features both approaches, http://bit.ly/bfhnZz." CreationDate="2010-09-23T10:58:16.690" UserId="930" />
  <row Id="3834" PostId="2989" Text="No problem, kwak; remember only that it does premultiplication, though modifying the algorithm to do postmultiplication shouldn't be too hard." CreationDate="2010-09-23T11:45:26.313" UserId="830" />
  <row Id="3835" PostId="3002" Score="2" Text="Monotonic *and increasing*, of course." CreationDate="2010-09-23T13:32:59.260" UserId="919" />
  <row Id="3836" PostId="2733" Text="Besides that fact that the former link wasn't really related to *mathematical* statistics, the latter one is neither: &quot;Data Mining with STATISTICA Video Series&quot; - http://www.youtube.com/user/StatSoft#g/c/B804A810436AFB03" CreationDate="2010-09-23T14:00:53.827" UserId="653" />
  <row Id="3838" PostId="2971" Score="1" Text="Best is the model with lowest future error and cross-validation gives you that estimate. The reason for c(Complexity)+e(Error) formulas is because you could use error on training data as estimate of future error, but that's overly optimistic, so you add a term to make this estimate unbiased, which is usually some function of model complexity" CreationDate="2010-09-23T14:32:33.153" UserId="511" />
  <row Id="3839" PostId="2959" Text="Thanks for the pointers. Will need to hit the library to make sense of it all." CreationDate="2010-09-23T14:53:10.090" UserId="1393" />
  <row Id="3840" PostId="2961" Text="Thanks, that was an interesting paper." CreationDate="2010-09-23T14:53:50.333" UserId="1393" />
  <row Id="3841" PostId="2852" Text="Thank you,Thylacoleo.These are DNA divergence data of six different groups.I want to know the difference between the first column band each one of others with P value." CreationDate="2010-09-23T15:00:35.967" />
  <row Id="3842" PostId="3020" Score="1" Text="@whuber +1, Really excellent points. At some point I would have suggested a wiki as a way to document progress. Perhaps, we need one right now before the threads get out of hand. Can someone volunteer to set one up for this project? " CreationDate="2010-09-23T15:07:09.837" />
  <row Id="3843" PostId="3020" Score="1" Text="I can set one up, yes." CreationDate="2010-09-23T15:20:54.230" UserId="5" />
  <row Id="3844" PostId="3020" Text="@Shane Great. Please let us know the link once it is up and running." CreationDate="2010-09-23T15:23:22.110" />
  <row Id="3845" PostId="3020" Text="@Srikant: It's here: http://www.statalgo.com/polystatsproject/wiki/.  I'll try to make it a little more attractive, etc. as time goes on.  I may also install the R mediawiki plugin.  But this should do the trick for now." CreationDate="2010-09-23T15:39:52.127" UserId="5" />
  <row Id="3846" PostId="2997" Text="Having CSV format is the best overall, although it would be good for R users to also have Rdata format available.  This compressed and can store complex data structures." CreationDate="2010-09-23T16:34:46.687" UserId="5" />
  <row Id="3847" PostId="2917" Text="I am not sure if most CIs are really computed via t-values or rather via z-values. However, on bigger ns (&gt; 30) this shouldn't make much of a difference." CreationDate="2010-09-23T16:45:44.500" UserId="442" />
  <row Id="3848" PostId="2891" Text="A typical ratio would be 1042:42. 1042 turn left, 42 turn right." CreationDate="2010-09-23T17:13:32.613" />
  <row Id="3849" PostId="2989" Text="@J.M.:&gt; it should be okay: i only need Q (i.e. qmult(eye(p)))" CreationDate="2010-09-23T20:23:54.853" UserId="603" />
  <row Id="3850" PostId="3005" Text="@chl:&gt; thanks. (i also recommend the lars paper www-stat.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf)" CreationDate="2010-09-23T20:27:50.763" UserId="603" />
  <row Id="3851" PostId="3021" Text="I can make a citeulike group library with which to store suggested references if people would like me to. You don't need an account to see the library and follow the DOI, but you do need an account to post articles to the group. I have found plenty of work examine all sorts of aspects of online communities in just a short time span. I think this article looks pretty interesting, http://portal.acm.org/citation.cfm?id=1341549 PDF here http://www.cs.cornell.edu/~lars/wsdm08.pdf" CreationDate="2010-09-23T20:32:58.420" UserId="1036" />
  <row Id="3852" PostId="3021" Text="Also does anybody know of a Sociological/Psychological/Anthropological literature that is applicable to this topic (keywords or specific scholars)?" CreationDate="2010-09-23T20:35:10.897" UserId="1036" />
  <row Id="3853" PostId="2975" Text="It seems to me no valid comparison is possible among these series unless there is some persistent spatial phenomenon producing the &quot;temperature signature.&quot;  Otherwise they all measure different things.  If there is a spatial pattern, then you really have a problem of spatial interpolation, not of time series normalization." CreationDate="2010-09-23T21:56:58.190" UserId="919" />
  <row Id="3854" PostId="2901" Text="@Kwak: Thank you for clarifying your meaning.  Of course, the same response could be applied to any odd moment: they measure asymmetry further and further out in the tails." CreationDate="2010-09-23T22:07:46.273" UserId="919" />
  <row Id="3855" PostId="2901" Text="@Whuber:&gt; Of course. Note that even for a fair tailed distribution like the gaussian, by the 7th moment you are already in effect comparing the max to the min." CreationDate="2010-09-23T23:12:54.533" UserId="603" />
  <row Id="3856" PostId="2945" Text="If overfitting is the tendency of a model fitting procedure to fit noise in addition to signal, then we can look at a given procedure to see where such tendencies could arise.  Perhaps due to a lack of imagination or knowledge, while considering a few different procedures, I couldn't boil this down to something that can't be restated as &quot;number of parameters&quot; (or &quot;effective number of parameters&quot;).  We could flip this on its head and ask: all else equal, what happens when we introduce noise to our data?  Then we arrive at measures such as Ye's GDF." CreationDate="2010-09-23T23:22:11.097" UserId="251" />
  <row Id="3857" PostId="2945" Text="@ars:&gt; suppose you compare two models for $y$ a) $y=\alpha_1*xz$ and $y=\alpha_1*x$. They have the same number of parameters, but the first one is sensitive to noise in both x and z." CreationDate="2010-09-23T23:29:34.983" UserId="603" />
  <row Id="3858" PostId="2945" Text="@kwak: That doesn't meet my &quot;all else equal&quot; case -- sorry if that was unclear.  I'm really trying to arrive at an answer to Yaroslav's question, perhaps restated: is there something other than dimensionality to account for tendency to fit noise in addition to signal?  And keeping the considered case very simple, I'm failing ..." CreationDate="2010-09-23T23:36:08.993" UserId="251" />
  <row Id="3859" PostId="3002" Text="Of course. Thanks. I've updated the answer." CreationDate="2010-09-24T03:06:23.310" UserId="159" />
  <row Id="3860" PostId="3032" Text="wow. i am an embarrassed nOOb. thanks for the pointer.  " CreationDate="2010-09-24T03:33:32.593" UserId="1410" />
  <row Id="3861" PostId="3034" Text="This is great Dason, thanks so much. Rob's code was great but your example made it obvious to me what was happening -- thanks for taking the time to do that!" CreationDate="2010-09-24T03:39:19.963" UserId="1410" />
  <row Id="3862" PostId="3042" Text="THanks! That works! I tried to read the help documentation, but still not sure what is the difference between what=&quot;logical&quot; and what=logical(0)? Thanks!" CreationDate="2010-09-24T06:51:21.023" UserId="1005" />
  <row Id="3863" PostId="3042" Text="Basically, it wants an object of the actual type to which values should be coerced.  You could also just say `scan(file='1.txt', what=TRUE)` and it would work the same.  `logical(0)` is a cleaner way, IMO." CreationDate="2010-09-24T07:00:45.793" UserId="251" />
  <row Id="3864" PostId="3038" Text="what do you mean by &quot;this assumes the null hypothesis to be true&quot; ?&#xA;" CreationDate="2010-09-24T08:39:27.173" UserId="223" />
  <row Id="3865" PostId="3038" Text="If you want to be able to control the probability of declaring wrongly &quot;there is a difference&quot;  you need to separate the two hypothesis (did I already mentionned I love this quote: http://stats.stackexchange.com/questions/726/famous-statistician-quotes/728#728 ;) )&#xA;  &#xA;" CreationDate="2010-09-24T08:40:19.950" UserId="223" />
  <row Id="3866" PostId="2775" Text="@gappy:&gt; the problem with this method is that you can't control the ratio of smallest to largest eigenvalue (and i think that as the size of your randomly generated dataset goes to infinity, this ratio will converge to 1)." CreationDate="2010-09-24T09:28:03.940" UserId="603" />
  <row Id="3867" PostId="3045" Text="@dmuir:&gt; the problem with this method is that you can't control the ratio of smallest to largest eigenvalue (and i think that as the size of your randomly generated dataset goes to infinity, this ratio will converge to 1)." CreationDate="2010-09-24T09:28:29.650" UserId="603" />
  <row Id="3868" PostId="2783" Text="@Simon:&gt; Not random: two matrices generated from the same Whishard will not be independant from one another. If you plan to change the Whishart at each generation, then how do you intend to generate those Whishart in the first place?" CreationDate="2010-09-24T09:32:47.007" UserId="603" />
  <row Id="3869" PostId="3045" Text="@dmuir:&gt; besides, the method is not very efficient (from a computationnal point of view)" CreationDate="2010-09-24T09:36:01.417" UserId="603" />
  <row Id="3870" PostId="3045" Text="Your &quot;random matrix&quot; is a specially structured one called a &quot;diagonal plus rank-1 matrix&quot; (DR1 matrix), so not really a good representative random matrix." CreationDate="2010-09-24T10:08:08.447" UserId="830" />
  <row Id="3871" PostId="3038" Text="@Robin the p value of a null hypothesis significance test is the probability of seeing as or more extreme data than that observed assuming the null hypothesis is true; but perhaps I could word the statement above better." CreationDate="2010-09-24T10:26:31.300" UserId="183" />
  <row Id="3872" PostId="3038" Text="@Robin I modified the question to try to make my point clearer" CreationDate="2010-09-24T10:32:39.133" UserId="183" />
  <row Id="3873" PostId="2783" Text="@kwak: I don't understand your question: the Bartlett decomposition will give independent draws from the same Wishart distribution." CreationDate="2010-09-24T11:03:38.877" UserId="495" />
  <row Id="3874" PostId="3047" Text="I was wondering if someone would provide a Bayesian approach. Excellent. Thanks." CreationDate="2010-09-24T11:41:32.317" UserId="183" />
  <row Id="3875" PostId="2783" Text="@Simon:&gt; Let me rephrase this, where do you get the scale matrix of your whishart distribution from ?" CreationDate="2010-09-24T12:16:44.293" UserId="603" />
  <row Id="3876" PostId="2971" Text="On the other hand reasoning in light of Runge phenomenon (Physics inspirations again) drives to a conclusion that future error is something about Complexity/Train_Error." CreationDate="2010-09-24T12:25:26.540" UserId="88" />
  <row Id="3877" PostId="2783" Text="@kwak: it's a parameter of the distribution, and so is fixed. You select it at the start, based on the desired characteristics of your distribution (such as the mean)." CreationDate="2010-09-24T12:40:23.173" UserId="495" />
  <row Id="3878" PostId="3021" Text="@Andy I would go ahead with whatever you think makes sense. If you could collect a few relevant references then perhaps we can divide up the task of reading the relevant articles and providing a summary on the wiki." CreationDate="2010-09-24T13:20:54.640" />
  <row Id="3879" PostId="3021" Text="@all: Once we have some reading materials we need some volunteers to take up the task of reading some of that material. Could you please add your name as a comment or in the answer so that we can decide how to divide the readings?" CreationDate="2010-09-24T13:21:39.060" />
  <row Id="3880" PostId="2901" Text="@Kwak: Two quick follow-up questions; no need to respond if you don't want.  (1) &quot;Fair tailed&quot;??  (2) What are the min and max of a Gaussian?" CreationDate="2010-09-24T13:29:44.777" UserId="919" />
  <row Id="3883" PostId="3051" Score="4" Text="Have you seen [this](http://rss.acs.unt.edu/Rdoc/library/TTR/html/MovingAverages.html)?" CreationDate="2010-09-24T14:52:24.400" UserId="830" />
  <row Id="3884" PostId="3051" Text="Can you give some background on this &quot;slide&quot; idea?  " CreationDate="2010-09-24T15:01:01.503" UserId="5" />
  <row Id="3885" PostId="3055" Text="Thank you Shane, but this seem to find local minima (or maxima) - i.e. a single point in a region. My data ( as any biological data) IS NOISY&gt; I don't really care about point minima themselves but about larger regions which are low." CreationDate="2010-09-24T15:53:49.667" UserId="634" />
  <row Id="3886" PostId="3055" Text="If you have local maximum and minimum points, you can easily calculate the differences.  So you want to know instances when the differences are both large in magnitude and in &quot;duration&quot;?  Is this time series data?" CreationDate="2010-09-24T15:59:20.717" UserId="5" />
  <row Id="3887" PostId="3055" Text="@david Perhaps, you can iteratively use this function. Use the function to identify a minima. Drop that point and surrounding points (say x points within some tolerance level). You can choose a tolerance level (e.g., +- 10 counts) which would define a flat region for your application.  Find a new minima on the new dataset. Will that work? " CreationDate="2010-09-24T16:01:07.127" />
  <row Id="3888" PostId="3055" Text="@shane The analogy that comes to mind is that of valleys in a mountainous region. I think the goal is to identify all the valleys and the issue is some valleys are 'deeper' and some are 'shallow' relative to the mountains." CreationDate="2010-09-24T16:02:57.350" />
  <row Id="3889" PostId="3055" Text="@Shane It's not a time series, these are coordinate along the genome (chromosome)." CreationDate="2010-09-24T16:03:43.913" UserId="634" />
  <row Id="3890" PostId="3055" Text="@srikant: this function gives you all the local maximum and minimum indexes.  You can easily use these to calculate the depth of the valleys.  What I'm missing in this question is whether there is some other factor to be considered (e.g. &quot;width&quot;)?" CreationDate="2010-09-24T16:05:04.203" UserId="5" />
  <row Id="3891" PostId="3052" Text="Could you provide a small data sample?" CreationDate="2010-09-24T16:20:43.850" UserId="5" />
  <row Id="3892" PostId="3021" Text="@all, I've gone ahead and made a group library on Citeulike, which can be found here, http://www.citeulike.org/groupfunc/14031/home , you can make suggestions for articles to post either here or at the actual site. And if you have a citeulike account you can post articles yourself. So make some recommendations, as it is pretty sparse at the moment." CreationDate="2010-09-24T16:40:05.393" UserId="1036" />
  <row Id="3893" PostId="3052" Text="@Shane see update" CreationDate="2010-09-24T16:43:03.593" UserId="634" />
  <row Id="3894" PostId="3052" Text="@David Thanks.  As both the answers imply, time series analysis can be applied here since you have ordered observations." CreationDate="2010-09-24T16:50:52.710" UserId="5" />
  <row Id="3895" PostId="2901" Text="@Whuber; the explanation for (2) is quiet subtile. The basic idea is based upon 'a quantile alternative to kurtosis' (J.J.A. Moors). Just as the kurtosis is determined by the normalized sum of (E_7-E_5) and (E_3-E_1) (where E_i is the ith octile), the 8^th central moment is determined by the normalized sum of (G_15-G_14) and (G_3-G_1) (where G_i is the ith 100/16 percentile) -the top and bottom 5% of the data. " CreationDate="2010-09-24T17:39:26.417" UserId="603" />
  <row Id="3896" PostId="2901" Text="The same applies to the odd moments." CreationDate="2010-09-24T17:45:00.873" UserId="603" />
  <row Id="3897" PostId="3052" Text="This is kind of hard to answer without know exactly what you're looking for. Can you maybe circle the points on the plots that you're looking to capture? What do you consider a &quot;valley&quot;? how low does it have to go and what are you looking to return? It's hard to formulate a solution without knowing the question, ie thresholds and such." CreationDate="2010-09-24T17:48:37.710" UserId="1409" />
  <row Id="3898" PostId="2935" Text="Gaetan:&gt; yes linear regression can handle slope dummy." CreationDate="2010-09-24T17:56:04.143" UserId="603" />
  <row Id="3899" PostId="2901" Text="@Kwak: I'm not grasping the subtlety of attributing a min and max to a distribution that doesn't have one! ;-)  I suspect you may be referring to sampling theory, whereas I read the original question as one about theoretical distributions." CreationDate="2010-09-24T18:00:51.377" UserId="919" />
  <row Id="3900" PostId="2901" Text="@whuber:&gt; Oh, then indeed i missunderstood your comment. Yes min (max) was a poor choice of terms. " CreationDate="2010-09-24T18:13:22.357" UserId="603" />
  <row Id="3901" PostId="3053" Text="This does not return what the asker wanted, but 5.33 5.00 6.33. However, it looks quite interesting. Can you explain your idea, because I don't get it." CreationDate="2010-09-24T18:14:07.487" UserId="442" />
  <row Id="3902" PostId="3057" Text="@cxr Thank for your response. I has a look at `surveillance` and `DCluster `, but could you please be a bit more specific? They are both relatively large packages and their aim seems quite specific. I'm not sure where to begin." CreationDate="2010-09-24T18:37:44.347" UserId="634" />
  <row Id="3903" PostId="3052" Text="@ Shane♦ Thank you. As I have no experience with time-series analysis also, could you leave a few pointers of where should I start?" CreationDate="2010-09-24T18:39:08.233" UserId="634" />
  <row Id="3904" PostId="3061" Text="Duplicate? http://stats.stackexchange.com/questions/3062/how-do-we-find-concordance-between-two-sets-of-non-linear-non-monotonic-data" CreationDate="2010-09-24T19:13:01.753" UserId="5" />
  <row Id="3905" PostId="3061" Text="@user1417: You should edit a question rather than posting a new one." CreationDate="2010-09-24T19:13:28.303" UserId="5" />
  <row Id="3906" PostId="3061" Text="You should give a more precise reference than 'Lin's concordance correlation coefficient'." CreationDate="2010-09-24T19:24:28.720" UserId="603" />
  <row Id="3907" PostId="3052" Text="@David: Why not start by looking at the drawdowns function that I posted.  Walk through the example and look at the algorithm.  It's very simple." CreationDate="2010-09-24T19:41:12.803" UserId="5" />
  <row Id="3908" PostId="3052" Text="@David: By the way, I hate to state the obvious, but just looking at the regions you selected in your graph, it looks like you chose the area in the lowest quartile.  Is that a sufficient criteria?  " CreationDate="2010-09-24T19:44:51.190" UserId="5" />
  <row Id="3909" PostId="3064" Text="Oh, and feel free to retag - my browser isn't working with the auto-suggest, so I'm having a hard time seeing what's out there." CreationDate="2010-09-24T19:55:12.150" UserId="71" />
  <row Id="3910" PostId="3058" Text="Please don't downvote without providing a comment.  How am I supposed to know what's wrong?" CreationDate="2010-09-24T19:59:02.007" UserId="71" />
  <row Id="3911" PostId="3065" Text="Obviously you can apply the same approach using something else than the moving average, this was just to give an idea." CreationDate="2010-09-24T20:38:43.047" UserId="582" />
  <row Id="3912" PostId="3065" Text="+1 Thank you very much, nico. Let me see if I got you right: at the end, this is basically like setting some global threshold and defining any point with value &lt; threshold as a part of a valley. The sampling etc. is just used to get some meaningful measure (quantile) to set the threshold. Why can't we use a single threshold for the entire points, I mean, if we did enough simulations we would get straight (read and yellow) lines. Also, correct me if I'm mistaken, but this does not take into account the surrounding environment but examines the absolute value of each point." CreationDate="2010-09-24T21:29:29.190" UserId="634" />
  <row Id="3913" PostId="3065" Text="@David B: of course, you could use a global threshold and that would probably save you some calculation time. I guess choosing something like 1/3 of the global mean could be a start. This swapping process is probably more helpful if you use some other statistics than the moving average, it was mostly to give an idea. Anyway the moving average will take into account the surrounding, in the example it will take into account a window of 10 points." CreationDate="2010-09-24T21:35:14.623" UserId="582" />
  <row Id="3914" PostId="2945" Text="ars: for maximum likelihood estimation, curvature of the parametric manifold affects sample complexity" CreationDate="2010-09-24T21:58:55.737" UserId="511" />
  <row Id="3915" PostId="2945" Text="Yaroslav Bulatov:&gt; even more intuitively, the median is unaffected by shifts concerning &lt;50% of the sample, yet it's still a one parameter location model (just like the mean)." CreationDate="2010-09-24T22:08:31.837" UserId="603" />
  <row Id="3916" PostId="3054" Text="BTW, I once wrote about a usage of this function to implement the notion of &quot;quantile loess&quot; : http://www.r-statistics.com/2010/04/quantile-loess-combining-a-moving-quantile-window-with-loess-r-function/" CreationDate="2010-09-24T22:11:50.897" UserId="253" />
  <row Id="3917" PostId="3052" Text="it sounds like you want to build histograms of depths of coverage for each span from 1 to 100k (or so).  Then you probably might like to color-code your overall genomic chart where each vertical bar's intensity depends on for how many histograms it fell into the bottom percentile (or so)." CreationDate="2010-09-24T23:13:47.350" UserId="486" />
  <row Id="3918" PostId="3060" Text="Seems interesting. Do you know of a simple R implementation?" CreationDate="2010-09-25T01:54:38.520" UserId="104" />
  <row Id="3919" PostId="3008" Text="Do you have a worked case in R?" CreationDate="2010-09-25T02:07:25.970" UserId="104" />
  <row Id="3920" PostId="2988" Text="Let me suck a dropout rate out of my thumb - 20%.  We will indeed collect many variables for instance, age, trauma score but I wanted to keep things as simple as possible for the power calculation. I have often found it useful to discuss a primary model and then secondary models that are loaded with more finesse and nuance." CreationDate="2010-09-25T02:11:12.877" UserId="104" />
  <row Id="3921" PostId="2766" Text="@shabbychef: Regarding the wikipedia link that you gave which requires data to be normally distributed, however my data is not presumed to be normal or of any other distribution. Further, what is the best way to compare distributions if you think the mentioned method is not? Thanks !" CreationDate="2010-09-25T03:49:38.800" UserId="1102" />
  <row Id="3922" PostId="2766" Text="@Harpreet: true enough, however, if your distribution has a finite variance, the central limit theorem ( http://en.wikipedia.org/wiki/Central_limit_theorem ) describes the asymptotic distribution of the sample mean. Since you presumably do not know the population variance, you would have to estimate that as well. see also http://en.wikipedia.org/wiki/Student's_t-statistic#Prediction" CreationDate="2010-09-25T03:58:17.563" UserId="795" />
  <row Id="3923" PostId="3070" Text="+1, good points.  On this: &quot;unpleasant ideas about memory management&quot; .. interesting, can you elaborate?" CreationDate="2010-09-25T05:27:26.547" UserId="251" />
  <row Id="3924" PostId="3070" Score="1" Text="my memory is going _somewhere_; my experience with Java outside of Matlab usage indicate it is the likely culprit, and running in `-nojvm` appears to help..." CreationDate="2010-09-25T05:42:58.403" UserId="795" />
  <row Id="3925" PostId="3070" Text="Thanks shabby!!" CreationDate="2010-09-25T05:44:23.953" UserId="1102" />
  <row Id="3926" PostId="3071" Text="Nice one too. Thanks ars!" CreationDate="2010-09-25T05:45:06.543" UserId="1102" />
  <row Id="3927" PostId="3070" Text="@shabbychef: ah, thanks." CreationDate="2010-09-25T05:56:21.760" UserId="251" />
  <row Id="3928" PostId="2975" Text="It is complex and messy real-world data. The &quot;thermal anomaly&quot; is (I hypothesize) variable in time but consistent in location. Overlaid on this is a different large-scale regional temperature gradient, seasonal and day/night variations, noise, instrument variability, etc. The flight paths come and go in different directions (so the mean temp of two flight paths should not be equal, even if season+time-of-day are removed, because of the large-scale regional gradient, etc.)... " CreationDate="2010-09-25T07:14:29.190" UserId="957" />
  <row Id="3929" PostId="2978" Text="This is helpful, thank you. And addresses the simplified problem stated above. Unfortunately, as commented above, the real world is much more complex than what I initially described. I'm still working on it..." CreationDate="2010-09-25T07:15:46.247" UserId="957" />
  <row Id="3930" PostId="2988" Text="@Farrel Ok, but usually the expected % dropout, the number of covariates, and whether covariates are measured with errors (see e.g., http://j.mp/9fJkhb) enter the formula (in all case, it will increase the sample size)." CreationDate="2010-09-25T07:37:32.087" UserId="930" />
  <row Id="3931" PostId="3053" Text="@Henric I use this trick frequently, yet user1414's code return this roll with slide 1, not 2, as intended by OP. Check out `(c(0,0,x)+c(0,x,0)+c(x,0,0))/3` to see what I mean (and how does it work). The proper formula would be: `(c(0,0,x)+c(0,x,0)+c(x,0,0))[1:(length(x)-3)*2+1]/3` (we must cut 0-padding at the beginning and select even elements then.  " CreationDate="2010-09-25T08:00:54.957" UserId="88" />
  <row Id="3932" PostId="3058" Text="@Matt It wasn't me, but this is slow (but not much slower than `rollapply`)." CreationDate="2010-09-25T08:06:01.900" UserId="88" />
  <row Id="3933" PostId="3072" Text="+1 Not tada, rv/windowsize ;-)" CreationDate="2010-09-25T08:24:54.523" UserId="88" />
  <row Id="3934" PostId="3072" Text="This marg... comment box is too narrow for this code, so I've posted a new answer." CreationDate="2010-09-25T08:31:46.993" UserId="88" />
  <row Id="3935" PostId="3054" Text="You may add a 0 at the end of x (`x&lt;-c(x,0)`) to get the last element of answer." CreationDate="2010-09-25T08:36:22.007" UserId="88" />
  <row Id="3936" PostId="3070" Text="My favorite example of MATLAB strange built-in codes is shuffle, which reorders the data with the ordering returned by sorting a freshly created random vector." CreationDate="2010-09-25T08:49:05.917" UserId="88" />
  <row Id="3937" PostId="3054" Score="1" Text="@mbq; that is making a strong assumption that the observation is 0. I had been mulling this point and T-Burns is making the same assumption (an unobserved 0). I would prefer perhaps to pad with NA and pass in the `na.rm = TRUE` argument to `mean`. The answer won't be the same as what the OP requested, but it seems more useful. I'll edit my answer to include this." CreationDate="2010-09-25T08:59:38.317" UserId="1390" />
  <row Id="3938" PostId="3054" Text="@ucfagls Yet this is easy to change and as you said this assumption was made by the OP. On the other hand, I would be even more restrictive and removed the last average." CreationDate="2010-09-25T09:24:45.050" UserId="88" />
  <row Id="3939" PostId="3058" Score="1" Text="@Matt; wasn't me either, but as mentioned by yourself, pre-allocation of the result object will help with the speed issue. One trick, if you don't know, or it is tedious/difficult to determine, the size of the result object you need. Allocate something reasonable, perhaps pre-filling with NA. Then fill in with your loop, but add a check that if you are approaching the limit of the preallocated object, allocate another big chunk, and continue filling." CreationDate="2010-09-25T09:43:32.883" UserId="1390" />
  <row Id="3940" PostId="3058" Score="1" Text="@mbq; Speed of results, whilst important, isn't the only consideration. Instead of having to reinvent the while and handle all the indexes etc in the custom solutions, the one-linear that is `rollapply` is much easier to understand and grep the intention of. Also, `rollapply` is likely to have had many more eyeballs checking its code than something I might cook up one afternoon. Horses for courses." CreationDate="2010-09-25T09:45:50.687" UserId="1390" />
  <row Id="3941" PostId="3069" Score="1" Text="This should be community wiki, IMO. " CreationDate="2010-09-25T12:01:46.490" UserId="5" />
  <row Id="3942" PostId="175" Score="1" Text="Sharpie, I wonder if it's time to accept a &quot;best answer&quot;or at least explain why you are not satisfied with the answers on offer so far. " CreationDate="2010-09-25T13:42:30.757" UserId="521" />
  <row Id="3943" PostId="3058" Text="ucfagls, thanks for adding some info on pre-allocating when you don't know the ultimate size of the object - that's helpful.  And while I agree that rollapply is probably almost definitely the way to go, I a) wanted to provide an example that would calculate the edge cases with less than three values, and b) just wanted to use `while()`, which I've yet to have a use for ;)" CreationDate="2010-09-25T14:43:16.413" UserId="71" />
  <row Id="3944" PostId="3075" Text="Thanks for your answer.  Regarding not knowing the true status: we do have extensive questionnaires and are well aware of the BCG vaccine issue with the skin test - in fact, these blood tests are supposed to resolve that issue because they use a different set of antigens than the PPD you're used to.  That's almost a separate question, however, and one we're going to be working on a bit later - right now, my interest is in making this test 'longitduinally aware'." CreationDate="2010-09-25T14:50:42.067" UserId="71" />
  <row Id="3945" PostId="3075" Text="... especially because some individuals do flip from negative to positive, and that's often a product of their typical nil and TB results making small fluctuations - nil down a bit, TB up a bit, and suddenly they're positive.  Next test, they've gone back to being negative.  I can see that as I review individual results, but I'm not sure how to appropriately incorporate my intuition into a model." CreationDate="2010-09-25T14:54:01.327" UserId="71" />
  <row Id="3946" PostId="3075" Text="Finally, while I have tried taking the log results, that doesn't seem to be sufficient to get them even close to normality.  They are very, very skewed, and the truncation at the high end further complicates this by adding a noticeable blob of density at the ceiling.  Interestingly, however, the sample-wide nil and TB result distributions are quite similar, with the only difference being that that blob on the ceiling is much larger for the TB results." CreationDate="2010-09-25T14:57:14.367" UserId="71" />
  <row Id="3947" PostId="3075" Text="Thanks for taking the time to read and answer this beast of a question!" CreationDate="2010-09-25T14:57:47.267" UserId="71" />
  <row Id="3948" PostId="2915" Text="I don't know what starting a bounty mean?  I have been asked to start one.  What are its implications?" CreationDate="2010-09-25T16:06:10.577" UserId="1329" />
  <row Id="3949" PostId="1001" Text="@Ampleforth Indent your code with 4 spaces to get it converted into code. Or, equivalently, select it and use the button with binary data." CreationDate="2010-09-25T17:05:36.933" UserId="88" />
  <row Id="3950" PostId="1001" Text="mdb: Not sure what you mean. I would not ask if it would all be correct and without question. I am not sure what you mean with indenting. I hope it has nothing do to with using this website 'right' ;)" CreationDate="2010-09-25T19:22:40.460" UserId="608" />
  <row Id="3951" PostId="3078" Text="Thank you for your input. I will read more about the Anderson-Darling test, which I have not heard of . Regarding the Chi-Square test, doesn't this test require that the distribution is Chi Square? I agree that all other assumptions are very relaxed..." CreationDate="2010-09-25T19:28:40.227" UserId="608" />
  <row Id="3952" PostId="1001" Text="@Ampleforth I think it's just to remind you that this is a Markdown enabled website, so you can benefit from easy syntax highlighting (which facilitates reading) through md markup (http://j.mp/9bQHMC) -- or the utilities provided in the on-line editor." CreationDate="2010-09-25T21:38:14.443" UserId="930" />
  <row Id="3953" PostId="3069" Text="Would you care to explain why you couldn't also look at R?" CreationDate="2010-09-25T23:32:31.163" UserId="334" />
  <row Id="3954" PostId="3069" Text="@DirK: I've hardly heard of R. Moreover I wanted to learn some programming language like Python, and then again too I don't think R is anywhere close to python, IMO. I hope it answers your question." CreationDate="2010-09-25T23:37:38.127" UserId="1102" />
  <row Id="3955" PostId="3069" Score="1" Text="Poke around a little here and at StackOverflow in terms of what people recommend for *statistical analysis and programming*. Many of us feel that there is no real alternative to R.  But just like  beauty, this is in the eye of the beholder, so good luck." CreationDate="2010-09-25T23:41:07.090" UserId="334" />
  <row Id="3956" PostId="2897" Text="@mark: the result you state for [sub]gaussian tails does not look right. according to the bound [$A\sqrt{p}$] you cite, the $p^{th}$ norm of a centered gaussian variable would not [in the limit] exceed 1. but the $p^{th}$ norm of a rv tends to its ess sup, which is $+\infty$ for a gaussian variable." CreationDate="2010-09-26T02:00:25.923" UserId="1112" />
  <row Id="3957" PostId="3078" Text="Ampleforth, I am not sure that is the case.  I think you can test all sorts of data with the Chi Square test." CreationDate="2010-09-26T04:18:22.340" UserId="1329" />
  <row Id="3958" PostId="3076" Text="+1 I like the second point (I use roxygen + git). The first point makes me think also of the possibility to give your code to another statistician that will be able to reproduce your results at a later stage of the project, without any help." CreationDate="2010-09-26T09:52:17.847" UserId="930" />
  <row Id="3959" PostId="3082" Score="3" Text="I just can't understand why people think this swapping one is 'simpler' or 'more naive' than FY... When I was solving this problem for a first time I have just implemented FY (not knowing it has even a name), just because it seemed the simplest way to do it for me." CreationDate="2010-09-26T09:56:18.700" UserId="88" />
  <row Id="3960" PostId="3044" Text="@Jeromy Thanks for the link and the pointer to the `equivalence` package." CreationDate="2010-09-26T09:59:49.067" UserId="930" />
  <row Id="3961" PostId="3082" Text="@mbq: personally, I find them equally easy, although I agree that FY seems more &quot;natural&quot; to me." CreationDate="2010-09-26T10:29:21.267" UserId="582" />
  <row Id="3962" PostId="3079" Text="I´ll look into the matching package. My first analysis does control for the same factors as implemented in the match procedure in a multivariate linear regression model. Thx for the feedback.." CreationDate="2010-09-26T11:50:40.637" UserId="1291" />
  <row Id="3963" PostId="3082" Text="When I researched shuffling algorithms after writing my own (a practice I have since abandoned), I was all &quot;holy crap, it's been done, and **it has a name**!!&quot;" CreationDate="2010-09-26T13:55:04.440" UserId="830" />
  <row Id="3964" PostId="3081" Text="I particularly like the likelihood ratio as a means of aggregating evidence in meta-analysis; if you have sufficient data to compute them for each study, you simply compute the product across studies to represent the aggregate evidence for/against a hypothesis." CreationDate="2010-09-26T14:36:21.077" UserId="364" />
  <row Id="3965" PostId="3086" Text="How many individuals are used to estimate the item intercorrelations? What software is used for computing both correlations?" CreationDate="2010-09-26T15:42:05.733" UserId="930" />
  <row Id="3966" PostId="3073" Text="thanks for translating. I figured it would be an easy exercise, and I learned some R from it" CreationDate="2010-09-26T15:59:52.410" UserId="795" />
  <row Id="3967" PostId="3089" Text="@shabbychef Yes, it's the MLE of correlation in a two-way table, see Hamdam (1970), http://j.mp/9SN7Lk, or Brown &amp; Benedetti (1977), http://j.mp/aJjjzu." CreationDate="2010-09-26T17:11:20.817" UserId="930" />
  <row Id="3968" PostId="3088" Text="@Cyrus But aren't these steps already part of meta-analysis?" CreationDate="2010-09-26T17:14:57.470" UserId="930" />
  <row Id="3969" PostId="3086" Text="matrix is 5000 x 300. Software is SAS. PROC CORR and PLCORR. Convergence was not reached so the number of iterations have been changed to 100 iterations." CreationDate="2010-09-26T17:29:12.980" UserId="1154" />
  <row Id="3970" PostId="3091" Score="1" Text="You may be interested in this related question, http://stats.stackexchange.com/q/2628/930." CreationDate="2010-09-26T18:30:37.800" UserId="930" />
  <row Id="3971" PostId="3093" Text="Thanks for both of your responses. I will dig a little further into some of the references you suggested chi. I guess what I was getting at in my question, was aside from the issue of spatial effects, if I actually did have population-wide independent data, would classical hypothesis tests suffice - since they are based on sampling theory? Is multi-level modelling the only approach here? Forests aren't really nested in counties, rather I'd say they make up a measured proportion of each county, and those measurements are correlated in space..." CreationDate="2010-09-26T20:16:16.410" />
  <row Id="3972" PostId="3093" Text="@kip Would you mind explaining &quot;they make up a measured proportion of each county&quot;? I initially thought there was a % of Lyme disease recorded at each site (in your case all the forests in each country)." CreationDate="2010-09-26T20:29:16.593" UserId="930" />
  <row Id="3973" PostId="3093" Text="@kip About your second point, well if you consider you have a set of measurements collected on a fixed period of time on all possible statistical units (this is what you called your population), you can apply inferential procedures provided you are willing to assume there is an evolving generating model underlying the observed data (but see the related question I put in comment for a thorough discussion, e.g. by @ars, on this point)." CreationDate="2010-09-26T20:29:53.787" UserId="930" />
  <row Id="3974" PostId="3089" Text="@chl thanks, I thought I that was the case, but it always seems like a too remarkable fact because the cutoffs are unknown." CreationDate="2010-09-26T20:31:35.547" UserId="795" />
  <row Id="3975" PostId="3089" Score="1" Text="@shabbychef Yes, the threshold remains unknown. To my opinion, John Uebersax provides a great overview on TCs correlation, http://j.mp/bnhem7. " CreationDate="2010-09-26T20:34:57.947" UserId="930" />
  <row Id="3976" PostId="3088" Score="3" Text="@chl: True, but the point is that these steps get to the essence of the question.  A meta-analysis would be helpful only when there are many studies (not just two) and their merits have already been carefully evaluated.  The question before us is really asking how one goes about evaluating the quality of a study, or pair of conflicting studies, in the first place.  Cyrus has pointed to some of the many aspects of this; a reasonable treatment usually requires one or two semesters of university-level study.  In this light I think his use of the term &quot;heroic&quot; is somewhat understated!" CreationDate="2010-09-26T20:38:33.363" UserId="919" />
  <row Id="3977" PostId="3095" Score="2" Text="@whuber Thanks for your (always) enlightened remarks! You provide a more thorough response than mine wrt. OP (I must admit I'm a little bit biased toward epidemiology). [I'll +1 ASAP]" CreationDate="2010-09-26T20:40:18.207" UserId="930" />
  <row Id="3978" PostId="3081" Text="I commented on the (ir)relevance of meta analysis after Cyrus's answer, but upvoted this response for everything else, especially the bullet points." CreationDate="2010-09-26T20:40:59.527" UserId="919" />
  <row Id="3979" PostId="3080" Text="Maybe this should be CW?  There will not be a unique answer to this question and multiple perspectives and approaches might emerge." CreationDate="2010-09-26T20:42:23.330" UserId="919" />
  <row Id="3980" PostId="3088" Score="1" Text="@whuber Yes, I agree with you and @Cyrus. Of course, assessing the quality and trustiness of previous studies is a mandatory step (and it takes time to review every studies, especially when we have to contact authors because informations are missing in the MS); I just thought this was part of the meta-analysis, and the &quot;statistical part&quot; reduces to bringing a quantitative summary of trustworthy results." CreationDate="2010-09-26T20:50:34.073" UserId="930" />
  <row Id="3981" PostId="849" Text="+1 For the example." CreationDate="2010-09-26T21:27:40.270" UserId="776" />
  <row Id="3982" PostId="3093" Text="@chi Sorry for the confusion, I meant a proportion for forest cover, and a binary indicator for presence or absence at each country. I think I get what you are saying here, would the assumption of not having an evolving generating model essentially be an assumption of 'stationarity'? " CreationDate="2010-09-26T21:41:44.703" />
  <row Id="3983" PostId="3095" Text="@whuber Thanks a lot for this explanation. It really helps. The distinction between process and population is what I was missing (among other stuff!).. that the process is stochastic, and will vary from place to place and time to time" CreationDate="2010-09-26T21:47:13.227" />
  <row Id="3984" PostId="3080" Score="2" Text="@whuber I would vote against CW because even if there are different perspectives there is likely to be one *best* approach. This is similar to how the same hypothesis can be tested using different frameworks/models but there is likely to be one best approach.  " CreationDate="2010-09-26T22:17:32.030" />
  <row Id="3985" PostId="2761" Text="The unpaired student t test has two formulas.  The large sample formula is applied to samples with more than 30 observations.  The small sample formula is applied to samples with less than 30 observations.  The main difference in those formulas is how they calculate the pooled standard error.  The small sample formula is much more complicated and counterintuitive.  And, in reality it really makes very little difference.  I have tested that several times.  That's why I think most people have forgotten about this distinction.  And, they use most of the time the large sample formula.    " CreationDate="2010-09-26T22:45:11.677" UserId="1329" />
  <row Id="3986" PostId="3066" Text="@suncoolsu: many thanks! I have just followed your advice and ran prcomp. I also stored the loadings matrix it produced. But how can I use this matrix to group together the pages?" CreationDate="2010-09-26T23:00:03.900" UserId="1007" />
  <row Id="3987" PostId="3087" Score="1" Text="&quot;Let's be generous, too, and suppose you are actually selecting distinct pairs of indexes uniformly at random for your shuffles&quot;. I don't understand why that assumption can be made, and how it is generous. It does seem to discard possible permutations, resulting in an even less random distribution." CreationDate="2010-09-27T01:14:05.660" UserId="1421" />
  <row Id="3988" PostId="3085" Score="1" Text="+1. That demonstrates that the probability for a given card to end up in a given position approximates the expected ratio as the number of shuffles increases. However, the same would also be true of an algorithm that just rotates the array once by a random amount: All cards have an equal probability to end up in all positions, but there is still no randomness at all (the array remains sorted)." CreationDate="2010-09-27T01:30:05.487" UserId="1421" />
  <row Id="3990" PostId="3081" Text="@whuber @Gaetan's question assumes that one study is closer to the truth. I try to take a step back and situate variations in results between studies within a meta-analytic framework, acknowledging the possibility that the studies may be of equal quality, but that random sampling or substantive differences may be the explanation. " CreationDate="2010-09-27T02:43:35.087" UserId="183" />
  <row Id="3991" PostId="3081" Text="@whuber Even with two-studies it would be possible to form a meta-analytic estimate of the effect of interest. Of course, the confidence interval of the estimate of effect may be large. But a high degree of uncertainty is to be expected if only two studies have been conducted and they are giving conflicting results." CreationDate="2010-09-27T02:44:04.557" UserId="183" />
  <row Id="3992" PostId="3090" Text="Thanks a lot for the answer, I'll do the inquiry when I'll have more time " CreationDate="2010-09-27T04:56:38.863" UserId="223" />
  <row Id="3993" PostId="3070" Score="1" Text="@mbq: `shuffle` might be in a toolbox, is not stock matlab. could hardly be worse than builtin `randperm` which returns sort index of a random vector. Again, this is probably the wrong algorithm (I just learned about the Knuth-Fisher-Yates shuffle here on stats.SE).." CreationDate="2010-09-27T05:14:06.270" UserId="795" />
  <row Id="3994" PostId="3070" Text="@shabbychef True, I was trying to write about `randperm` and I've missed the name. And it is not a bad algorithm (it is correct and quite elegant), it is just blatantly inefficient because of a redundant sort." CreationDate="2010-09-27T07:19:12.820" UserId="88" />
  <row Id="3995" PostId="3089" Text="Thanks for explanation and references. But what i would know is when values obtained from tetrachoric are slightly different of person coefficient. Because they're supposed to be close. Indeed, in many programs, for estimation, Pearson r are used as start values for algorithm which compute tetrachoric correlation." CreationDate="2010-09-27T07:33:07.207" UserId="1154" />
  <row Id="3996" PostId="3085" Text="@Thilo: Sorry I don't follow your comment. An &quot;algorithm  rotates by a random amount&quot; but there's still &quot;no randomness&quot;? Could you explain further?" CreationDate="2010-09-27T08:21:59.907" UserId="8" />
  <row Id="3997" PostId="3085" Text="If you &quot;shuffle&quot; an N-element array by rotating it between 0 and N-1 positions (randomly), then every card has exactly the same probability to end up in any of the N positions, but 2 is still always located between 1 and 3." CreationDate="2010-09-27T08:26:39.787" UserId="1421" />
  <row Id="3998" PostId="3085" Score="1" Text="@Thio: Ah, I get your point. Well you can work out the probability (using exactly the same idea as above), for the Pr(A in position 2) and Pr(A in position 3) - dito for cards B and C. You will see that all probabilities tend to 1/3. Note: my answer only gives a particular case, whereas @whuber nice answer gives the general case." CreationDate="2010-09-27T08:46:09.197" UserId="8" />
  <row Id="3999" PostId="3082" Text="+1 A really nice question" CreationDate="2010-09-27T08:46:29.600" UserId="8" />
  <row Id="4000" PostId="3093" Text="@kip Finally, I like @whuber point of view: &quot;you're modeling a process, not a population&quot;, which seems pretty close to what I had in mind." CreationDate="2010-09-27T09:27:46.160" UserId="930" />
  <row Id="4001" PostId="3102" Text="@kwak Thanks. The idea of modelling the two curves together seems like a good one. However, I have theoretical reasons for wanting to quantify the degree to which the shape of the curves for the 100 and 400 metres are the same. Does the approach you mention do this?" CreationDate="2010-09-27T09:28:06.533" UserId="183" />
  <row Id="4002" PostId="3102" Score="1" Text="@Jeromy:&gt; I'm afraid not. On the other hand, i would pay attention to the issue of sample size and model complexity as 200 observations ain't a lot. " CreationDate="2010-09-27T11:42:14.847" UserId="603" />
  <row Id="4003" PostId="3080" Text="@Srikant: In any particular case I can imagine you could amass a strong defense to support your assertion.  In general, though--which is the present situation--the best answer will depend on the context.  As a simple (and incomplete) example, contemplate the differences between evaluating a pair of designed physical experiments (such as measuring the speed of light, where historically most of the confidence intervals have missed the truth!) and an observational study in the social sciences." CreationDate="2010-09-27T15:03:40.153" UserId="919" />
  <row Id="4004" PostId="3087" Score="1" Text="@Thilo: Thank you.  Your comment deserves an extended answer, so I placed it in the response itself.  Let me point out here that being &quot;generous&quot; does not actually discard any permutations: it just eliminates steps in the algorithm that otherwise would do nothing." CreationDate="2010-09-27T15:23:04.257" UserId="919" />
  <row Id="4005" PostId="3116" Score="2" Text="I agree with you and about a decade ago collected a bunch of such problems for a course (see http://www.quantdec.com/envstats/homework/class_03/paradox.htm ).  However, there is a strong pedagogical counter-argument: Probability itself can be confusing, so if you start off with counter-intuitive examples, you risk losing your audience forever (like Augustus DeMorgan, a pioneering probabilist, who later in life completely gave up on probability as hopelessly difficult!).  So caution is in order here, especially if you want to *motivate* people in an *introductory* setting." CreationDate="2010-09-27T15:32:34.597" UserId="919" />
  <row Id="4006" PostId="3080" Text="@whuber Perhaps, we should continue this conversation on meta. I admit that I am still fuzzy about when to use CW and when not to but to take up your point: the very best answer to this question would then be that the answer is context dependent and explain why via a few examples. In any case, I somehow feel that this question should not be CW but I am unable to articulate any more reasons beyond the ones I have outlined above." CreationDate="2010-09-27T15:55:20.627" />
  <row Id="4007" PostId="3115" Score="1" Text="Have a look at my question here: http://stats.stackexchange.com/questions/1881/analysis-of-cross-correlation-between-point-processes " CreationDate="2010-09-27T16:35:52.950" UserId="582" />
  <row Id="4008" PostId="3070" Score="1" Text="@mbq: the other good part about `randperm` is that it is affected by the seeding of `randn`, whereas a mex'ed version of Knuth-Fisher-Yates perhaps cannot access the randn seed 'internally', and a pure .m version of shuffle would probably be too slow." CreationDate="2010-09-27T16:38:31.493" UserId="795" />
  <row Id="4009" PostId="3100" Text="When are two curves &quot;consistent,&quot; Jeromy?  (Without a clear definition, many different answers are possible.)  Your second initial thought is suggestive, but not dispositive.  For example, unless you choose just the right way to express the results--should they be times, speeds (miles per hour), or inverse speeds (hours per mile), for instance?--then you might fail to identify and quantify a &quot;consistency&quot; that is really present.  In particular, I would **expect** $\theta_2$ to be smaller for the shorter race." CreationDate="2010-09-27T16:45:07.457" UserId="919" />
  <row Id="4010" PostId="2897" Text="Thanks for catching that.  I forgot the exponent on the RHS; it's corrected now." CreationDate="2010-09-27T16:52:17.833" UserId="89" />
  <row Id="4011" PostId="3119" Text="Please elaborate : do you want to scale x2 so it lies between 0 and 0.5 like x1, or do you want to be able to predict x1 from x2?" CreationDate="2010-09-27T17:00:17.347" UserId="1124" />
  <row Id="4012" PostId="3116" Text="I think it causes polarization. The students who are not interested in mathematics/probability will become confused, and the inquisitive/interested students will be inspired to learn more. Like you said, it might be best to exercise caution. Nothing could be worse than a confusing teacher presenting a confusing example!" CreationDate="2010-09-27T17:12:43.163" UserId="1118" />
  <row Id="4013" PostId="3087" Text="This problem can be analyzed fully as a Markov chain on the Cayley graph of the permutation group.  Numerical calculations for k = 1 through 7 (a 5040 by 5040 matrix!) confirm that the largest eigenvalues in size (after 1 and -1) are exactly $(k-3)/(k-1) = 1 - 2/(k-1)$.  This implies that once you have coped with the problem of alternating the sign of the permutation (corresponding to the eigenvalue of -1), the errors in *all* probabilities decay at the rate $(1 - 2/(k-1))^n$ or faster.  I suspect this continues to hold for all larger $k$." CreationDate="2010-09-27T17:48:33.853" UserId="919" />
  <row Id="4014" PostId="3119" Text="x1 and x2 are predictions from 2 different classifiers. I want to scale x2 so it lies between 0 and 0.5 like x1. " CreationDate="2010-09-27T17:57:12.317" />
  <row Id="4015" PostId="3121" Score="1" Text="I'm not sure what this means; do you have the numbers 1-100 in a bag, and are going to pick them out one at a time without replacement?" CreationDate="2010-09-27T18:05:22.537" UserId="795" />
  <row Id="4016" PostId="3119" Text="try squaring your x2. it will look more like x1, then." CreationDate="2010-09-27T18:07:25.700" UserId="795" />
  <row Id="4017" PostId="3121" Score="1" Text="@shabbychef: The formula in that case is n = 0, because the average and median are both 50.5! :-)" CreationDate="2010-09-27T18:13:22.507" UserId="919" />
  <row Id="4018" PostId="3121" Score="2" Text="@whuber: yes, this is why I'm not sure what is being asked. although the question could be: how many numbers must I pick until I pick one of 49,50,51 or 52, with 95% probability..." CreationDate="2010-09-27T18:32:22.263" UserId="795" />
  <row Id="4019" PostId="3108" Score="1" Text="@csgillespie Is there an explanation for the appearance of 7?" CreationDate="2010-09-27T18:52:29.477" />
  <row Id="4020" PostId="3119" Text="Let me explain the problem in another way. After adjusting one solution to the scale of the other I want to fit linear regression so that intercept=0 and slope=1." CreationDate="2010-09-27T19:16:39.193" />
  <row Id="4021" PostId="3119" Text="@Bob run a linear regression to get $x_1 = m x_2 + b$, then transform   as $\hat{x_2} \leftarrow m x_2 + b$. Then a linear regression of $\hat{x_2}$ vs $x_1$ will have intercept 0 and slope 1." CreationDate="2010-09-27T19:28:36.470" UserId="795" />
  <row Id="4022" PostId="3119" Text="@shabbchef: You're right, but the relationship between $x_1$ and $x_2$ will still be curvilinear.  One needs to find *nonlinear* re-expressions of the variables before removing any &quot;tilt&quot; via linear regression." CreationDate="2010-09-27T19:49:00.890" UserId="919" />
  <row Id="4023" PostId="3121" Score="2" Text="@shabbychef: I completely agree with you; my comment was just an amusing way to emphasize the vagueness.  The *answers* are obvious--sequential methods or standard experimental design methods, with a reference to the literature on the relative power of robust methods to handle the case of the median--but exactly *which* answer is appropriate (or best) will depend on how the numbers are being obtained and assumptions about their possible distribution.  E.g., if the numbers are in the range $[1,100]$ *and that's all you know,* I can guarantee 2% relative error with 95% confidence in 9228 draws." CreationDate="2010-09-27T20:23:30.093" UserId="919" />
  <row Id="4024" PostId="3121" Score="1" Text="(continued)...But if you use a sequential method and assume normality and it turns out all the values are around 98 - 100, then you can stop after about a half dozen draws!" CreationDate="2010-09-27T20:26:05.950" UserId="919" />
  <row Id="4025" PostId="3125" Score="2" Text="You assumed a uniform distribution, so why do a simulation?  An exact answer is straightforward to obtain.  (Indeed, my facetious comment still applies: by assuming a uniform, you know *a priori* that the mean is 50.5, so you don't need to pick *any* numbers at all!)  But if *all* you know is that the numbers are in the range [1,100], you have to plan for the worst case, which will require up to 9000+ draws (when half the values are 1 and half are 100).  First we need to hear from the OP concerning what assumptions are valid to make here." CreationDate="2010-09-27T20:30:14.263" UserId="919" />
  <row Id="4026" PostId="3119" Text="(Sorry about misspelling your handle: it's the cleverest one around and so deserves to be typed correctly!)" CreationDate="2010-09-27T20:31:50.283" UserId="919" />
  <row Id="4027" PostId="3125" Text="@whuber, I assumed the numbers where NOT put back in the &quot;pool&quot; (that is why with 100 draws you have 100% of the trials giving the good result). Also, the simulation works for ANY set of 100 numbers, I just used 1:100 because that's what the question asked (well, I guess at least), but it can be any set of 100 numbers :)" CreationDate="2010-09-27T20:49:32.763" UserId="582" />
  <row Id="4028" PostId="3121" Score="1" Text="@whuber: I would think this was a homework question, but it was so ambiguously worded." CreationDate="2010-09-27T20:50:35.070" UserId="795" />
  <row Id="4029" PostId="3126" Text="@ars Thanks for the link on the R-help mailing list. There is also this one on s-news: http://j.mp/8Zol8W." CreationDate="2010-09-27T20:52:31.620" UserId="930" />
  <row Id="4030" PostId="3125" Text="Anyway, you can easily modify the script to allow for resampling the same number multiple times (add `replace=TRUE` in the `sample` call). The result turns out to be around 5000 in that case" CreationDate="2010-09-27T21:06:16.690" UserId="582" />
  <row Id="4032" PostId="3119" Text="@whuber the OP states he wants to 'scale x2' to get a linear fit with given slope and intercept. I live to serve." CreationDate="2010-09-27T21:37:00.250" UserId="795" />
  <row Id="4033" PostId="3119" Score="2" Text="@shabbychef: Ditto.  I think I got lucky this time in guessing the intent.  We have to recognize that non-statisticians often are unaware of the technical distinctions we make.  After all, a great deal of statistical consulting amounts to figuring out what someone might *really* be asking you!  Here the best clue is that the OP himself suggested a nonlinear re-expression (&quot;link&quot;).  The scatterplot he posted helps immensely in understanding the situation, too.  The combination of those useful clues seemed sufficient to venture an answer rather than to keep probing for clarifications." CreationDate="2010-09-27T21:43:39.537" UserId="919" />
  <row Id="4034" PostId="3121" Score="2" Text="@shabbychef: Yep, the ambiguity is certainly there.  The followup about the median could make for some difficult homework, though: you have to make a lot of assumptions and the analysis is not found in elementary courses.  (Under some distributional assumptions the median is a more efficient estimator of central tendency than the mean, but most of the time in practice it's less efficient.)" CreationDate="2010-09-27T21:47:05.547" UserId="919" />
  <row Id="4035" PostId="3108" Text="My general hand-waving explanation is this: people avoid {1, 5, 10} because they are too obvious and therefore &quot;not random&quot;. Numbers less than 5 - well who wants a small RN! People then tend to go for the middle number between 5 and 10. I've tried this example six times now (in classes of size ~100) and it's worked each time." CreationDate="2010-09-27T21:49:28.800" UserId="8" />
  <row Id="4037" PostId="3125" Score="1" Text="@nico: Now you're getting to the heart of the matter: the trick is to find a method that works for the full range of possible pool contents consistent with one's assumptions.  All a simulation can do is indicate what can be achieved by sampling when one particular assumption is true.  Your modification already reveals what's at stake: the difference between a sample of 98 and one of 5000 is enormous.  This suggests looking a Bayesian and/or sequential sampling methods instead of designing a fixed sample size based on a frequentist assumption." CreationDate="2010-09-27T22:09:29.823" UserId="919" />
  <row Id="4038" PostId="3108" Score="1" Text="And of course, 17 is the least random number.  http://www.catb.org/~esr/jargon/html/R/random-numbers.html  but my favorite random number is 37: http://jtauber.com/blog/2004/07/09/37_is_a_psychologically_random_number/  (though, also see http://scienceblogs.com/cognitivedaily/2007/02/is_17_the_most_random_number.php )" CreationDate="2010-09-27T23:17:01.850" UserId="251" />
  <row Id="4039" PostId="3111" Score="1" Text="This is my favorite example too (HIV test), but unsure if conditional probability is too &quot;advanced&quot; given the introductory nature (plenty of studies showing that it's not too intuitive).  If you do teach this, I recommend perusing Gigerenzer and the frequency method: http://library.mpib-berlin.mpg.de/ft/gg/GG_How_1995.pdf " CreationDate="2010-09-27T23:21:00.407" UserId="251" />
  <row Id="4040" PostId="3111" Text="@ars:&gt; maybe first you state them all the relevant informations in table form, then the problem &quot;what do you think is  p(AIDS|test=1)?&quot;, then the counter intuitive punchline, only then  you show them the problem re-casted as a 'tree' (where the final 4 nodes are all possible cases) and the branches show the respective probability. In my experience, the last leg need not be understood by everybody, but it has to convey the importance of having a principled way of thinking about these issues." CreationDate="2010-09-28T00:08:12.337" UserId="603" />
  <row Id="4041" PostId="3100" Text="@whuber Thanks for the point about $\theta_2$. The analysis is motivated by a theoretical and qualitative interest in consistency. Thus, my question is asking &quot;what is a good way to quantify consistency&quot;. I.e., what is a good definition of consistency?" CreationDate="2010-09-28T00:51:03.830" UserId="183" />
  <row Id="4043" PostId="3066" Text="Hello Laramichaels,&#xA;please find my answer below.&#xA;" CreationDate="2010-09-28T04:05:03.050" UserId="1307" />
  <row Id="4044" PostId="3101" Text="Can you explain why this is a pyramind and not a triangle?  Or a triangle and not a ladder?  I'm not being snarky -- I'm wondering if I'm missing something significant that this visualization is trying to convey." CreationDate="2010-09-28T05:03:31.033" UserId="251" />
  <row Id="4045" PostId="3125" Text="@whuber: OK, I understand what you're saying now. Actually, to clarify on my previous comment: the code works for any 100 number set, but it does not give the same result. If we only use numbers between 90 and 100, for instance, 2 or 3 draws are enough to estimate the mean. Still, an interesting problem" CreationDate="2010-09-28T07:03:40.353" UserId="582" />
  <row Id="4046" PostId="3089" Text="@shabbychef Sorry, I didn't read your updated answer before posting mine. The bias would increase when the cutoff depart from the mean, but it will be even worse when the departure is assymmetrical wrt. joint density." CreationDate="2010-09-28T07:43:22.707" UserId="930" />
  <row Id="4047" PostId="3089" Text="Thanks a lot. Great word. It will be helpful for me." CreationDate="2010-09-28T08:06:34.747" UserId="1154" />
  <row Id="4048" PostId="3136" Score="2" Text="@Skarab Maybe I'm totally off, but wouldn't you expect that the frequency of any word will be inversely proportional to its rank in the frequency table of words, according to Zipf's law (http://j.mp/9er2lv)? In this case, check out the `zipfR` package." CreationDate="2010-09-28T09:29:57.893" UserId="930" />
  <row Id="4049" PostId="3136" Text="I agree with @chl - it would be minor miracle if your data was normally distributed. Perhaps another question about what you want to do with the data would be worthwhile. Don't reinvent the wheel!" CreationDate="2010-09-28T09:32:54.513" UserId="8" />
  <row Id="4052" PostId="3129" Score="1" Text="@Rob I agree with you, although most of the sample size calculation that are done when devising RCTs are based on parametric models. I like the bootstrap approach, but it appears that very few studies rely on it. I just found those papers that might be interesting: http://bit.ly/djzzeS, http://bit.ly/atCWz3, and this one goes in the opposite direction http://bit.ly/cwjTHe for health measurement scales." CreationDate="2010-09-28T11:22:16.953" UserId="930" />
  <row Id="4059" PostId="3140" Text="Can you please make your question a bit more understandable? It seems as you want to use Cross-Validation to make classification itself, but this is nonsense." CreationDate="2010-09-28T14:50:24.703" UserId="88" />
  <row Id="4060" PostId="411" Text="Thank you for this great overview :) Are there books you know that especially discuss such distance measures for distributions in detail? I am particularly interested in non-parametric measures that operate with minimal assumptions. " CreationDate="2010-09-28T15:03:02.660" UserId="608" />
  <row Id="4061" PostId="3136" Score="1" Text="How could your data be distributed according to a model that gives non zero probability to negative occurrence? " CreationDate="2010-09-28T15:05:49.790" UserId="603" />
  <row Id="4062" PostId="3100" Score="1" Text="@Jeromy: That's not a statistical question, LOL!  But (to stave off possible objections) I agree that its answer can be usefully informed by statistical thinking.  &quot;Consistency&quot; depends on what you are studying and what use you will make of a decision that two response curves are &quot;consistent&quot; or not.  In some applications it would be enough that they are both increasing or decreasing; in others it would amount to a test of equality of all parameters.  Without more information, one can only guess where along this spectrum your needs fall." CreationDate="2010-09-28T15:06:45.227" UserId="919" />
  <row Id="4064" PostId="3130" Score="1" Text="It's *binomial* estimation.  (There is no marking or recapturing at all. which leads to hypergeometric estimation.)" CreationDate="2010-09-28T15:11:05.110" UserId="919" />
  <row Id="4065" PostId="3136" Text="What is the reason for doing this test?" CreationDate="2010-09-28T15:12:36.353" UserId="919" />
  <row Id="4066" PostId="3134" Score="1" Text="The answer of 25000, which is obviously ridiculous no matter what the true state of affairs may be, indicates that a power calculation is not appropriate here.  The answer of 9228 I obtained guarantees that no matter what the distribution may be (provided only that its values all lie between 1 and 100), 95% of all simple random samples (without replacement) with n=9228 will lie within 2% of the mean.  Power is not relevant!" CreationDate="2010-09-28T15:31:33.217" UserId="919" />
  <row Id="4067" PostId="3123" Score="1" Text="The asymptotic distribution of the median is Normal with variance inversely proportional to the square of the pdf at the median.  Thus, experimental designs for median estimation have to make some specific assumptions about the nature of the distribution near its middle.  (See http://mathworld.wolfram.com/StatisticalMedian.html for example.)  Usually people study a range of parametric alternatives to compare the efficiency of the median to that of the mean; relative efficiencies translate via the usual square-root law into relative sample sizes." CreationDate="2010-09-28T15:35:51.293" UserId="919" />
  <row Id="4069" PostId="3089" Score="1" Text="@chl: not a problem. I like that you posted R code (I am slowly learning R); my version is in homebrew Matlab, which I cannot share. Having code that the OP can test is definitely the way to go." CreationDate="2010-09-28T16:40:17.440" UserId="795" />
  <row Id="4070" PostId="3145" Score="1" Text="@shabbychef Huh, factors are well handled using the `factor` command, so no need to rely on dummy coding (otherwise, it's accessible through `model.matrix(lm(x~a*b))` for my toy example)." CreationDate="2010-09-28T18:01:10.887" UserId="930" />
  <row Id="4071" PostId="3145" Score="1" Text="@chl thanks. I know very little R, but suspected this simple algebraic trick could be applied in any case, or at least illustrate what is going on 'under the hood'" CreationDate="2010-09-28T18:05:48.227" UserId="795" />
  <row Id="4072" PostId="3145" Text="@shabbychef Yes, definitely! this was just a precision. (and you have my +1)" CreationDate="2010-09-28T18:14:52.873" UserId="930" />
  <row Id="4073" PostId="3144" Text="Thanks! It did work, question: it did not force the condition that coefficients sum up to 0, but minimized the sums for both factors. Is there a way to force the ==0 condition, or it simply does not do that since it would affect the error ?" CreationDate="2010-09-28T19:26:09.390" UserId="1439" />
  <row Id="4074" PostId="3148" Text="I've performed a Chi-Squared test, but that was based on the subjective results that I received from user surveys. They sound quite similar though?" CreationDate="2010-09-28T19:37:35.640" UserId="1441" />
  <row Id="4075" PostId="3148" Text="Yes, you could do a chi-squared test as well if that is familiar to you. Both the test I recommended and the chi-squared test will give you the same answer." CreationDate="2010-09-28T19:43:23.337" />
  <row Id="4076" PostId="3066" Text="@suncoolsu: I am dealing with a similar problem, but I want to &quot;cluster&quot; the individuals that have the same &quot;dynamics&quot; (actually I have a huge number of timeseries per regions and I want to model them). I was thinking to use pam with the correlation distance (1-rho). Is this a recommended way? Could you please suggest some paths to explore?   " CreationDate="2010-09-28T19:59:12.917" UserId="1443" />
  <row Id="4077" PostId="3125" Text="I think this is what I was looking for nico. The numbers arent put back into the pool." CreationDate="2010-09-28T20:05:38.950" />
  <row Id="4078" PostId="3148" Text="Well I'd rather use a different method because I have already implemented the Chi-Squared test on other results..I'm just not sure if this test is too similar to the Chi-Squared one or not..?" CreationDate="2010-09-28T20:34:14.790" UserId="1441" />
  <row Id="4079" PostId="3125" Text="by the way, how can I copy the formula and change the numbers?" CreationDate="2010-09-28T20:44:53.360" />
  <row Id="4080" PostId="3143" Score="1" Text="Please **do not crosspost** simultaneously here and on SO." CreationDate="2010-09-28T20:47:17.103" UserId="334" />
  <row Id="4082" PostId="3101" Text="Hi ars, it could well be the triangle of evidence. The reason for the word pyramid is probably historical, I am simply using it as it was taught to me.  BTW, it shouldn't be a leader, since the width of the triangle also reflects the abundance of such an evidence in practice (Or at least, that's my guess :) )" CreationDate="2010-09-28T21:09:48.540" UserId="253" />
  <row Id="4083" PostId="3125" Text="@Greg, I am not completely sure what you are asking... if you want to change the numbers just change the `numbers` variable. For instance `numbers &lt;- c(1, 5, 18, 32, 36, 42, 95, 97)` or whatever you want to use :) The rest of the script will run for any set of numbers." CreationDate="2010-09-28T21:10:06.900" UserId="582" />
  <row Id="4084" PostId="3144" Score="1" Text="@Vytautas: I think you're forgetting the reference level for your factor which is not shown (since it's the baseline for the others)." CreationDate="2010-09-28T21:14:14.897" UserId="251" />
  <row Id="4085" PostId="3125" Text="@nico, Im asking how I can run the script for myself. Like is there a website I can cut and paste it to? I tried on here but it didnt display a graph." CreationDate="2010-09-28T21:17:02.977" />
  <row Id="4086" PostId="3101" Text="@Tal: I hadn't come across this before, so just curious for any insight into what it was saying.  Your point about abundance of evidence makes sense.  Thanks.  :)" CreationDate="2010-09-28T21:32:23.253" UserId="251" />
  <row Id="4087" PostId="3144" Score="1" Text="@Vytautas or take a look at *Interpreting model matrix columns when using contr.sum*, http://j.mp/9pNFQe." CreationDate="2010-09-28T21:43:49.100" UserId="930" />
  <row Id="4088" PostId="3152" Score="1" Text="Hi csgillespie, the code for the example you gave has a lot of &quot;before and after&quot; additions to the code (http://www.stat.uni-muenchen.de/~leisch/Sweave/example-2.Snw) including the fact that I will need to learn (some) latex+sweave.  Which is something to do one day, but I am looking if there is a simpler alternative." CreationDate="2010-09-28T22:03:49.940" UserId="253" />
  <row Id="4090" PostId="3152" Text="@Tal I'll add to @csgillespie `odfSweave` and `brew` (probably less flexible). Check out also *Sweave and LaTeX*, http://j.mp/dah7Hr." CreationDate="2010-09-28T22:12:53.950" UserId="930" />
  <row Id="4092" PostId="3152" Score="2" Text="@Tal I guess you can always write some code (Python, Ruby or whatever) to store R output in temp file, as well as images, calling R in BATCH mode, and then compile all of them in a tex file after inserting the temp files. Actually, this is what is done for embedding R code in ConTeXt file. " CreationDate="2010-09-28T22:22:10.707" UserId="930" />
  <row Id="4093" PostId="1931" Text="@Kwak: A continuous mapping from R^n to R cannot induce a total order.  Technically, I was wrong, because any bijection from R^n to R will &quot;order&quot; R^n via the order on R, but this order will not be compatible with any of the metric structure on R^n (which is an essential part of the concept of a &quot;median&quot;), and that was the spirit of my comment." CreationDate="2010-09-28T22:31:51.093" UserId="919" />
  <row Id="4094" PostId="3152" Text="@Tal Huh, I browse through your `r2wd` example: it's awful (ok, I never use Micro$oft tools, and my appreciation is certainly biased)! I'm sure there's a better way to handle this with Python scripting and tex/html export... Just to call for other responses..." CreationDate="2010-09-28T22:37:59.900" UserId="930" />
  <row Id="4096" PostId="3156" Text="Hi. Please see my edit above. Thanks." CreationDate="2010-09-28T23:25:34.897" UserId="1102" />
  <row Id="4097" PostId="3156" Text="What's p?  what's h?  Start at the beginning and just say what you're measuring, why the distributions are different, and what you want to accomplish." CreationDate="2010-09-28T23:28:48.340" UserId="601" />
  <row Id="4098" PostId="3156" Text="Perhaps if I guess some things it would help??  Are the p's the probability of the mean showing up in each distribution?  You want to say what one is farthest fromthe mean?... closest to it?" CreationDate="2010-09-28T23:46:38.250" UserId="601" />
  <row Id="4099" PostId="3156" Text="p is the probability, under the null hypothesis, of observing a value as extreme or more extreme of the test statistic. This might be of help to you: http://www.mathworks.com/help/toolbox/stats/ttest.html ... I used Matlab's 'ttest' function." CreationDate="2010-09-28T23:53:44.320" UserId="1102" />
  <row Id="4100" PostId="3156" Text="@John: This is my question (http://stats.stackexchange.com/questions/2639/how-to-compare-different-distributions-with-reference-truth-value-in-matlab) where I have defined my problem about what I want to accomplish. Thanks." CreationDate="2010-09-28T23:55:29.797" UserId="1102" />
  <row Id="4101" PostId="3066" Text="@Musa .. Can you be bit clearer. I don't think I understand the &quot;dynamics&quot; mentioned by you. Definitely _pam_ is OK for clustering. But you can also try the R packages pvclust and hopach as mentioned by me. Also, SOM (self organizing maps) are a different way of looking at clustering. Please see Ripley and Venable (2002) book - MASS for further details. The book offers a thorough treatment of clustering." CreationDate="2010-09-29T00:26:35.203" UserId="1307" />
  <row Id="4102" PostId="3157" Text="@suncoolsu: Thanks for your answer. Basically I wanted to compare my distributions using CI, for which I guess we don't need any assumptions, and I found ttest function on Matlab which does that, so I used it. Can you suggest how to compare them? " CreationDate="2010-09-29T00:54:24.883" UserId="1102" />
  <row Id="4103" PostId="3148" Text="@Mark The two tests give you the same information in terms of what you would conclude regarding the effectiveness of method 1 vs method 2 but the way they approach the issue is different. Any particular reason to avoid the chi-square? Feel free to choose either one." CreationDate="2010-09-29T01:28:14.663" />
  <row Id="4104" PostId="3134" Text="Conceded whuber. I'll modify the answer to reflect this, although the question appears to have changed in the interim. BTW it's &quot;with replacement&quot;." CreationDate="2010-09-29T02:31:20.320" UserId="521" />
  <row Id="4105" PostId="3100" Text="@whuber Thanks. I see the translation of a theoretical question into a statistical question as one of the most important skills that a data analyst can acquire. In most areas of statistics, there are multiple ways of making the translation, with a body of knowledge existing on when and why you would apply one approach over another. Thus, I'm interested in knowing whether any standard methods exist for quantifying consistency of two fitted curves and when and why you would apply one approach over another. I'll have a think about how I can edit the question to make my specific aims clearer." CreationDate="2010-09-29T04:14:33.540" UserId="183" />
  <row Id="4106" PostId="1013" Text="you mean the standard deviation..." CreationDate="2010-09-29T04:16:36.990" UserId="795" />
  <row Id="4107" PostId="3151" Text="@Tal great question. This would be really useful." CreationDate="2010-09-29T06:53:01.927" UserId="183" />
  <row Id="4108" PostId="3066" Text="@suncoolsu: Sorry for the poor formulation! I have 200 timeseries that I want to model (i.e. to simulate). I think that I can cluster &quot;similar&quot; (i.e. having the same behavior over time: the straight forward approach is to use the correlation) timeseries and simulate only the cluster timeseries..." CreationDate="2010-09-29T06:57:13.590" UserId="1443" />
  <row Id="4109" PostId="3161" Score="1" Text="@romunov How would images be automatically included?" CreationDate="2010-09-29T07:07:38.767" UserId="183" />
  <row Id="4110" PostId="3159" Text="@ars @grautur I'd like to add this one too: http://mkweb.bcgsc.ca/circos/tableviewer/" CreationDate="2010-09-29T07:26:35.843" UserId="930" />
  <row Id="4112" PostId="3136" Text="I want to estimate if the huge result of the Information Extraction is  correct. I want to check if the distribution of the entities found in the text follows my expectations (I know the domain and the text corpus). " CreationDate="2010-09-29T09:06:59.363" UserId="1389" />
  <row Id="4113" PostId="3066" Text="@Musa as you say that your data is a time series, the first step should be removing all the trends from the data .. as you don't mention the size and type of your data (ie size -- moderate, large etc and type -- count data, continous data?), I will assume your data is moderate sized and continuous. There are better methods than clustering if you just want to model the time series. Please check out the (one of the many) time series package in R named zoo. There are great books: http://bit.ly/dmsWtl and http://amzn.to/9DmG6b to accomplish this task. Clustering may not be v.informative in ur case" CreationDate="2010-09-29T09:37:01.803" UserId="1307" />
  <row Id="4114" PostId="3163" Text="@Misha What's wrong with @Joris's response on stackoverflow where you cross-posted?" CreationDate="2010-09-29T10:08:12.330" UserId="930" />
  <row Id="4115" PostId="3163" Text="link to StackOverflow Question: http://stackoverflow.com/questions/3817182/vcovhc-and-confidence-interval" CreationDate="2010-09-29T11:09:25.627" UserId="183" />
  <row Id="4116" PostId="3125" Text="@Greg: Ah, OK! Well, you need to install R, you can download it from http://cran.r-project.org" CreationDate="2010-09-29T11:24:11.803" UserId="582" />
  <row Id="4117" PostId="3164" Text="Hi mbq, do you mean to wrap every line of code, or to wrap the entire section ?" CreationDate="2010-09-29T11:25:40.087" UserId="253" />
  <row Id="4118" PostId="3164" Text="@Tal Rather a custom `print` and `plot` that formats ans redirect their output to the report file. R does not have any output handler to hook to :-(" CreationDate="2010-09-29T11:33:50.357" UserId="88" />
  <row Id="4119" PostId="3164" Text="@mbq Isn't this close to what `R2HTML` actually does (more or less)? " CreationDate="2010-09-29T11:37:47.827" UserId="930" />
  <row Id="4120" PostId="3163" Text="@chl &amp; @Jori:I'm sorry for crossposting but I looked at stackoverflow immediately before doing so and could only find the comment by @Bertelsen suggesting a crosspost. Looking at the time I cannot explain why I didnt find @Joris post. The answer by @Jori was just what I was looking for.  " CreationDate="2010-09-29T11:38:28.907" UserId="1291" />
  <row Id="4121" PostId="3164" Text="@chl R2HTML is R object to HTML converter, Tal wanted to capture textual output and images only, to, as I understand, make something like static Mathematica or Sage notebook. " CreationDate="2010-09-29T11:51:11.723" UserId="88" />
  <row Id="4122" PostId="3151" Text="Thank you for the kind words Jeromy :)" CreationDate="2010-09-29T12:00:39.047" UserId="253" />
  <row Id="4123" PostId="3164" Text="mbq: textual output includes tables/data.frames/lm.summary and so on.  It is true that outputting it as text (ascii) is fine.  Although the better solution would be something like R2HTML, but without the need to add all those lines of code around **every** output statement. " CreationDate="2010-09-29T12:03:16.220" UserId="253" />
  <row Id="4124" PostId="3164" Text="@mbq Ok, the `ascii` package I proposed works along the same way..." CreationDate="2010-09-29T12:04:53.937" UserId="930" />
  <row Id="4125" PostId="3164" Text="@Tal BTW, with `ascii` you not only get a readable plain text file but also an HTML page (asciidoc will format the txt file); less powerful than Sweave, but you work with just a text file, so..." CreationDate="2010-09-29T12:07:52.637" UserId="930" />
  <row Id="4126" PostId="3164" Text="Hi chl, thanks - the ascii package does look nice.  Though I still hope for a simpler solution for smaller snippets of code :)" CreationDate="2010-09-29T12:16:48.423" UserId="253" />
  <row Id="4127" PostId="3164" Score="1" Text="@chl Conversion is one thing, catching output (especially images) is something other. @Tal Again -- you'll not find it, print and plot cannot be hooked in any low-level enough way.  " CreationDate="2010-09-29T12:36:46.880" UserId="88" />
  <row Id="4128" PostId="3161" Text="@Jeromy with data uri scheme, see my answer." CreationDate="2010-09-29T12:38:33.630" UserId="88" />
  <row Id="4129" PostId="3164" Score="1" Text="@mbq (+1) Yes, I agree. This is why I initially suggested to rely on an external scripting language to call R in batch mode; now, if we look for a simpler solution, I do not know how we can do that." CreationDate="2010-09-29T12:42:11.950" UserId="930" />
  <row Id="4130" PostId="3165" Text="In your Q2, do you mean `cex.lab` instead of `cex.axis`?" CreationDate="2010-09-29T12:43:17.143" UserId="1390" />
  <row Id="4131" PostId="3166" Text="@ucfagls It won't fit text labels to the maximum available space in the margins (which is what the OP ask for, to my understanding)." CreationDate="2010-09-29T12:54:22.453" UserId="930" />
  <row Id="4132" PostId="3166" Score="1" Text="@chl; my opening line says I can't think of an answer to Q1, but Q2 *can* be done via my example, substituting in the maximum values for cex.lab in place of where I have `2` and `3` now. That was just an example of how to label the plot separately and never claimed to answer Q1. Maybe I misread/misinterpretted what Tal wanted with his Q2 though? Your reading of Q2's meaning seems obvious if Q1 is TRUE, hence I thought Tal was aksing the Q2 I answered...?" CreationDate="2010-09-29T13:11:01.580" UserId="1390" />
  <row Id="4133" PostId="3166" Text="@ucfagls (+1) That's right." CreationDate="2010-09-29T13:19:08.450" UserId="930" />
  <row Id="4134" PostId="3168" Text="@Shane Ouaouh! You're just summarizing in a few words a lot of R good practices!" CreationDate="2010-09-29T13:49:11.140" UserId="930" />
  <row Id="4136" PostId="3165" Text="Why would you want to do this? Surely when you include your graph within another document, the text sizes will now look too big." CreationDate="2010-09-29T14:18:30.887" UserId="8" />
  <row Id="4137" PostId="3170" Text="That's not working for me (you probably need to edit out the `cex.lab = mycex` part of the call to `plot()` to get that example to run) as the y-axis label is half outside the plot window and the x-axis label extends across the axis and into the plot region." CreationDate="2010-09-29T14:21:44.340" UserId="1390" />
  <row Id="4138" PostId="3134" Text="@Thylacoleo: Thanks about the BTW.  I mis-typed and intended to say *with* replacement." CreationDate="2010-09-29T14:33:06.620" UserId="919" />
  <row Id="4139" PostId="1013" Text="@shabbychef: You are right. Fixed." CreationDate="2010-09-29T14:35:05.990" UserId="56" />
  <row Id="4140" PostId="411" Text="@Ampleforth: I'm not familiar with any one source that discusses many of these distances in general, and what I have seen is very much from a mathematician's and not a statistician's perspective (hence my question).  I just happened to see a reference to a book *Probability for Statisticians* by Shorack which apparently discusses many such distances." CreationDate="2010-09-29T14:46:53.530" UserId="89" />
  <row Id="4141" PostId="3169" Text="Yes, I am aware of TeXexample.net. Besides showing all fancy features, I'm a bit disappointed in the number of 'normal' line plots they have (showing different styles)" CreationDate="2010-09-29T15:02:22.787" UserId="190" />
  <row Id="4142" PostId="3168" Text="This is not completely what I was looking for but thanks! The command in point 2 is very useful, only why use a bitmap format (png) when the result is pdf? Isn't it way better to have svg, pdf or pgf output?" CreationDate="2010-09-29T15:03:59.653" UserId="190" />
  <row Id="4144" PostId="3168" Text="@Peter See my updated response.  " CreationDate="2010-09-29T15:24:17.380" UserId="5" />
  <row Id="4145" PostId="3163" Text="@Misha Since you have obtained the answer on SO perhaps you can post that answer (with proper attribution etc) here and accept it for the benefit of future readers. That way the question will be self-contained here and readers do not have to click on the SO link to find out the answer." CreationDate="2010-09-29T15:54:02.340" />
  <row Id="4146" PostId="3157" Score="1" Text="@Harpreet When constructing confidence intervals, say a 95% CI for a difference of means, you are explicitely assuming a sampling distribution (t or z, i.e. that of your test statistic under $H_0$)." CreationDate="2010-09-29T16:28:10.380" UserId="930" />
  <row Id="4148" PostId="3163" Text="Sorry Misha. You're caught in the growing pains of the 'R' overlap between stats.stackexchange.com and SO. " CreationDate="2010-09-29T18:32:14.643" UserId="776" />
  <row Id="4149" PostId="3163" Text="@Brandon @Misha Yes, but it's just a little bit annoying, so maybe the complete answer can now be pasted here as suggested by @Srikant." CreationDate="2010-09-29T19:08:06.447" UserId="930" />
  <row Id="4152" PostId="3165" Text="This is really an R-code question, not a stats question." CreationDate="2010-09-29T19:18:32.653" UserId="601" />
  <row Id="4153" PostId="3064" Text="Is your dependent variable continuous or discrete? Or, perhaps, the underlying test result is continuous and it is converted into a discrete answer (i.e., 'positive', 'negative') depending on some cut-off? Could you also clarify why an individual would flip from negative to positive despite not being exposed to TB? A specific example (with some numbers thrown in) of such a flip may help." CreationDate="2010-09-29T19:21:19.027" />
  <row Id="4154" PostId="2611" Text="@Rok A very interesting question. I look forward to hearing from your results, if you want to share them..." CreationDate="2010-09-29T19:25:24.387" UserId="930" />
  <row Id="4156" PostId="3163" Text="@chl, annoying for you. But you have to remember that other people who are new to the community are going to be oblivious to what's wrong." CreationDate="2010-09-29T19:45:51.787" UserId="776" />
  <row Id="4157" PostId="3171" Text="I am not sure I see what the issue is. Don't you associate the responses of each respondent over time with some sort of dummy id (to preserve anonymity)? If so, you just have a repeated measures design with missing data. The missing data would be for those respondents who responded to a survey in the past but do not do respond to a later survey. Whether a respondent's answer changes or does not change over repeated measurements is not a data or sampling artifact but an indication of the stability or unstability of response for that individual.  " CreationDate="2010-09-29T19:57:21.750" />
  <row Id="4158" PostId="3171" Text="Thanks Srikant. I would agree in theory, but in practice the physical well-being of the subjects depends on their anonymity and we can't risk anything that might identify them from their answers. I expect there probably exists a robust system to solve this, but we're not security experts and neither can we afford one." CreationDate="2010-09-29T20:17:01.260" UserId="1343" />
  <row Id="4159" PostId="3163" Text="@Brandon Oh, no! It wasn't intended to be read as a reproach (the word &quot;annoying&quot; was perhaps too excessive, sorry I'm a French guy and I certainly did not find the good term); just to say that we often end up with posts on stackoverflow and this SE, with partly overlapping answers or no response at all..." CreationDate="2010-09-29T20:24:24.580" UserId="930" />
  <row Id="4160" PostId="3169" Text="@Laurent Nice that you cited TeXexample.net! Maybe too much for that purpose, anyway `tikz` is very powerful for illustrations and diagram." CreationDate="2010-09-29T20:38:03.697" UserId="930" />
  <row Id="4161" PostId="3177" Text="@shabbychef: Thanks! It is solved now, though with **hist** only." CreationDate="2010-09-29T22:14:39.953" UserId="1102" />
  <row Id="4162" PostId="3176" Text="why would you want to do that ? " CreationDate="2010-09-29T22:36:49.400" UserId="603" />
  <row Id="4163" PostId="3171" Text="So, as I understand it, you have no way to tell which response belongs to which respondent as you do not record any id consistently across respondents. Is that correct? I would imagine that you do know the sample size of each poll though, right?" CreationDate="2010-09-29T22:48:06.247" />
  <row Id="4164" PostId="3176" Score="2" Text="General remark -- try to give questions reasonable, informative titles and make them clear and understandable for people not working in your field. Also good question has a bigger chance for a satisfying answer." CreationDate="2010-09-29T23:35:58.257" UserId="88" />
  <row Id="4165" PostId="3064" Score="1" Text="@Matt The examples are really helpful to visualize the data. Another question regarding your caveat: &quot;the values clump at the floor and the ceiling and that the data are not normal.&quot; Can you tell me if (a) the data on the lower end of the scale look normal and (b) the data on the upper end of the scale look normal? " CreationDate="2010-09-30T00:23:22.700" />
  <row Id="4166" PostId="3178" Score="3" Text="Thank you for providing this comment!  But unless you can quantify the amount of skewness, that fact by itself is not very useful.  Plenty of distributional families are skewed but have practical normal approximations, such as the Chi-square (Gamma) and Poisson, and plenty more can be strongly skewed but rendered close to (or exactly) Normal through a simple re-expression of the variable, such as the Lognormal.  Could you perhaps amplify your answer to explain how the knowledge of the skewness could be used to estimate p-values from reported ORs?" CreationDate="2010-09-30T03:57:26.437" UserId="919" />
  <row Id="4167" PostId="3179" Text="&quot;clear, polished, and most importantly, succinct manner&quot; Sounds like ggplot2 to me. " CreationDate="2010-09-30T04:49:08.917" UserId="776" />
  <row Id="4168" PostId="3179" Text="I'm looking not just for R libraries, but also any specific types of graphs. My knowledge of graphs is limited to scatter, box, qq, histograms, violins, kernel density estimations, etc. Any slightly more obscure graphs that can reveal more about the data than those would be fantastic." CreationDate="2010-09-30T04:52:27.967" UserId="1118" />
  <row Id="4169" PostId="3163" Text="@chl C'est pas grave mon ami. Quel que fois je ne trouve pas les mots non plus - encore en anglais! :) " CreationDate="2010-09-30T04:57:17.310" UserId="776" />
  <row Id="4170" PostId="3159" Text="@chl:  very handy, I hadn't seen it.  Thanks for pointing it out!" CreationDate="2010-09-30T06:08:01.957" UserId="251" />
  <row Id="4171" PostId="3180" Text="@Jeromy Does this study involve only one disorder (with low prevalence as I understand) or are there multiple diagnoses assessed by multiple indicators?" CreationDate="2010-09-30T06:18:42.450" UserId="930" />
  <row Id="4172" PostId="3182" Score="1" Text="@ars Let's add Hadley's R tools to enhance GGobi experience, e.g. `DescribeDisplay` and `clusterfly`." CreationDate="2010-09-30T06:28:38.703" UserId="930" />
  <row Id="4173" PostId="3180" Text="@chl Just one disorder" CreationDate="2010-09-30T06:43:50.650" UserId="183" />
  <row Id="4174" PostId="3180" Text="@Jeromy Do the scales overlap to some extent (i.e. shared constructs across the questionnaires)?" CreationDate="2010-09-30T06:45:31.350" UserId="930" />
  <row Id="4175" PostId="3180" Text="@chl they correlate but they are conceptually distinct; I've updated the question a little bit to reflect your two queries." CreationDate="2010-09-30T06:49:26.497" UserId="183" />
  <row Id="4176" PostId="3182" Text="Hi ars, as I wrote on my answer - my experience with ggobi is that it doesn't handle large datasets well.  Do you have another experience with that ?" CreationDate="2010-09-30T08:09:00.373" UserId="253" />
  <row Id="4177" PostId="3179" Score="1" Text="parallel coordinates sounds like another one to mention.  Also methods of dimensionality reduction might be helpful." CreationDate="2010-09-30T08:09:48.437" UserId="253" />
  <row Id="4178" PostId="3181" Score="4" Text="I think this might need to be a wiki question :) ..." CreationDate="2010-09-30T08:13:12.163" UserId="253" />
  <row Id="4179" PostId="3170" Text="Nice answer John.  Why do you think this is not a good idea ?" CreationDate="2010-09-30T08:27:20.857" UserId="253" />
  <row Id="4180" PostId="3165" Text="John - you are right. But when I get to these, it is always borderline for me if to put them here or stackoverflow.  Thanks for the answer either way." CreationDate="2010-09-30T08:28:16.347" UserId="253" />
  <row Id="4181" PostId="3182" Text="@Tal The problem comes from not relying on glyph for screen display/rendering, which is common to R base graphics. This was discussed at the latest DSC conference (http://j.mp/bpOhBH). Actually, there is an ongoing project with Qt as a backend, and a new port of GGobi, to enhance interactive display of large data sets." CreationDate="2010-09-30T08:37:56.540" UserId="930" />
  <row Id="4182" PostId="3188" Text="@Tal Thanks for the links about `rflowcyt` and Acinonyx." CreationDate="2010-09-30T08:43:11.107" UserId="930" />
  <row Id="4183" PostId="3188" Text="@Tal BTW, `rflowcyt` has been deprecated with recent releases of Bioconductor, it is now recommended to use `flowViz`. Anyway, both rely on `lattice`." CreationDate="2010-09-30T08:53:19.970" UserId="930" />
  <row Id="4184" PostId="3066" Text="@suncoolsu another formulation: what I have is some panel data with very short time series (9 points for each of them). I want to model them, but not all of them (I will do a Monte Carlo simulation and do not want to simulate 200 time series). I was thinking that by clustering I can reduce the number of time series to simulate to manageable order... " CreationDate="2010-09-30T08:58:11.463" UserId="1443" />
  <row Id="4185" PostId="3169" Text="@chl Thanks for fixing the typo." CreationDate="2010-09-30T10:00:33.213" UserId="1355" />
  <row Id="4186" PostId="3170" Text="the sizes on each side could vary enough that that would be distracting.  I think a good idea could be made of it by simply combining the min() of each separate axis to the min of all of the variables.  But that isn't what you asked for.  It could then be wrapped into a nice function you use frequently called getMaxLabelCEX()" CreationDate="2010-09-30T10:28:55.807" UserId="601" />
  <row Id="4187" PostId="3156" Text="None of this answers why you're comparing these distributions to a mean and why simple analysis of the confidence interval could possibly help.  You don't compare a distribution to a mean just because it has some numeric property.  That's why you can't get an answer for your question." CreationDate="2010-09-30T10:43:28.977" UserId="601" />
  <row Id="4188" PostId="3066" Text="@musa .. is there a specific reason that you don't want to simulate 200 time series? Using hierarchical modeling (using WinBUGS, OpenBUGS or JAGS), you can easily model such data sets. It will be definitely better than clustering and then proceeding to inference (and/or simulation)." CreationDate="2010-09-30T11:17:44.233" UserId="1307" />
  <row Id="4189" PostId="3066" Text="@suncoolsu I have some time constraints (I am working) and want to have a quick solution. At a later stage I am also thinking to do a complete inference/simulation. My goal is to be able to forecast from the model. I have 2 questions:1)is it reasonable to proceed, for now, with clustering+inference or should I concentrate on some other models e.g. random effects model? 2)assuming that I want to model the whole time series what is the best way of doing it, could please give me a reference that I can quickly access?" CreationDate="2010-09-30T12:15:19.520" UserId="1443" />
  <row Id="4190" PostId="3066" Text="I am looking, as a first try, for a solution where the simulation is not needed or it should run very quickly (less than 5 minutes)..." CreationDate="2010-09-30T12:41:46.947" UserId="1443" />
  <row Id="4191" PostId="3188" Text="chl, it is good to know - thank you." CreationDate="2010-09-30T13:50:44.960" UserId="253" />
  <row Id="4192" PostId="3182" Text="chl, it's great to know.  After seeing the latest release was on 2008: http://www.ggobi.org/downloads/ I thought this project was dormant, but looking at their blog I see they are having some activity - http://ggobi.blogspot.com/  I hope they'll grow even more.  Best, Tal" CreationDate="2010-09-30T13:53:58.280" UserId="253" />
  <row Id="4193" PostId="3198" Text="@Thylacoleo: All the answers are similar, but slightly different. I don't think it really matters." CreationDate="2010-09-30T13:58:47.103" UserId="8" />
  <row Id="4194" PostId="3195" Text="@Thylacoleo. Point taken about CW! I made it that way because stats is not my field so I assumed my question would require some correction or extension. Will consider this more carefully next time. Thanks!" CreationDate="2010-09-30T14:08:42.453" UserId="1343" />
  <row Id="4195" PostId="3199" Text="2x2 matrix's upper left corner is only one element... Can you reformulate the question?" CreationDate="2010-09-30T14:11:44.297" UserId="88" />
  <row Id="4196" PostId="3199" Text="@mbq: I tried to reformulate, let me know if it's still unclear (high values and low values refer to block matrices inside the big matrix)" CreationDate="2010-09-30T14:17:40.323" UserId="900" />
  <row Id="4197" PostId="3171" Text="@Srikant, more-or-less correct; individuals that conduct the polling *may* remember who said what, but there is no physical record between response and respondent. Polling is more continuous than punctual: roughly 6 interviews are conducted per week though this can vary wildly as external factors that disrupt access to the group prohibit a more consistent sampling. We have 3 years of data. Since I arrived I've been enforcing putting dates on the results, though many old records are un-dated.&#xA;" CreationDate="2010-09-30T14:22:05.467" UserId="1343" />
  <row Id="4198" PostId="3199" Text="@Xodarap Much better now, thanks. Still, the more details you'll put here (Are those clusters sharp or smooth? Is it all noisy? How large should be those clusters?) the more useful answer you get. " CreationDate="2010-09-30T14:27:10.220" UserId="88" />
  <row Id="4199" PostId="3191" Text="Great job chl!  Would it be o.k. by you if I where to publish this on my blog? (I mean, this text is cc, so I could, but I wanted you permission any way :) )&#xA;Cheers,&#xA;Tal" CreationDate="2010-09-30T14:49:02.897" UserId="253" />
  <row Id="4200" PostId="3191" Text="@Tal No problem. It's far from being an exhaustive list, but maybe you can aggregate other useful links at a later time. Also, feel free to adapt or reorganize in a better way." CreationDate="2010-09-30T15:07:36.027" UserId="930" />
  <row Id="4201" PostId="3183" Text="Thanks for the two novel libraries. My main conflict with these two is that I am submitting my report via paper copy, so interactive graphics might not be fully used. The graphics of Mondrian look pretty complex. I'll give it a look-see." CreationDate="2010-09-30T15:20:13.930" UserId="1118" />
  <row Id="4202" PostId="3183" Text="@Christopher For Mondrian, you have the &quot;equivalent&quot; R version through `iplots` cited by @Tal. About Paraview, you have the option to save a screenshot of your viz. `DescribeDisplay` is the way to go for exporting dynamic visualization from GGobi, http://cran.r-project.org/web/packages/DescribeDisplay/index.html." CreationDate="2010-09-30T15:27:23.300" UserId="930" />
  <row Id="4203" PostId="3191" Text="+1 This is a nice list.  You might consider &quot;accepting this&quot; so that it's always on top; given that it's CW, anyone can keep it updated." CreationDate="2010-09-30T15:34:48.407" UserId="5" />
  <row Id="4204" PostId="3201" Text="It can't be done unless you come to some objective criterion; probably most of possible ratings can be constructed with some combination of your parameters. " CreationDate="2010-09-30T15:39:21.887" UserId="88" />
  <row Id="4205" PostId="3191" Text="@Shane Well, I am indebted to you for providing a first answer with so useful links. Feel free to add/modify the way you want." CreationDate="2010-09-30T15:45:02.120" UserId="930" />
  <row Id="4206" PostId="3205" Text="@whuber (+1) The 1st display really conveys a lot of information, and it will allow to see clusters (wrt. time or individual), if any." CreationDate="2010-09-30T15:51:42.530" UserId="930" />
  <row Id="4207" PostId="3205" Text="@chl Thanks.  But I think it may illustrate some of the problems to which the OP is reacting: it does not clearly show the extent to which units may be active for less than one day.  It could be improved in this regard (e.g., by coloring all streaks less than a day long), but it still does not lend itself well to gauging the extent of the problem.  What it does provide is the ability to distinguish one unit from another.  Displaying the two together on a common time scale suggests that if one single graphic won't do, maybe a collection of related graphics will." CreationDate="2010-09-30T15:57:10.147" UserId="919" />
  <row Id="4208" PostId="3191" Text="I republished it here.  Great list!  http://www.r-statistics.com/2010/09/managing-a-statistical-analysis-project-guidelines-and-best-practices/" CreationDate="2010-09-30T16:03:47.190" UserId="253" />
  <row Id="4209" PostId="3066" Text="@ musa .. in genomics there is a specific modeling strategy for &quot;time course experiments&quot;. You can use it to get to see if the modeling strategy fits your data. Genes correspond to pages and individuals correspond to patients. I also listed 2 references in my previous comment." CreationDate="2010-09-30T16:18:15.263" UserId="1307" />
  <row Id="4210" PostId="3209" Score="1" Text="This looks nice, but won't it just pick highest-variance attributes and biggest clusters of cross-correlated ones?" CreationDate="2010-09-30T16:21:38.610" UserId="88" />
  <row Id="4211" PostId="3196" Score="2" Text="+1 for the reference: it's a lengthy treatise on practical die testing.  Halfway down the author suggests using a KS test and then goes into ways to identify specific forms of deviation from fairness.  He's also well aware that chi-square is an approximation for small numbers of rolls per face (e.g., for 100 rolls of a 20-sided die), that power varies, etc., etc.  In short, anything the OP might like to know is clearly laid out." CreationDate="2010-09-30T16:25:31.363" UserId="919" />
  <row Id="4212" PostId="3188" Text="Very thorough answer, Tal! Plot generation time should not be a huge issue. I've been doing most of my graphs with the base package, and the issue of having the graphs look nicer was for when I decide to use a graph for the paper. I had considered using a scatterplot matrix for the numerical variables, but since many of them are of different units (some are in dollars, others in sqft), the only valuable information I'd get would be general trends, but with ~8 numeric variables, an 8x8 SPM is a bit cluttered. " CreationDate="2010-09-30T16:49:58.013" UserId="1118" />
  <row Id="4213" PostId="3211" Text="@Gaetan Well, for PCA you have to find a suitable numerical coding for variable such as &quot;textual content&quot;..." CreationDate="2010-09-30T17:02:17.817" UserId="930" />
  <row Id="4214" PostId="3148" Text="I want to avoid it because I have already used a chi-squared test for another part of the study. However, I don't know if I'm being stupid by saying that. I guess it wouldn't really matter how many times I used a test within a paper." CreationDate="2010-09-30T17:08:41.457" UserId="1441" />
  <row Id="4215" PostId="3209" Text="@gd047 Alternatively, one can perform *multiple correspondence analysis* or *multiple factor analysis* for mixed data (if numerical recoding happens to be not realistic for some variables), and the rest of your idea (computing factor scores and looking at variable loadings on the 1st dimension) applies as well." CreationDate="2010-09-30T17:16:08.697" UserId="930" />
  <row Id="4216" PostId="3182" Score="1" Text="@Tal: My experience is that it's quite slow when refreshing/repainting the views, e.g. when adding a variable or dragging to rearrange displays in the PCP.  Still, it is usable though not as interactive as advertised with large data.  @chl: That's really good to know, thanks!" CreationDate="2010-09-30T17:16:45.277" UserId="251" />
  <row Id="4217" PostId="3182" Score="1" Text="@ars @Tal Here are the links on Qt interface for R (http://j.mp/d1AJp7) and GGobi (http://j.mp/cUOvfp). See also Hadley's Github repository!" CreationDate="2010-09-30T17:37:19.883" UserId="930" />
  <row Id="4218" PostId="3202" Score="4" Text="I don't mind the downvote, but a comment would be appreciated (so I can learn myself, understand what was wrong, and improve my future responses)." CreationDate="2010-09-30T17:53:17.233" UserId="930" />
  <row Id="4219" PostId="3212" Score="1" Text="str(variable) is your best friend." CreationDate="2010-09-30T18:21:15.193" UserId="776" />
  <row Id="4220" PostId="3202" Score="2" Text="+1, good answer and also seconding chl's comment about leaving an explanation for downvotes to help both the author and readers understand what might be lacking.  Thanks." CreationDate="2010-09-30T18:31:16.577" UserId="251" />
  <row Id="4221" PostId="3211" Text="That's not the issue I am raising.  PCA can handle dummy variables as you suggest.  PCA is incredibly powerful and flexible that way.  But, it is the interpretation of the principal components that gets really challenging.  Let's say the first principal component starts like this: 0.02 years of experience - 0.4 textual content of reviews + 0.01 associations...  Maybe you can explain it.  An expert performance is proportional to years of experience, but inversely proportional to textual content of reviews?  It seems absurd.  But, PCA often does generate counter-intuitive results. " CreationDate="2010-09-30T18:42:19.873" UserId="1329" />
  <row Id="4222" PostId="3214" Text="Can you provide an example of the outcomes you consider in your experiments? What is your sample size (in each group)?" CreationDate="2010-09-30T18:43:03.467" UserId="930" />
  <row Id="4223" PostId="3211" Text="@Gaetan Still, I reiterate my opinion that the problem lies in how you choose to represent your variables (or how you find a useful metric). I agree with you about the difficulty of interpreting a linear combination of variables when dealing with non-continuous measurements or a mix of data types. This is why I suggested in another comment to look for alternative factorial methods. Anyway, developing scoring rules based on user preferences or expert reviewing (as is done in clinical assessment) also calls for some kind of statistical validation (at least to ensure scores reliability)." CreationDate="2010-09-30T19:01:03.413" UserId="930" />
  <row Id="4224" PostId="3202" Text="Thank you for the excellent answer. " CreationDate="2010-09-30T19:03:37.073" UserId="1458" />
  <row Id="4225" PostId="3206" Text="Thanks! If you do not have access to the original data but only a table of regression coefficients, is the Bonferroni adjustment your only choice?&#xA;" CreationDate="2010-09-30T19:08:18.570" UserId="1458" />
  <row Id="4226" PostId="3206" Score="2" Text="Presumably you also have the p-values :-).  But with only those and the coefficients, it's hard to imagine what else you might do besides a Bonferroni adjustment.  (I always make such an adjustment whenever reading any paper with multiple tests: it's a quick way to winnow out the results that are likely to be junk.)  Most people also provide summary statistics for the variables: you can use ranges or sds along with the coefficients to estimate how much effect each explanatory variable might have on the predictand." CreationDate="2010-09-30T19:17:13.900" UserId="919" />
  <row Id="4227" PostId="3206" Text="@whuber Thanks for your explanation, esp. on cross-validation. I appreciate your last argument, i.e. that we also have to look for theoretical relevance (beyond p-values)." CreationDate="2010-09-30T19:20:13.547" UserId="930" />
  <row Id="4228" PostId="3215" Score="1" Text="Could you confirm that (a) d1 and d2 are the side lengths (and not angles); (b) that you are assuming the angle between them is a right angle (for otherwise the atan formula is suspect); and (c) that you are interested in the distribution of one of the other angles of this right triangle?  Also, presumably, the SD of each length distribution is much smaller than its expectation because the triangle shouldn't have any appreciable probability of a negative side length :-)." CreationDate="2010-09-30T19:25:25.873" UserId="919" />
  <row Id="4229" PostId="3215" Text="Exact. I've rephrased the problem to make it a bit clearer. And yes, the SD will be small relative to the dimensions." CreationDate="2010-09-30T19:47:32.383" UserId="77" />
  <row Id="4230" PostId="3215" Text="Using formulas for multiplication and addition, you can try Taylor expansion." CreationDate="2010-09-30T20:04:10.813" UserId="88" />
  <row Id="4231" PostId="3211" Text="@Gaetan, Yes some of your comments make a lot of sense, and you're right in saying that it is not merely a statistical exercise but involves elements that are more subjective. The reason being that the intent from a user/customers standpoint might differ. Assuming he's doing a search for an expert, then i just add filters to allow him to select experts &gt;X number of years of experience and so on But let's say he's narrowed down to 2 experts, and wants an independent comparison. So i'm just looking for a generic method to compare any two experts." CreationDate="2010-09-30T20:19:52.127" UserId="1459" />
  <row Id="4232" PostId="3211" Text="@Gaetan, So yes i'm assuming i can somehow quantify all of the attributes, which again might be subjective. But anyhow i see some interesting points in this discussion and will think some more on that before commenting further." CreationDate="2010-09-30T20:26:10.080" UserId="1459" />
  <row Id="4233" PostId="3211" Text="@chl numerical coding for &quot;textual content&quot;, could be as simple as some sort of sentiment analysis and giving them a binary value of good/bad?" CreationDate="2010-09-30T20:30:33.727" UserId="1459" />
  <row Id="4234" PostId="3211" Text="@Sidmitra, your comments make sense.  Maybe you will derive good use out of PCA after all.  I would suggest you may run PCA several times by deriving the simplest and most explainable models that still explains most of the variance between experts.  By doing so, hopefully you may end up with a PCA model with principal components that are readily explainable to others and to yourself. " CreationDate="2010-09-30T20:43:29.537" UserId="1329" />
  <row Id="4235" PostId="3217" Text="Great idea! I have latitudes and longitudes of all datapoints already, so such a task would be relatively elementary. I was thinking the maps library would be a good way to go, unless there's something better." CreationDate="2010-09-30T20:43:39.287" UserId="1118" />
  <row Id="4236" PostId="3211" Score="1" Text="+1 for pointing out this is not a statistical exercise.  At best, PCA can describe relationships within a particular data set and, conceivably, simplify the data by identifying near-collinearities.  It is not apparent how it can inform us about how to *rank* the experts." CreationDate="2010-09-30T21:03:53.277" UserId="919" />
  <row Id="4237" PostId="3209" Score="1" Text="It seems to me the first component will merely point out a strong direction of commonality among the experts.  How could it possibly tell us who is better and who is worse, though?  That requires additional information concerning the relationships between these variables and the quality of being a &quot;good&quot; or &quot;bad&quot; expert.  If we believe all the variables are monotonically associated with goodness or badness, then perhaps PCA can help us explore the frontier of extreme (or maybe just outlying!) experts.  Watch out though--even the monotonicity assumption is suspect." CreationDate="2010-09-30T21:09:03.570" UserId="919" />
  <row Id="4238" PostId="3217" Score="2" Text="@Christopher You can also do this with `ggplot2` (esp. if you don't need to draw country boundaries), http://had.co.nz/ggplot2/coord_map.html. Otherwise, `maps`, `gmaps` are better. There's also `GeoXp` and an R interface to GRASS. BTW, Mondrian has a plugin for geographical data :)" CreationDate="2010-09-30T21:12:38.490" UserId="930" />
  <row Id="4239" PostId="3209" Text="@whuber I see the point, thanks. Maybe you could add this in your own response (which is very welcomed)?" CreationDate="2010-09-30T21:41:02.990" UserId="930" />
  <row Id="4240" PostId="3216" Text="There was never any chance of it being normally distributed.  It is an angle!  It only takes values on $[-\pi, \pi)$." CreationDate="2010-10-01T00:06:22.883" UserId="352" />
  <row Id="4241" PostId="3198" Score="1" Text="@csgillespie: I was just amazed how quickly people jump in and answer!" CreationDate="2010-10-01T00:24:23.820" UserId="521" />
  <row Id="4242" PostId="3224" Text="Oh neat, I didn't know plot could do that.  " CreationDate="2010-10-01T00:31:50.840" UserId="251" />
  <row Id="4243" PostId="3224" Text="Thanks! In examples like plot(sin, -pi, 2*pi), is there a default spacing for sampling the range in domain?" CreationDate="2010-10-01T01:16:11.087" UserId="1005" />
  <row Id="4244" PostId="3224" Text="The default is 101 points. See help(plot.function)." CreationDate="2010-10-01T01:21:02.630" UserId="159" />
  <row Id="4245" PostId="3216" Score="1" Text="P(Y/X $\le$ q) = P(Y $\le$ qX) is not correct if X is a normal r.v. - X *can* be negative too." CreationDate="2010-10-01T02:38:42.110" UserId="1112" />
  <row Id="4246" PostId="3216" Text="@ronaf: actually, since $X$ and $Y$ are the side lengths of a physical triangle, we should *not* have negative $X$!" CreationDate="2010-10-01T04:26:34.060" UserId="795" />
  <row Id="4247" PostId="3216" Text="d'accordo shabbychef! - but you can't eat your cake and have it. if X and Y are normal r.v.s. they will be negative occasionally. perhaps one solution is to think of the triangle as pointing down into the third quadrant if X &gt; 0 but Y &lt; 0 [etc.]  in any case, one can still consider the behavior of arctan(X/Y) - which will then range in ($-\frac{\pi}{2}, \frac{\pi}{2}$)." CreationDate="2010-10-01T04:49:12.223" UserId="1112" />
  <row Id="4248" PostId="3218" Text="@whuber (+1) and thanks for the references. " CreationDate="2010-10-01T06:50:47.000" UserId="930" />
  <row Id="4249" PostId="3220" Text="Very helpful, but John Wiley &amp; Sons may come after you for copyright violation..." CreationDate="2010-10-01T07:18:12.547" UserId="449" />
  <row Id="4250" PostId="3170" Text="Thanks John, that makes sense - I'll wrap that later and possibly republish it (with credit).  Thanks again :)    Tal" CreationDate="2010-10-01T08:27:29.527" UserId="253" />
  <row Id="4251" PostId="3196" Text="Thanks whuber - the power of google searching  :)" CreationDate="2010-10-01T08:28:40.490" UserId="253" />
  <row Id="4252" PostId="3202" Text="I second chl's recommendation to use some kind of penalized regression (e.g., the Lasso)." CreationDate="2010-10-01T09:28:34.730" UserId="1352" />
  <row Id="4253" PostId="3202" Score="4" Text="@chl: I'm unhappy with recommending stepwise predictor selection. Usually, this is based on p-values (&quot;exclude a predictor with p&gt;.15, include it if p&lt;.05&quot;) and leads to biased estimates and bad predictive performance (Whittingham et al., 2006, Why do we still use stepwise modelling in ecology and behaviour? J Anim Ecol, 75, 1182-1189). However, AIC-based stepwise approaches have the same weakness - Frank Harrell discussed this in a post to R-help on Mon, 09 Aug 2010 16:34:19 -0500 (CDT) in the thread &quot;Logistic Regression in R (SAS -like output)&quot;." CreationDate="2010-10-01T09:40:13.540" UserId="1352" />
  <row Id="4254" PostId="3202" Text="Is there any way to include links in comments?" CreationDate="2010-10-01T09:40:33.337" UserId="1352" />
  <row Id="4255" PostId="3202" Text="@Stephan +1 Thanks for this. I know Frank Harrell's point of view. His book is a &quot;salvation&quot; for biostatistics, as well as the more recent one by EW Steyerberg. This is why I referred to his work at the end of my response, assuming the interested reader would look for a more thorough explanation. I also wrote a quick &amp; dirty memo on that (http://j.mp/cViAUX). My initial thought was just to point to different ways of approaching the problem. As @whuber nicely pointed it out, there's room for improvement in my response, esp. with cross-validation and conceptual issues around predictive modeling." CreationDate="2010-10-01T09:49:30.420" UserId="930" />
  <row Id="4256" PostId="3202" Text="@Stephan Now the problem is that often we want to keep some variables in our model (whatever their p-values), as @whuber said; this is why I suggested to look at the `penalized` package from J Goeman, because it allows to penalize only a subset of the covariates. From my experience, I found results from Lasso/Ridge regression a little bit difficult to explain to an external audience, esp. when data don't really called for shrinkage/regularization (e.g. &quot;ideal&quot; case with 500 subjects and 10 variables, no collinearity or mediation issues). So when we can keep things simple without overfitting..." CreationDate="2010-10-01T10:07:33.010" UserId="930" />
  <row Id="4257" PostId="3230" Text="@Thylacoleo I was thinking of confidence intervals, but the question is not clear to me because there seems to be different outcomes each time (but may I don't understand the question) which would prevent from pooling anything at all." CreationDate="2010-10-01T10:56:55.520" UserId="930" />
  <row Id="4258" PostId="2818" Text="@Thylacoleo Thanks for the links (esp. the genetic related one). So, what Stata command would you recommend?" CreationDate="2010-10-01T11:07:58.510" UserId="930" />
  <row Id="4259" PostId="2818" Text="The question concerns Cox regression. I'll post an expmple (above) provided with Stata 11. " CreationDate="2010-10-01T12:37:53.337" UserId="521" />
  <row Id="4260" PostId="2818" Text="@Thylacoleo Ok, this was just to be sure that the `stcox` command was the correct approach." CreationDate="2010-10-01T12:39:56.313" UserId="930" />
  <row Id="4261" PostId="2749" Score="1" Text="Also as a note to the poster you can not estimate the shared frailty in SPSS as in Thylacoleo's example. You can only include covariates in the Cox regression (either time varying or static)." CreationDate="2010-10-01T14:54:45.633" UserId="1036" />
  <row Id="4262" PostId="2818" Text="@Thylacoleo, do you know if you included fixed effects for all the matched groups (ie dummy variables) would you get the same covariate coefficients when you define shared frailty?" CreationDate="2010-10-01T14:57:50.777" UserId="1036" />
  <row Id="4263" PostId="3239" Text="I had come accross cointegration a few years back - but it did seem terribly complicated to me (I didn't understand it!). I was hoping there would be a less theoretical (i.e. more practical) solution ..." CreationDate="2010-10-01T15:08:16.437" UserId="1216" />
  <row Id="4264" PostId="3239" Score="2" Text="The Engle-Granger method is not especially complicated: you just take the residuals of a regression between the two series and determine if it has a unit root.  This is certainly practical: it's used regularly for a broad spectrum of problems.  That said, I imagine that any answer to your question will require some statistical knowledge (for instance, you should understand things like stationarity, independence, etc.)..." CreationDate="2010-10-01T15:19:35.990" UserId="5" />
  <row Id="4265" PostId="3214" Text="I'm a bit confused what the goal is as well. Is there a reason examining a plot of each of the 5 hazard functions (and/or their confidence intervals) is insufficient? Do you need a test statistic to state where the curves intersect? As chl suggested in a comment to Thylacoleo's answer pooling seems inappropriate with different outcomes." CreationDate="2010-10-01T15:29:21.947" UserId="1036" />
  <row Id="4266" PostId="3216" Score="1" Text="@ronaf: That's the right idea.  If one uses signed side lengths and also considers the angle as a real value (rather than its value modulo $2\pi$), there is no inconsistency with normality in either case.  Your point about the inequality possibly being wrong is excellent.  All I can do in response is to claim that the equation is an excellent approximation under the assumptions made because the chance of X or Y being negative is negligible." CreationDate="2010-10-01T16:01:38.573" UserId="919" />
  <row Id="4267" PostId="3239" Text="@Shane, is there a better way to do this than to test all pair-wise series for co-integration (with the same ideal in mind to cluster series together?) Also wouldn't this suggestion be dependent on the fact that the series themselves are integrated at the onset?" CreationDate="2010-10-01T16:34:55.210" UserId="1036" />
  <row Id="4268" PostId="3239" Text="@Andy: I'm sure that there is a better way, and I look forward to hearing about it.  This is a pretty basic approach." CreationDate="2010-10-01T16:38:29.650" UserId="5" />
  <row Id="4269" PostId="3238" Score="1" Text="You may also be interested in the responses to this question, http://stats.stackexchange.com/q/2777/1036" CreationDate="2010-10-01T16:39:38.900" UserId="1036" />
  <row Id="4270" PostId="3217" Text="Assigning a best answer can be difficult when there's several great suggestions, but I feel this is the right direction, keeping &quot;succinct&quot; in mind. I will give ggplot2 a try, and take a look at maps, GeoXp, and Mondrian. Thanks for the idea of graphing spatially!" CreationDate="2010-10-01T17:34:52.817" UserId="1118" />
  <row Id="4271" PostId="3244" Text="Could you link to the spreadsheet Mike Lawrence provided?" CreationDate="2010-10-01T17:43:53.083" UserId="1036" />
  <row Id="4272" PostId="3215" Text="Thanks for both your excellent answers, which (as far as I can tell with my limited stats expertise) are both intuitive and sound." CreationDate="2010-10-01T18:03:21.920" UserId="77" />
  <row Id="4274" PostId="3246" Text="I second this. There are many examples of something that looked like a power law, but when examined a little more rigorously turned out not to be....and no, the high R^2 on the chart is not enough." CreationDate="2010-10-01T18:32:38.170" UserId="247" />
  <row Id="4275" PostId="2888" Text="@Yaroslav Thanks for the additional link. Still for me the problem is with the small $n$ and the heterogenous predictors. It seems to me that the $n\ll p$ case is now increasingly well-studied in genetics, neuroimaging studies, or when we can assume an exponential relationship between $n$ and $p$, but at the moment I never found any evidence of the relevance or predictive power of boosting in the particular study I presented. I am currently running MC simulations to see how RFs and sparse regression perform in this case. I'll let you know all of any progress in this direction." CreationDate="2010-10-01T19:08:27.593" UserId="930" />
  <row Id="4276" PostId="3244" Text="Here is the URL https://spreadsheets.google.com/ccc?key=0Ap2N_aeyRMGHdHJxUnVNeEl5VGtvY1RVLVc5UjU4Vmc&amp;hl=en#gid=0 It was related to his question http://stats.stackexchange.com/questions/2956/have-i-computed-these-likelihood-ratios-correctly" CreationDate="2010-10-01T19:13:11.320" UserId="1329" />
  <row Id="4277" PostId="2525" Text="Great recommendation, thank you -- I got a copy recently based on this and it really is quite good." CreationDate="2010-10-01T19:14:28.050" UserId="251" />
  <row Id="4278" PostId="2525" Text="I'm glad to hear someone else appreciates this book!" CreationDate="2010-10-01T20:14:08.543" UserId="919" />
  <row Id="4279" PostId="3198" Text="Thanks for the answer.  I accepted this because it included all the newbie stuff about p values and rejecting." CreationDate="2010-10-01T20:22:05.867" UserId="1456" />
  <row Id="4280" PostId="3236" Text="Don't you mean `mean(replicate(N, system.time(f(...))[3]), trim = 0.05)` ?" CreationDate="2010-10-01T20:29:55.107" UserId="46" />
  <row Id="4281" PostId="3236" Text="Yup. Thanks for catching that." CreationDate="2010-10-01T20:31:00.143" UserId="334" />
  <row Id="4282" PostId="3248" Text="@Tal, thanks for your feedback.  This is the exact type of comments I was interested in.  You have really leveraged the sharing and importing component of Google docs.  Good for you.  I'll read your material to learn more about it. " CreationDate="2010-10-01T20:43:16.037" UserId="1329" />
  <row Id="4283" PostId="3246" Text="&quot;So you think...&quot; is an excellent reference.  Points 1-6 (out of 7) directly address the question posed here." CreationDate="2010-10-01T20:45:26.157" UserId="919" />
  <row Id="4284" PostId="3236" Score="1" Text="Thinko.  I like it." CreationDate="2010-10-01T20:56:48.717" UserId="5" />
  <row Id="4285" PostId="3241" Score="1" Text="Chris, regular clustering won't cut it. You either have to acknowledge that a series is highly correlated with it's own past by putting each $y_{1,t}$ as a dimension of it's own (i.e. resulting in N*T dimensions) or you trow all the dimensions together, but then (given the high correlation inside a series) you will always end up with a single cluster. Also most clustering methods are ill-suited/advised for highly correlated variables (there is a more or less bending assumption of spherical clusters)." CreationDate="2010-10-01T21:01:15.290" UserId="603" />
  <row Id="4286" PostId="3239" Text="@Shane:&gt; i can't suggest anything else, but cointegration is both very fragile ('parametric assumptions' gone wild series) in practice and ill suited for the task at hands: at each step, it amounts to doing hierarchical clustering, at most merging two series unto one (the co-integrated mean)." CreationDate="2010-10-01T21:09:58.127" UserId="603" />
  <row Id="4287" PostId="3246" Text="But a power-law *distribution* isn't the same thing as fitting a power law relationship between two separate variables. I'd assumed the question was about the latter, though i'm not certain." CreationDate="2010-10-01T21:39:59.267" UserId="449" />
  <row Id="4288" PostId="3251" Score="2" Text="Actually, the arima function in R will not fit your model (1). arima() does regression with ARIMA errors and your equation (1) is an ARMAX model." CreationDate="2010-10-01T23:56:20.467" UserId="159" />
  <row Id="4289" PostId="3255" Text="Hi Kwak and Rob.  Thanks for looking at this.  I wanted to use exponential smoothing because this is what I'm more familiar with.&#xA;I'm thinking that I need to learn on how to use the ARIMA framework.  Could you recommend a good book that would help me to learn enough about the ARIMA framework to apply such a dummy variable approach?&#xA;&#xA;I have Bowerman's &quot;Forecasting, Time Series, and Regression&quot; and Levenbach &quot;Forecasting: Practice and Process for Demand Management&quot;, which I used to learn about exponential smoothing.   I don't know if these go in sufficient detail for what I'd need.&#xA;Thanks!&#xA;" CreationDate="2010-10-02T00:25:27.677" UserId="1479" />
  <row Id="4290" PostId="1011" Text="excellent, +1 from me." CreationDate="2010-10-02T00:58:03.273" UserId="438" />
  <row Id="4291" PostId="3246" Text="Non-expert's question: apart from &quot;robustness&quot;, are there other reasons why one should check goodness-of-fit with Kolmogorov-Smirnov instead of $\chi^2$ in this case?" CreationDate="2010-10-02T02:02:56.740" UserId="830" />
  <row Id="4292" PostId="3220" Text="I think I could convince them that this is good advertising for the book :)" CreationDate="2010-10-02T02:51:41.507" UserId="352" />
  <row Id="4293" PostId="3246" Score="2" Text="@J.M.: not really, chi-square is sensitive to binning and tail fluctuations complicate that.  I think even with the KS, they reweigh the statistic for extremal points, and there's some discussion of other tests.  @onestop: I assumed the other way, and on re-reading, you could be right.  I'm not really sure .." CreationDate="2010-10-02T05:51:33.583" UserId="251" />
  <row Id="4294" PostId="3251" Text="Rob:&gt; i've edited equation one. Can you point to a source where the difference(s) between armax and regression with arima errors are explain (or alternatively provide an intuitive explanation). Also, would you know of a R package that implements ARMAX models ? Thanks in advance." CreationDate="2010-10-02T07:46:42.523" UserId="603" />
  <row Id="4295" PostId="3254" Text="@onestop +1 Thanks for the link (and the retag). I'll look into this direction." CreationDate="2010-10-02T08:05:26.010" UserId="930" />
  <row Id="4296" PostId="2818" Text="@Andy W. No as frailty is a the equivalent of a random effect in survival analysis. So the cathetar model is equivalent to a mixed model with both fixed and random coefficients. To show this I'll add the dummy variable example above. Perhaps it could be interpreted as an &quot;order effect&quot; - second infection versus first." CreationDate="2010-10-02T08:11:37.257" UserId="521" />
  <row Id="4297" PostId="3248" Text="Dear Gaetan, I am delighted by your response - thank you for the kind words.  Best,  Tal." CreationDate="2010-10-02T08:17:52.557" UserId="253" />
  <row Id="4298" PostId="3151" Text="Jeromy, see the answer I gave- most interesting!" CreationDate="2010-10-02T08:22:40.667" UserId="253" />
  <row Id="4299" PostId="3152" Text="Hi chl,  I have seem to have found a solution doing what you are describing!  Please have a look at the answer I just posted.  Best, Tal" CreationDate="2010-10-02T08:23:43.367" UserId="253" />
  <row Id="4300" PostId="3228" Text="You mean, notepad++ with the use of npptor :)" CreationDate="2010-10-02T08:33:16.710" UserId="253" />
  <row Id="4301" PostId="3258" Text="@Tal Oh, I seem to remember browsing the webiste some time ago, but definitively forgot about it. It remind me of the technique used on http://bm2.genes.nig.ac.jp to display complete R output. Let us know if you can manage to only display the output (text+img) and not the R command... " CreationDate="2010-10-02T08:34:27.897" UserId="930" />
  <row Id="4302" PostId="2818" Text="@Thylacoleo (+1) Thanks for all of this. It looks definitively better with your annotated Stata example." CreationDate="2010-10-02T08:37:50.653" UserId="930" />
  <row Id="4303" PostId="3240" Text="mod-tip: post the second part as a comment to the Dirk's answer." CreationDate="2010-10-02T09:28:27.997" UserId="88" />
  <row Id="4304" PostId="3247" Text="@Gaetan Aside from my response, I gave my +1 to the question because I think it is very relevant for debating about statistical practice and project management." CreationDate="2010-10-02T09:35:43.670" UserId="930" />
  <row Id="4305" PostId="3258" Text="Hi chl, I already contacted the author of that plugin.  I hope that this could be accomplished (and actually much more).  More (I hope) to come..." CreationDate="2010-10-02T10:08:20.590" UserId="253" />
  <row Id="4306" PostId="3258" Text="Hi chl, regarding the link you gave, I forget about it.  Interestingly enough, it doesn't solve the issue, since he has all the ?sink output and then all the ?png output (I am not sure if that is how he implements it, but that's the result).  This indeed could be a good idea for upgrading such a website." CreationDate="2010-10-02T10:15:28.440" UserId="253" />
  <row Id="4307" PostId="3260" Text="zenna:&gt; do you have a set of pre-identified break points ? is the dates of points were 'interesting changes across different datasets' did occur ?" CreationDate="2010-10-02T13:06:49.490" UserId="603" />
  <row Id="4308" PostId="2818" Text="Yes Thank you very much Thylacoleo" CreationDate="2010-10-02T13:14:49.963" UserId="1036" />
  <row Id="4309" PostId="3260" Text="no, I have to find them also.  They are (normally) local minima within each dataset, the problem is there are lots of local minima which are not 'interesting points' which is why I am trying to combine multiple datasets to find the meaningful local minima." CreationDate="2010-10-02T13:23:29.660" UserId="809" />
  <row Id="4310" PostId="3236" Score="2" Text="If the f() call is long then it's fine.  However, if the f() call is short then any timing call overhead will likely be increasing error measurement.  With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster)." CreationDate="2010-10-02T16:07:29.607" UserId="601" />
  <row Id="4311" PostId="2400" Text="Here's something interesting I found about divergences recently -- each step of Belief Propagation can be viewed as a Bregman Projection http://www.ece.drexel.edu/walsh/Walsh_TIT_10.pdf" CreationDate="2010-10-02T17:05:35.383" UserId="511" />
  <row Id="4312" PostId="3257" Text="This looks like a good, tractable starting point. thanks for the links." CreationDate="2010-10-02T17:17:38.017" UserId="1216" />
  <row Id="4313" PostId="3235" Text="Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow." CreationDate="2010-10-02T17:22:02.910" UserId="71" />
  <row Id="4314" PostId="3264" Text="Good leads, thanks chl :)" CreationDate="2010-10-02T17:48:35.787" UserId="253" />
  <row Id="4315" PostId="3264" Text="@chl: I agree that permutation tests and other explicit manifestations of randomness can be quite educational.  This suggests showing dynamic simulations to the class, so they can watch the permutations being done and see the effects on the statistics.  Just to tweak you a little bit (apropos a different thread): one of the best tools available for that is...Excel!  (It helps that the students will have access to this and be familiar with it, unlike a better platform like Mathematica.)" CreationDate="2010-10-02T18:13:52.550" UserId="919" />
  <row Id="4316" PostId="3264" Score="1" Text="@whuber Thanks. Even before using any software, I like discussing Phillip Goud example (updated in my answer) and let them do the calculation by hand. Then, I think any software will do the job, provided they feel involved and do it themselves." CreationDate="2010-10-02T18:27:20.373" UserId="930" />
  <row Id="4317" PostId="3260" Text="Statistics works by training. The idea is that you need to first identify (by yourself) a set of points satisfying what you are looking for, then try to train a statistical procedure to do this job for you." CreationDate="2010-10-02T18:51:32.927" UserId="603" />
  <row Id="4318" PostId="3266" Text="@paul Thanks for the link. What about imputation when data are not missing at random? It seems to me the problem lies in the fact that there is a complete block of measurements that is missing, and we cannot assume MAR or MCAR, nor are these data missing by design (in which case ML estimation yields unbiased parameters estimates)." CreationDate="2010-10-02T19:10:27.867" UserId="930" />
  <row Id="4319" PostId="3008" Score="1" Text="@Farrel - here's a very short script, which assumes [0,1]-uniformly distributed covariates, an OR of 2 between the first and third quartile of the covariate and standard normal noise, leading to power .34 for n=100. I'd play around with this to see how sensitive everything is to my assumptions: runs &lt;- 1000; nn &lt;- 100; set.seed(2010);&#xA;detections &lt;- replicate(n=runs,expr={covariate &lt;- runif(nn);&#xA;outcome &lt;- runif(nn)&lt;1/(1+exp(-2*log(2)*covariate+rnorm(nn)));&#xA;summary(glm(outcome~covariate,family=&quot;binomial&quot;))$coefficients[&quot;covariate&quot;,&quot;Pr(&gt;|z|)&quot;] &lt; .05})&#xA;cat(&quot;Power:&quot;,sum(detections)/runs,&quot;\n&quot;)&#xA;" CreationDate="2010-10-02T20:04:27.407" UserId="1352" />
  <row Id="4320" PostId="3008" Score="1" Text="@Stephan You can attach your code as a pastie (http://pastebin.com/) or a Gist (http://gist.github.com/) if you feel it's more convenient, and link back to it in your comment." CreationDate="2010-10-02T20:20:43.807" UserId="930" />
  <row Id="4321" PostId="3266" Score="1" Text="@chi MAR is exactly the sort of assumption that Manski's approach is meant to avoid. Perhaps a simple example will make it clearer. This is straight out of the paper I linked above. Suppose you want to estimate the average of some function of a variable y, E[g(y)]. Let z=1 for people with y observed, and z=0 otherwise. Also, suppose g is bounded between g0 and g1. Then you know that E[g(y)|z=1]P(z=1) + g0 P(z=0) &lt; E[g(y)] &lt; E[g(y)|z=1]P(z=1) + g1 P(z=0). E[g(y)|z=1] and P(z) can be estimated as the sample mean of the observed y, and the portion of the sample with y observed." CreationDate="2010-10-02T20:23:52.823" UserId="1229" />
  <row Id="4322" PostId="3266" Text="@paul Thanks for the explanation. I have no access to Manski's paper, but I will try to get it at work. (I have not vote left, so I'll +1 your response ASAP)" CreationDate="2010-10-02T20:31:52.957" UserId="930" />
  <row Id="4323" PostId="3008" Text="@chl: +1, thanks a lot! Here's the gist: http://gist.github.com/607968" CreationDate="2010-10-02T20:41:40.577" UserId="1352" />
  <row Id="4324" PostId="3255" Text="Bowerman O'Connell and Koehler is quite good for introducing ARIMA models but I don't think it includes ARIMA with covariates. You could try my 1998 textbook which covers ARIMA modelling and regression with ARIMA errors at an introductory level. See http://robjhyndman.com/forecasting/ for details." CreationDate="2010-10-02T21:41:39.857" UserId="159" />
  <row Id="4325" PostId="3251" Text="A first order ARMAX model with one covariate is&#xA;y_t = a + bx_t + cy_{t-1} + e_t&#xA;where e_t is iid zero mean.&#xA;The corresponding regression with ARIMA error is&#xA;y_t = a + bx_t + n_t where n_t = phi*n_{t-1}+z_t&#xA;and z_t is iid zero mean." CreationDate="2010-10-02T21:44:55.933" UserId="159" />
  <row Id="4326" PostId="3266" Text="@chi This ungated paper by Kline and Santos might also be useful. It focuses on quantile regression, but has references to similar papers about other models. http://www.econ.berkeley.edu/~pkline/papers/missing4.pdf" CreationDate="2010-10-02T22:25:22.967" UserId="1229" />
  <row Id="4328" PostId="3251" Text="Thanks Rob, but i'm still a bit confused with the n_t in 'y_t = a + bx_t + n_t where n_t = phi*n_{t-1}+z_t and z_t is iid zero mean'. It seems that 'n_t' follows a moving average process (i.e. not autoregressive). Can you confirm that a regression with ARMA errors is actually a 'MAX' model (moving average + eXogeneous) ? Also, does it mean that regressions with ARIMA errors cannot be fitted by OLS ?" CreationDate="2010-10-02T23:02:18.317" UserId="603" />
  <row Id="4330" PostId="3251" Text="Also, I get exactly the same estimates when fitting 'arima(y,order= c(1, 0, 0),xreg=x)' from package stats and 'arimax(y,order= c(1, 0, 0),xreg=x)' (using the arimax function in package TSA). Does that mean that arimax() does not fit ARIMAX models ?" CreationDate="2010-10-02T23:26:31.530" UserId="603" />
  <row Id="4332" PostId="3251" Score="1" Text="@kwak. First, n_t = phi*n_{t-1} + z_t is AR(1). A moving average process of order 1 would be n_t = theta*z_{t-1} + z_t. Second, an regression with MA errors is equivalent to a MAX model. But once you add AR terms in the error process, there is no equivalence between the two classes. Third, the arimax() function in TSA fits transfer function models, a special case of which is a regression with ARIMA errors. It does not fit ARIMAX models. I might write a blog post about this as it is hard to find the various model classes compared and discussed anywhere." CreationDate="2010-10-03T03:38:12.633" UserId="159" />
  <row Id="4333" PostId="3274" Text="one such $f$ is based on EM, as described in Little &amp; Rubin (I was somewhat put off by their treatment of regression, but perhaps I should revisit)." CreationDate="2010-10-03T04:19:38.183" UserId="795" />
  <row Id="4334" PostId="3276" Text="Similar question at http://stats.stackexchange.com/questions/2957/ols-is-blue-but-what-if-i-dont-care-about-unbiasedness-and-linearity/2961#2961. A nice review paper on the topic is http://webee.technion.ac.il/Sites/People/YoninaEldar/Download/67j-04490210.pdf" CreationDate="2010-10-03T07:30:40.507" UserId="352" />
  <row Id="4335" PostId="3261" Text="What is $\epsilon$?" CreationDate="2010-10-03T07:49:16.740" UserId="352" />
  <row Id="4337" PostId="2270" Text="@Freya (+1) Nice review, thanks for sharing." CreationDate="2010-10-03T10:20:19.910" UserId="930" />
  <row Id="4338" PostId="3143" Text="Certainly, Sir :)" CreationDate="2010-10-03T11:04:56.197" UserId="1439" />
  <row Id="4339" PostId="3251" Text="@ Rob:&gt; Ok, thanks for all the clarifications." CreationDate="2010-10-03T11:05:01.240" UserId="603" />
  <row Id="4340" PostId="3247" Text="A comment for the downvote would be greatly appreciated." CreationDate="2010-10-03T12:25:48.637" UserId="930" />
  <row Id="4341" PostId="3278" Text="What a wonderful synopsis!" CreationDate="2010-10-03T12:57:04.367" UserId="919" />
  <row Id="4342" PostId="3282" Text="@Jeromy Nice book!" CreationDate="2010-10-03T13:20:11.830" UserId="930" />
  <row Id="4343" PostId="2749" Text="@Andy Thanks for the tips. In fact, I don't really use SPSS (only from time to time for Factor Analysis, and to exchange with colleagues), but it might be of interest for SPSS users. (my +1)" CreationDate="2010-10-03T13:23:25.520" UserId="930" />
  <row Id="4344" PostId="3282" Text="Thank you! Any other materials are welcome." CreationDate="2010-10-03T15:08:02.297" UserId="1250" />
  <row Id="4345" PostId="3278" Text="Thanks @chl this is a great description (merci bien pour le link egalement. Le premier link as une bonne explication des lignes et de collones). I have one more question. What would you say about the relationship between people with red hair and people with green eyes?" CreationDate="2010-10-03T18:13:26.300" UserId="776" />
  <row Id="4346" PostId="3278" Score="1" Text="@Brandon The 1st axis is an axis of &quot;dominance&quot; (light -&gt; dark) for both modalities, but we can also see that the 1st axis opposes blue and green eyes to brown and hazel eyes (their coordinates are of opposite signs), and red hair/green eye combination--which is quite uncommon--contribute mostly to the 2nd factor axis. As this axis only explains 9.5% of the total inertia, it is rather difficult to draw firm conclusions (esp. wrt. genetic hypotheses)." CreationDate="2010-10-03T18:35:01.597" UserId="930" />
  <row Id="4347" PostId="3278" Score="1" Text="@Brandon Two further references (in english this time): the PBIL course (http://j.mp/cHZT7X) and Michael Friendly's resources (http://j.mp/cYHyVn + `vcd` and `vcdExtra` R packages, the latter including a nice vignette)." CreationDate="2010-10-03T18:37:19.003" UserId="930" />
  <row Id="4348" PostId="1470" Text="@propofol (+1) Good references! Thanks." CreationDate="2010-10-03T19:26:09.587" UserId="930" />
  <row Id="4349" PostId="3278" Text="When you refer to modalities, are you referring to &quot;hair colour&quot; and &quot;eye colour&quot;? Also, using the &quot;ca&quot; package, I see clearly how inertia for the dimensions is stated but I'm not clear on which numbers you used to identify that red hair/green eyes contributed to the 2nd factor. Are you looking at the &quot;cor&quot; column (high/low) where k=2? " CreationDate="2010-10-03T19:29:22.967" UserId="776" />
  <row Id="4350" PostId="3278" Score="1" Text="@Brandon Yes, one modality = one category for your variable. For your 2nd question, `cor` is the squared correlation with the axis, and `ctr` is the contribution (it has to be divided by 10 to be read as a %). So &quot;red hair&quot; contributes 55.1% of the inertia of the 2nd axis. In a certain sense I found the FactoMineR output more &quot;intuitive&quot; (`CA(tab, graph=FALSE)$row$contrib` gives you directly the %)." CreationDate="2010-10-03T19:52:17.793" UserId="930" />
  <row Id="4351" PostId="3283" Text="Thanks for the help!  I do have different probabilities for each game, which makes this method inefficient for calculating outcome probabilities.  However, I have come up with a way to enumerate all outcomes without having to check all 3^10 possibilities by using dynamic programming!" CreationDate="2010-10-03T19:57:04.227" />
  <row Id="4352" PostId="1470" Text="@propofol This post has also useful references related to RCTs: http://j.mp/bAgr1B." CreationDate="2010-10-03T20:06:52.653" UserId="930" />
  <row Id="4353" PostId="3278" Text="Here's another example for follow-up. http://stats.stackexchange.com/questions/3287/interpreting-2d-correspondence-analysis-plots-part-ii" CreationDate="2010-10-03T21:24:49.827" UserId="776" />
  <row Id="4354" PostId="3286" Score="2" Text="please, read the highest graded answer (i.e. mbq's) here: http://stats.stackexchange.com/questions/1001/spearmans-correlation-coefficient-to-compare-distributions" CreationDate="2010-10-03T21:33:34.677" UserId="603" />
  <row Id="4355" PostId="2247" Text="@Shane, could you provide a link to what you feel is the best tutorial that you've seen using R and survival?" CreationDate="2010-10-03T21:45:56.817" UserId="776" />
  <row Id="4356" PostId="3261" Text="$\epsilon$ is a zero-mean error term. " CreationDate="2010-10-04T01:49:59.700" UserId="795" />
  <row Id="4357" PostId="3274" Text="It seems like imputation would not work for covariance estimation." CreationDate="2010-10-04T01:51:07.453" UserId="795" />
  <row Id="4358" PostId="3283" Text="If you're enumerating all outcomes you're looking at all 3^10 possibilities.  There's no free lunch here ;-)." CreationDate="2010-10-04T02:05:43.500" UserId="919" />
  <row Id="4359" PostId="618" Score="1" Text="A colleague of mine looked at the data for this in the post-2000 period, and found that the relationship held fairly well 'out-of-sample', which is even more disturbing..." CreationDate="2010-10-04T04:12:47.273" UserId="795" />
  <row Id="4360" PostId="3283" Text="You're right, I'm not actually enumerating all individual game outcomes, just overall outcomes AFTER 10 games.  I'm looking at the possible outcomes for only one category, and then adding the new possible outcomes with each new additional category (one at a time).  With this method, you're able to find the probability of all (W,L,T) outcomes after X games in polynomial time: O(n^3) :-)" CreationDate="2010-10-04T05:09:30.730" />
  <row Id="4361" PostId="3236" Text="@John: Thanks but I don't quite get what you said. I am still wondering which is better, repeating f() inside or outside system.time()?" CreationDate="2010-10-04T05:59:47.657" UserId="1005" />
  <row Id="4362" PostId="3284" Score="1" Text="There's also an example of using Mfuzz in my [tutorial paper](http://www.mas.ncl.ac.uk/~ncsg3/microarray/)" CreationDate="2010-10-04T08:51:19.263" UserId="8" />
  <row Id="4363" PostId="3294" Score="2" Text="You should probably make your post a community wiki since there isn't a correct answer." CreationDate="2010-10-04T09:09:16.930" UserId="8" />
  <row Id="4364" PostId="3294" Score="1" Text="I've just converted it." CreationDate="2010-10-04T09:14:39.793" UserId="88" />
  <row Id="4365" PostId="3236" Text="Every call to the system.time() command has some variable time it takes to call that causes some amount of measurement error. This is a small amount.  But what if f() is a very brief call?  Then this error can be conflated with the time taken to call f().  So, when you call f() 1e5 times inside a single system.time() call the error gets divided down into 1e5 chunks.  When you call system.time() for every f() it's impact could be meaningful if time for f() is small.  Of course, if all you need is relative timing it doesn't much matter." CreationDate="2010-10-04T10:13:52.917" UserId="601" />
  <row Id="4366" PostId="3236" Text="Oh, and the second part is that it would be faster to just call system.call() once." CreationDate="2010-10-04T10:14:36.733" UserId="601" />
  <row Id="4367" PostId="3288" Score="2" Text="To make it clearer, here you will rather need Kolomogorov-Smirnov distance than KS-test. " CreationDate="2010-10-04T10:22:05.020" UserId="88" />
  <row Id="4368" PostId="3286" Text="Wikipedia has some sort of overview of distribution distances here: http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_distance#Other_probability-distance_measures" CreationDate="2010-10-04T10:22:48.173" UserId="88" />
  <row Id="4370" PostId="3251" Score="2" Text="I've tried to summarise the various models at http://robjhyndman.com/researchtips/arimax/" CreationDate="2010-10-04T12:41:45.107" UserId="159" />
  <row Id="4371" PostId="3267" Text="+1.  Absolutely agree with stressing the real-world relevance of stats and focussing on fundamental concepts." CreationDate="2010-10-04T12:53:34.967" UserId="266" />
  <row Id="4372" PostId="3254" Text="@onestop Just found this article in *The Lancet*: Data trawling: to fish or not to fish, http://j.mp/aestPN." CreationDate="2010-10-04T13:16:17.483" UserId="930" />
  <row Id="4373" PostId="3247" Text="@chl: although I didn't downvote this answer, I think I understand why one would downvote it. The information you have provided is correct, very very important and think-provoking. HOWEVER, most of it (except for the last two paragraphs) do not answer the question. Ideally, one would write this large disclaimer elsewhere and give a link to it." CreationDate="2010-10-04T13:23:51.490" UserId="1496" />
  <row Id="4374" PostId="3247" Text="@chl: despite what I said in my comment, I love your answer and up-vote it" CreationDate="2010-10-04T13:24:35.593" UserId="1496" />
  <row Id="4375" PostId="3283" Text="Are you sure it's not O(3^n)?" CreationDate="2010-10-04T13:34:50.743" UserId="919" />
  <row Id="4376" PostId="3289" Text="What is the statistical motivation for this question?  (It seems like it would be more appropriate to pose it on a math forum.)" CreationDate="2010-10-04T13:37:06.733" UserId="919" />
  <row Id="4378" PostId="3247" Text="@bgbg Thanks for your comment. Maybe I didn't answer the CW question. However, I never intended to give a purely provocative answer. The OP asked about potential &quot;bugs and flaws&quot; in GDocs: I provide illustrations about what I know from Excel, acknowledging the fact I don't know how it would translate to GDocs. I also understand part of the question as &quot;what are the benefits of using GDocs for data analysis&quot;, and I just gave some arguments against the use of spreadsheet for large scale projects, or analysis at the bleeding edge (still, I acknowledged at the beginning that this would be biased)." CreationDate="2010-10-04T14:21:58.300" UserId="930" />
  <row Id="4379" PostId="3293" Score="1" Text="'Elements of...' largely covers the same topics as 'Modern Multivariate...'." CreationDate="2010-10-04T14:30:50.113" UserId="603" />
  <row Id="4380" PostId="3293" Text="@kwak Well, some savings for @shabbychef! I found Izenman's book more an applied textbook featuring some methods not covered in ESL (e.g. correspondence analysis), but you're right they are pretty written in the same spirit one each other. (I'll update my answer)" CreationDate="2010-10-04T15:11:03.217" UserId="930" />
  <row Id="4381" PostId="3303" Text="@Carlos (+1) Thx for the link. I like the PGP framework too." CreationDate="2010-10-04T15:53:07.730" UserId="930" />
  <row Id="4382" PostId="3296" Text="@bgbg Could you indicate what classification method(s) is/are used?" CreationDate="2010-10-04T16:56:11.790" UserId="930" />
  <row Id="4383" PostId="3288" Text="Bonus marks for explaining how to find out stuff for myself." CreationDate="2010-10-04T17:03:03.273" UserId="1492" />
  <row Id="4384" PostId="3295" Text="@csgillespie .. Yes, but one of the difficulties I had was in extracting the clusters from the heatmap.2 object. Is there an easy way of extracting the clusters? I am aware of the `cutree` command which can be used to extract clusters from the heatmap.2 object." CreationDate="2010-10-04T17:33:26.677" UserId="1307" />
  <row Id="4385" PostId="3311" Text="This is just a recap' of what I read so far. Obviously, *I will not accept my own answer*. Any other thoughts would be much appreciated." CreationDate="2010-10-04T17:58:01.817" UserId="930" />
  <row Id="4386" PostId="3296" Text="@chl: see update to the question" CreationDate="2010-10-04T18:52:28.690" UserId="1496" />
  <row Id="4387" PostId="3310" Text="Thank you &quot;Extreme value theory&quot; was what I was looking for!" CreationDate="2010-10-04T18:59:45.897" UserId="253" />
  <row Id="4388" PostId="3308" Text="when I say they are misclassified, I mean that when I test different alternative models on the training set, all those samples are misclassified by all or most of those models" CreationDate="2010-10-04T19:00:36.237" UserId="1496" />
  <row Id="4389" PostId="3293" Text="@chl:&gt; i agree with you btw (i own both books). On the other hand, if budget is a binding constraints, you might want to diversify. " CreationDate="2010-10-04T19:27:55.740" UserId="603" />
  <row Id="4391" PostId="3293" Text="@kwak For sure :) Add your must-have references! (thx for the undelete)" CreationDate="2010-10-04T19:58:53.140" UserId="930" />
  <row Id="4393" PostId="3159" Text="@ars another funny website: http://www.visualizing.org/" CreationDate="2010-10-04T20:15:24.103" UserId="930" />
  <row Id="4394" PostId="3314" Score="1" Text="@kwak: Thanks for the reference.  It's always fun to see early work (even when it was published in one's own lifetime, LOL!). The results are simple because they combine two simple results: (1) we know the pdf of the Normal distribution (which allows us to give a name to its cdf, the Gaussian integral,which has no closed form evaluation).  (2) the standard expressions for distributions of the order statistics of any pdf.  Thus you do obtain &quot;exact&quot; expressions for the order statistics, but in they end they must be polynomials in the Gaussian integral. (Computing them in 1961 wasn't easy.)" CreationDate="2010-10-04T20:18:24.217" UserId="919" />
  <row Id="4395" PostId="3317" Text="Cont'd:                                                          2. Cook R.J. and farewell V.T. Multiplicity considerations in the design and analysis of clinical trials. Journal of the Royal Statistical Society, Series A 1996; Vol. 159, No. 1 : 93-110      " CreationDate="2010-10-04T20:41:43.223" UserId="1501" />
  <row Id="4396" PostId="3317" Text="Thank you for your comments, Brenden, especially the last one about prediction vs. causal explanation.  And welcome to the site!  I hope to see many more of your contributions in the future." CreationDate="2010-10-04T20:44:50.773" UserId="919" />
  <row Id="4397" PostId="3317" Text="Cont'd:                                                           3. Rothman K.J. No adjustments are needed for multiple comparisons. Epidemiology 1990; Vol. 1, No. 1 : 43-46            4. Marshall J.R. Data dredging and noteworthiness. Epidemiology 1990; Vol. 1, No. 1 : 5-7                                        5. Greenland S. and Robins J.M. Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Epidemiology 1991;&#xA;Vol. 2, No. 4 : 244-251 " CreationDate="2010-10-04T20:45:48.947" UserId="1501" />
  <row Id="4398" PostId="3314" Text="@whuber:&gt; Yes. the link will probably be less useful than the Gumbel approximation (except perhaps for the &lt;strike&gt; min and max &lt;\Strike&gt; *largest* and *smallest* draw) but i cited the paper nonetheless because, as you said, it has this 'head in the cloud' cachet about it :)" CreationDate="2010-10-04T20:59:04.617" UserId="603" />
  <row Id="4399" PostId="3295" Score="1" Text="@suncoolsu: I've updated my answer. Does that help?" CreationDate="2010-10-04T21:13:41.803" UserId="8" />
  <row Id="4400" PostId="3317" Text="@Brenden (+1) You may be interested in the following thread: http://stats.stackexchange.com/questions/3252/how-to-cope-with-exploratory-data-analysis-and-data-dredging-in-small-sample-stud. It seems we share a lot of links in common :-)" CreationDate="2010-10-04T21:14:57.903" UserId="930" />
  <row Id="4402" PostId="3319" Text="@Shane Nice tips, thanks." CreationDate="2010-10-04T21:43:57.563" UserId="930" />
  <row Id="4405" PostId="3322" Text="Wow, starting right at me.  Thanks." CreationDate="2010-10-04T23:18:37.907" UserId="569" />
  <row Id="4406" PostId="3289" Text="@whuber: i was hoping that some clever chap, such as you, would have some insights about how to approach this. btw - there is a longish list of related problems on the RHS of the screen, to many of which your question could be applied." CreationDate="2010-10-04T23:50:36.020" UserId="1112" />
  <row Id="4407" PostId="3278" Score="1" Text="@chl: wow, for someone who knows nothing about CCA or the &quot;French way&quot;, this was a great read!  Many thanks.  I also found this with some googling that might be of interest: http://www-stat.stanford.edu/~susan/papers/dfc.pdf" CreationDate="2010-10-05T00:53:11.253" UserId="251" />
  <row Id="4408" PostId="3324" Text="From your question you may be reading the code slightly wrong. The syntax is using the differences as the &quot;instruments&quot; to estimate the lag of the dependent variable." CreationDate="2010-10-05T01:39:55.210" UserId="1036" />
  <row Id="4410" PostId="3326" Text="Thank you for answering, many voted, but not so many answered =) I have been looking at Multi Hypothesis Tracking for a while now. I will check out your suggestions, have not heard of two of them!" CreationDate="2010-10-05T07:47:02.787" UserId="1411" />
  <row Id="4411" PostId="3331" Score="2" Text="you might get a few ideas from an earlier question on clustering individual longitudinal data trajectories http://stats.stackexchange.com/questions/2777/modelling-longitudinal-data-where-the-effect-of-time-varies-in-functional-form-be" CreationDate="2010-10-05T07:52:45.383" UserId="183" />
  <row Id="4412" PostId="3285" Score="1" Text="@chl: The `timecourse` package isn't really for determining clusters, rather it's for calculating which genes are differentially expressed. " CreationDate="2010-10-05T09:41:49.263" UserId="8" />
  <row Id="4413" PostId="3285" Text="@csgillespie (+1) Thanks. I thought it might be used to isolate genes with varying temporal profiles across biological conditions, or as a first step before using a clustering procedure (in fact, I was thinking of `kml` but I'm not really an expert in that domain)." CreationDate="2010-10-05T09:53:59.010" UserId="930" />
  <row Id="4414" PostId="3285" Text="@chl: You are correct in that you would tend use to isolate interesting genes before any clustering - basically thin down your list of genes. I suppose it does perform clustering of a sort, i.e.differentially expressed vs non-differentially expressed." CreationDate="2010-10-05T10:09:48.373" UserId="8" />
  <row Id="4415" PostId="3278" Score="1" Text="@ars (+1) Thanks for the link (didn't know about this monograph, it looks interesting). My best recommendations for recent developments are actually ALL papers from Jan de Leeuw and these two books: *Multiple Correspondence Analysis And Related Methods* from Greenacre, and *Geometric Data Analysis: From Correspondence Analysis to Structured Data Analysis* from Le Roux &amp; Rouanet (the french way)." CreationDate="2010-10-05T10:46:46.753" UserId="930" />
  <row Id="4416" PostId="2635" Score="1" Text="... and maybe a better topic?" CreationDate="2010-10-05T10:52:27.663" UserId="88" />
  <row Id="4417" PostId="3331" Score="1" Text="@Jeromy Anglin Thanks for the link.  Did you have any luck with `kml`?" CreationDate="2010-10-05T12:24:07.557" UserId="179" />
  <row Id="4418" PostId="3334" Text="Sometimes I just want to thwack myself upside the head.  Excellent suggestion." CreationDate="2010-10-05T12:28:18.390" UserId="1499" />
  <row Id="4420" PostId="3342" Text="see http://stackoverflow.com/ for R question" CreationDate="2010-10-05T13:04:33.243" UserId="1154" />
  <row Id="4421" PostId="3342" Score="5" Text="@pbneau R questions are acceptable on here.  See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions and the FAQ.  See also the recent discussion: http://meta.stats.stackexchange.com/questions/474/printhello-world-n" CreationDate="2010-10-05T13:10:23.157" UserId="5" />
  <row Id="4422" PostId="3341" Text="interesting ! there are testing images to try it, exercices to learn how to use it... " CreationDate="2010-10-05T13:16:52.360" UserId="223" />
  <row Id="4423" PostId="3343" Score="3" Text="I did not know about this. Now I can manipulate the `h$count` vector to my nefarious purposes. Marvelous. Thanks" CreationDate="2010-10-05T13:21:57.397" UserId="1492" />
  <row Id="4424" PostId="3339" Score="1" Text="@csgillepsie: there is catch here. The answers about comparing distributions assume the draws from the distributions are iid. From the tone of the questions (..&quot;mapped to line chart&quot;) this may not be the case in Tristan's application. That would invalidate all the solutions (k-s, chi-2,...) proposed." CreationDate="2010-10-05T13:26:26.503" UserId="603" />
  <row Id="4425" PostId="3339" Text="@kwak: Very good spot and something that I missed. Just to clarify it invalidates the *k-s* and *chi-2* solutions, not all **all** the answers given, i.e. my answer and the answer by Joris still stand." CreationDate="2010-10-05T13:31:59.310" UserId="8" />
  <row Id="4426" PostId="3339" Text="@kwak @csgillepsie: yeah just to clarify the data i'm dealing with is not iid at all." CreationDate="2010-10-05T13:40:58.620" UserId="1505" />
  <row Id="4427" PostId="3334" Text="I also just thought of using the `kmeans` function in R.  I really shouldn't ask questions between midnight and 4am." CreationDate="2010-10-05T13:44:33.200" UserId="1499" />
  <row Id="4428" PostId="127" Text="I agree with @PeterR." CreationDate="2010-10-05T13:49:57.157" UserId="1499" />
  <row Id="4431" PostId="3347" Text="Unfortunately, the Valencia meetings are no more. The last one was this year! Instead, we're just going to have general ISBA meetings." CreationDate="2010-10-05T16:23:44.217" UserId="8" />
  <row Id="4432" PostId="3345" Text="Thanks Kwak. I have ordered the first book from Amazon. I could still do with a few lines to get me started though ..." CreationDate="2010-10-05T16:36:42.660" UserId="1216" />
  <row Id="4434" PostId="3347" Text="Oh!  I didn't know.  Crap." CreationDate="2010-10-05T17:08:35.347" UserId="1499" />
  <row Id="4435" PostId="3347" Text="yeah Dr. Bernardo has apparently told others that he is too old to take care of the valencia conference. He wants young guns to take care of it now." CreationDate="2010-10-05T18:08:48.767" UserId="1307" />
  <row Id="4436" PostId="3353" Text="Thanks for the note. I updated my question to include the priors on beta_0 and tau; I had omitted these for simplicity" CreationDate="2010-10-05T21:07:04.240" UserId="1381" />
  <row Id="4438" PostId="3355" Text="Sorry, M. Tibbits updated the answer later, but my comments are still valid for David's question." CreationDate="2010-10-05T21:59:18.247" UserId="1307" />
  <row Id="4439" PostId="3327" Text="If I have two possible models to fit to my data, the sine wave as described in my original question and a LMS straight line fit, could I simply compare the average squared deviation from the true data values of the sine wave with the residuals of the LMS fit line and then choose the model with the lower overall value on the grounds that this model exhibits a more accurate fit to the data? If so, would it also be valid to perhaps split the data into halves and do the same with each half separately, using the same sine wave/LMS fits to see how each model may be improving/getting worse with time?" CreationDate="2010-10-05T22:24:15.647" UserId="226" />
  <row Id="4440" PostId="3283" Text="Yes I'm sure :-)" CreationDate="2010-10-06T02:11:04.877" />
  <row Id="4441" PostId="3283" Text="(see answer below for clarification)" CreationDate="2010-10-06T02:24:10.200" />
  <row Id="4442" PostId="3357" Text="Thank you for the clarification.  The point is that at each stage you only have to consider the probability distribution over (W,L,T) and not how it was derived." CreationDate="2010-10-06T02:44:23.820" UserId="919" />
  <row Id="4443" PostId="3331" Text="@fmark I've had a quick look, but for the moment I'm using a customised cluster analysis based on selected features of the individual time series (e.g., mean, initial, final, variability, presence of abrupt changes, etc.)." CreationDate="2010-10-06T02:59:14.890" UserId="183" />
  <row Id="4444" PostId="3327" Text="I'm not sure.  My suggestion was to use a Least Squares metric, but I wasn't saying to run linear regression.  You might check out [Periodic Regression](http://www.google.com/search?client=ubuntu&amp;channel=fs&amp;q=Periodic+Regression&amp;ie=utf-8&amp;oe=utf-8)." CreationDate="2010-10-06T03:41:21.377" UserId="1499" />
  <row Id="4445" PostId="3327" Text="As to your other question, could you cut the data in half, I would be very cautious in doing so -- because that would double the minimum frequency you could consider.  I think you may end up needing to look at Fourier coefficients (take an [FFT](http://en.wikipedia.org/wiki/Fast_Fourier_transform) or a [DCT](http://en.wikipedia.org/wiki/Discrete_cosine_transform) and regress on them?!? -- __Not sure__).  Or perhaps periodic regression as mentioned above." CreationDate="2010-10-06T03:47:50.440" UserId="1499" />
  <row Id="4446" PostId="2611" Text="@chl: I can send you the tables with the results when I'm done, but I won't be inventing anything new really. So far I'm just planning to compare MI under a two-level imputation model (R package pan) to MI under a simple normal model (ignoring the two-level structure, R package norm) and listwise deletion. Under different sample sizes, values of the variance component etc. This should be enough for the seminar (I'm a PhD student), but not exactly groundbreaking. If you have any ideas on how to &quot;jazz up&quot; the simulation study, i'd love to hear.&#xA;" CreationDate="2010-10-06T07:37:47.373" UserId="1266" />
  <row Id="4447" PostId="2611" Score="1" Text="One other thing: i'm not sure that a proper analytical solution to this problem even exists. I've looked at some additional literature, but this problem is elegantly looked over everywhere. I've also noticed that yucel&amp;demirtas (in the article i mentioned, page 798) write:“These multiply imputed datasets were used to estimate the model […] using the R package lme4 leading to 10 sets of (beta, se(beta)), (sigma_b, se(sigma_b)) which were then combined using the MI combining rules defined by Rubin.” " CreationDate="2010-10-06T07:40:59.840" UserId="1266" />
  <row Id="4448" PostId="2611" Text="It seems they used some kind of shortcut to estimate the SE of the variance component (which is, of course, inappropriate, since the CI is asymmetrical) and then applied the classic formula." CreationDate="2010-10-06T07:47:05.907" UserId="1266" />
  <row Id="4449" PostId="3311" Text="Thanks for accepting my answer chi, though your own reference list is much better and more recent. I really should have thought of a couple of them myself as I've got them on my hard drive, and may have even read parts of them..." CreationDate="2010-10-06T08:18:51.513" UserId="449" />
  <row Id="4450" PostId="2611" Text="@Rok Ok, thx for that. Can you put your comments into an answer so that it can be voted?" CreationDate="2010-10-06T09:25:40.647" UserId="930" />
  <row Id="4452" PostId="3360" Score="1" Text="For record keeping: The DOI of the &quot;The distribution of product of independent beta random variables with application to multivariate analysis&quot; is 10.1007/BF02480942." CreationDate="2010-10-06T13:59:29.430" UserId="1512" />
  <row Id="4453" PostId="3360" Score="1" Text="The other one is 10.1137/0118065 :)" CreationDate="2010-10-06T14:00:17.970" UserId="1512" />
  <row Id="4454" PostId="3358" Text="@Rok I appreciate you came back to share your experience with this problem. Unfortunately, I have no real solution but maybe other suggestions will come up." CreationDate="2010-10-06T14:30:17.857" UserId="930" />
  <row Id="4455" PostId="3072" Text="Thanks, but MATLAB isn't free!!" CreationDate="2010-10-06T15:33:25.953" UserId="1024" />
  <row Id="4456" PostId="3051" Text="@J.M - I hadn't! Thank you! I'm about to see how it works." CreationDate="2010-10-06T15:36:57.103" UserId="1024" />
  <row Id="4457" PostId="3051" Text="@Shane - Yes! I'm sorry that wasn't clear. The slide is the number of positions/indices you move to start computing the next window of averages. So rather than the next window starting after the end of the last there is some overlap when the slide is smaller than your window size. The idea is to smooth out the data points a bit." CreationDate="2010-10-06T15:39:42.370" UserId="1024" />
  <row Id="4458" PostId="3362" Text="I find the question interesting. However, it would help if you could briefly summarize the idea in the paper so that users need not download and read the paper to answer your question. I think it would help a lot if your question is self-contained to get quick and quality answers." CreationDate="2010-10-06T15:46:05.883" />
  <row Id="4459" PostId="3054" Text="Thanks! Especially for noting the last value as zero assumption, I hadn't considered that. I definitely care about that last window!! " CreationDate="2010-10-06T15:59:54.633" UserId="1024" />
  <row Id="4460" PostId="203" Score="1" Text="@Jeromy You may be interested in this recent article published in PARE, *Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon*, http://j.mp/biLWrA." CreationDate="2010-10-06T17:05:16.600" UserId="930" />
  <row Id="4461" PostId="3072" Text="@T-Burns: octave is free, however; also R is close enough to Matlab that this code can easily be translated. In fact, @mbq did that.." CreationDate="2010-10-06T17:24:20.507" UserId="795" />
  <row Id="4462" PostId="3361" Text="The last page alone is worth $1.20! maybe we can wedge that into @Carlos Accioly's knapsack by buying one of the books used.." CreationDate="2010-10-06T17:26:47.897" UserId="795" />
  <row Id="4463" PostId="3361" Text="The last page comes from this article: http://www.math.wm.edu/~leemis/2008amstat.pdf.  I've posted a condensed version with links for footnotes here: http://www.johndcook.com/distribution_chart.html" CreationDate="2010-10-06T17:28:55.197" UserId="319" />
  <row Id="4464" PostId="3054" Text="This is perfect! Thanks!!" CreationDate="2010-10-06T17:51:30.810" UserId="1024" />
  <row Id="4465" PostId="1618" Text="@Srikant (+1) Interesting links, thanks." CreationDate="2010-10-06T19:17:45.087" UserId="930" />
  <row Id="4466" PostId="3348" Text="This looks very useful! Just to be clear (since I'm not fluent in R): this calculates spatial correlation, but weights more highly correlations between things that are closer together. Is that correct?" CreationDate="2010-10-06T19:44:02.333" UserId="900" />
  <row Id="4467" PostId="3348" Score="1" Text="Precisely. It uses the inverse distance between the pixels (as measured in pixels) to weight the measure of spatial autocorrelation. Note that I generated a grayscale image, but you could similarly apply this to a color image either treating the colors separately or taking some combined score." CreationDate="2010-10-06T19:46:05.587" UserId="1499" />
  <row Id="4468" PostId="3368" Text="Please consider adding some references to this question (e.g. for the Nyquist frequency)." CreationDate="2010-10-06T19:57:33.723" UserId="5" />
  <row Id="4469" PostId="3362" Text="Good suggestion.  I'm not sure I understand the method well enough yet to summarize how it works accurately... in the mean time, I've excerpted the brief summary from the R implementation." CreationDate="2010-10-06T20:46:30.953" UserId="644" />
  <row Id="4470" PostId="3356" Text="Hi Andy. I don't know stata code. That is why i do not mention the code snipped in my answer, which has to be understood as a response to the part of the question that is formulated in english." CreationDate="2010-10-06T21:17:38.833" UserId="603" />
  <row Id="4471" PostId="3324" Text="lara: could you edit your question to explain in plain terms the meaning of the stata code snipped ?" CreationDate="2010-10-06T21:19:16.257" UserId="603" />
  <row Id="4472" PostId="3372" Text="Kaelin:&gt; do you mean whether there exists 'on the fly' method for computing summary stats such as median and quartiles ? If this is what you want i could give you links to papers detailing them. You could also give more details about the platforms you are working on as efficient GNU implementation of these methods likely exists in R." CreationDate="2010-10-06T21:24:31.530" UserId="603" />
  <row Id="4473" PostId="3349" Text="Thanks, Srikant! Sorry, I somehow missed your comment earlier. The upper cluster is really just a spike right at the ceiling - there's no variability there except for the long stretch of uniformity that links it to the lower distribution, which is basically as you describe.  It'll take me some time to parse out your answer (especially since I'm stuck in IE and can't see the LaTeX properly right now), but I really appreciate your dedication to this odd little question." CreationDate="2010-10-06T23:09:06.687" UserId="71" />
  <row Id="4474" PostId="3064" Text="Note: I apparently missed the deadline to actually award the bounty, so I'm setting up another so that I can properly reward Srikant for his help. More answers are always welcome, but the bounty is for him." CreationDate="2010-10-06T23:13:29.863" UserId="71" />
  <row Id="4475" PostId="3372" Text="@kwak: Yes, that sounds like what I am looking for. I would greatly appreciate those links. :-) I am working on Mac OS X… I can use R for post-processing data, but can't link GPL code into my company's product for the usual reasons." CreationDate="2010-10-06T23:20:49.910" UserId="1515" />
  <row Id="4476" PostId="731" Text="Me too. +1 for the paper." CreationDate="2010-10-06T23:29:24.227" UserId="1515" />
  <row Id="4477" PostId="3331" Text="Is this a duplicate? http://stats.stackexchange.com/questions/3238/time-series-clustering-in-r" CreationDate="2010-10-07T01:32:55.087" UserId="159" />
  <row Id="4478" PostId="3238" Text="And this one: http://stats.stackexchange.com/questions/3331/is-it-possible-to-do-time-series-clustering-based-on-curve-shape" CreationDate="2010-10-07T01:33:23.713" UserId="159" />
  <row Id="4479" PostId="3368" Text="What do you mean by `I have 3 time series'? Do you have 1 time series that has been divided it into 3 pieces? Or do you have 3 devices all independently measuring the same `signal' at the `same' time?" CreationDate="2010-10-07T01:55:12.883" UserId="352" />
  <row Id="4480" PostId="3356" Text="@kwak - I was not criticizing your post, I agree with everything you said. I was simply wondering if there was some logic as to why someone would use the differences as instruments that I was unaware of. I can't imagine any situation in which the differences would meet any of the requirements for such a procedure." CreationDate="2010-10-07T02:04:20.827" UserId="1036" />
  <row Id="4482" PostId="3356" Text="Hi Andy:&gt; i didn't take you're comment as a critic. Your post is highlighting a key aspect of the question that neither Rob nor I (admittedly) understood. If anything, it illustrates the importance of collaboration." CreationDate="2010-10-07T02:49:50.533" UserId="603" />
  <row Id="4483" PostId="3336" Text="Thank you for those references." CreationDate="2010-10-07T03:00:35.717" UserId="1036" />
  <row Id="4484" PostId="3378" Text="Wow, really interesting approach. I didn't know about that, and I will keep it in mind. Unfortunately, in this case it will not work, since I have really restrictive requirements from the point of view of memory usage, so M should be really small, and I guess there would be too much precision loss." CreationDate="2010-10-07T05:26:10.993" UserId="667" />
  <row Id="4485" PostId="3331" Text="@Rob This question doesn't seem to assume irregular time intervals, but indeed they are close one each other (I didn't remind of the other question at the time of my writings)." CreationDate="2010-10-07T06:11:08.007" UserId="930" />
  <row Id="4486" PostId="3365" Text="@Chi Thanks! This works, but means I have to plot each item individually (and combine them with par). What I was hoping for was to simply obtain the same result by using par(mfrow=c(3,3)) and simply plot(fit1, 'some edits to 'main='&quot;) which gives me all the plots (9 in my case). " CreationDate="2010-10-07T06:14:54.317" UserId="913" />
  <row Id="4488" PostId="3365" Text="@Tormod I'm afraid you can't. Look at the R code for that function (just type `plot.grm` at the R prompt or `edit(plot.grm)`) near line 52 and 91: Dimitri R. automatically prefixes the title if missing and append &quot;Item&quot;." CreationDate="2010-10-07T06:25:55.320" UserId="930" />
  <row Id="4489" PostId="3365" Text="@Chi Thanks again! Still, your solution works fine so I am still able to achieve what I want, albeit with a few more lines of code. " CreationDate="2010-10-07T06:31:47.643" UserId="913" />
  <row Id="4490" PostId="3374" Text="@Brett Note that you will get the same results with `principal(attitude, 2, rotate=&quot;none&quot;)` from the `psych` package and that Kayser's rule (ev &gt; 1) is not the most recommended way to test for dimensionality (it overestimates the number of factors).&#xA;" CreationDate="2010-10-07T06:36:59.833" UserId="930" />
  <row Id="4491" PostId="3365" Text="@Tormod The same applies for the `eRm` package. It's difficult to provide users with both a friendly and generic interface :(" CreationDate="2010-10-07T06:40:03.617" UserId="930" />
  <row Id="4493" PostId="3363" Score="1" Text="Thanks for your very detailed answer chi.  I've got `kml` running on my data, but as you suggested it is clustering mostly based on magnitude rather than curve shape, so I'm trying a few pre-processing steps to see if I can improve matters.  The work by Sangalli et al. looks very promising for what I want to do - I cannot find an implementation of their approach however.  I probably do not have time to create my own implementation of their work for this project, hwoever.  Are you aware of any FOSS implementations?" CreationDate="2010-10-07T06:45:18.550" UserId="179" />
  <row Id="4494" PostId="3363" Text="@fmark No OSS implementation to my knowledge (the work is quite recent, though); they use k-means and k-medoids which are both available in R. To my opinion, the most critical parts are to generate template curves and implement the warping function. For that, you could find additional infos by looking at morphometry/procruste analysis, or lookup the code of the Matlab PACE toolbox (but this should be full of EM or things like that). My best recommendation would be: Ask the author for any free of charge implementation of their algorithm." CreationDate="2010-10-07T06:53:49.593" UserId="930" />
  <row Id="4495" PostId="3363" Score="2" Text="I'll report back if I get an affirmative :)  Their paper [k-mean alignment for curve clustering](http://mox.polimi.it/it/progetti/pubblicazioni/quaderni/13-2008.pdf) has some more implementation details that also might be useful to someone wanting to do this themselves.&#xA;" CreationDate="2010-10-07T07:10:19.817" UserId="179" />
  <row Id="4496" PostId="3376" Text="+1: nice answer. Thanks." CreationDate="2010-10-07T08:16:54.630" UserId="8" />
  <row Id="4497" PostId="3247" Text="@chi: I can see why someone would downvote this, but I think your answer is a valid response.  A list of circumstances when using spreadsheets is not a good idea is a handy addition to a discussion of the advantages of using them." CreationDate="2010-10-07T08:27:38.400" UserId="266" />
  <row Id="4498" PostId="3376" Text="+1 Right; I was still in dark ages of making approximation from histogram." CreationDate="2010-10-07T08:50:21.420" UserId="88" />
  <row Id="4499" PostId="3368" Text="Shane, by references do you mean tagging? If so, it seems I'm not allowed to add new tags as I'm a newbie." CreationDate="2010-10-07T10:08:32.523" UserId="1513" />
  <row Id="4500" PostId="3368" Text="No, the three time series are not divided from one. Although independently sampled, they are strongly correlated with each other." CreationDate="2010-10-07T10:15:30.717" UserId="1513" />
  <row Id="4501" PostId="3373" Score="1" Text="Really appreciate your help! Using your cell tower as an example, my scenario is more like the following: three cell phones at three equally-spaced locations receive signal from the same cell tower. Would this be considered &quot;in phase&quot;? Thanks again!" CreationDate="2010-10-07T10:33:02.617" UserId="1513" />
  <row Id="4503" PostId="3373" Score="1" Text="I'm not sure.  If you look at your data, does the first sample collected in each of the three time series correspond to the same point in time in the underlying process?  If so, then they're &quot;in phase&quot;.  Or is there a lag between when one series is collected and the other two are collected?  If there is a lag, then they would be out of phase. There's a cute picture on [this Wikipedia](http://en.wikipedia.org/wiki/Phase_%28waves%29) page which might provide some insight." CreationDate="2010-10-07T11:28:20.737" UserId="1499" />
  <row Id="4504" PostId="3381" Score="6" Text="What's wrong with conventional kernel density estimation?" CreationDate="2010-10-07T12:08:17.477" UserId="449" />
  <row Id="4505" PostId="3375" Text="Thanks Srikant, yes I think that summarizes the method well.  The examples I've seen have applied the method to an aggregate dependent variables like GDP or cigarette sales per capita (estimated from tax revenues).  Is there anything special I'd need to do when using a survey-based dependent variable instead?  I guess add confidence bands to my time-series plots??  Then do some power estimates based on what divergence between treated and control groups I wouldn't want to miss by chance, then extrapolate/guesstimate how many control-groups responses I need to form useful synthetic control??  " CreationDate="2010-10-07T12:53:56.537" UserId="644" />
  <row Id="4506" PostId="3375" Text="Are there other things I'd need to do to take into account the fact that my dependent variable is an estimate?" CreationDate="2010-10-07T12:55:49.853" UserId="644" />
  <row Id="4507" PostId="3383" Text="Can you distinguish what pre-treatment weight is assigned to each different treatment group? (i.e. I know these 16 observations are pre-treatment weights for treatment A). Also were treatments randomly assigned?" CreationDate="2010-10-07T13:03:32.487" UserId="1036" />
  <row Id="4508" PostId="3383" Text="The 16 observations are 16 fishes in a tank. The treatment is given to the tank. So the 16 fishes in the tank were weighed before and after treatment. But as the fishes weren't marked, it's impossible to link the weights to a specific individual." CreationDate="2010-10-07T13:08:06.660" UserId="1124" />
  <row Id="4509" PostId="3385" Text="I've been thinking about that as well, but the initial mean weight differs significantly between tanks, and using all t=0 observations as one control makes for a very unbalanced design. On top of that, the hypothesis is formulated in difference between treatments, which cannot be formally tested using a control as reference group. The multi-level framework doesn't help me either for the same reason. " CreationDate="2010-10-07T13:55:38.403" UserId="1124" />
  <row Id="4510" PostId="3385" Text="I don't see why the unbalanced design is a problem, but the mean weight difference is obviously a problem with my suggestion. I would still think the multi-level framework would allow you to assess differences between treatments somehow (I do not know how offhand though). The only other thing I can think of would be to simply graph each control/comparison like this response did (minus the connecting lines) http://stats.stackexchange.com/questions/2067/follow-up-in-a-mixed-within-between-anova-plot-estimated-ses-or-actual-ses/2138#2138 , although I imagine this is not entirely satisfactory." CreationDate="2010-10-07T14:10:36.467" UserId="1036" />
  <row Id="4511" PostId="3374" Text="Yes, I know psych principal wraps this up.  My purpose was to show what SPSS &quot;factor analysis&quot; was doing when using the principal components extraction method.  I agree that the eigenvalue rule is a poor way to select the number of factors.  But, that is exactly what SPSS does by default and this was what I was demonstrating. " CreationDate="2010-10-07T14:21:18.797" UserId="485" />
  <row Id="4512" PostId="3383" Score="1" Text="Your issues with independence of observations go beyond repeated measurements.  Your 16 fish within a tank are not independent and the treatment was applied to the tank.  You do not have 16 independent pre and post observations.  In fact, you have 6 observations assigned to one of 6 treatments.  Any analysis that uses the fish as the analytical unit will grossly overestimate your degrees of freedom.  At very least, you need to account for the nested structure and the intra-tank correlations.  Needless to say, you need to increase your N, which is 6 right now (see pseudo-replication). " CreationDate="2010-10-07T14:41:28.690" UserId="485" />
  <row Id="4513" PostId="3382" Text="well to console you i have just started out and was just looking around some. i have no interest in getting in your way (: its just something that has been bugging me for a while. there should be some kind of normalization possible with sequences" CreationDate="2010-10-07T15:08:36.047" UserId="1516" />
  <row Id="4514" PostId="3383" Score="1" Text="@Brett : I am aware of the fact that my observations aren't independent. But I don't agree with your statement that N=6. That would mean that a clinical study with 4 hospitals involved would have N=4, which obviously doesn't make sense. It's impossible to give fish in the same tank different treatments, as the treatment is in the water. The 16 fish are independent, but the measurements at t=0 and t=1 aren't. I correct for this by calculating the df using Satterthwaite and then divide them by 2, which gives me a df of appx. 14 for the t-tests. " CreationDate="2010-10-07T15:09:33.383" UserId="1124" />
  <row Id="4515" PostId="3375" Text="@heather Why do you call your DV as an estimate? If you call it as an estimate because you are taking the mean/sum of your likert scales then I do not see how that matters conceptually. As far as power is concerned, that would be dependent on the model/estimation specifics and I am unable to offer any suggestions on that issue. Perhaps, you can just email the authors for some suggestions or do a simulation to see how much sample size you need for desired power?" CreationDate="2010-10-07T15:11:03.453" />
  <row Id="4516" PostId="3383" Text="In your hospital example, it depends on how the patients are randomized.  If the hospitals are assigned to the treatments, then your N is the number of hospitals.  If patients are randomly assigned within hospitals, then your N is the number of patients.  Again, I refer you to the literature on pseudoreplication which you'll find easily.  Once you say &quot;the treatment is given to the tank&quot; you are done.  Level of treatment implies appropriate level of analysis.  Here is the original article http://www.masterenbiodiversidad.org/docs/asig3/Hurlbert_1984_Pseudoreplication.pdf There's plenty more." CreationDate="2010-10-07T15:31:17.050" UserId="485" />
  <row Id="4517" PostId="3388" Score="1" Text="@gaetan I do not understand the remark about a single column vs multiple columns. Are you suggesting that categorical variables should be coded as 1, 2, 3 etc in a single column instead of using dummy variables? I am not sure that makes sense to me as you are then imposing an implicit constraint that the difference in the effect on dv between leve1s 1 and 2 is the same as the difference in the effect on dv between levels 2 and 3. Perhaps, I am missing something." CreationDate="2010-10-07T16:07:12.280" />
  <row Id="4518" PostId="3375" Text="Great, Srikant, thanks.  The fact that you don't think it matters conceptually answers it for me!  I wanted confirmation of that I guess." CreationDate="2010-10-07T16:13:46.760" UserId="644" />
  <row Id="4519" PostId="3378" Text="@gianluca: it sounds like you have 1. a lot of data, 2. limited memory resources, 3. high precision requirements. I can see why this problem is freaking you out! Perhaps, as mentioned by @kwak, you can compute some other measure of spread: MAD, IQR, standard deviation. All of those have approaches which might work for your problem." CreationDate="2010-10-07T16:22:27.510" UserId="795" />
  <row Id="4520" PostId="3383" Text="@Brett Magill - I completely disagree with the statement &quot;Level of treatment implies appropriate level of analysis&quot;. Your units of analysis will be determined both by the question at hand and restrictions on your data. Now I agree that this design causes nesting complications, but it doesn't make it unreasonable to make assumptions about the independence of units within treatment groups and make adjustments to degrees of freedom accordingly. Your statement is essentially refuting the methodology of multi-level models and that entire body of work (as well as any non-experimental study.)" CreationDate="2010-10-07T16:45:27.370" UserId="1036" />
  <row Id="4521" PostId="3383" Text="Notice, I said implies rather than dictates or controls.  Also notice the mention that you need to at least account for intra-tank correlation--a nod to a multi-level approach.  In fact, in this case, this is exactly what multi-level models will do--penalize the d.f. according to the degree of intra-class correlation, effectively adjusting the N downward to account for the structure.  By the way, I've got slides from George Casella's Experimental Design session at ASA that talk about this specific problem--randomizing tanks rather than fish--as an example of pseudo-replication. " CreationDate="2010-10-07T17:00:37.870" UserId="485" />
  <row Id="4523" PostId="3383" Text="If you don't want to read the original article and the ensuing literature that I cited previously, take a look at this 1.5 page writeup from the statistical consulting unit at Cornell. It's pretty easy to follow. http://cscu.cornell.edu/news/statnews/stnews75.pdf" CreationDate="2010-10-07T17:24:08.353" UserId="485" />
  <row Id="4525" PostId="3388" Score="1" Text="@Gaetan I am not sure I follow you. How exactly does XLStat transform the 'text' values of cold, mild or hot into numerical values for the purpose of estimation? If there is a method that will let you estimate the effects of categorical variables without using dummy variables surely that should be independent of the software you use as there should be some underlying conceptual/model based logic. " CreationDate="2010-10-07T17:35:04.523" />
  <row Id="4526" PostId="3391" Text="@Irish Thanks for the link!" CreationDate="2010-10-07T17:48:50.517" UserId="930" />
  <row Id="4527" PostId="3378" Text="gianluca:&gt; Give us more quantitative idea about the size of memory, arrays and accuracy you want. It may well be that your question will be best answered @ stackoverflow though. " CreationDate="2010-10-07T18:28:03.013" UserId="603" />
  <row Id="4528" PostId="3388" Text="@Gaetan I don't follow your point unless you consider that your ordinal variable is treated as a continuous one (this might make sense sometimes, although we clearly assume that the variable can inherit the property of an interval scale as pointed by @Skrikant). Usually, a variable with $k$ levels is represented in the design matrix as $k-1$ columns, and I think this is quite independent of the software used (surely, XLStat takes care of constructing the correct design matrix as R, SPSS or Stata does)." CreationDate="2010-10-07T18:42:08.173" UserId="930" />
  <row Id="4530" PostId="3388" Text="@Srikant.  XLStat in its demo used an example a model where the dependent variable was probability of renewing a subscription.  And, one categorical variable within a single column had 6 different age ranges of subscribers.  Using a maximum likelihood algorithm, it can interpret each specific age range as a separate data set equivalent to a separate dummy variable in its own column.  When you choose that specific column, you just have to state it is a &quot;qualitative&quot; variable (instead of a &quot;quantitative&quot; one).  From everyone comments, I gather this is not something you can code in SPSS. " CreationDate="2010-10-07T19:01:26.587" UserId="1329" />
  <row Id="4531" PostId="3382" Text="@tarrash Nah, this fourth point is rather too fresh to be tested enough. And as I wrote, normalization is either in aligning or in getting into spectra. " CreationDate="2010-10-07T19:10:21.303" UserId="88" />
  <row Id="4532" PostId="3388" Score="1" Text="@Gatean Ok, in this case, the same can be done in SPSS (you have the choice between numerical/ordinal/nominal for each variable) -- then, the design matrix is constructed accordingly." CreationDate="2010-10-07T19:21:50.903" UserId="930" />
  <row Id="4533" PostId="3388" Score="2" Text="@Gaetan @chl To summarize my understanding: The features of SPSS and XLStat whereby you can specify the measurement scale (nominal, ordinal etc) decreases the data file size. However, in both instances, the software uses the correct coding scheme (e.g., expand a nominal variable with J categories into J-1 dummy variables) as part of the estimation process in the background. Would that be a fair assessment of the situation?" CreationDate="2010-10-07T19:32:54.027" />
  <row Id="4534" PostId="3388" Text="@Skrikant It seems you are summarizing the situation much better than me!" CreationDate="2010-10-07T19:49:18.240" UserId="930" />
  <row Id="4535" PostId="3397" Text="@Stephan (+1) You are nicely echoing the discussion on Medstats about the need to keep a record of data edit and analysis (http://j.mp/dAyGGY)! Thx." CreationDate="2010-10-07T20:01:03.863" UserId="930" />
  <row Id="4536" PostId="3383" Text="@Brett : I did read the article and I understand what you're getting at, but this is not an ecological experiment. Following Hurlbert, each fish should have its own tank, and the water in these tanks should not come from the same source. But the experiment done here is not equivalent to the ecological studies he describes. Plus, this approach is impossible if you can't collect water in 192 different lakes... Furthermore, his paper is not a statistical result, but a logical -and in a number of cases valid- argument that not everybody agrees upon." CreationDate="2010-10-07T20:11:23.440" UserId="1124" />
  <row Id="4537" PostId="3383" Text="Ok, how about Casella in hist Statistical Design of Experiments Book.  Here's a link the relevant page in Google books.  http://books.google.com/books?id=sqnbSUtryVAC&amp;pg=PA5&amp;lpg=PA5&amp;dq=casella+pseudoreplication&amp;source=bl&amp;ots=634YgjM_h7&amp;sig=hf8C_vWd03hELis6bC-nxyltTWA&amp;hl=en&amp;ei=3ymuTOeDJdT-nAfis4DuBQ&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=2&amp;ved=0CBYQ6AEwAQ#v=onepage&amp;q&amp;f=false" CreationDate="2010-10-07T20:14:30.780" UserId="485" />
  <row Id="4538" PostId="3396" Text="Thanks for your help!!! Do you know if anything of the sort exists for Matlab by any chance? I am not that familiar with R.." CreationDate="2010-10-07T20:15:00.437" />
  <row Id="4539" PostId="3397" Text="@Stephan: Saving your &quot;work and blind alleys&quot; isn't any harder to do with Excel than with R.  It's just a matter of actually doing it.  The main problem with Excel is related to its strength: it's all too easy to change something inadvertently.  But for EDA--the focus of the OP--we rarely if ever save everything we do.  EDA, after all, is supposed to be *interactive.*" CreationDate="2010-10-07T20:18:51.347" UserId="919" />
  <row Id="4540" PostId="3383" Text="@Brett : As I said, it is a point of view. One can easily argument that every fish has its own physiology. Biologically spoken, adding the weights of all fish in 1 tank doesn't even make sense, as you have males and females, different age groups and the likes. If I would have done the experiment, I would have marked the fish so it would be a truly repeated design. But having 4 seperate tanks filled with exactly the same water and given exactly the same treatment, does not mean for me that I suddenly have more degrees of freedom. It merely means I have split up 1 tank in 4. " CreationDate="2010-10-07T20:19:45.760" UserId="1124" />
  <row Id="4541" PostId="3396" Text="@BobJones: I am unfamiliar with MatLab but quite familiar with a lot of the geostatistical software out there; I haven't run across anything specifically mentioning MatLab.  It's not hard to create an empirical variogram, though: it involves binning the squared differences of values associated with all distinct pairs of (X,Y) points, summarizing the values by bin, and graphing those summaries.  It's so easy, though, to learn to import a simple dataset (your X, Y, Z values) and call an R routine that you could be up and running quickly if you want." CreationDate="2010-10-07T20:25:34.460" UserId="919" />
  <row Id="4542" PostId="3383" Text="@Brett : the main idea behind Hurlbert, is that you can't distinguish between the effect of &quot;tank&quot; and the effect of &quot;treatment&quot; theoretically spoken. Practically, it's proven in the lab that the used procedures do not cause a &quot;tank&quot; effect beyond natural variation between fish. Hence... edit : for the record, I do appreciate your input, it makes me think about the setup and it gives me something to tell the lab people not to repeat this setup any more in their life :-)" CreationDate="2010-10-07T20:31:03.667" UserId="1124" />
  <row Id="4543" PostId="3398" Score="3" Text="@whuber A nice and handy overview of pros and cons!" CreationDate="2010-10-07T20:32:22.593" UserId="930" />
  <row Id="4544" PostId="3388" Text="@Skrikant, you are summarizing the situation correctly.  I may have confused everyone with an earlier comment whereby I used &quot;nominal&quot; incorrectly.  I thought this adjective related to real numbers so to speak.  I realize that nominal means just a label which is what it should be when dealing with qualitative categorical variables." CreationDate="2010-10-07T20:36:32.897" UserId="1329" />
  <row Id="4545" PostId="3394" Text="I like the nuance about &quot;different levels of discussion.&quot;" CreationDate="2010-10-07T20:39:50.200" UserId="919" />
  <row Id="4546" PostId="3398" Score="1" Text="+1 nice and balanced.  I especially like the point about &quot;immediacy of interacting directly&quot; which I think is Excel's (or really, the spreadsheet's) biggest selling point.  Declarative programming for the masses -- which explains why some people think that 80% of the world's business logic is written in Excel (worth pointing out to programmers and statisticians who argue about R v SAS or Java v C++, etc)." CreationDate="2010-10-07T20:42:44.387" UserId="251" />
  <row Id="4547" PostId="3386" Score="2" Text="This is just a partial answer: even when you create the dummies explicitly (rather than using the software's implicit capabilities), keep them together in all analyses.  In particular, they should all enter together and all leave together in a stepwise regression, with the p-value computed appropriately for the total number of variables involved.  (This is Hosmer &amp; Lemeshow's recommendation, anyway, and it makes a lot of sense.)" CreationDate="2010-10-07T20:53:03.170" UserId="919" />
  <row Id="4549" PostId="3398" Score="1" Text="I heard that Microsoft hired some numerical analysts several years ago to fix the broken functions in Excel.  Do you know whether the problems with Excel are still there in the 2007 or 2010 versions?" CreationDate="2010-10-08T00:44:30.780" UserId="319" />
  <row Id="4550" PostId="3398" Text="John, the McCullough and Heiser paper that Carlos references addresses Excel 2007.  I will not adopt Excel 2010 for various reasons, so I haven't had the opportunity to test it." CreationDate="2010-10-08T02:29:18.043" UserId="919" />
  <row Id="4551" PostId="3399" Text="@Gaetan: Thanks for the additional points and for sharing your perspective." CreationDate="2010-10-08T02:31:07.517" UserId="919" />
  <row Id="4552" PostId="3403" Score="4" Text="@ars: Everything is correct and nicely stated.  But one thing seems to be missing: the standard deviation of the &quot;best approximation&quot; j/n depends on the *true* proportion of redheads, not the estimated one.  The problem, of course, is that we don't know the true proportion.  But the fact remains that the standard error does not actually equal the standard deviation of the approximation except when the estimate happens to be exactly correct.  I know you don't need reminding of this subtlety, nor will most readers, but it's rather relevant to the original question." CreationDate="2010-10-08T02:39:29.737" UserId="919" />
  <row Id="4553" PostId="3405" Score="3" Text="This is far and away the aspect of Excel that infuriates me the most.  Data storage needs explicit data types, not formatting." CreationDate="2010-10-08T02:45:13.517" UserId="71" />
  <row Id="4554" PostId="3404" Text="@propofol, I fully agree with you.  An observational study may be good to uncover some associations that in turn one can test using a much more rigorous framework (randomized trial as you suggest). " CreationDate="2010-10-08T03:42:40.613" UserId="1329" />
  <row Id="4555" PostId="3405" Score="1" Text="Actually, this is something about MS software in general that annoys me: it changes your input into what it believes you actually meant, and you usually don't even see it happening." CreationDate="2010-10-08T04:17:49.740" UserId="666" />
  <row Id="4556" PostId="3405" Text="@csgillespie (+1) Good to hear of Excel from this perspective!" CreationDate="2010-10-08T06:15:51.263" UserId="930" />
  <row Id="4557" PostId="3403" Text="@whuber: good point, thanks for noting that. " CreationDate="2010-10-08T08:47:32.220" UserId="251" />
  <row Id="4558" PostId="3397" Text="@Stephan it is possible to keep a reproducable record of your methods if you do it in VB, but the GUI focus of Excel doesn't encourage that behaviour." CreationDate="2010-10-08T10:06:19.590" UserId="229" />
  <row Id="4559" PostId="3381" Text="Correct me, conventional KDE uses *all* the input data points for *each* estimate(x) -- quite unpractical for many points. I'm looking for ways to combine a) get some nearby points (a general problem), b) weight the values from this small sample" CreationDate="2010-10-08T10:54:06.927" UserId="557" />
  <row Id="4560" PostId="3396" Text="Ok cool. I have played around with R a bit so maybe I'll try to combine both somehow. Thanks again for your help. " CreationDate="2010-10-08T12:17:07.883" />
  <row Id="4561" PostId="3412" Text="@John Your models are not nested, what would be the rationale for using an LRT between the two?" CreationDate="2010-10-08T12:52:42.560" UserId="930" />
  <row Id="4563" PostId="3405" Score="1" Text="My favorite error occurred when Excel used to quietly truncate fields during export to other formats.  In a file of pesticide concentrations in soil, it converted a value of 1,000,050 (extraordinarily toxic) to 50 (almost inconsequential) by clipping off the initial digit!" CreationDate="2010-10-08T13:04:25.073" UserId="919" />
  <row Id="4566" PostId="3414" Text="Thank you! Do you know any Matlab / Mathematica based packages?" CreationDate="2010-10-08T14:12:08.797" UserId="1250" />
  <row Id="4567" PostId="3414" Text="No, I'm afraid not.  But R is fully open source so all the algorithms above will be very transparent if you want to re-implement. " CreationDate="2010-10-08T14:20:03.580" UserId="5" />
  <row Id="4568" PostId="3416" Score="1" Text="I was thinking that this wouldn't be appropriate because the two would actually be highly correlated.  As the center is far away so will the near tend to be." CreationDate="2010-10-08T14:34:55.313" UserId="601" />
  <row Id="4569" PostId="3416" Text="@John Good point." CreationDate="2010-10-08T14:41:46.183" />
  <row Id="4570" PostId="3414" Score="1" Text="and if you google for matlab hurst exponent you'll find several." CreationDate="2010-10-08T14:42:09.200" UserId="247" />
  <row Id="4571" PostId="3414" Text="First result from google: [Hurst Exponential](http://www.mathworks.com/matlabcentral/fileexchange/9842)" CreationDate="2010-10-08T14:59:39.040" UserId="1499" />
  <row Id="4572" PostId="3416" Text="I think your point is good as well... I'm actually not sure it matters.  I know it's high but under 0.8... still analyzable." CreationDate="2010-10-08T15:03:32.863" UserId="601" />
  <row Id="4573" PostId="3417" Text="I would probably add, that the difference between regions in terms of occaisonal smokers is largely restricted to the second axis of variation and as we can see in the stats this is such as small component as to not really be worth interpreting. The structure in the data is really the contrast between BC and the other regions in terms of daily smokers and former smokers." CreationDate="2010-10-08T16:25:56.173" UserId="1390" />
  <row Id="4574" PostId="3403" Text="@whuber: This clarification left me a bit confused. Given a $j$ and an $n$, what would be the standard error, described by $j$ and $n$? (In contrast to being dependent on the *true* proportion of redheads, which we can't know.)" CreationDate="2010-10-08T16:42:05.280" UserId="1523" />
  <row Id="4575" PostId="3403" Score="2" Text="@cool-RR: ars is correct about the standard error.  The point is that the standard error itself is an estimate of how accurate the statistic j/n estimates the true proportion.  For example, suppose 10% of all people are redheads.  Then in many cases it can happen that j=0 when n=10.  You would obtain an SE of Sqrt(0(1-0)/10) = 0.  This obviously underestimates the actual precision of your statistic p = j/n = 0/10.  The true precision is Sqrt(0.10(1-0.90)/n), even though you don't know that!" CreationDate="2010-10-08T17:19:14.910" UserId="919" />
  <row Id="4576" PostId="3411" Text="Thanks for the insight!" CreationDate="2010-10-08T17:20:40.817" UserId="1499" />
  <row Id="4577" PostId="3396" Score="1" Text="@BobJones: You can see http://www.mathworks.com/matlabcentral/fileexchange/20355 for variograms or http://mgstat.sourceforge.net/ for a full-out matlab geostatistics toolkit" CreationDate="2010-10-08T17:29:22.580" UserId="900" />
  <row Id="4578" PostId="3395" Text="I asked a similar question that you might find useful: http://stats.stackexchange.com/questions/3199/clustering-of-a-matrix-homogeneity-measurement" CreationDate="2010-10-08T17:32:34.273" UserId="900" />
  <row Id="4579" PostId="3396" Text="Cool thanks Xodarap!! Appreciate it!" CreationDate="2010-10-08T18:21:20.963" />
  <row Id="4580" PostId="3396" Text="@Xodarap: many thanks for sharing that tip." CreationDate="2010-10-08T20:11:54.723" UserId="919" />
  <row Id="4581" PostId="153" Score="1" Text="John Tukey wrote otherwise (back in 1960) in a monograph &quot;Data Analysis and Behavioral Science&quot; (published in Collected Works v. III).  One result he obtained is that if you're getting better than about 10% test-retest agreement, your scale isn't narrow enough!" CreationDate="2010-10-08T20:17:35.517" UserId="919" />
  <row Id="4582" PostId="663" Text="Michael Lavine is a clear lecturer and thinker, so I have little doubt his book is worth a look." CreationDate="2010-10-08T20:26:34.293" UserId="919" />
  <row Id="4583" PostId="3403" Text="Again: I am interested in what I *can* know, not in what I *can't* know. Let's take your example where $j=0$ and $n=10$. The most likely proportion of redheads is 0%, but there's good chance it's 2% or 5% or 10%. So my question is: Given that $j=0$ and $n=10$, what is the probability distribution function of the proportion of redheads, *from the information that I know, not the information that I don't know?*" CreationDate="2010-10-08T20:50:22.417" UserId="1523" />
  <row Id="4585" PostId="3422" Text="@ars Thanks for the clarification." CreationDate="2010-10-08T21:11:35.550" UserId="930" />
  <row Id="4586" PostId="3403" Score="1" Text="@cool-RR: for small samples, use the Agresti-Coull interval specified in the Wikipedia link on confidence intervals.  Based on your observations, you will obtain a 95% interval for estimate.  Then, what you will know, based on what you observed, is inherent in the definition of a 95% CI." CreationDate="2010-10-08T21:26:40.960" UserId="251" />
  <row Id="4587" PostId="3424" Text="Yeah, I thought about using a log scale. Unfortunately, I don't think our selected charting package (highcharts) supports it. Also, the data is such that even with a log scale it's still pretty extreme. I guess I was hoping there'd be some obscure viz type that I haven't heard of. :)" CreationDate="2010-10-08T21:31:18.143" UserId="1531" />
  <row Id="4588" PostId="3368" Text="@user1513 By references I mean links to more information." CreationDate="2010-10-08T21:31:34.307" UserId="5" />
  <row Id="4589" PostId="3424" Score="1" Text="@sprugman I wouldn't accept this just yet.  There are plenty of other possible answers.  But I do think that scaling the data so that it's more linear between small and large values would be a good approach (you don't have to use a log scale, it can be something else...).  And scaling can be done on the data before it is plotted, so you can do that with any package." CreationDate="2010-10-08T21:33:22.120" UserId="5" />
  <row Id="4590" PostId="3424" Text="ok, shane, you talked me out of accepting your answer. Thanks. :)" CreationDate="2010-10-08T21:43:10.357" UserId="1531" />
  <row Id="4591" PostId="3296" Text="It is not clear to me: are the independent (i.e. right hand side) variable continuous or discrete ?" CreationDate="2010-10-08T22:02:54.020" UserId="603" />
  <row Id="4592" PostId="3427" Score="8" Text="+1. Also, in case it's helpful, the term to google when comparing strings is &quot;edit distance&quot;: http://en.wikipedia.org/wiki/Edit_distance" CreationDate="2010-10-08T22:05:39.320" UserId="251" />
  <row Id="4593" PostId="3427" Text="@ars:&gt; thanks, that's a handy list to feed unto a R search engine and see what comes out!" CreationDate="2010-10-08T22:09:46.757" UserId="603" />
  <row Id="4594" PostId="3424" Score="3" Text="+1, log scale sounds good to me -- @sprugman: couldn't you just preprocess your data and log-transform it before dispatching it to your charting routine?" CreationDate="2010-10-08T22:22:45.080" UserId="251" />
  <row Id="4595" PostId="3432" Text="@Whuber:&gt; thanks, first time i've eared of this idea. " CreationDate="2010-10-09T00:26:53.660" UserId="603" />
  <row Id="4596" PostId="663" Text="I'm teaching out that book this semester.  It may be a good *statistics* book, but I'm having second thoughts regarding its light probability content." CreationDate="2010-10-09T01:21:48.787" UserId="319" />
  <row Id="4597" PostId="3425" Score="1" Text="Tal, could you show us some of the data? Or tell us a little bit more about how the strings do or do not match? There are a number of options here most of which are dependent on how you match it up visually. " CreationDate="2010-10-09T02:28:06.740" UserId="776" />
  <row Id="4598" PostId="3422" Text="I didn't specify that I had been fitting with REML = FALSE.  I'm still in a bit of a quandry though... The AIC gives me a measurement of the whole likelihood including the random effects.  That's a large component. And of course, the AIC's are extremely unlikely to be exactly the same.  Therefore, it seems unwise to just select the larger value with out some analytic way of telling how much larger it is." CreationDate="2010-10-09T07:49:55.163" UserId="601" />
  <row Id="4599" PostId="3422" Text="@ars @John This lecture highlights interesting point about REML vs. ML and AIC (and points to what you said, John), http://j.mp/bhUVNt. Bolker's review of GLMM is also worth to take a look at: http://j.mp/cAepqA." CreationDate="2010-10-09T08:29:22.517" UserId="930" />
  <row Id="4600" PostId="3436" Score="1" Text="Agree with last para. The Gaussian kernel uses all the data points but other commonly-used kernels (Epanechnikov, biweight, cosine, Parzen, triangular, ...) are finite-width, i.e. they are defined to be zero outside a finite interval. R's density() function defaults to Gaussian but Stata's -kdensity- command defaults to Epanechnikov." CreationDate="2010-10-09T09:38:31.357" UserId="449" />
  <row Id="4601" PostId="3427" Text="+1 for RecordLinkage." CreationDate="2010-10-09T12:01:21.647" UserId="88" />
  <row Id="4602" PostId="3425" Text="Hi Brandon, I will now update my question with some data.  Thanks" CreationDate="2010-10-09T13:12:14.780" UserId="253" />
  <row Id="4603" PostId="3433" Text="Hi Brandon - I added a sample of the data.  Thanks!" CreationDate="2010-10-09T13:17:59.283" UserId="253" />
  <row Id="4604" PostId="3440" Text="(I have version 7.) I have no problem loading the Statistics package. But what's the function in there called? Because I get the impression that this `Quantile` line will do the calculation manually instead of using a formula." CreationDate="2010-10-09T14:13:25.680" UserId="1523" />
  <row Id="4605" PostId="346" Score="1" Text="@Srikant:&gt; it's a pretty active area of research in statistics :) The solution closest to the lower theoretical bounds in terms of storage involve some pretty clever probability constructs as well. All in all i was surprised when i first looked unto it a couple of month ago; there is more stats here than meets the eye." CreationDate="2010-10-09T14:21:03.117" UserId="603" />
  <row Id="4606" PostId="3439" Score="1" Text="@Tal Quantile *vs.* percentile (it's merely a matter of terminology), http://j.mp/dsYz9z. " CreationDate="2010-10-09T14:22:20.277" UserId="930" />
  <row Id="4607" PostId="3440" Text="Evaluate it with symbolic parameters (i.e. don't assign values to `mu`, `sigma`, and `q`); you should get an expression involving the inverse error function." CreationDate="2010-10-09T14:24:37.310" UserId="830" />
  <row Id="4608" PostId="3425" Score="2" Text="Hi Tal:&gt; Given that these seems to be typo-free scientific names, i would try the Levenshtein metric first (in the context of a 92-by-55 distance matrix) and see how it comes out. " CreationDate="2010-10-09T14:31:27.313" UserId="603" />
  <row Id="4609" PostId="3439" Score="1" Text="@Tal While we are in, in R Wald-adjusted CIs (e.g. Agresti-Coull) are available in the `PropCIs` package. Wilson's method is the default in `Hmisc::binconf` (as suggested by Agresti and Coull)." CreationDate="2010-10-09T14:36:02.540" UserId="930" />
  <row Id="4610" PostId="3424" Score="1" Text="I agree with @ars -- log the data first, then plot and adjust the axis labels." CreationDate="2010-10-09T15:41:05.537" UserId="88" />
  <row Id="4611" PostId="3444" Text="@ars Thanks for adding a real illustration with ppf." CreationDate="2010-10-09T16:22:52.177" UserId="930" />
  <row Id="4612" PostId="2371" Text="Looks interesting, thanks.  I have similar problem to the OP trying penalized + coxph on a big data set, wonder if this helps." CreationDate="2010-10-09T16:48:11.853" UserId="251" />
  <row Id="4613" PostId="3445" Score="3" Text="Could you clarify whether the Ms Hudson to whom you refer is Kate Hudson or Jennifer Hudson??" CreationDate="2010-10-09T17:16:13.840" UserId="449" />
  <row Id="4614" PostId="3445" Text="@onestop This is a software for population genetics, http://j.mp/a75Z4Y" CreationDate="2010-10-09T18:07:00.527" UserId="930" />
  <row Id="4615" PostId="3447" Text="@ars Ouaouh, this is again an JIT answer :) " CreationDate="2010-10-09T18:15:32.050" UserId="930" />
  <row Id="4616" PostId="3443" Text="@Charlie Thanks for adding the original reference." CreationDate="2010-10-09T18:24:43.120" UserId="930" />
  <row Id="4617" PostId="3448" Text="@chl: I just noticed, funny. :)" CreationDate="2010-10-09T18:24:49.097" UserId="251" />
  <row Id="4618" PostId="3450" Text="@Cyrus Good that you cite the whole book from Pearl." CreationDate="2010-10-09T19:26:33.330" UserId="930" />
  <row Id="4619" PostId="1957" Text="@whuber I like this &quot;original&quot; approach (wrt. other entries). Still, I always find difficult to explain why bootstrap works in practice..." CreationDate="2010-10-09T19:38:21.807" UserId="930" />
  <row Id="4620" PostId="3412" Text="restated things as per your comment" CreationDate="2010-10-09T19:56:16.933" UserId="601" />
  <row Id="4621" PostId="3427" Score="2" Text="Levenshtein edit distance is implemented as part of the base package via agrep()" CreationDate="2010-10-09T20:05:57.667" UserId="776" />
  <row Id="4622" PostId="3427" Text="@Brandon:&gt; thanks, i couldn't find it...." CreationDate="2010-10-09T20:11:02.890" UserId="603" />
  <row Id="4623" PostId="3453" Text="@ccgillespie:&gt; i think my question may have been poorly worded. In the package i see (GPUtools, magma) double precision seems to be used as standard (with the loss of performance you describe). I was wondering why single precision is not offered as an option. " CreationDate="2010-10-09T20:46:34.330" UserId="603" />
  <row Id="4624" PostId="3453" Text="@kwak: The double precision values must be converted to single precision by the wrapper. The wrapper was just trying to be helpful. " CreationDate="2010-10-09T20:51:10.617" UserId="8" />
  <row Id="4625" PostId="3453" Text="@ccgillespie:&gt; yes, but it seems the wrapper comes with  performance costs exceeding the factor 2 you cite (again, correct me if i'm wrong on this) and in some cases no tangible benefits (i can think of many application in stat were SP FP arithmetics would be okay). I was wondering whether it makes sense to ask for an option to switch off said wrapper." CreationDate="2010-10-09T20:54:58.997" UserId="603" />
  <row Id="4626" PostId="3453" Score="2" Text="@kwak: Glancing at the GPUtools help file, it seems that `useSingle=TRUE` seems to be the default in the functions.  Am I missing something here?" CreationDate="2010-10-09T20:56:27.733" UserId="251" />
  <row Id="4627" PostId="3453" Text="@csgillespie: Remember, until relatively recently most nvidia cards simply **couldn't** do double precision computation. The factor of 2 hit is what I observed using raw C/CUDA code. Having a python/R wrapper may make this worst." CreationDate="2010-10-09T21:08:08.437" UserId="8" />
  <row Id="4628" PostId="3453" Text="@ars:&gt; argh. I didn't have a NVdia cart at home, so i didn't bother to read the manual. The white papers (and the GPU+R site) are all about double precision arithmetic. Would you be so kind to post you comment as a response so i can close the question ?" CreationDate="2010-10-09T22:36:36.610" UserId="603" />
  <row Id="4629" PostId="3381" Score="2" Text="@Denis. If you use a kernel with a finite support (e.g., the quadratic Epanechnikov kernel), then only nearby points contribute anything to a kde. Even with a Gaussian kernel, the contribution of points more than 3 bandwidths from x is negligible. I would definitely use kde rather than your proposed approach." CreationDate="2010-10-09T22:39:23.083" UserId="159" />
  <row Id="4630" PostId="3453" Text="@kwak: ah, got it.  I posted a separate answer." CreationDate="2010-10-09T22:57:15.440" UserId="251" />
  <row Id="4631" PostId="3428" Text="Nice paper and suggestion -- thanks." CreationDate="2010-10-10T00:11:19.950" UserId="251" />
  <row Id="4632" PostId="3453" Text="As noted above, the Fermi cards support double precision.  The GTX 2xx series also support double precision -- but the Fermi cards GTX 4xx have approximated twice the double precision performance of the 2xx series.  __However__, the Tesla 20xx product is four times faster still.  Here is a [discussion](http://www.vizworld.com/2010/04/geforce-gtx-480-18supthsup-double-precision-performance/) about the crippled double perf. of GTX 480.  [Specs for Tesla 20xx](http://www.nvidia.com/object/product_tesla_C2050_C2070_us.html) (515 GFlops DP Perf) " CreationDate="2010-10-10T00:12:41.527" UserId="1499" />
  <row Id="4633" PostId="3453" Text="[Specs for GTX 480](http://www.nvidia.com/object/product_geforce_gtx_480_us.html) -- 168 GFlops Peak DP Perf  [Specs for GTX 285](http://www.nvidia.com/object/product_geforce_gtx_285_us.html) -- 88.5 GFlops Peak DP Perf.  The performance numbers are not listed on the nvidia website for the GTX products, I found them [here.](http://techreport.com/articles.x/18682)" CreationDate="2010-10-10T00:18:10.857" UserId="1499" />
  <row Id="4634" PostId="3439" Text="Thanks for the comments chl " CreationDate="2010-10-10T04:11:21.127" UserId="253" />
  <row Id="4635" PostId="3454" Text="@kwak: I find the answer above helpful, but it really doesn't answer the question posed - &quot;is single precision so bad?&quot; Perhaps you should reword your question?" CreationDate="2010-10-10T07:57:45.337" UserId="8" />
  <row Id="4636" PostId="3433" Text="Hi Brandon - your solution worked great - thank you." CreationDate="2010-10-10T08:23:07.410" UserId="253" />
  <row Id="4637" PostId="3435" Text="Thank you Andy - I will have a look at it in the future." CreationDate="2010-10-10T08:23:27.813" UserId="253" />
  <row Id="4638" PostId="3432" Text="Thank you Whuber - I never heard of it." CreationDate="2010-10-10T08:23:43.927" UserId="253" />
  <row Id="4639" PostId="3455" Text="Can you elaborate a bit? It seems some iterative algorithm (matrix invert, QR decomposition) seem to work well. I'm also curious as to whether the inaccuracy of SP becomes more of a problem for operations involving larger arrays." CreationDate="2010-10-10T08:23:43.977" UserId="603" />
  <row Id="4640" PostId="3427" Text="Great answer Kwak - I will have a look at it in the future!" CreationDate="2010-10-10T08:24:00.807" UserId="253" />
  <row Id="4641" PostId="3427" Text="Personally, I feel that this is a more complete answer to Tal's question. +1 for pointing our RecordLinkage - I'll definitely have to try that out." CreationDate="2010-10-10T08:40:15.807" UserId="776" />
  <row Id="4642" PostId="3433" Text="+1 for the link to the previous question on the subject in S.E. (thaks for the pointer to agrep())." CreationDate="2010-10-10T08:50:58.123" UserId="603" />
  <row Id="4643" PostId="3296" Text="@kwak the variables are continuous (fixed the question)" CreationDate="2010-10-10T09:43:57.340" UserId="1496" />
  <row Id="4645" PostId="3463" Score="2" Text="Are the time series both stationnary ? www.econ.ohio-state.edu/dejong/note1.pdf" CreationDate="2010-10-10T11:54:40.560" UserId="603" />
  <row Id="4646" PostId="3461" Text="+1 for kernlab, really makes life easier." CreationDate="2010-10-10T13:20:27.330" UserId="88" />
  <row Id="4647" PostId="3458" Score="2" Text="Don't bother with neural networks, this is an obsolete technology." CreationDate="2010-10-10T13:22:11.813" UserId="88" />
  <row Id="4648" PostId="3466" Text="When you say &quot;condition&quot;, do you mean group assignment?" CreationDate="2010-10-10T13:49:45.367" UserId="561" />
  <row Id="4649" PostId="3464" Score="4" Text="I would have given almost the same answer but here it is and well stated, too.  Let me add just two things based on experience.  First, I have found it's almost always worthwhile re-running any analysis I possibly can: it serves to check my understanding and, more often than one might expect, it exposes errors in the paper itself.  Second, it's essential to locate the key references and to find references of your own by searching the Web for phrases in the paper.  A substantial number of contributions recently are (auto-)plagiarisms or bald attempts to get another paper out of old work." CreationDate="2010-10-10T15:28:07.633" UserId="919" />
  <row Id="4651" PostId="3446" Score="2" Text="I confess to being mystified by this, despite struggling several times to make sense of it: is there a question here?  &quot;So bad&quot; is vague and has no referent.  What exactly are you seeking to understand or find out?" CreationDate="2010-10-10T16:19:55.700" UserId="919" />
  <row Id="4652" PostId="3462" Text="@kwak (+1) Nice concrete recommendation! I was about to suggest the use of the `mvoutlier` package (which relies on mahalanobis distance and displays outlying obs. in an interactive display)." CreationDate="2010-10-10T16:27:41.813" UserId="930" />
  <row Id="4655" PostId="3465" Score="2" Text="kernlab also uses libsvm for the optimization, so there isn't a big difference in that sense (although it is much more flexible)." CreationDate="2010-10-10T16:44:26.677" UserId="5" />
  <row Id="4662" PostId="3461" Text="@Shane @Tal Here is a fair (or I think so) review of SVM *vs.* RFs: A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification, http://j.mp/ab7U8V. I also prefer `kernlab` to `e1071`." CreationDate="2010-10-10T19:25:28.780" UserId="930" />
  <row Id="4663" PostId="3307" Score="1" Text="@Edward Can you elaborate a little bit about the purpose/context of this particular analysis, or why do you seek an alternative measure of association?" CreationDate="2010-10-10T19:33:06.003" UserId="930" />
  <row Id="4665" PostId="3461" Text="@chl: Thanks for the reference." CreationDate="2010-10-10T21:00:12.353" UserId="5" />
  <row Id="4666" PostId="3459" Text="+1 Great answer.  I also agree with the caret recommendation." CreationDate="2010-10-10T21:00:48.300" UserId="5" />
  <row Id="4670" PostId="3455" Text="There are two parts to it: 1) What does the data represent? 2) How do you process the data?  If you're looking at thousands of points of data from a medical study, single precision would likely be plenty for quantifying patient wellness, and I doubt you would ever need double.  Geometry, on the other hand, could require either single or double precision depending on your scale &amp; zoom.  Calculating the trajectory of a probe to Saturn would always require doubles, as even small errors could drastically effect the result.  You need to look at the data and decide what your tolerances are." CreationDate="2010-10-10T22:55:50.203" UserId="1539" />
  <row Id="4671" PostId="3473" Text="This site is for asking questions about data analysis. You haven't asked a question so I'm closing this. If you can ask something specific instead, I'm sure someone will be able to help." CreationDate="2010-10-10T23:02:44.087" UserId="159" />
  <row Id="4672" PostId="3466" Text="@propofol: yes. apologies if my language is not clear." CreationDate="2010-10-11T00:47:50.343" UserId="183" />
  <row Id="4677" PostId="3464" Text="I've added an additional question. If it's not too much hassle, would you update your answer?" CreationDate="2010-10-11T08:29:16.833" UserId="8" />
  <row Id="4678" PostId="3478" Text="I've added an additional question. If it's not too much hassle, would you update your answer?" CreationDate="2010-10-11T08:29:52.717" UserId="8" />
  <row Id="4679" PostId="3455" Text="It will depend on the numerical stability of the algorithm you are using and how well-conditioned the problem is. Remember that double precision gives you access to smaller numbers as well as larger ones." CreationDate="2010-10-11T11:18:17.260" UserId="229" />
  <row Id="4680" PostId="3462" Text="@chl:&gt; actually, i wonder whether your solution would not ultimately be better. I think you should post it, either for the benefit of bgbg or that a future readers (though it is mentioned in the paper i cited). The main point, in my opinion, is to outline the fact that in this setting (binary dependent, continuous independent) it is quiet satisfactory to look for the outliers on the design space." CreationDate="2010-10-11T11:43:49.863" UserId="603" />
  <row Id="4681" PostId="3462" Text="@kwak I would be equally happy with this second option added to your own response." CreationDate="2010-10-11T12:01:08.807" UserId="930" />
  <row Id="4682" PostId="3448" Text="Thanks for your detailed answer!  Actually, I do not know how to convert my binary data into genotype data? Do you have any idea about that?" CreationDate="2010-10-11T12:05:59.327" UserId="1534" />
  <row Id="4683" PostId="3448" Text="@jacki Well, actually it is not very clear to me what's the repeated lines of binary value mean: Does it stand for an haplotype block (1=mutant, 0=ancestral allele) on different chromosomes, simulated for different individuals (each new line starting with //), because in this case the haplotype are of varying size... Is it possible to compute haplotype frequencies easily from your software? Maybe I need to reread some papers and come back to you, but if you have any further information (how do we read the output in terms of haplotype, allele marker, etc.), pls update your original question." CreationDate="2010-10-11T12:28:49.873" UserId="930" />
  <row Id="4685" PostId="3454" Text="@csgellespie: you are totally correct. I will reword this question so as it can be used by future readers. Indeed, the wording was particularly poor." CreationDate="2010-10-11T14:01:49.117" UserId="603" />
  <row Id="4686" PostId="3446" Text="@Whuber:&gt; My question was poorly worded. It probably was due to it being borne out of ignorance: i had read some white papers on use of GPU , (although, unfortunately it turns out, not the R command reference of GPUtools) and could not understand why all the tests were carried out in DP. I will re-phrase the question (and the title)." CreationDate="2010-10-11T14:02:30.793" UserId="603" />
  <row Id="4689" PostId="3474" Score="1" Text="@kwak I'm not sure why changing the title was necessary...the &quot;homework&quot; tag should be sufficient." CreationDate="2010-10-11T14:10:03.100" UserId="5" />
  <row Id="4690" PostId="3474" Text="@Shane:&gt; ok changed back to the original. " CreationDate="2010-10-11T15:18:48.490" UserId="603" />
  <row Id="4691" PostId="3464" Score="1" Text="@csgillespie: I guess I'm too early in my career to answer that, as I probably do not get asked to review as many papers as someone with more experience then me. I think @whuber answer makes a lot of sense though." CreationDate="2010-10-11T16:03:25.597" UserId="582" />
  <row Id="4693" PostId="3381" Text="@Rob, thanks; I'll try to clarify the question, and split off &quot;adaptive histograms&quot; as a separate question." CreationDate="2010-10-11T16:40:54.847" UserId="557" />
  <row Id="4694" PostId="3448" Text="@chl:  I am trying to do programming in R for haplotype frequencies, for number of segregating sites, etc. &#xA;          SNPs&#xA;Ind1   0 0 0 0 1&#xA;Ind2   1 0 1 0 0&#xA;Ind3   0 1 0 1 0&#xA;Ind4   1 1 1 0 0&#xA;&#xA;In above data, each column-wise is SNP and row-wise is individual(Ind1, Ind2,...). As I said before it was simulated on whole genome. The position is normalize random variable from uniform distribution [0,1]. &#xA;Thanks for your help!&#xA; " CreationDate="2010-10-11T17:28:14.557" UserId="1534" />
  <row Id="4695" PostId="3448" Text="@jacki If I read it correctly, 00001 means 5 markers/alleles with one variant, and you have $n$ such vectors with varying locations for SNP (it was not clear `ms` manual)?" CreationDate="2010-10-11T17:36:18.367" UserId="930" />
  <row Id="4696" PostId="3487" Text="I've heard about World Programming System (WPS), http://j.mp/9GjesM. Don't know how comparable it is, though. There was an old GNU project, called Dap (http://j.mp/955DqP), but it never reaches a mature state. In biomedical research, SAS is still considered as THE reference software, although FDA is on her way to (progressively) accept R-based analyses for clinical trials. " CreationDate="2010-10-11T17:57:20.487" UserId="930" />
  <row Id="4697" PostId="3487" Text="Interesting.  I've never heard of WPS.  I'll check it out - it mentions the ability to read SAS datasets..." CreationDate="2010-10-11T18:01:22.710" UserId="1499" />
  <row Id="4698" PostId="3491" Text="Thanks for your answer. Yes, the distribution does represent the scores for the population that should be rejection by my biometric system but are instead authenticated. However, I'm not sure how to choose my threshold in this case, so I need to determine that before I can proceed. " CreationDate="2010-10-11T18:04:16.200" UserId="1224" />
  <row Id="4699" PostId="3483" Text="@whuber excluding bounty and +15 rep for accepted answer :)" CreationDate="2010-10-11T18:05:25.543" UserId="930" />
  <row Id="4700" PostId="3491" Score="2" Text="I'd bet that the threshold would be a design parameter set by the client.  Basically it will come down to which is more important for a specific client Type I or Type II error (probably Type II - Falsely accepting someone who should've been denied access).  With a knowledge of what the client wants, you could then state that under a specific threshold, the probability of falsely admitting someone who should be denied access is 1 / 10,000,000 or something." CreationDate="2010-10-11T18:09:49.417" UserId="1499" />
  <row Id="4701" PostId="3490" Text="Is getting certified actually gold?  I've heard lots of mixed reports about the impact of technical certifications in general, but I don't know how that applies to SAS.  Can you expand on that a little bit?" CreationDate="2010-10-11T18:09:59.077" UserId="71" />
  <row Id="4702" PostId="3478" Score="1" Text="@Stephan Interestingly, most researchers in genetics studies are encouraged or pleased (it depends on the review) to make data available. I also remind of @csgillepsie nice answer about *reproducible research*, http://stats.stackexchange.com/questions/1980/complete-substantive-examples-of-reproducible-research-using-r/1993#1993" CreationDate="2010-10-11T18:10:25.740" UserId="930" />
  <row Id="4703" PostId="3491" Text="I could however envision a client who must employ a certain biometric authentication device due to government law or otherwise -- and said individuals might hate when their scanner goes wonky and won't give anyone access -- hence, they might care more about Type I error - not admitting someone who __should be allowed__ (because they probably have six other security measures and this one is only a deterrent for *wandering eyes...*" CreationDate="2010-10-11T18:12:16.453" UserId="1499" />
  <row Id="4704" PostId="3490" Score="1" Text="I've also heard mixed reports -- I taught the SAS class at PSU for 4 semesters, had a bunch of student report that companies were happy that they'd simply had exposure to SAS.  Several students mentioned being drilled on *which* Procs they knew during interviews, but mostly the feeling I got was that employer want a familiarity with SAS and accept that you'll need to learn specific procedures or methods after you've been hired.  Within the programming community, employer will sometimes view Certs as a red flag -- a waste of time, or a book knowledge over practical/experience-oriented info." CreationDate="2010-10-11T18:20:29.867" UserId="1499" />
  <row Id="4705" PostId="3490" Score="1" Text="Here's a link to a certification discussion over on [Not Programming Related](http://programmers.stackexchange.com/questions/44/are-certifications-worth-it) -- also a Stack Exchange website." CreationDate="2010-10-11T18:21:03.340" UserId="1499" />
  <row Id="4706" PostId="3491" Text="So, does the threshold need to be the minimum score that needs to be obtained in order to be authenticated?" CreationDate="2010-10-11T18:53:02.207" UserId="1224" />
  <row Id="4707" PostId="3491" Text="I think so, but this is highly dependent on your implementation." CreationDate="2010-10-11T18:56:44.357" UserId="1499" />
  <row Id="4709" PostId="3483" Score="1" Text="@chl of all the people here you should be the least concerned about voting enough!" CreationDate="2010-10-11T20:06:08.107" UserId="919" />
  <row Id="4710" PostId="1151" Text="@ars I upvoted a long ago but re-reading your answer it remind me that I always liked *Plane Answers to Complex Questions* from Christensen (http://j.mp/atRp9w)." CreationDate="2010-10-11T21:14:48.740" UserId="930" />
  <row Id="4711" PostId="3491" Score="1" Text="@rohanbk Did you look at Signal Detection Theory or ROC curve analysis (http://j.mp/b49wDl in French, http://j.mp/aTjobH in English)? Determining the cut-off is exactly finding the best compromise between sensitivity and 1-specificity, as M. Tibbits said." CreationDate="2010-10-11T21:23:54.363" UserId="930" />
  <row Id="4712" PostId="3455" Text="Not necessarily smaller or larger numbers; remember, we're dealing with floating point.  Rather, it lets you use larger and smaller numbers in relation to each other, while preserving the significant digits." CreationDate="2010-10-11T21:54:47.987" UserId="1539" />
  <row Id="4713" PostId="3491" Text="Excellent link @chl.  My brain balked on that one." CreationDate="2010-10-11T21:57:56.630" UserId="1499" />
  <row Id="4714" PostId="3481" Text="Ahh, it's sort of coming back to me from multivariable calculus. I remember doing problems like that. How do I find the radius as a function of the remaining variable? It still seems like I'm going to have some sort of monster integral left over." CreationDate="2010-10-11T22:13:13.907" UserId="1545" />
  <row Id="4715" PostId="3494" Text="Hey Shankar, try the `hopach package` in Bioconductor. It gives you an estimate of the number of clusters in your group. Also, you should try `pvclust package` in R/BioC, it gives you the &quot;statistically significant&quot; clusters." CreationDate="2010-10-11T22:20:25.033" UserId="1307" />
  <row Id="4716" PostId="3481" Score="2" Text="Let the remaining variable be y.  Then x^2 &lt;= 1-y^2 describes the region over which you have to integrate.  Evidently the radius equals Sqrt(1 - y^2), whence the cross-sectional area equals \pi * (1 - y^2)/2.  That's a pretty simple formula :-).  (Remember, the theme here is geometry, not calculus...)" CreationDate="2010-10-11T22:29:45.320" UserId="919" />
  <row Id="4717" PostId="3481" Text="Oh, right. That crossed my mind, but it seemed too simple. I guess I was determined for it to be complicated. Thanks!" CreationDate="2010-10-11T22:40:12.100" UserId="1545" />
  <row Id="4718" PostId="3481" Text="I forgot to ask: how does c figure into this?" CreationDate="2010-10-11T22:49:48.457" UserId="1545" />
  <row Id="4719" PostId="3448" Text="@chl: Now it is clear to you? How we can covert this binary data into genotype data?" CreationDate="2010-10-12T00:42:20.273" UserId="1534" />
  <row Id="4720" PostId="1151" Text="@chl: cool, definitely going to check it out then.  :)" CreationDate="2010-10-12T02:30:51.990" UserId="251" />
  <row Id="4721" PostId="3483" Score="1" Text="@chl: you set a high bar in every way! :) Maybe our first polystats project should be to set up some scripts to maintain and update a set of charts like these: http://meta.stats.stackexchange.com/questions/314/vote-early-vote-often/317#317  " CreationDate="2010-10-12T02:44:02.130" UserId="251" />
  <row Id="4722" PostId="3490" Text="I suppose my comment was stated with little backing evidence, and in retrospect, cannot be backed fully. I have seen a few jobs in the San Francisco Bay Area that, in the qualifications section, ask that the individual be familiar with using a data analytics package, and add that SAS Certification is recommended or highly recommended. It's very likely they are instead just looking for some way for you to show you are comfortable with SAS and not just putting it on your resume because it is &quot;one of the most common requirement is experience of SAS&quot;." CreationDate="2010-10-12T05:52:42.927" UserId="1118" />
  <row Id="4723" PostId="3506" Text="@chl I am sorry chl, I think we were almost answering this question together at the same time. Basically, you and me are pointing to the same fact." CreationDate="2010-10-12T07:52:46.110" UserId="1307" />
  <row Id="4724" PostId="3505" Text="Hi @suncoolsu, there are n=6 replicates in each of the 12 treatments. It was set up as an RCBD with three blocks with n=2 reps/treatment/block because there were three trays with for 24 samples each, but there was no block effect so I dropped it from my analysis. Sorry for the omission" CreationDate="2010-10-12T08:21:33.420" UserId="1381" />
  <row Id="4725" PostId="3506" Text="@chi The effect of A was expected, while the effects of B and A*B were both main hypotheses. And I am going to show the effect sizes in a figure. Thank you for reminding me not to make the error of accepting Ho." CreationDate="2010-10-12T08:32:55.220" UserId="1381" />
  <row Id="4726" PostId="3505" Text="Hello @David, I will be a little uncomfortable with saying - &quot;there was no block effect&quot;. Please see my updated reply for your answer." CreationDate="2010-10-12T08:43:29.517" UserId="1307" />
  <row Id="4727" PostId="3481" Text="In my opinion, Whuber's answer deserves to be upvoted for two reasons. First it answers the question asked, second as a model for how we could in the future handle (explicitly stated) homework questions: this type of answers actually contributes to the learning process and could be a better policy with respect to homework question than that adopted at MO/SO." CreationDate="2010-10-12T08:50:32.957" UserId="603" />
  <row Id="4728" PostId="3478" Text="@chl: yes, making data available very much depends on the discipline, and I would love to see more of this in &quot;mainstream&quot; psychology - I just can't recall having seen a single instance of a psych paper that actually did give out the data. " CreationDate="2010-10-12T09:42:04.303" UserId="1352" />
  <row Id="4729" PostId="3504" Score="4" Text="In papers, when interaction effects aren't significant I still mention in passing that I looked for them to stop reviewers asking about interaction effects. This is obviously different from &quot;accepting H_0&quot; and making a big thing about no effect." CreationDate="2010-10-12T10:56:00.077" UserId="8" />
  <row Id="4730" PostId="3513" Text="@kwak @wok Of note, the scikit.learn package also offers efficient implementation in Python for this kind of stuff." CreationDate="2010-10-12T12:03:50.520" UserId="930" />
  <row Id="4731" PostId="3461" Score="1" Text="@chl I don't like this paper while it made from the SVM learning perspective -- making one repetition of a stochastic algorithm (RF) is just a junk; also appendix 2 shows how bad it may be to apply SVM workflow to RF. Yet I agree that almost always SVM  can be tuned to outperform RF because of the kernel trick (which plain RF does not have, while it doesn't mean it can't have it in general), but with exponentially growing optimization effort." CreationDate="2010-10-12T12:13:13.917" UserId="88" />
  <row Id="4732" PostId="3510" Score="1" Text="+1 Very good point." CreationDate="2010-10-12T12:18:14.110" UserId="88" />
  <row Id="4733" PostId="3515" Text="Ok, let me correct my question. How do you know if a binary variable follows the binomial distribution?" CreationDate="2010-10-12T12:42:15.150" />
  <row Id="4734" PostId="3481" Text="@Jarrod: you can use c to determine the normalization constant for the marginal density.  Alternatively, forget c, derive the functional form of the marginal (that is, up to a constant factor), then integrate the marginal to find its normalizing constant." CreationDate="2010-10-12T12:45:48.243" UserId="919" />
  <row Id="4735" PostId="3515" Text="See http://en.wikipedia.org/wiki/Binomial_test . Talking as a moderator now: so can you please make a new, good question? You'll get a better answer and the wiki spirit of this site will be held." CreationDate="2010-10-12T12:49:40.767" UserId="88" />
  <row Id="4736" PostId="3461" Text="@mbq Indeed, this a good point." CreationDate="2010-10-12T13:03:05.217" UserId="930" />
  <row Id="4737" PostId="3481" Text="@Kwak: Thanks for pointing out there is a policy concerning homework questions!  I continued this discussion on meta at http://meta.stats.stackexchange.com/q/12/919 ." CreationDate="2010-10-12T13:12:55.210" UserId="919" />
  <row Id="4738" PostId="3515" Text="Sure. Let me do that." CreationDate="2010-10-12T13:14:50.837" />
  <row Id="4739" PostId="3514" Score="2" Text="Since you have made a new question I think you should accept mbq's answer and link to your new question." CreationDate="2010-10-12T13:19:58.520" UserId="1036" />
  <row Id="4740" PostId="3517" Score="1" Text="@whuber (+1) The second point is very interesting." CreationDate="2010-10-12T13:38:58.233" UserId="930" />
  <row Id="4741" PostId="2739" Text="@Henrik Yes, as a base of rule-based classifiers." CreationDate="2010-10-12T13:41:15.413" UserId="88" />
  <row Id="4742" PostId="3514" Text="vote to close.  (but I'm below 500r)" CreationDate="2010-10-12T13:47:57.743" UserId="1499" />
  <row Id="4743" PostId="3508" Text="+1 Wooooooowwww  (It had to be 15 characters long...)" CreationDate="2010-10-12T13:52:21.743" UserId="1499" />
  <row Id="4744" PostId="3496" Text="I suspect that the answer is domain specific (i.e., specific to spam filters). If you can calculate the components P(A|B) etc then you should be able to calculate the simpler P(A) as you stated. Or, perhaps the answer is related to pedagogy so that readers understand the relationship between P(A) and its decomposition in terms of P(A|B), P(B) etc." CreationDate="2010-10-12T14:06:35.957" />
  <row Id="4745" PostId="3519" Score="2" Text="R is based on Scheme but is pretty good in pretending to the beginners that it is rather C based." CreationDate="2010-10-12T14:31:24.267" UserId="88" />
  <row Id="4746" PostId="3519" Score="2" Text="@Roo:&gt; I will not post this as an answer because there are many people more knowledgeable in these things than me. However, i  think that concerning speed you have to distinguish between R base and the mix good programmer/great packages. A good programmer can leverage some the tools in cran such as multicore, GPUtools&amp;magma (soon but not yet usefull for MCMC), Rcpp,... to make a pretty fast code. I don't think matlab central has anything comparable to offer." CreationDate="2010-10-12T14:56:49.033" UserId="603" />
  <row Id="4747" PostId="3519" Text="@kwak -- I probably should've posted my answer as a comment too.  Sorry about that." CreationDate="2010-10-12T15:02:41.683" UserId="1499" />
  <row Id="4748" PostId="3521" Text="+1: very practical answer with good explanation." CreationDate="2010-10-12T15:06:28.033" UserId="603" />
  <row Id="4749" PostId="3505" Text="Hi @suncoolsu, Thank you again for your response. re: 1. This is an interesting point, I will look at Venable's reference. re: 2. The blocking and replication within blocks was based on the fact that the trays held 24 samples, so I blocked them in case there was any effect of tray positon or time of measurement. re: 3. Thanks for again for reminding me not to incorrectly interpret my findings, e.g. by saying that there was no block effect. It is a bad habit. re: 5. I should have clarified that by 12 treatments, I meant the 12 permutations of three levels of A and four of B. " CreationDate="2010-10-12T15:28:00.363" UserId="1381" />
  <row Id="4750" PostId="3519" Score="2" Text="MT, that's crazy - your answer is great.  If anything, kwak should make his a proper answer, too.  Let those more knowledgeable vote or answer as they see fit." CreationDate="2010-10-12T15:33:50.570" UserId="71" />
  <row Id="4751" PostId="3522" Text="@Roger You still have the option to choose a non-euclidean metric for your (dis)similarity matrix (e.g. Jaccard, Mahanattan, etc). BTW, this should be a new question, not a follow-up to your initial question in the same thread." CreationDate="2010-10-12T15:46:01.477" UserId="930" />
  <row Id="4752" PostId="3523" Score="2" Text="(+1) The exchangeability assumption is also at the heart of permutation tests." CreationDate="2010-10-12T15:53:46.153" UserId="930" />
  <row Id="4753" PostId="3522" Score="2" Text="@Roger, chl's right.  If you want to formulate this as a new question, please provide some additional information about the other variables you will use and the purpose of the analysis (because half the battle is figuring out the right question to ask!)." CreationDate="2010-10-12T15:55:58.717" UserId="919" />
  <row Id="4754" PostId="3517" Score="1" Text="@chl: Yes, it occurred to me as I was writing about the CLT that there could be problems with the variances converging to zero, for then the Lindeberg condition could be violated.  It's easy to see why the resulting limit distribution might be non-normal, for if (for example) the chance of observing a 1 approaches zero sufficiently rapidly, then the distribution of the mean might remain strongly skewed and never get close to normal." CreationDate="2010-10-12T16:00:11.377" UserId="919" />
  <row Id="4755" PostId="3521" Text="+1 for C++; while it is quite easy to embed C/C++ to R I often wrap my codes and run them inside R -- then it is nicer to pass parameters, do live visualisation and obviously analyse results without thinking of output file format." CreationDate="2010-10-12T16:42:55.483" UserId="88" />
  <row Id="4756" PostId="3521" Text="well put; MC will eventually require one to move to C/C++. I do not have enough experience in R to comment, but have had a number of headaches using C/C++ with Matlab due to different versions of shared object libraries (under Linux) being pulled in by the Matlab executable than what I want to link with my code.  " CreationDate="2010-10-12T16:54:29.177" UserId="795" />
  <row Id="4757" PostId="3525" Text="I completely agree @Henrik.  If you're concerned about profiling, Matlab has an **excellent** profiling tool (even back in version 7.1!!).  Rprof on the otherhand leaves a lot to be desired.  I end up profiling by executing each command several times in a for loop and comparing the `system.time` difference amongst different versions.  [Here is an interesting case study](http://jeromyanglim.blogspot.com/2010/02/case-study-in-optimising-code-in-r.html)" CreationDate="2010-10-12T16:57:24.667" UserId="1499" />
  <row Id="4758" PostId="3463" Text="@kwak: No, the series are both NOT stationary." CreationDate="2010-10-12T17:49:19.967" UserId="1216" />
  <row Id="4760" PostId="3503" Text="Hello Gaten, Thank you for looking into this. I am still not sure that I understand the logic. The full sample beta.hat is and estimate of the true beta. My sample.mean.beta.hat is an estimate of beta.hat is it not? Is the CLT argument that both beta.hat and sample.mean.beta.hat converge towards each other?" CreationDate="2010-10-12T18:24:59.633" />
  <row Id="4761" PostId="3507" Text="Hello suncoolsu. Yes, I think that the exchangeability assumption is crucial. Thank you for pointing that out. Do you know of any results on rates of convergence?" CreationDate="2010-10-12T18:29:47.580" />
  <row Id="4762" PostId="304" Score="1" Text="A test of normality is usually too severe.  Often it suffices to obtain symmetrically distributed residuals.  (In practice, residuals tend to have strongly peaked distributions, partly as an artifact of estimation I suspect, and therefore will test out as &quot;significantly&quot; non-normal no matter how one re-expresses the data.)" CreationDate="2010-10-12T19:02:17.600" UserId="919" />
  <row Id="4763" PostId="301" Score="1" Text="@cgillespie: Concentrations, yes; but age?  That is strange." CreationDate="2010-10-12T19:02:52.670" UserId="919" />
  <row Id="4764" PostId="3526" Score="1" Text="Perhaps [this discussion](http://groups.google.co.bw/group/sci.stat.edu/browse_thread/thread/c247a85a83c5ae00) is relevant as -- it isn't often used because it is very conservative (much like [Scheffe's Method](http://en.wikipedia.org/wiki/Scheff%C3%A9%27s_method))?" CreationDate="2010-10-12T19:06:18.953" UserId="1499" />
  <row Id="4765" PostId="3530" Score="1" Text="@whuber +1 Great answer/summation.  You're on a roll today." CreationDate="2010-10-12T19:23:57.020" UserId="5" />
  <row Id="4766" PostId="3495" Text="This does not give the exact answer, for the following reason: $\sqrt{\sum_i x_i} \ne \sum_i \sqrt{x_i}$. You are computing the former, while the OP wants the latter." CreationDate="2010-10-12T19:45:10.130" UserId="795" />
  <row Id="4767" PostId="3512" Text="Thanks. I am reading this and thinking about it. However, this would be a huge modification of the current algorithm." CreationDate="2010-10-12T19:53:06.370" UserId="1351" />
  <row Id="4768" PostId="3513" Text="The coordinate descent algorithm is interesting. Thanks. Still thinking about it." CreationDate="2010-10-12T19:53:46.397" UserId="1351" />
  <row Id="4769" PostId="301" Text="@whuber: I suppose it's very data dependent, but the data sets I used, you would see a big difference between a 10 and 18 yr old, but a small difference between a 20 and 28 yr old. Even for young children the difference between a 0-1 yr old isn't the same as the difference between a 1-2." CreationDate="2010-10-12T20:00:38.490" UserId="8" />
  <row Id="4770" PostId="3507" Score="1" Text="Hello Joseph, as with most (_not in general_) results in classical statistics: parametric bootstrap converges around rates $n^{\frac{1}{2}}$, where$n$ is the sample size. In your case $n$ corresponds to $M$, as you are taking average of $M$ bootstrap estimates. This is equivalent to the asymptotic normality (or _CLT_). The assumptions and the details of the result can be found in: Hall, P. 1988. Rate of Convergence in Bootstrap Approximations. Annals of Probability." CreationDate="2010-10-12T20:02:07.850" UserId="1307" />
  <row Id="4771" PostId="3505" Text="Hello @David, I understand your point of blocking on trays to remove the variation because of them. This is perfectly justified and you should block if you can. But what I am trying to point out is: as you said, there are 2 replicates for a particular block and treatment level (say: tray 1 had two replicates of first level for trt A) IMHO is _not correct_. If you really want to block for positions as well, probably a more efficient design like **latin square** or **split plot** designs may be the way to go. The choice of specific design depends on your resources, your parameter of intrst. etc." CreationDate="2010-10-12T20:14:35.550" UserId="1307" />
  <row Id="4772" PostId="3481" Text="In case anyone was dying to know if there was a way to solve this with calculus, you can instead recognize that $x = \sqrt{1-y^2}$ is of the form $u = \sqrt{a-b^2}$, which has the special integral $\int udx = \frac{1}{2}(xu + a^2 arcsin \frac{x}{a})$, which gives you the same result as wheber's method when integrated from $-\sqrt{1-y^2}$ to $\sqrt{1-y^2}$. My professor discussed this in class today, but I prefer wheber's method for obvious reasons." CreationDate="2010-10-12T20:19:28.303" UserId="1545" />
  <row Id="4773" PostId="3507" Text="Addendum: when I say $n^{1/2}$, I mean error goes to zero with that rate $O(n^{-1/2})$." CreationDate="2010-10-12T20:32:46.480" UserId="1307" />
  <row Id="4774" PostId="551" Text="Very good summary Prof. Cookson! I like it a lot." CreationDate="2010-10-12T20:34:57.243" UserId="1307" />
  <row Id="4775" PostId="3523" Score="2" Text="Given the question of the when &amp; why of exchangeability, chl's pointer to permutation tests may merit a few additional words. Permutation tests are a nonparametric technique used when normality and similar assumptions are untenable - instead one uses the much weaker &quot;null assumption&quot; of exchangeability, approximates the distribution of a test statistic under this null assumption (by permuting) and looks whether the actually observed test statistic is extreme compared to this null distribution. There is an accessible book by P. Good, &quot;Permutation, Parametric, and Bootstrap Tests of Hypotheses&quot;." CreationDate="2010-10-12T20:43:34.077" UserId="1352" />
  <row Id="4776" PostId="3527" Score="3" Text="I like the part about &quot;My current working practice is to use R to prototype and use C when I need an extra boost of speed.&quot; - it sounds like the job description of my poor unworthy self and the C++ developers in the next office... and I think it really captures basically *any* situation involving R, C/C++ and an issue in statistical computation." CreationDate="2010-10-12T20:48:37.787" UserId="1352" />
  <row Id="4777" PostId="3523" Text="@Stephan I like this book! Still, exchangeability is weaker than independence..." CreationDate="2010-10-12T21:01:42.570" UserId="930" />
  <row Id="4778" PostId="3481" Text="@Jarrod: Thank you for the follow-up.  If the purpose of your course is to learn applications of calculus, then your professor's solution has pedagogical merit.  If it's to learn about probability, then seeing multiple solutions--including some relatively simple ones--can help you identify the key *probability* ideas and learn them better.  (The calculus stuff might then just be a distraction...)" CreationDate="2010-10-12T21:36:44.923" UserId="919" />
  <row Id="4779" PostId="301" Text="@csgillespie: Yes it will be data dependent: your ability to conduct an insightful and effective analysis is the ultimate arbiter of this issue, not my preconceptions.  I was just trying to envision situations where age as an *independent* variable would merit such a strong transformation.  Some strange things will happen with newborns, too ;-)." CreationDate="2010-10-12T21:40:22.480" UserId="919" />
  <row Id="4780" PostId="3507" Text="Hello suncoolsu. Thank you for the reference. I very much appreciate it. I'll do my homework." CreationDate="2010-10-12T21:47:05.337" />
  <row Id="4782" PostId="3526" Text="Surely you mean &quot;after rejecting the null&quot; not &quot;after failing to reject the null&quot;? And it seems there's only one L in 'Marascuilo' (NIST's error, not yours):&#xA;&#xA;Leonard A. Marascuilo. Large-sample multiple comparisons. Psychological Bulletin, 1966; 65(5): 280-290. http://dx.doi.org/10.1037/h0023189.&#xA;&#xA;" CreationDate="2010-10-12T21:58:37.800" UserId="449" />
  <row Id="4783" PostId="3526" Text="@onestop Thanks. I corrected the typo." CreationDate="2010-10-12T22:39:40.813" />
  <row Id="4784" PostId="3536" Text="@chl Thanks for the pointers to R packages/functions. I will take a look at them." CreationDate="2010-10-12T22:40:13.260" />
  <row Id="4785" PostId="3495" Text="I agree that the method is inexact.  However, I disagree with your diagnosis of the inexactness.  Welford's method for calculating variance, which does not even contain a sqrt, has a similar error.  However, as `n` gets large, the `error/n` gets vanishingly small, suprisingly quickly." CreationDate="2010-10-12T23:06:18.687" UserId="179" />
  <row Id="4786" PostId="3495" Text="Welford's method has no sqrt because it is computing the variance, not the standard deviation. By taking the sqrt, it seems like you are estimating the standard deviation, not the mean absolute deviation. am I missing something?" CreationDate="2010-10-12T23:33:53.820" UserId="795" />
  <row Id="4787" PostId="3529" Text="+1. Josh, that's a pretty complete reference sample -- thanks!" CreationDate="2010-10-13T00:06:21.587" UserId="251" />
  <row Id="4788" PostId="3541" Text="@Shankar, It is not prudent to relate causality and gene expression of microarray data. It is close to saying &quot;correlation $\Rightarrow$ causation&quot;" CreationDate="2010-10-13T00:19:06.383" UserId="1307" />
  <row Id="4790" PostId="3542" Text="+1 Great question; will be interested to see the answers." CreationDate="2010-10-13T02:00:42.150" UserId="5" />
  <row Id="4791" PostId="3542" Score="1" Text="@Jeromy, I don't know if Andrew Gelman will be happy with the question, but definitely for small tables this question needs to be addressed. +1" CreationDate="2010-10-13T02:04:15.647" UserId="1307" />
  <row Id="4792" PostId="3523" Text="Thanks to both of you, a very important component and right on the mark." CreationDate="2010-10-13T03:11:32.503" UserId="1108" />
  <row Id="4793" PostId="3495" Text="@shabbychef Each iteration of Welfords is calculating the contribution of the new datapoint to the absolute deviation, squared.  So I take the square root of each contribution squared to get back to the absolute deviance.  You might note, for example, that I take the square root of the delta before I add it to the deviance sum, rather than afterward as in the case of the standard deviation." CreationDate="2010-10-13T05:07:59.337" UserId="179" />
  <row Id="4794" PostId="3542" Score="2" Text="@suncoolsu I suppose any good resource on table design should talk about the pros and cons of tables versus graphics." CreationDate="2010-10-13T05:25:13.613" UserId="183" />
  <row Id="4795" PostId="3542" Score="1" Text="@Jeromy Just to point to the `apsrtable` R package which offers an alternative display of Tables, compared to `xtable`, and `reporttools` described in the JSS, http://j.mp/97GXWV" CreationDate="2010-10-13T06:17:35.793" UserId="930" />
  <row Id="4796" PostId="3537" Text="I think it would be excellent if you could give more detail (see suncoolsu's answer). Furthermore, with design, do you mean how to analyze the data? " CreationDate="2010-10-13T08:59:28.513" UserId="442" />
  <row Id="4797" PostId="3521" Text="@m-tibbits: How do you generate RN's when using openMP?" CreationDate="2010-10-13T09:05:34.927" UserId="8" />
  <row Id="4798" PostId="3463" Text="Here: http://stats.stackexchange.com/questions/1881/analysis-of-cross-correlation-between-point-processes I was proposing a Monte Carlo approach to determine confidence limits. The idea was to do this for two point processes, but I guess it could be easily adapted for your situation." CreationDate="2010-10-13T10:53:44.703" UserId="582" />
  <row Id="4800" PostId="3538" Text="Hi all,&#xA;Thanks for you answers!&#xA;&#xA;1) I want check the effect of treatment(control, stress) with time (week1, week2,...,week12) and with varieties (var1,var2,...,var12).&#xA;&#xA;2) difference b/w replicates is because of source available&#xA;											3)Some description of data is given below:&#xA;&#xA;All Citrus plants were grown in pots with same soil and all condition same. Then only' Stress' labelled plants were treated with chemical 'NO3' and other plants 'Control' were untreated.													&#xA; Chlorophyll content (Data given) was recorded at day 1 and then each week for exactly the same plants.	&#xA;" CreationDate="2010-10-13T11:27:00.330" UserId="1534" />
  <row Id="4802" PostId="3521" Text="Right now, I'm not.  The most expensive parts of my MCMC algorithms are computing several likelihoods so I try to bunch them together best as possible and compute them in parallel.  But all of the setup, RN generation (for proposals), are done on a single cpu core.  For parallel RNGs, I'd start with [DC](http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/DC/dc.html) for the Mersenne Twister -- but I've never personally used it beyond a trivial translation to CUDA for GPUs (more as an exercise really)." CreationDate="2010-10-13T11:35:31.243" UserId="1499" />
  <row Id="4803" PostId="3546" Score="1" Text="Three additional references:  1.  [Beyond Kappa: A review of interrater agreement measures](http://onlinelibrary.wiley.com/doi/10.2307/3315487/abstract) by Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, &amp; Debajyoti Sinha  2.  [Interrater reliability and agreement of performance ratings: A methodological comparison](http://www.springerlink.com/content/76887531n77808n4/) by John W. Fleenor, Julie B. Fleenor &amp; William F. Grossnickle" CreationDate="2010-10-13T12:28:39.690" UserId="1499" />
  <row Id="4804" PostId="3546" Text="3. [Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine.](http://www.ncbi.nlm.nih.gov/pubmed/9820922) by Atkinson G &amp; Nevill AM.  The first reference is specific to ordinal data and discusses other measures beyond kappa for ordinal data.  The second and third are specific to interval data." CreationDate="2010-10-13T12:30:37.377" UserId="1499" />
  <row Id="4805" PostId="3555" Score="1" Text="+1:&gt; faily good way to make it intuitive." CreationDate="2010-10-13T13:17:08.423" UserId="603" />
  <row Id="4806" PostId="3546" Text="(+1) Much Thanks M. Tibbits! I generally provide a lot of references and examples during my lectures in psychometrics, including the first you cited, but I didn't know the two others." CreationDate="2010-10-13T13:22:58.753" UserId="930" />
  <row Id="4807" PostId="3555" Score="3" Text="@csgillespie (+1) Very nice example!" CreationDate="2010-10-13T13:25:22.073" UserId="930" />
  <row Id="4809" PostId="2467" Text="@pbneau Could you confirm that &quot;classification&quot; is not meant as &quot;cluster analysis&quot; (which is called *classification* in French), that is do you really seek to apply a supervised method following your FA?" CreationDate="2010-10-13T13:59:10.147" UserId="930" />
  <row Id="4810" PostId="3544" Text="@tosonb Would you mind explaining why such procedure should be preferred?" CreationDate="2010-10-13T14:20:44.713" UserId="930" />
  <row Id="4811" PostId="3538" Text="@Jacki, Your experimental design looks like a special case of Split plot design, but to decide the special case, I need extra details. Can you please represent your design inform of a picture (see my update) ? Currently, I don't see how you use the varieties? Is the difference between varieties of interest or not?" CreationDate="2010-10-13T15:11:30.000" UserId="1307" />
  <row Id="4812" PostId="3558" Text="This one is *much* older than xkcd, LOL!  Did you read J.M.'s contribution (August 9)?  Same joke, different animal." CreationDate="2010-10-13T15:21:18.877" UserId="919" />
  <row Id="4814" PostId="3559" Text="Srikant's answer is certainly a good one (and I have nothing to contribute), but you might want to wait a bit longer than 24 minutes to mark a best answer to see if anybody else has any ideas." CreationDate="2010-10-13T17:05:07.650" UserId="71" />
  <row Id="4815" PostId="3495" Score="2" Text="I see the problem; Welfords obscures the problem with this method: the online estimate of the mean is being used instead of the final estimate of the mean. While Welford's method is exact (up to roundoff) for variance, this method is not. The problem is *not* due to the `sqrt` imprecision. It is because it uses the running mean estimate. To see when this will break, try `xs &lt;- sort(rnorm(n.testitems))`  When I try this with your code (after fixing it to return `a.dev / n`), I get relative errors on the order of 9%-16%. So this method is not permutation invariant, which could cause havoc..." CreationDate="2010-10-13T17:05:59.583" UserId="795" />
  <row Id="4818" PostId="3558" Text="http://stats.stackexchange.com/questions/1337/statistics-jokes/1436#1436" CreationDate="2010-10-13T17:34:34.270" UserId="88" />
  <row Id="4821" PostId="1388" Text="I do not have enough reputation to downvote this!" CreationDate="2010-10-13T17:41:41.090" UserId="1077" />
  <row Id="4822" PostId="3560" Score="1" Text="A &quot;perfect fit&quot; is so far from being attainable in any realistic logistic regression that it seems unfair to use it as a reference or a standard." CreationDate="2010-10-13T17:47:46.273" UserId="919" />
  <row Id="4823" PostId="3565" Score="4" Text="Linear transformations (like these) never change correlation coefficients.  The point to standardization is to improve the conditioning of the normal matrix." CreationDate="2010-10-13T17:57:00.530" UserId="919" />
  <row Id="4824" PostId="3563" Text="Nice idea.  (It doesn't generalize well to categorical predictors, though.)  I suspect that many other strategies could be interpreted from this point of view, too.  For example, selecting a subset of $k$ of the predictors could be interpreted as approximating a basis for the span of the $k$ largest eigenvectors in a PCA." CreationDate="2010-10-13T17:59:45.683" UserId="919" />
  <row Id="4825" PostId="3565" Text="Yes, I agree. As my last line says: _this is not a remedy_" CreationDate="2010-10-13T18:02:21.847" UserId="1307" />
  <row Id="4826" PostId="3565" Text="Sorry I see, I said reduce correlation. Its edited now." CreationDate="2010-10-13T18:03:00.353" UserId="1307" />
  <row Id="4827" PostId="3563" Text="@Mike In an explanatory approach, then you have to interpret how your linear combination(s) of the $p$ variables relate to the outcome, and this might sometimes be tricky." CreationDate="2010-10-13T18:05:59.427" UserId="930" />
  <row Id="4828" PostId="3565" Text="Standardizing the variables will not affect the correlations among the independent variables and will not &quot;reduce the effect of correlation&quot; in any way that I can think of with respect to this problem." CreationDate="2010-10-13T18:15:24.217" UserId="485" />
  <row Id="4829" PostId="3567" Score="1" Text="@Brett (+1) Now, the problem is that the OP didn't indicate how many variables enter the model, because in case they are numerous it might be better to do both shrinkage and variable selection, through e.g. the *elasticnet* criterion (which is combination of Lasso and Ridge penalties)." CreationDate="2010-10-13T18:22:26.763" UserId="930" />
  <row Id="4830" PostId="3565" Score="2" Text="@Brett, a typical example where standardization helps is _Polynomial Regression_. It is always recommended to standardize the regressors. Standardizing doesn't change the correlation matrix, but makes the var cov matrix (which is now the correl matrix) well behaved (called conditioning by @whuber pointing to the condition number of the matrix, IMHO).  " CreationDate="2010-10-13T18:23:23.650" UserId="1307" />
  <row Id="4831" PostId="3565" Text="Agreed.  Centering is useful when entering higher order terms, like polynomial or interaction terms.  That doesn't seem to be the case here and will not otherwise help with the problem of correlated predictors. " CreationDate="2010-10-13T18:28:56.520" UserId="485" />
  <row Id="4832" PostId="3562" Text="@whuber (+1) I was initially thinking of expanding my response (that came just after yours), but definitely your answer is self-sufficient." CreationDate="2010-10-13T18:31:14.580" UserId="930" />
  <row Id="4833" PostId="3560" Text="@whuber True but you could use the standard to compare the relative performance of two competing models. Your points of low R^2 in your answer and its implications are good points but if you *have* (e.g., reviewers demand it etc) to use some form of R^2 then Nagelkerke is preferable." CreationDate="2010-10-13T18:32:23.877" />
  <row Id="4834" PostId="3560" Text="@Skridant Yes, still the problem of reviewers that want to see $R^2$ and Bonferroni correction everywhere..." CreationDate="2010-10-13T19:32:18.657" UserId="930" />
  <row Id="4835" PostId="3560" Text="@Srikant, @chl: A cynical reading of this thread would suggest just picking the largest R^2 among all those the software reports ;-)." CreationDate="2010-10-13T19:44:32.320" UserId="919" />
  <row Id="4836" PostId="3562" Text="@chl I wondered why your response wasn't as encyclopedic as usual :-)." CreationDate="2010-10-13T19:45:10.697" UserId="919" />
  <row Id="4837" PostId="3560" Text="@chl Offering push-back to reviewers/clients is of course necessary but sometimes we have to be pragmatic as well. If readers do not mis-interpret low R^2 as lack of adequate model performance then the issues raised by @whuber will be mitigated to some extent. " CreationDate="2010-10-13T19:46:09.110" />
  <row Id="4838" PostId="3561" Score="1" Text="I'm sorry see @Suncoolsu's answer was deleted.  It and the comments that followed clarified a difference between multicollinearity and ill conditioning.  Also, in a comment Suncoolsu pointed out how preliminary standardization can help with polynomial regression.  If it happened to reappear I would vote it up ;-)." CreationDate="2010-10-13T19:49:11.747" UserId="919" />
  <row Id="4840" PostId="3573" Text="I second this answer... and this really *is* my area (or at least I tell myself it is)." CreationDate="2010-10-13T20:03:36.193" UserId="1352" />
  <row Id="4841" PostId="3564" Text="Could you tell us a little more about your problem? Why exactly do you want to use KDE? I happen to be rather active in the area of forecasting &amp; replenishment, and I have never seen anyone use KDE. As onestop below notes, at first glance one would much rather use some kind of time series analysis method." CreationDate="2010-10-13T20:05:24.857" UserId="1352" />
  <row Id="4842" PostId="3573" Text="I've retagged the Q to add 'time-series'. Turns out there's a max of 5 tags, so i took out 'pdf'." CreationDate="2010-10-13T20:10:29.473" UserId="449" />
  <row Id="4843" PostId="1377" Text="Was his name Ronald Coase? http://en.wikipedia.org/wiki/Ronald_Coase#Quotes" CreationDate="2010-10-13T20:26:15.290" UserId="449" />
  <row Id="4844" PostId="3565" Text="+1 for the conversation this one inspired." CreationDate="2010-10-13T20:33:12.550" UserId="919" />
  <row Id="4845" PostId="3563" Text="@chl Good point.  But since the principal components are linear combinations, it's straightforward (although sometimes a bit of a pain) to compose the fitted regression model (=one linear transformation) with the projection onto the components (=another linear transformation) to obtain an interpretable *linear* model involving all the original variables.  This is somewhat akin to orthogonalization techniques.  Note, too, that Srikant's latest proposals (sum or average the regressors) essentially approximate the principal eigenvector yet induce similar explanatory difficulties." CreationDate="2010-10-13T20:37:38.437" UserId="919" />
  <row Id="4846" PostId="3569" Score="1" Text="Completely agreed, +1.  But the characterization of PCA as a &quot;mathematical trick&quot; unfairly disparages it, IMHO.  If you agree (I'm not sure you do) that summing or averaging groups of regressors, as Srikant suggests, would be acceptable, then PCA should be just as acceptable and it usually improves the fit.  Moreover, the principal components can provide insight into which groups of predictors are correlated and how they correlate: that's an excellent tool for the thinking you are advocating." CreationDate="2010-10-13T20:40:44.353" UserId="919" />
  <row Id="4847" PostId="3570" Text="Thanks for pointing out that paper; somehow I missed it (and it appeared when I was in the middle of a big logistic regression project!)." CreationDate="2010-10-13T20:44:17.940" UserId="919" />
  <row Id="4848" PostId="3561" Text="@Ηλίας : The product is likely to be unstable in many applications.  It can be plagued by many zeros if the individual regressors have some zeros; its absolute value is likely to have strong positive skew, giving rise to some high-leverage points; it might amplify outlying data, especially simultaneous outliers, further adding to their leverage.  It might be rather difficult to interpret, too, especially if the regressors already are re-expressions of the original variables (like logs or roots)." CreationDate="2010-10-13T20:47:57.727" UserId="919" />
  <row Id="4849" PostId="3576" Text="+1.  Beat me by 22 seconds... :-)" CreationDate="2010-10-13T20:51:25.247" UserId="919" />
  <row Id="4850" PostId="3576" Text="That's a relief.  You've been fast on the draw recently.  :)" CreationDate="2010-10-13T20:55:16.873" UserId="5" />
  <row Id="4851" PostId="3569" Score="1" Text="@whuber, I see and agree with your point, and I don't want to disparage PCA, so definitely +1. I just wanted to point out that blindly using PCA without looking at and thinking about the underlying problem (which no one here is advocating) would leave me with a bad feeling..." CreationDate="2010-10-13T21:22:18.010" UserId="1352" />
  <row Id="4852" PostId="3571" Text="This makes sense if the regressors are all measured on the same scale. In psychology, various subscales are often measured on different scales (and still correlated), so a weighted sum or average (which is really the same here) would be appropriate. And of course, one could view PCA as providing just this kind of weighting by calculating axes of maximum variance." CreationDate="2010-10-13T21:24:28.820" UserId="1352" />
  <row Id="4853" PostId="3564" Text="Another question: how does a Gaussian *distribution* enter into KDE? Are you sure you are not looking at a Gaussian *kernel*?" CreationDate="2010-10-13T21:27:12.503" UserId="1352" />
  <row Id="4854" PostId="3573" Text="I suspect you are right. Maybe I should recast my question &quot;How do I know I have a problem where KDE can help&quot;?" CreationDate="2010-10-13T21:33:43.313" UserId="1574" />
  <row Id="4855" PostId="3564" Text="Mostly because I read KDE generalizes histograms, and our current program essentially buckets sales and calculates an average, which seemed like it was related to me.  &#xA;&#xA;I probably meant Gaussian kernel ... apologies, I'm not up on the terminology. " CreationDate="2010-10-13T21:35:06.613" UserId="1574" />
  <row Id="4857" PostId="3572" Text="My initial idea was to add take into account the pairwise  interaction of the regressors" CreationDate="2010-10-13T22:17:12.480" UserId="1077" />
  <row Id="4858" PostId="3538" Text="@suncoolsu: Thanks for you detailed answer and help!&#xA;The design you made is fine. Yes I am also interested to see the differences between varieties.&#xA;Do you know how to do in SPSS?" CreationDate="2010-10-13T22:40:26.643" UserId="1534" />
  <row Id="4862" PostId="3565" Text="I deleted it because I didn't want to confuse people with wrong answer. Probably the moderators brought it up again." CreationDate="2010-10-14T00:14:45.607" UserId="1307" />
  <row Id="4863" PostId="3538" Text="@Jacki, you would do this in SPSS using the mixed model options. I am sorry, but this is the end of my knowledge in SPSS (not a regular user :-( ). I can point out some good reference for R or SAS, in case you are interested." CreationDate="2010-10-14T00:17:12.640" UserId="1307" />
  <row Id="4864" PostId="3538" Text="Also, please modify your question above so that we can see the full detail. The comment space is too small for such discussions." CreationDate="2010-10-14T00:39:47.077" UserId="1307" />
  <row Id="4865" PostId="3556" Text="Can you give alittle more context as to what you mean by &quot;what works&quot; or the particular goals of your project at hand. I've used them for visualizing spatial point processes but I doubt that is what you had in mind when asking this question." CreationDate="2010-10-14T01:11:10.957" UserId="1036" />
  <row Id="4867" PostId="3582" Score="2" Text="Loess is a variable kernel REGRESSION method. The question asked about variable kernel DENSITY estimation." CreationDate="2010-10-14T02:04:15.347" UserId="159" />
  <row Id="4868" PostId="2103" Text="@Srikant: Done.  Thank you for the excellent guidance." CreationDate="2010-10-14T02:31:45.483" UserId="919" />
  <row Id="4869" PostId="3503" Text="@Joseph.  I am not sure I understand your comment.  We just use a slightly different syntax.  I don't know what beta.hat means.  My point was that a greater sample N will give you greater statistical significance (lower standard error, higher t stat, lower p value) on all regression coefficients within a single run.  Meanwhile, the greater number of iterations M will give you greater statistical significance for the Mean of each specific coefficients across all iterations.  They are two different things.     " CreationDate="2010-10-14T03:34:22.003" UserId="1329" />
  <row Id="4870" PostId="3495" Text="Ah, well spotted :)  I guess the utility of this algorithm will depend on the dataset then..." CreationDate="2010-10-14T04:07:56.833" UserId="179" />
  <row Id="4871" PostId="3585" Score="5" Text="This model assumes the response is an additive function of traveling to each place, which is highly unlikely.  It can still be made to work by including interaction terms.  A full set of all possible interactions might be needed (beyond just the two-way interactions).  (That would be mathematically identical to providing a separate dummy for each possible combination of destinations.)" CreationDate="2010-10-14T04:59:39.657" UserId="919" />
  <row Id="4872" PostId="3563" Text="@whuber Yes, I agree with both of your points. I extensively used PLS regression and CCA, so in this case we have to deal with linear combinations on both side (st. a max. covariance or correlation criteria); with a large number of predictors, interpreting the canonical vectors is painful, so we merely look at the most contributing variables. Now, I can imagine that there is not so much predictors so that all of your arguments (@Stephan, @Mike) make sense." CreationDate="2010-10-14T05:56:18.487" UserId="930" />
  <row Id="4873" PostId="3574" Score="2" Text="@onestop The original HL $\chi^2$ GoF test is not very powerful for it depends on categorizing the continuous predictor scale into an arbitrary number of groups; H &amp; L proposed to consider decile, but obviously it depends on the sample size, and under some circumstances (e.g. IRT models) you often have very few people at one or both end of the scale such that cutoffs are unevenly spaced. See A comparison of goodness-of-fit tests for the logistic regression model, Stat. Med. 1997 16(9):965, http://j.mp/aV2W6I" CreationDate="2010-10-14T06:36:24.097" UserId="930" />
  <row Id="4874" PostId="3582" Text="Oops, you're right. Misread the question." CreationDate="2010-10-14T06:56:25.680" UserId="1569" />
  <row Id="4875" PostId="3585" Score="4" Text="Better have a *lot* of data if you use all interactions (15 parameters) rather than just the main effects (4 parameters)..." CreationDate="2010-10-14T06:56:40.920" UserId="1352" />
  <row Id="4876" PostId="3587" Text="I'm using the riskratio function from the EpiTool package, manual attached here:&#xA;http://bm2.genes.nig.ac.jp/RGM2/R_current/library/epitools/man/riskratio.html&#xA;&#xA;I'm still digesting your answer, thanks again!" CreationDate="2010-10-14T07:01:20.257" UserId="588" />
  <row Id="4877" PostId="3589" Text="This does not sound like you have observations (or &quot;cases&quot;) on which you observe both an X and a Y realization. How do you find out which X is associated to which Y?" CreationDate="2010-10-14T07:01:47.680" UserId="1352" />
  <row Id="4879" PostId="3589" Score="1" Text="I suppose I forgot to mention this. X and Y are stock prices. Company X has been public for a much shorter time period than Y. I wanted to tell how correlated the prices of X and Y are. I could definitely get a correlation for the period of time that X and Y both exist. I wanted to know if knowing the stock prices for several extra years of Y that X did not exist yielded me any additional information." CreationDate="2010-10-14T07:10:00.317" UserId="1118" />
  <row Id="4880" PostId="3574" Text="Thanks chi, that's a useful ref, though your j.mp link took me to a BiblioInserm login prompt. Here's a doi-based link:&#xA;http://dx.doi.org/10.1002/(SICI)1097-0258(19970515)16:9&lt;965::AID-SIM509&gt;3.0.CO;2-O" CreationDate="2010-10-14T07:20:21.123" UserId="449" />
  <row Id="4881" PostId="3558" Text="I just noted that I got it via xkcd forum. I didn't mean this is its origin. Thank you for pointing out the deer hunting skit. Also funny. :)" CreationDate="2010-10-14T07:25:31.703" UserId="144" />
  <row Id="4882" PostId="3586" Text="I think the condition of 5 is for the actual cell count and I see that one of the cell has cell count = 3. Further, the p-value calculated by the ChiSq test is based on pchisq( .. , lower.tail=F), if I understand the code correctly. Therefore, the p-value is based on one sided test, where as the CI is two sided, hence the discrepancy." CreationDate="2010-10-14T07:25:56.070" UserId="1307" />
  <row Id="4883" PostId="3574" Text="@onestop Sorry for the incorrect link... I seem to remember Frank Harrell's `Design` package features the alternative H&amp;L 1 df test." CreationDate="2010-10-14T07:31:47.527" UserId="930" />
  <row Id="4884" PostId="3586" Text="I'll investigate on the &quot;one-sided/two-sided&quot; issue, but for condition of 5, I think this should be referred to the EXPECTED FREQUENCY as from http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test: &quot;Expected Cell Count – Adequate expected cell counts. Some require 5 or more, and others require 10 or more. A common rule is 5 or more in all cells of a 2-by-2 table, and 5 or more in 80% of cells in larger tables, but no cells with zero expected count. When this assumption is not met, Yates' correction is applied.&quot;" CreationDate="2010-10-14T07:53:58.303" UserId="588" />
  <row Id="4885" PostId="3591" Text="actually I have used the small-sample adjustment, and it then CI and Chi-Square fits perfectly, but how can I justify that it is a &quot;small sample&quot;?" CreationDate="2010-10-14T07:55:09.813" UserId="588" />
  <row Id="4886" PostId="3534" Text="Thanks for the Windows warning." CreationDate="2010-10-14T08:10:10.553" UserId="8" />
  <row Id="4887" PostId="3535" Text="Geyer's slides are really helpful. Thanks." CreationDate="2010-10-14T08:10:51.847" UserId="8" />
  <row Id="4888" PostId="3593" Text="@ars (+1) Both responses provide a handy summary of possible discrepancies between all three statistics." CreationDate="2010-10-14T08:11:47.547" UserId="930" />
  <row Id="4889" PostId="3589" Score="1" Text="@Christopher I'd recommend that you update your question to reflect your above comment. Also, for correlation to be meaningful, more than just equal dimensions are required; the actual measurements have to come from the same cases, which in your case is presumably the same time points." CreationDate="2010-10-14T08:13:20.753" UserId="183" />
  <row Id="4890" PostId="3591" Score="2" Text="The asymptotics (and power) depends much more on the total number of positives than the overall total (strictly, the total number of the whichever outcome is rarer). Your total number of positives is 14. That's pretty small. 'Expected value 5 or more in all cells' is only a rule of thumb. &#xA;&#xA;I'm not sure what the small sample adjustment used by epitools is doing exactly (can't find any explanation in the documentation), but it's certainly something more sophisticated than Yates' correction." CreationDate="2010-10-14T08:22:41.480" UserId="449" />
  <row Id="4892" PostId="2697" Text="What about the eigenvectors &amp; eigenvalues?" CreationDate="2010-10-14T08:29:12.460" UserId="1077" />
  <row Id="4893" PostId="3492" Text="I think it's wrong to say that R would not be helpfull in most corporate settings.  Knowing R won't really help you learn SAS if that's what you're stuck on, but that's like saying you need a hammer when faced with a box of screws....." CreationDate="2010-10-14T08:34:30.327" UserId="114" />
  <row Id="4894" PostId="3589" Score="1" Text="I second Jeromy's comment on updating the question..." CreationDate="2010-10-14T08:53:30.187" UserId="1352" />
  <row Id="4895" PostId="1441" Text="Since we cannot migrate, closed." CreationDate="2010-10-14T09:00:32.217" UserId="88" />
  <row Id="4896" PostId="3589" Text="Another question: you mention that X and Y have the same number of columns. Would that be one each? Or do you have multiple series for both X and Y (prices at different stock exchanges or some such)?" CreationDate="2010-10-14T09:00:46.130" UserId="1352" />
  <row Id="4897" PostId="3565" Text="@suncoolsu: yes, standardization helps in polynomial regression - but I would go the whole way and transform the data into appropriate polynomial basis functions. Numerical people have been thinking a lot about stuff like this." CreationDate="2010-10-14T09:04:30.863" UserId="1352" />
  <row Id="4899" PostId="1462" Text="@stats Consider accepting some answer." CreationDate="2010-10-14T09:10:46.340" UserId="88" />
  <row Id="4900" PostId="3591" Text="@onestop @lokheart In fact, the &quot;expected counts greater than 5&quot; has been shown to be somewhat too stringent, at least in the case of the Pearson $\chi^2$ test, see Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations, Stat. Med. 2007 26:3661, and Campbell's webpage, http://bit.ly/dbPfOO." CreationDate="2010-10-14T09:18:44.200" UserId="930" />
  <row Id="4901" PostId="2467" Text="Yes i think to cluster analysis rather than classifcation. " CreationDate="2010-10-14T09:55:37.543" UserId="1154" />
  <row Id="4902" PostId="3591" Text="Thanks for the ref chl. Wald CIs have worse finite-sample properties than the Pearson chi-square test though, so it may well not be stringent enough when it comes to appropriateness of Wald CIs." CreationDate="2010-10-14T10:12:59.693" UserId="449" />
  <row Id="4903" PostId="3595" Score="5" Text="Can you pls be a bit more specific? What do you actually want to data mine and how do you plan to do it? I have used R to analyze similar size records as yours, and it wasn't a bad experience at all. " CreationDate="2010-10-14T10:31:53.547" UserId="1307" />
  <row Id="4904" PostId="3595" Score="2" Text="This should be community wiki. " CreationDate="2010-10-14T10:49:33.373" UserId="5" />
  <row Id="4905" PostId="3595" Text="I agree with @Shane; this is merely open to equally valid answers, unless it is clearly stated what kind of analyses or questions come with this data set. (Shouldn't have post an answer without further information in fact!)" CreationDate="2010-10-14T12:20:20.977" UserId="930" />
  <row Id="4906" PostId="3585" Text="@whuber and @Stephen, Thanks for the responses, and I agree completely with each of you. I personally would be ok with the main effects dummy variable approach if multiple responses weren't all that common, which may not be a tenable assumption given the original posters concerns. I would maybe propose other designs if the original poster was interested in the risk of travelling to A vs B (such as some type of matching procedure). And I agree additive risk does not make sense except if some selection bias is occurring." CreationDate="2010-10-14T12:31:35.670" UserId="1036" />
  <row Id="4907" PostId="3566" Text="@chl, could you elaborate on the distinction between &quot;goodness of fit&quot; and strength of association or predictive ability?" CreationDate="2010-10-14T13:12:04.013" UserId="1036" />
  <row Id="4908" PostId="3492" Text="Let me rephrase: I don't think R will help anyone get a SAS programming position in most corporate settings, primarily because I doubt most people in corporate SAS shops even know what R is." CreationDate="2010-10-14T13:20:59.863" UserId="71" />
  <row Id="4909" PostId="3511" Text="Still could not get a better convergence by adapting IRLS. :'(" CreationDate="2010-10-14T13:29:45.227" UserId="1351" />
  <row Id="4910" PostId="3594" Text="For basic financial time series references, you can see my answer here: http://stats.stackexchange.com/questions/328/resources-for-learning-about-the-statistical-analysis-of-financial-data/329#329.  The Tsay text is one of the most popular." CreationDate="2010-10-14T13:34:08.087" UserId="5" />
  <row Id="4911" PostId="3566" Text="@Andy Thanks for pointing that. I realize afterwards that my first sentence does not sound well indeed. I'll update my answer, pls let me know if this ok with you." CreationDate="2010-10-14T14:00:07.663" UserId="930" />
  <row Id="4912" PostId="3605" Text="Clarification: I didn't mention GARCH to deal with the missing data problem (which of course would not make sense) - but to improve on a simple calculation of correlation between the time series at times where both exist." CreationDate="2010-10-14T14:12:55.287" UserId="1352" />
  <row Id="4913" PostId="2095" Text="And what if x=12 and y=1/2? Then your answer will have  F(12, 1/2)=12+1/2-1=23/2." CreationDate="2010-10-14T14:15:44.917" UserId="247" />
  <row Id="4914" PostId="3599" Text="@Misha:&gt; *in that values of B above a threshold will change C* Do you know that threeshold in advance or does it need to be estimated as well ? Also, when you write B*t do you actually mean $B_t$ ?" CreationDate="2010-10-14T14:20:12.987" UserId="603" />
  <row Id="4915" PostId="3603" Text="How did you deduce that B and C are correlated?  They might be, but this does not seem to be necessarily the case.  In particular, if B rarely exceeds that threshold, B and C might be approximately independent." CreationDate="2010-10-14T14:23:22.533" UserId="919" />
  <row Id="4916" PostId="3605" Text="@Stephan: OK.  I mentioned it mainly to show I wasn't ignoring you!" CreationDate="2010-10-14T14:24:38.210" UserId="919" />
  <row Id="4917" PostId="3603" Text="@whuber There is no issue at all if B rarely exceeds the threshold. We may as well assume that they are independent and use standard logistic models. I was implicitly assuming that there is some correlation between B and C given that there are threshold effects, the fact that changes in C impact B and that B and C are continuous variables." CreationDate="2010-10-14T14:30:54.077" />
  <row Id="4918" PostId="3588" Text="Thanks Kwak. &quot;... for Gaussian distributed random variables&quot;; would you know of newer work for &quot;clumpy&quot; distributions ?" CreationDate="2010-10-14T14:38:07.343" UserId="557" />
  <row Id="4919" PostId="2095" Text="@PeterR good point! Fixed the error." CreationDate="2010-10-14T14:44:31.593" />
  <row Id="4920" PostId="3505" Text="Hi @suncoolsu. Your points are good but, as the experiment is complete, the design is fixed." CreationDate="2010-10-14T14:46:21.043" UserId="1381" />
  <row Id="4921" PostId="3603" Text="@Srikant You're right.  The concern lies with intermediate situations.  In those cases the correlation is probably not great enough to warrant special measures to deal with correlated regressors.  Moreover, although we speak of &quot;correlation&quot; the relationship likely is not linear, so a lot of care is needed in dealing with it." CreationDate="2010-10-14T14:48:29.277" UserId="919" />
  <row Id="4922" PostId="3505" Text="Re the first point (1) - I read Venables' article that you referenced about main effects being an artifact of experimental design (the levels chosen of each factor) when interactions are present (p. 13), but I believe my study is one of the 'rare and special' exceptions, because the main effect of A on Y is very large, i.e., Y is an order of magnitude different at each level of A, while any effect of B would have been on the order of a 10-50% change or so; of course the point is moot since the interaction effect was not significant." CreationDate="2010-10-14T14:54:22.817" UserId="1381" />
  <row Id="4923" PostId="3505" Text="&quot;Three solid gold (significance) stars on the main effects will do very nicely, thank you, and if there are a few little stars here and there on the interactions, so much the better!&quot; -Venables 2000 [N.B. said in jest, but this attitude is so pervasive in research, peer review, and graduate training is so great I thought it would be edifying to include it here." CreationDate="2010-10-14T15:05:54.693" UserId="1381" />
  <row Id="4924" PostId="3597" Text="@chl: Have you yet found an effective parallel computing solution for 64-bit R?  When I last looked (late this summer) the only non-commercial ones appeared to work only in 32-bit R." CreationDate="2010-10-14T15:07:42.383" UserId="919" />
  <row Id="4925" PostId="3582" Text="@Rob, excuse my naive questions: if varying kernel width is (sometimes) good for local regression / Kernel smoothing, why is it bad for density estimation ? Isn't density estimation a case of f() estimation for f() == density() ?" CreationDate="2010-10-14T15:13:04.727" UserId="557" />
  <row Id="4926" PostId="3582" Text="@Hong Ooi, how many points in what Ndim have you used ? Thanks" CreationDate="2010-10-14T15:15:06.263" UserId="557" />
  <row Id="4927" PostId="3589" Text="Stephan: Thanks for editing my question to reflect the new information. I actually have a few columns, because I have date, opening price, highest price of the day, lowest price, and closing value." CreationDate="2010-10-14T15:28:23.673" UserId="1118" />
  <row Id="4928" PostId="3605" Text="Thank you, whuber. This is in line with what I was looking for. I don't think the backcasting will be of much use (or feasibility) to add a couple extra weeks of X when the mutual time frame between X and Y is about 16 years already." CreationDate="2010-10-14T15:36:47.693" UserId="1118" />
  <row Id="4929" PostId="3503" Text="@Joseph, using your language.  I am not sure that the CLT argument suggests that both beta.hat and sample.mean.beta.hat will converge towards each other.  But, that their respective distributions of outcome (defined by their standard error around the mean) will be normally distributed.  I think the two beta.hat(s) will converge towards each other simply because they will each become more firmed up or statistically significant as you use greater N and greater M." CreationDate="2010-10-14T15:49:41.200" UserId="1329" />
  <row Id="4930" PostId="730" Score="2" Text="And this is an actual quote, as opposed to something &quot;attributed to&quot; Box. It appears, e.g., in Box &amp; Draper (1987), *Empirical model-building and response surfaces*, Wiley, on page 424. Yes, I did go and look it up before using it in a paper." CreationDate="2010-10-14T15:53:20.163" UserId="1352" />
  <row Id="4931" PostId="3608" Text="@Shane Wonderful finding!" CreationDate="2010-10-14T15:56:20.517" UserId="930" />
  <row Id="4932" PostId="3608" Text="Very interesting Shane, thank you." CreationDate="2010-10-14T16:05:59.167" UserId="253" />
  <row Id="4933" PostId="3605" Score="1" Text="@Christopher: !! With 16 years (of daily closings?) you have enough data not only to find a correlation, but also to explore how it has been changing over time.  (This I believe is the spirit of @Stephan Kolassa's reply.)" CreationDate="2010-10-14T16:08:41.903" UserId="919" />
  <row Id="4934" PostId="3495" Text="@fmark nevertheless, your method seems to work reasonably well when the data are i.i.d. I suspect, however, that using Welford's trick somewhat obscures the approach, and that it is unnecessary for this approximation. The simplified version would then have `a.dev &lt;- a.dev + abs(x - mean)` as the update." CreationDate="2010-10-14T16:41:04.653" UserId="795" />
  <row Id="4935" PostId="3597" Text="@whuber Nope. I had to switch to 64 bits last year to manage large genetic data sets, but the statistical models we used do not call for parallelization (as far as I know). I thought there was an OpenMP binding for R but did not investigate this further. I know Revolution Analytics have made effort in this sense (http://j.mp/d7dFb5), but still in 32 bits (this is probably what you referred to). I found R/parallel (http://www.rparallel.org/) in the meantime, but I don't know how reliable/mature it is. " CreationDate="2010-10-14T16:47:50.797" UserId="930" />
  <row Id="4936" PostId="555" Score="4" Text="ANOVA can be seen as &quot;syntactic sugar&quot; for a special subgroup of linear regression models. ANOVA is regularly used by researchers who are not statisticians by training. They are now &quot;institutionalized&quot; and its hard to convert them back to using the more general representation ;-)" CreationDate="2010-10-14T16:52:51.193" UserId="1307" />
  <row Id="4937" PostId="3560" Text="@Skridant There are alternative measures of prediction performance that makes more sense or are more intuitive (e.g. ROC area, Somers D) and that can be reported together with pseudo $R^2$. " CreationDate="2010-10-14T16:55:48.357" UserId="930" />
  <row Id="4938" PostId="3597" Text="@chl I tried them all but couldn't get any of them to work." CreationDate="2010-10-14T17:53:29.933" UserId="919" />
  <row Id="4939" PostId="825" Text="@chl: Thanks for bumping this up.  In fact, I checked out all the non-commercial references from this thread shortly after it appeared but couldn't find anything that works on Win 7 x64." CreationDate="2010-10-14T18:07:11.927" UserId="919" />
  <row Id="4940" PostId="3588" Text="@Denis:&gt; 'Clumpy'=?concentrated=?with narrower tails than the gaussian?" CreationDate="2010-10-14T18:28:07.767" UserId="603" />
  <row Id="4942" PostId="3607" Text="That is quite analogous to what I´m looking into. I´ll look into their work. Thx for the input." CreationDate="2010-10-14T19:05:22.870" UserId="1291" />
  <row Id="4943" PostId="3599" Text="Unfortunately we do not know the threshold and the threshold itself might not be constant over time. " CreationDate="2010-10-14T19:09:03.470" UserId="1291" />
  <row Id="4944" PostId="3612" Text="@chi: Thanks a lot. I'll look the papers. Would you please comment the first question? Is hazard variable always time?" CreationDate="2010-10-14T19:12:54.947" UserId="1586" />
  <row Id="4945" PostId="3615" Text="@Srikant Excellent answer, but does that function resolve to something arithmetic (ie: not recursive)?" CreationDate="2010-10-14T20:06:46.517" UserId="1456" />
  <row Id="4946" PostId="3615" Text="@C. Ross Unfortunately I do not think so. But, I suspect that the recursion should not be that hard as long as are dealing with reasonably small n and small s. You could just build-up a lookup table and use that repeatedly as needed." CreationDate="2010-10-14T20:10:30.160" />
  <row Id="4947" PostId="3616" Text="What do you mean by &quot;different&quot;?" CreationDate="2010-10-14T20:21:50.570" UserId="5" />
  <row Id="4948" PostId="3616" Text="What do you mean with &quot;5-10 data points *per data set*&quot;?" CreationDate="2010-10-14T20:25:09.053" UserId="1352" />
  <row Id="4949" PostId="3612" Text="@yuk Not necessarily, as suggested by @whuber. I have in mind another application of Cox regression dealing with the treatment of systematic pattern of missing responses in educational testing, as it arises when a student has not enough time to complete the test (missing responses might then be considered as right-censored) -- in this case, this is item ordering that is considered as the time scale. I'll look at the original paper (although I think this was also the subject of a PhD)." CreationDate="2010-10-14T20:43:01.703" UserId="930" />
  <row Id="4950" PostId="3582" Text="@Denis. Great question. Can you please add it as a proper question on the site and we'll see what answers people can come up with." CreationDate="2010-10-14T20:58:14.657" UserId="159" />
  <row Id="4951" PostId="3605" Text="I agree. Using techniques to figure out what values X would've taken prior to its IPO seems prone to error. I might also question the relevance of data that's 16 years old to predict modern trends." CreationDate="2010-10-14T22:20:43.623" UserId="1118" />
  <row Id="4952" PostId="3618" Text="@whuber Will that mathematica code work with wolfram alpha?" CreationDate="2010-10-14T23:56:42.000" />
  <row Id="4953" PostId="3616" Text="I think he has a collection of several time series, each one with 5-10 observations." CreationDate="2010-10-15T01:07:37.033" UserId="159" />
  <row Id="4954" PostId="3613" Text="+1, thanks for pointing out the NADA package.  I noticed it makes it easier to handle left-censored data through the survival package -- is left-censored a common scenario with environmental data?" CreationDate="2010-10-15T02:11:07.610" UserId="251" />
  <row Id="4955" PostId="3612" Text="+1.  There are other papers, but I'm not sure they're necessarily better; I think Chalise does a pretty good job summing up the situation." CreationDate="2010-10-15T02:13:50.007" UserId="251" />
  <row Id="4958" PostId="3613" Text="@whuber: Thank you for the comment, NADA package looks very interesting." CreationDate="2010-10-15T05:23:24.287" UserId="1586" />
  <row Id="4959" PostId="3613" Text="@Andy: Thanks for the links. I think its worth to be an answer. I'd upvote." CreationDate="2010-10-15T05:25:07.700" UserId="1586" />
  <row Id="4960" PostId="3619" Text="@suncoolsu Do you have success with it? If yes, for what kind of application?" CreationDate="2010-10-15T05:50:47.757" UserId="930" />
  <row Id="4961" PostId="3588" Text="@Kwak, I'm no expert, but like &quot;data set clumpiness&quot; in the paper Lang et al., &quot;Insights on fast Kernel Density Estimation algorithms&quot;, 2004, 8p" CreationDate="2010-10-15T10:15:12.183" UserId="557" />
  <row Id="4962" PostId="3582" Text="@Rob, which part, can you help me formulate ? along the lines &quot;Is adaptive kernel width good for both kernel smoothing and KDE ?&quot;" CreationDate="2010-10-15T10:15:36.397" UserId="557" />
  <row Id="4964" PostId="3629" Text="Ok, now it's clear. Many thanks for your prompt reply. " CreationDate="2010-10-15T11:07:23.260" UserId="1219" />
  <row Id="4965" PostId="3613" Text="@Yuk, per your request I made my comment into an answer, and @whuber thanks for your example." CreationDate="2010-10-15T12:03:07.990" UserId="1036" />
  <row Id="4966" PostId="3631" Text="@Andy Easier to upvote now :)" CreationDate="2010-10-15T12:08:46.327" UserId="930" />
  <row Id="4967" PostId="3592" Text="Just as a note from what I can gather time varying in this context does not make sense, unless the criteria used to classify individuals in either before or after is time varying itself (in which case I would say you want those characteristics themselves as the covariates not the classification). " CreationDate="2010-10-15T12:50:18.157" UserId="1036" />
  <row Id="4968" PostId="3566" Text="@chl, Thanks for the update and it does clarify the distinction." CreationDate="2010-10-15T13:24:09.850" UserId="1036" />
  <row Id="4969" PostId="3613" Text="@ars: Yes, left censoring is characteristic of environmental data (and is a key concern of chemometrics in general).  It's a tricky and interesting problem.  Among the reasons are (1) the censoring limits are themselves determined by statistical estimates (through a calibration process), (2) censoring can occur in multiple ways--as limits of detection, limits of quantification, or &quot;reporting limits&quot;, (3) the thresholds often vary in response to covariates (&quot;matrix interferences&quot;) that can be strongly correlated with the original censored values, (4) data often are lognormally distributed." CreationDate="2010-10-15T14:11:52.090" UserId="919" />
  <row Id="4970" PostId="3618" Text="@Srikant I didn't even try it because usually alpha does not process Mathematica code correctly except for simple expressions.  However, see what happens when you type &quot;Expand[(x + x^2 + x^3 + x^4 + x^5 + x^6)^3]&quot; !  ( http://www.wolframalpha.com/ )" CreationDate="2010-10-15T14:15:17.037" UserId="919" />
  <row Id="4971" PostId="3632" Text="Did you really mean to include &quot;logistic&quot; in &quot;use logistic regression to estimate x&quot;?" CreationDate="2010-10-15T14:18:30.330" UserId="919" />
  <row Id="4972" PostId="3626" Score="1" Text="+1 for noting the nonlinearity.  But your use of the phase &quot;expected proportion&quot; seems to assume the data are a random sample of the distribution over which one is averaging and that's often not the case." CreationDate="2010-10-15T14:22:41.203" UserId="919" />
  <row Id="4973" PostId="3623" Text="Over what probability distribution do you intend to form an average, David?" CreationDate="2010-10-15T14:23:52.330" UserId="919" />
  <row Id="4974" PostId="3618" Score="1" Text="@whuber That works. I tried your earlier version but could not make any sense of the output." CreationDate="2010-10-15T14:24:16.927" />
  <row Id="4975" PostId="3632" Text="I think a bit more context (e.g., other variables, what kind of data you have, dependent and independent variables etc) would help someone to give a useful answer." CreationDate="2010-10-15T14:28:43.313" />
  <row Id="4976" PostId="3619" Text="Yes, RHIPE is great. Some of my friends use it to analyze internet traffic data. One of their aims is to model break-in attempts. Data is huge in such cases, petabytes is common!" CreationDate="2010-10-15T14:37:59.833" UserId="1307" />
  <row Id="4977" PostId="3592" Text="@ Andy W: yes, you're right. Maybe this was why I was uncomfortable with the idea of time varying covariates." CreationDate="2010-10-15T14:42:27.793" UserId="1573" />
  <row Id="4978" PostId="3581" Text="Thanks for the links. I actually requested your book from my local university. :)" CreationDate="2010-10-15T14:43:19.077" UserId="1574" />
  <row Id="4979" PostId="3604" Text="onestop, that's a good idea! Definitively valid with a clear conclusion to be drawn from the results. I don't have the final data until now, but hopefully I will have enough fluctuation between classes. If not, I will have a very small sample size in groups (c) and (d) and inference will be difficult. But that's the best strategy so far... :)" CreationDate="2010-10-15T14:47:09.627" UserId="1573" />
  <row Id="4980" PostId="3616" Text="I still think that this question is nearly impossible to answer without understanding what &quot;different&quot; means..." CreationDate="2010-10-15T15:30:42.697" UserId="5" />
  <row Id="4981" PostId="3383" Text="What sort of variance correction were you thinking about?  Any error in estimate as a consequence of using the sample mean will result in increased variance in the difference scores.  So isn't the variance already adjusted for?  If anything it is going to be high." CreationDate="2010-10-15T16:05:43.637" UserId="196" />
  <row Id="4982" PostId="3636" Score="1" Text="What are trying to do with the data, Robert?  How do you intend to interpret the standard deviation?" CreationDate="2010-10-15T16:12:47.757" UserId="919" />
  <row Id="4984" PostId="3641" Text="What's wrong with the moving average?" CreationDate="2010-10-15T16:55:19.890" />
  <row Id="4985" PostId="3641" Text="What's a &quot;period&quot;?  You can always add variables into your model (e.g. to prevent it going to unrealistic values)." CreationDate="2010-10-15T16:56:25.117" UserId="5" />
  <row Id="4986" PostId="3641" Text="@Shane - each period is 10 years. Age will be the only variable i have for those years" CreationDate="2010-10-15T16:57:40.873" UserId="59" />
  <row Id="4987" PostId="3641" Text="I understand that age is the only variable that you will have in your data, but you can add additional factors based on priori information (e.g. that population can never be negative).  This is especially true if you model this using a Bayesian approach." CreationDate="2010-10-15T16:59:47.783" UserId="5" />
  <row Id="4988" PostId="3641" Text="@Shane - I'm not that advanced yet as to do Bayesian analysis on the data set, but I'm a fast learner :)" CreationDate="2010-10-15T17:01:48.343" UserId="59" />
  <row Id="4989" PostId="3641" Text="@dassouki: Suppose that an average adult (say between 25 - 40 yrs) gives birth to 2 kids (something you can get from census). Then you can extrapolate using some assumptions: (a) Percentage of adults who have kids (say 60%) etc. Then you can perform a rolling lagged forecast using the number of kids born in 1930s who would be adults in 1960s etc Does that make sense?" CreationDate="2010-10-15T17:12:13.187" />
  <row Id="4990" PostId="3641" Text="@Sikrant Vadali - Changed image" CreationDate="2010-10-15T17:25:44.500" UserId="59" />
  <row Id="4991" PostId="3641" Text="@Shane - changed Image" CreationDate="2010-10-15T17:26:20.913" UserId="59" />
  <row Id="4992" PostId="3643" Text="a. I need to estimate the variance and the variance of the Cauchy distribution is not defined." CreationDate="2010-10-15T17:26:44.203" UserId="1381" />
  <row Id="4993" PostId="3643" Text="b. If I understand your second point, yes, I could assume that y-1 ~ N(mu, sigma), but I still need to calculate mu and sigma from the summary statistics given for y; also, I've chosen not to consider distributions with values &lt; 0 for variables only defined &gt; 0 (even though in many of the cases p(X&lt;0 | X~N(mu,s)) -&gt; 0 )" CreationDate="2010-10-15T17:32:32.323" UserId="1381" />
  <row Id="4994" PostId="3640" Text="should I post the Update question about calculating the variance on random draws from the Cauchy as a separate question?" CreationDate="2010-10-15T17:39:48.120" UserId="1381" />
  <row Id="4995" PostId="3644" Text="Thanks for the reference, that is where I found the Haaya 1975 reference and the equations in my question, although I'd appreciate reassurance that the equations are appropriate for my problem." CreationDate="2010-10-15T17:43:55.880" UserId="1381" />
  <row Id="4996" PostId="3636" Text="Is the data normally distributed and i.i.d.?" CreationDate="2010-10-15T17:52:45.883" UserId="5" />
  <row Id="4997" PostId="3643" Text="Doesn't the Cauchy apply for zero mean normals?" CreationDate="2010-10-15T17:57:04.253" UserId="251" />
  <row Id="4998" PostId="3644" Text="Taking a quick look at Haaya, it seems that they're concerned with obtaining a Normal approximation for the ratio and use simulations to determine when that applies (using the coefficient of variation, cv).  Does the cv in your case meet the criteria?  If so, the approximations apply." CreationDate="2010-10-15T17:58:56.107" UserId="251" />
  <row Id="4999" PostId="3613" Text="@whuber: Thanks, I usually encounter right censoring or left truncation and rarely anything else, so it's interesting to hear about other domains.  " CreationDate="2010-10-15T18:02:18.800" UserId="251" />
  <row Id="5000" PostId="3643" Text="@ars You are correct. The cauchy then may be of limited use." CreationDate="2010-10-15T18:15:34.753" />
  <row Id="5001" PostId="3643" Text="Ars: Yes, I believe the Cauchy result requires zero means. But that still means that at least in that special case, the variance that David is trying to estimate DOES NOT EXIST." CreationDate="2010-10-15T18:15:35.330" UserId="319" />
  <row Id="5002" PostId="3643" Text="@David: Simply invert y and compute the sample mean and sample standard deviation and use those as estimates of mu and sigma. A normal may approximate y^-1 well if sigma is relatively small." CreationDate="2010-10-15T18:17:24.907" />
  <row Id="5003" PostId="3643" Text="@John: true, good point; I missed David's first comment." CreationDate="2010-10-15T18:25:26.437" UserId="251" />
  <row Id="5005" PostId="3634" Score="1" Text="What is the reason for a non-parametric approach?" CreationDate="2010-10-15T19:01:32.327" UserId="930" />
  <row Id="5006" PostId="3356" Text="+1.  Didn't see all this earlier - Thanks for noting the issue as well as the crash/mini course on stata notation.  I took your first comment to imply the interpretation was mistaken and answered in a very general sense.  I'm glad you were more persistent and that kwak figured it out.  " CreationDate="2010-10-15T19:14:58.587" UserId="251" />
  <row Id="5007" PostId="3336" Text="+1 mystery solved, nicely done. :)" CreationDate="2010-10-15T19:15:43.403" UserId="251" />
  <row Id="5008" PostId="3644" Score="1" Text="@David: use Marsaglia 1965 instead as updated in the answer." CreationDate="2010-10-15T19:18:47.407" UserId="251" />
  <row Id="5009" PostId="3638" Text="@whuber I seem to remember that Median Polish can be used for a two-way layout; how does it extend to 5 factors, including nesting?" CreationDate="2010-10-15T19:19:14.240" UserId="930" />
  <row Id="5010" PostId="3643" Text="@Srikant I can't compute the sample standard deviation since I don't have the raw data... although I could do the calculation on simulated data sets, and take the average of these simulations." CreationDate="2010-10-15T19:27:39.217" UserId="1381" />
  <row Id="5011" PostId="3616" Text="My applogies for the poorly worded question.  By different I mean whether over the course of the time series (rather than at individual points) there is a difference between two treatment groups.  There would be inter-subject variation (which i guess would need to be accounted for) as well as inter-group variation (which is what I am interested in)." CreationDate="2010-10-15T19:32:41.607" UserId="1327" />
  <row Id="5012" PostId="3646" Score="2" Text="It's unclear how &quot;regression&quot; could be invoked in a *univariate* dataset.  Also, exactly what distinction are you suggesting between the two bulleted methods?" CreationDate="2010-10-15T19:35:50.620" UserId="919" />
  <row Id="5013" PostId="3638" Score="1" Text="@chl Median polish can work with as many factors as you might care to handle.  Tukey does some three-way examples (by hand!)." CreationDate="2010-10-15T19:37:15.550" UserId="919" />
  <row Id="5014" PostId="3646" Text="I accept that the question might make very little sense - I admit I'm confused! In terms of regression, I imagine that you would input the formula for the normal distribution into some application that can handle non-linear regression, with the mean and variance being parameters of that formula. You would then let the application attempt to derive the values for those parameters that maximise the fit of your data to the formula. So I see that as a completely different approach than just using the sample mean/variance as estimates." CreationDate="2010-10-15T19:42:59.653" UserId="1598" />
  <row Id="5015" PostId="3641" Score="1" Text="@Srikant: more detailed data of birth rates (by women grouped into five year age ranges) are readily available for exactly the purpose you propose.  It's best to obtain these data for the particular state in question rather than using nationwide averages." CreationDate="2010-10-15T19:44:52.697" UserId="919" />
  <row Id="5016" PostId="3647" Text="This is not a US state forecast. I can get the gender, race, income, birth/death data, for the last 10 years only, would that be enough?" CreationDate="2010-10-15T19:48:50.513" UserId="59" />
  <row Id="5017" PostId="3646" Text="if you have no predictor variables, there would be no way to fit a non-linear model, or any model other than 'data has a mean and variance' " CreationDate="2010-10-15T19:53:16.933" UserId="1381" />
  <row Id="5018" PostId="3647" Text="@dassouki: sorry, I saw the &quot;statewide&quot; on the graphic's title and presumed (incorrectly) that it referred to a US state.  The birth/death data for the last 10 years would be excellent, because it will be a reasonably good predictor of birth/death rates into the near future." CreationDate="2010-10-15T19:58:14.483" UserId="919" />
  <row Id="5019" PostId="3648" Text="Thanks David, I suppose I was thinking of a more elaborate non-linear model, e.g. Y = (whatever it would be if the data is in normally distributed, including things like pi and the two parameters of interest, the mean and variance) + epsilon ? Sorry again if this is meaningless!" CreationDate="2010-10-15T20:01:04.107" UserId="1598" />
  <row Id="5020" PostId="3648" Score="1" Text="Nice example.  It might be helpful to clarify what that regression model is.  You have an intercept only regression--so you have no predictor variables.  This is precisely equivalent to a one-sample t-test for difference from 0. More simply in R: x&lt;-rnorm(100); t.test(x); summary(lm(x~1)).  P values, SEs, estimates, intervals, etc. will all be the same." CreationDate="2010-10-15T20:01:18.460" UserId="485" />
  <row Id="5021" PostId="3646" Score="1" Text="@Bio.X2Y: one of the points your respondents are making is that one &quot;fits...a normal model&quot; to the data by estimating the mean and standard deviation.  These become the mean and sd of the fitted normal distribution.  The other point they are making is that &quot;regression&quot; in your case means performing a least-squares fit of a constant to the data (&quot;no covariates&quot;) and that's exactly the same as using the usual mean and sample standard deviation estimates.  So the short answer is, both approaches are reasonable and they're the same." CreationDate="2010-10-15T20:02:51.163" UserId="919" />
  <row Id="5022" PostId="3648" Score="1" Text="@David I think there's a typo and you invert `se` and `sd`; BTW your `mse` (I assume it is mean square error) is actually the SS." CreationDate="2010-10-15T20:03:24.210" UserId="930" />
  <row Id="5023" PostId="3646" Text="OK, am I right in thinking that the model you're suggesting is a linear one with no predictors, where the error is normally distributed? Isn't that different from a model where you use two predictors for mean and variance, regardless of the error?" CreationDate="2010-10-15T20:09:13.353" UserId="1598" />
  <row Id="5024" PostId="3646" Text="@Bio.X2Y I think you make a confusion between (a) estimating the mean and variance from an observed sample (which is strictly an univariate problem as @whuber said), while being happy with an underlying gaussian assumption, and (b) check how well your distribution fit a theoretical distribution, with unknown mean and variance. Neither of these cases call for a modeling approach. Maybe @David can update his response for (b), otherwise the first one was already suggested to you: just use the arithmetic mean (which is an unbiased estimator) and SD (I let others discuss the denominator issue)." CreationDate="2010-10-15T20:11:50.320" UserId="930" />
  <row Id="5025" PostId="3646" Text="thanks chl. Yes, I think I understand the (a) option (and the n-1 vs n in the denominator), and my confusion lies with the (b) option. I suppose I don't understand why (b) *doesn't* call for a modeling approach - surely since the theoretical distribution has an unknown mean and variance, these have to be 'estimated' before a fit can be established. So the output to (b) would be something like &quot;mean =1, sd=2, r^2=0.94&quot;? Thanks again" CreationDate="2010-10-15T20:19:53.190" UserId="1598" />
  <row Id="5026" PostId="3648" Text="@ chi, Thanks for pointing out my error - I think its fixed" CreationDate="2010-10-15T20:21:03.637" UserId="1381" />
  <row Id="5027" PostId="3644" Text="Thanks for the reference @ars" CreationDate="2010-10-15T20:23:35.160" UserId="1381" />
  <row Id="5028" PostId="3646" Text="@Bio.X2Y No, checking how an empirical distribution departs from an hypothetical/theoretical distribution is not the same than modeling a relationship between an outcome and a potential explanatory variable, or simply fit a regression with only an intercept. In the latest case, it is even better to simply use a Quantile-Quantile plot (observed vs. gaussian) to check if your data follows an expected normal distribution (as is the assumed distribution of the residuals in your LM)." CreationDate="2010-10-15T20:27:11.033" UserId="930" />
  <row Id="5029" PostId="3646" Text="ok, thanks again chl, looks like I have a lot more background reading to do!" CreationDate="2010-10-15T20:28:50.703" UserId="1598" />
  <row Id="5030" PostId="3651" Text="I could concede distributional assumptions for all of the variables. I will update my question." CreationDate="2010-10-15T20:34:14.497" UserId="1036" />
  <row Id="5031" PostId="3651" Text="@Skrikant For the 1st part: Wouldn't it be simply an hypergeometric distribution (and at the limitng case, a binomial)?" CreationDate="2010-10-15T20:39:29.477" UserId="930" />
  <row Id="5032" PostId="3651" Text="@chl True. As the total size of the urn is fixed. Answering a question too fast is not without its perils. :-)" CreationDate="2010-10-15T20:44:05.487" />
  <row Id="5034" PostId="3655" Text="This approach would work if we assume that our CI are of the form &#xA;$\hat{\beta}\pm Z_{\alpha}SE$. Unfortunately, sometimes asymmetric CI may be more sensible, for example the CI for a binomial proportion when it's close to 0. In that case pooling the SE like this may not help. " CreationDate="2010-10-15T20:52:17.863" UserId="1600" />
  <row Id="5035" PostId="1642" Text="@Thomas Andrew Gelman discussed type I and II errors before introducing S and M errors. I think this response is a valid and interesting one (wtr. other well-founded answers) since it allows to go beyond the traditional decision theory framework. I've upvoted this response." CreationDate="2010-10-15T20:56:49.637" UserId="930" />
  <row Id="5038" PostId="3655" Text="@user1600 Good point." CreationDate="2010-10-15T22:14:37.820" />
  <row Id="5039" PostId="3651" Text="@Srikant: &quot;Any&quot; deviation?" CreationDate="2010-10-15T22:58:50.270" UserId="919" />
  <row Id="5040" PostId="3651" Text="@whuber I meant some sort of chi-squared analysis implicitly. If you see the revisions you will see that I started off the answer with that idea. Would it help if I mention that explicitly?" CreationDate="2010-10-15T23:02:18.057" />
  <row Id="5041" PostId="3651" Text="@Srikant: I think it would help, given the elementary nature of the question.  If you like, add something about alternative tests, too.  For example, I suspect a one-sided KS test might be more powerful for the alternative suggested by the OP (&quot;higher frequencies of purple balls are disproportionately drawn&quot;)." CreationDate="2010-10-15T23:21:14.460" UserId="919" />
  <row Id="5042" PostId="3651" Score="1" Text="@whuber Reg chi-square tests: Done. I feel that any mention of KS test and alternatives to chi-square should be added to your answer instead of mine as the idea is yours. " CreationDate="2010-10-16T00:15:09.130" />
  <row Id="5043" PostId="3657" Text="Thank you so much for your response! " CreationDate="2010-10-16T02:37:15.150" UserId="834" />
  <row Id="5044" PostId="3643" Text="The variance is infinite for any normal distribution in the denominator, not just those with zero means. Similarly, the mean is undefined for any normal distribution in the denominator." CreationDate="2010-10-16T03:53:38.443" UserId="159" />
  <row Id="5045" PostId="3582" Text="@Denis. I meant the question you already asked, viz., &quot;If variable kernel widths are often good for kernel regression, why are they generally not good for kernel density estimation?&quot; I would post it myself, but then you would miss out on the rep points." CreationDate="2010-10-16T03:56:40.620" UserId="159" />
  <row Id="5046" PostId="3632" Text="@KalEl re: &quot;use logistic regression to estimate x&quot;.  Logistic regression is not applicable to estimating income (x).  It is for binary dependent variables only." CreationDate="2010-10-16T04:58:01.913" UserId="919" />
  <row Id="5047" PostId="825" Text="whuber, the solution I present works with win 7 and is non commercial (read the post I linked to for details).  It is bundled with a commercial environment but it can be separated from it (as my post shows how).  And the code itself is GPL..." CreationDate="2010-10-16T08:44:26.107" UserId="253" />
  <row Id="5048" PostId="3596" Text="beautiful link, thanks Jeromy." CreationDate="2010-10-16T08:45:48.810" UserId="253" />
  <row Id="5049" PostId="2952" Text="(My pleasure chl - thank you :) )" CreationDate="2010-10-16T08:47:09.363" UserId="253" />
  <row Id="5050" PostId="3660" Text="@Pedro Great... I wasn't aware of that!" CreationDate="2010-10-16T08:47:16.910" UserId="930" />
  <row Id="5051" PostId="3272" Text="Wonderful, Wonderful(!), answer!  Thank you Drury, I'll go through it a few more times to see what I can introduce to my teachings." CreationDate="2010-10-16T08:55:07.327" UserId="253" />
  <row Id="5052" PostId="3664" Score="1" Text="@onestop (+1) Thanks for mentioning the BA plot. You might be interested in an earlier response I made on a similar topic, http://stats.stackexchange.com/questions/527/what-ways-are-there-to-show-two-analytical-methods-are-equivalent/2834." CreationDate="2010-10-16T08:55:35.453" UserId="930" />
  <row Id="5053" PostId="3662" Text="@Stefania Thanks for the additional info. I was asking that because often people look for non-parametric alternatives as if it were to be an universal way to go, although the NP approach carries on its own assumption (e.g. equality of variance you cited). BTW you can just update your original question, instead of putting it as a response." CreationDate="2010-10-16T09:04:45.620" UserId="930" />
  <row Id="5054" PostId="3597" Text="@Whuber: are you on windows or a *nix box (mac, linux,...)" CreationDate="2010-10-16T09:07:21.790" UserId="603" />
  <row Id="5055" PostId="3508" Score="1" Text="The enclosed pictures were my first attempt at using Asymptote (http://j.mp/c8XUGq) instead of Metapost :-) Very sad idea, but I can share the code if you like." CreationDate="2010-10-16T09:10:05.467" UserId="930" />
  <row Id="5056" PostId="3508" Text="@chl: That's impressive; please share.  :)" CreationDate="2010-10-16T10:09:15.813" UserId="251" />
  <row Id="5057" PostId="3508" Score="3" Text="@ars Here it is (as Gist): http://gist.github.com/629642, http://gist.github.com/629644, http://gist.github.com/629645." CreationDate="2010-10-16T10:20:15.527" UserId="930" />
  <row Id="5058" PostId="3664" Text="Thanks chl, that was before my time here." CreationDate="2010-10-16T11:56:45.900" UserId="449" />
  <row Id="5059" PostId="3590" Text="@Seb Thanks for that additional information!" CreationDate="2010-10-16T13:31:18.817" UserId="930" />
  <row Id="5060" PostId="3597" Text="@kwak: Win 7.  (Thanks for asking.)" CreationDate="2010-10-16T14:15:15.070" UserId="919" />
  <row Id="5061" PostId="3666" Text="@Christopher (+1) I can imagine that with a 1400+ pages book the authors offer several chapters to AN(C)OVA :) BTW, there are SAS and Stata code for most of the chapters on UCLA, http://www.ats.ucla.edu/stat/sas/examples/alsm/" CreationDate="2010-10-16T16:18:15.870" UserId="930" />
  <row Id="5062" PostId="3508" Text="@chl: thank you!" CreationDate="2010-10-16T17:52:28.743" UserId="251" />
  <row Id="5063" PostId="3657" Text="Hello Sarah, you should close this question if you think it is answered. " CreationDate="2010-10-16T21:14:50.140" UserId="1307" />
  <row Id="5064" PostId="3664" Text="onestop and chi - thanks so much for the detailed explanations an dthe refs. I will try to go through the analyses as you described, and if I have more questions, I might pester you again - thanks again for the fast reply!" CreationDate="2010-10-16T21:56:14.883" UserId="1603" />
  <row Id="5065" PostId="3666" Text="Indeed, there are several chapters. I want to say that about half the book is dedicated to AN(C)OVA, while the first half is regression, so that's about 700 pages of analysis of variance. There are parts of the text (block designs, nested designs) that I felt were incredibly boring, and could've used some more work, but the regression sections were great." CreationDate="2010-10-17T00:08:13.107" UserId="1118" />
  <row Id="5066" PostId="3655" Text="This answer could be applied to any two distributions, it is just that the product of normals is a normal, giving a nice solution. MCMC simulation could be used with pairs of distributions without a closed form solution, using a Bayesian approach with one sample being the prior and the other the likelihood." CreationDate="2010-10-17T04:38:32.450" UserId="1381" />
  <row Id="5067" PostId="3671" Score="2" Text="@Christian The use of circular graphics for tabular data with Circos has been evoked here too, http://stats.stackexchange.com/questions/3158/what-is-this-type-of-circular-link-visualization-called/3159. My opinion is that this is a good way to reduce large symmetric tables, but it is less useful for summary tables with p-values and the like." CreationDate="2010-10-17T07:46:22.733" UserId="930" />
  <row Id="5068" PostId="422" Score="2" Text="I found Statistics in a Nutshell to be seriously flawed in terms of wrong/missing figures, mistakes in formulas, bad explanations and the book doesn't even have tables for critical values. This is especially bad at places where the authors write &quot;and since the critical value for this is foo, this is significant&quot;, leaving the reader totally unclear about where this foo value comes from. The book does have a good intro section but should be edited eventually to make it good. Just look at the errata page for the book and be stunned at all the errors." CreationDate="2010-10-17T09:07:29.810" UserId="1048" />
  <row Id="5069" PostId="3597" Text="@Whuber:&gt; there is a problem with winbox: it's not POSIX compliant, as a consequence there is no high prec. timing. This makes it difficult to do things like parallel generation of pseudo random numbers or organisation of tasks. *nix boxes are POSIX compliant. In *nix boxes all cores (as well as the GPU if it's an nvdia) can be easily accessed from within R thru packages such as multicore, gputools, magma,... (which you install as regular packages)." CreationDate="2010-10-17T10:40:55.403" UserId="603" />
  <row Id="5070" PostId="3597" Text=" I would advise you to install ubuntu (google 'download ubuntu') and to run your windows-only apps from within ubuntu via virtualbox (http://www.youtube.com/watch?v=KXgKnd-u2R4). R and latex editors run on ubuntu like a charm." CreationDate="2010-10-17T10:41:39.630" UserId="603" />
  <row Id="5071" PostId="3665" Text="Thanks. &quot;You should always aim to solve the problem at hand directly, rather than use a more general method and post-process the result&quot; - I was wondering if you have any intuitive/mathematical justification of the statement? Will be especially helpful with the current context." CreationDate="2010-10-17T11:14:29.023" UserId="994" />
  <row Id="5072" PostId="3672" Score="1" Text="A bit more details could be helpful; how they are obtained, are branches weighted?" CreationDate="2010-10-17T11:47:41.723" UserId="88" />
  <row Id="5073" PostId="3672" Text="Let me give you an example. I wrote an algo to cluster variables. Clustering can be represented by a dendrogram. This dendrogram changes over time (based on time series). The structure of the dendrogram slowly evolves and I am look for a measure to describe it (sort of descriptive statistics for dendrograms)." CreationDate="2010-10-17T11:59:04.740" UserId="1250" />
  <row Id="5075" PostId="3597" Text="@kwak: Many thanks for the suggestions and the information." CreationDate="2010-10-17T14:44:25.973" UserId="919" />
  <row Id="5076" PostId="3674" Score="2" Text="@mbq In R there is the `profdpm` package, and an overview of available indices (incl. Fowlkes &amp; Mallows's $B_k$) to compare HCs is available here, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.6189&amp;rep=rep1&amp;type=pdf." CreationDate="2010-10-17T16:09:08.590" UserId="930" />
  <row Id="5078" PostId="3683" Text="Is there an approachable textbook you could recommend that discusses how and why Manhattan Distances are useful in statistics?" CreationDate="2010-10-17T20:44:18.923" UserId="1515" />
  <row Id="5079" PostId="3684" Text="Of course, for discrete distributions, including distributions of finite support (like those in question here), the cf is just the probability generating function evaluated at x = exp(i t), making it a more complicated way of encoding the same information." CreationDate="2010-10-17T21:24:03.630" UserId="919" />
  <row Id="5080" PostId="3626" Text="Argh, yes, I realised this after some guys at work raised some objections about the categorical variables (interestingly, they didn't complain about the continuous variables!). The problem is even more complicated than I outlined, so I need to think about things a bit more." CreationDate="2010-10-17T21:51:02.387" UserId="1144" />
  <row Id="5081" PostId="3683" Score="1" Text="@Kaelin: Unfortunately I can't think of a text which discusses this in particular.  I can tell you that the L1 distance is preferred since it's less sensitive to outliers.  It's also related to distances between empirical distributions in probability theory ( L1 is twice the &quot;total variation distance&quot;: http://en.wikipedia.org/wiki/Total_variation_distance )." CreationDate="2010-10-17T22:00:39.113" UserId="251" />
  <row Id="5082" PostId="3687" Text="+1. Though I'm not sure an analysis book, even if it is Rudin, is &quot;approachable&quot;.  ;-)" CreationDate="2010-10-18T00:35:59.303" UserId="251" />
  <row Id="5086" PostId="3640" Text="david - since your variables are all positive, why do you want to fuss with $\mu = 0$?  btw - in your simulation, you seem to be generating variables per.c and per.n that are independent. is that correct - and if so, is that what you want?" CreationDate="2010-10-18T02:25:53.617" UserId="1112" />
  <row Id="5087" PostId="3684" Score="1" Text="@whuber: As you say, the cf, mgf, and pgf are more-or-less the same and easily transformable into one another, however Mathematica has a cf builtin that works with all the probability distributions it knows about, whereas it doesn't have a pgf builtin. This makes the Mathematica code for working with sums (and differences) of dice using cfs particularly elegant to construct, regardless of the complexity of dice expression as I hope I demonstrated above. Plus, it doesn't hurt to know how cfs, FTs, convolutions, and cross-correlations can help solve problems like this." CreationDate="2010-10-18T02:38:02.587" />
  <row Id="5088" PostId="3643" Text="@rob - how do you define the variance of Y = 1/X when X is normal, as EY is undefined?" CreationDate="2010-10-18T03:26:29.073" UserId="1112" />
  <row Id="5089" PostId="3684" Text="@Elisha: Good points, all of the them.  I guess what I wonder about the most is whether your ten or so lines of Mathematica code are really more &quot;elegant&quot; or efficient than the single line I proposed earlier (or the even shorter line Srikant fed to Wolfram Alpha).  I suspect the internal manipulations with characteristic functions are far more arduous than the simple convolutions needed to multiply polynomials.  Certainly the latter are easier to implement in most other software environments, as Glen_b's answer indicates.  The advantage of your approach is its greater generality." CreationDate="2010-10-18T03:35:44.683" UserId="919" />
  <row Id="5090" PostId="3687" Text="@ars: Yes, but I don't know of any one that really is.  That's why I pointed out the two Wikipedia articles." CreationDate="2010-10-18T03:37:59.607" UserId="919" />
  <row Id="5091" PostId="3643" Text="@ronaf. Good point. E[Y^2] is infinite whenever X is normal. The same goes for E[Y^m] for any even m. If m is odd, the result is undefined. Is that better?" CreationDate="2010-10-18T03:57:29.553" UserId="159" />
  <row Id="5092" PostId="3685" Text="I have also wondered about this problem, but (unfortunately) haven't found any convincing answers yet. I think there is no solution. There are R/BioC packages like `hopack` (and others) which can estimate the number of clusters, but that doesn't answer your question. " CreationDate="2010-10-18T04:02:09.440" UserId="1307" />
  <row Id="5093" PostId="3687" Text="I know, I liked it -- it's the right recommendation to make in case the OP wants to dig deeper." CreationDate="2010-10-18T04:12:20.807" UserId="251" />
  <row Id="5094" PostId="3618" Text="@Srikant: Expand[Sum[x^i,{i,1,6}]^3] also works in WolframAlpha" CreationDate="2010-10-18T07:04:52.270" />
  <row Id="5096" PostId="3696" Text="Great answer, thanks. As I now used the solution or ars I will accept his as the accepted answer." CreationDate="2010-10-18T09:21:51.507" UserId="190" />
  <row Id="5097" PostId="3697" Text="+1 Yup, this is the way it is mostly done." CreationDate="2010-10-18T09:44:20.690" UserId="88" />
  <row Id="5098" PostId="3698" Text="Hmm, that's what I get too: |O-E|/sqrt(E) = |17-16.5|/16.5 = 0.123. Can you add a link to the paper if it's available on the web?" CreationDate="2010-10-18T09:57:08.440" UserId="449" />
  <row Id="5100" PostId="3698" Text="Unfortunately it isn't. I have collected comparable data and wish to compare my findings with the publication. I have found that the R function prop.test gives results that approximately agree with the paper, but I wanted to understand how the author arrives at his exact numbers." CreationDate="2010-10-18T10:11:34.200" UserId="1614" />
  <row Id="5101" PostId="3698" Text="I've contacted the author. Perhaps he can shed light on the problem." CreationDate="2010-10-18T11:27:15.950" UserId="1614" />
  <row Id="5102" PostId="3636" Text="@whuber - the std dev would give me a number that tells me how much the temperature is varying." CreationDate="2010-10-18T12:17:21.790" UserId="1595" />
  <row Id="5103" PostId="3636" Text="@Shane - It's a temperature reading.  How would I know if it is normally distributed?  It should be i.i.d., but I'm just a lowly engineer trying to implement something management wants." CreationDate="2010-10-18T12:19:56.933" UserId="1595" />
  <row Id="5104" PostId="3698" Text="Are these paired proportions or two independent samples?" CreationDate="2010-10-18T12:32:14.880" UserId="930" />
  <row Id="5106" PostId="3701" Text="That was my understanding and like I say, the prop.test function of R gives the same direction of results. I just wondered if there was something that I was doing wrong to not get the z=.08." CreationDate="2010-10-18T12:55:33.780" UserId="1614" />
  <row Id="5109" PostId="3690" Text="Thanks for the answer! However, I'm interested in modeling individuals within groups where group-level effects are difficult to quantify or anomalous and can't easily be included in future predictions. To alter your example a little, suppose we're predicting minute usage for individual customers over the next month using the last three month's of data. Suppose that while customer age is a significant predictor of usage, a local concert last month made young users unusually likely to make phone calls. The 'age' coefficient may be unnaturally large and skew predictions. Does this make sense?" CreationDate="2010-10-18T13:20:46.280" UserId="1611" />
  <row Id="5110" PostId="3702" Score="2" Text="Could you try formating your data so that we can get a clearer idea of what you want. For example, I don't see an 8x11 matrix in your example." CreationDate="2010-10-18T13:52:17.997" UserId="8" />
  <row Id="5111" PostId="3702" Score="1" Text="More of a SO question, I think." CreationDate="2010-10-18T14:07:00.170" UserId="144" />
  <row Id="5112" PostId="3702" Text="Thanks for quick reply, I am trying format matrix in this windows but quit difficult. I describe my matrix in word: the 8x11 matrix comprise of 8 rows corresponding to ages (1, 1.25,1.5,1.75,2,3,4,5) and 11 columns (first column name &quot;age&quot;, then boy01...boy05, girl01...girl05) the values of matrix is the height of 5 boys and 5 girls." CreationDate="2010-10-18T14:07:04.090" UserId="1615" />
  <row Id="5113" PostId="3669" Text="Thanks. I will pose another question as I am not a stats expert and am not sure my question was clear. " CreationDate="2010-10-18T14:11:12.973" UserId="834" />
  <row Id="5114" PostId="3702" Text="I do not know how to present the height of group boy as &quot;$hgtm&quot;, group girl as &quot;hgtf&quot; and age column as &quot;$age&quot; as display above." CreationDate="2010-10-18T14:15:34.960" UserId="1615" />
  <row Id="5115" PostId="3690" Text="@danpelota That is an example of 'interactions'. I will edit the answer to clarify." CreationDate="2010-10-18T14:16:28.513" />
  <row Id="5116" PostId="3703" Text="Thank you very much for your kind help." CreationDate="2010-10-18T14:23:39.807" UserId="1615" />
  <row Id="5117" PostId="3702" Text="At this point, I'm a little confused about the data structure too.  Maybe describe what you're trying to do?  You have a set of boys and girls, along with their respective heights and ages?  Are the heights measured at different ages?" CreationDate="2010-10-18T14:29:14.443" UserId="5" />
  <row Id="5118" PostId="3673" Score="1" Text="Thanks chl. I think the problem is that many of the answers were too good - e.g. a search for &quot;patient cluster analysis&quot; in Google scholar returns 650,000 results. So rather than saying &quot;here are some good directions to go in,&quot; I wanted a more discrete bibliography." CreationDate="2010-10-18T14:39:03.467" UserId="900" />
  <row Id="5119" PostId="3702" Score="1" Text="As it hasn't been mentioned before, there are *introductory manuals* that come with R. The *Introduction* will teach you the basics of data structure use (and more); the *Data Import/Export* is helpful for data transfer questions." CreationDate="2010-10-18T14:44:16.050" UserId="334" />
  <row Id="5120" PostId="3704" Score="1" Text="Please explain why you're not satisfied with suncoolsu's answer to your previous version of the same question http://stats.stackexchange.com/q/3653/449.&#xA;Also please indicate what software the code in your question is for, and add an appropriate tag. That way, someone with experience in that software package might be able to suggest explicit code." CreationDate="2010-10-18T14:52:05.733" UserId="449" />
  <row Id="5121" PostId="3657" Text="Hi - Thanks again for your answer. I forgot to mention that I am using Stata. When I add two coefficients together (using the output from Stata), can I also just add the standard errors? If so, then I should be able to obtain the standard errors by dividing the sum of the coefficients by the sum of the standard errors. Do you agree? Thanks again. " CreationDate="2010-10-18T15:13:34.937" UserId="834" />
  <row Id="5122" PostId="3704" Text="Sorry for posting a second question. For some reason I can't figure out how to edit my question. I'm new to the site so please bear with me. And thanks again to those who have provided answers so far!" CreationDate="2010-10-18T15:15:52.340" UserId="834" />
  <row Id="5123" PostId="3708" Text="What are the geographic units and do you expect them to have any geographic dependency (i.e. spatial autocorrelation?) Although we would always like to have more data you have a reasonable amount of observations to project estimates." CreationDate="2010-10-18T15:28:59.090" UserId="1036" />
  <row Id="5124" PostId="3709" Score="2" Text="It seems dassouki standardized the distributions, and that is why it ranges between -2 to 4." CreationDate="2010-10-18T15:30:18.243" UserId="1036" />
  <row Id="5125" PostId="3707" Text="Since their is some confusion over what we are seeing can you state how you standardized the distributions and why you standardized the distributions? Also some greater context as to your motivation might be nice although not necessary." CreationDate="2010-10-18T15:33:28.003" UserId="1036" />
  <row Id="5126" PostId="3708" Text="The 100 regions belong to the same country, so I expect that they are correlated in time and in space (in fact I began with a correlation clustering exercise that shows that they are correlated)..." CreationDate="2010-10-18T15:35:34.960" UserId="1443" />
  <row Id="5127" PostId="3657" Text="Sarah, In Stata, use the 'lincom' function. Suppose you have variables var1 and var2 and want to add 3 times the coefficient on var1 and 2 times the coefficient on var2. Type 'lincom 3 * var1 + 2 * var2'. This gives the standard error and confidence interval for this estimate." CreationDate="2010-10-18T15:42:56.263" UserId="401" />
  <row Id="5128" PostId="3708" Text="What is the frequency (i.e. is it weekly,monthly, quarterly, annual data) ?" CreationDate="2010-10-18T15:44:28.393" UserId="603" />
  <row Id="5129" PostId="3711" Text="I'm not sure this is right. U^n corresponds to multiplying the *same* random value by itself n times. dassouki's procedure will multiply n *different* random values. As an example, the product distribution of two standard normal r.v.'s is *not* chi-squared on 2 d.f : http://dx.doi.org/10.1137%2F0118065" CreationDate="2010-10-18T15:45:34.117" UserId="449" />
  <row Id="5130" PostId="3708" Text="Annual data, so I have only 9 points per region..." CreationDate="2010-10-18T15:45:52.540" UserId="1443" />
  <row Id="5131" PostId="3706" Score="1" Text="@sarah I did not downvote but you should see the other thread as to why this is not ok. " CreationDate="2010-10-18T15:47:05.240" />
  <row Id="5132" PostId="3704" Score="1" Text="@sarah You can edit the question by clicking 'edit' which appears below the tags for your question. If some aspect of suncoolsu's answer is not clear you should ask for clarification via commenting to the answer. There is a 'add comment' link below every answer." CreationDate="2010-10-18T15:49:16.903" />
  <row Id="5133" PostId="3711" Text="@onestop hmm your are right but I think the reasoning carries over to the general case. I will think about it and fix the answer." CreationDate="2010-10-18T15:50:32.890" />
  <row Id="5134" PostId="3712" Text="Thanks so much Charlie. This is very useful. I didn't know about the lincom command so this will save me lots of time. However, if I want to get the impact of treatment on the outcome for quintile 2 should i not add treat + treatXquin2 (rather than adding quintile2 + treatXquin2)? " CreationDate="2010-10-18T15:53:39.783" UserId="834" />
  <row Id="5135" PostId="3704" Text="Thanks Srikant. I appreciate your help. " CreationDate="2010-10-18T15:57:39.703" UserId="834" />
  <row Id="5136" PostId="3711" Text="@Srikant: The conclusion about convergence to a delta function isn't terribly useful or insightful.  (Analogously, if one weren't to adopt the right standardization of sums of random variables, the CLT would vanish into thin air: you would often just conclude that the sums &quot;converge&quot; to an improper uniform distribution over all the reals!)  A better way to investigate such phenomena is to seek some standardization that assures convergence to something non-trivial if that is at all possible." CreationDate="2010-10-18T16:03:29.143" UserId="919" />
  <row Id="5137" PostId="3705" Text="Many thanks for your nice commends" CreationDate="2010-10-18T16:04:56.673" UserId="1615" />
  <row Id="5138" PostId="3709" Text="Yes as Andy said. Isn't that what I was supposed to do ?" CreationDate="2010-10-18T16:06:32.507" UserId="59" />
  <row Id="5140" PostId="3711" Score="2" Text="@Srikant: The distribution of the product of n uniforms is the exponential of a (negated) Gamma(n) variate." CreationDate="2010-10-18T16:08:16.203" UserId="919" />
  <row Id="5141" PostId="3709" Text="I don't know what you are _supoposed_ to do, because I don't kow what you are _trying_ to do. That is why others suggested explaining the motivation..." CreationDate="2010-10-18T16:13:55.183" UserId="247" />
  <row Id="5142" PostId="3711" Text="@whuber You are correct about standardization but I did not interpret the question that way (or at least it was not clear from the question context that the OP wanted to know the behavior of the product of different standardized uniform random variates. Thanks for the pointer on the product of n uniforms. I will update the answer to reflect that info." CreationDate="2010-10-18T16:14:29.067" />
  <row Id="5143" PostId="3709" Text="What I can say, is that you are reporting a plot and some code (i.e. the rand()*rand()...), but the code you supplied can not possibly create the plot you display. In other words, you need to supply more details in order to get useful help." CreationDate="2010-10-18T16:15:21.647" UserId="247" />
  <row Id="5144" PostId="3712" Text="Charlie - I'm not sure I understand why you multiply treatXquin2 by 1 for yes. If that is the case then how do I obtain the estimate for the untreated (i.e. treat==0)? It doesn't seem to make sense that I would multiply treatXquin2 by 0. Any help would be much appreciated! " CreationDate="2010-10-18T16:17:59.827" UserId="834" />
  <row Id="5145" PostId="3647" Text="@Whuber, I just discovered that the data I have covers 2 periods in the last 10 years: 2 counts at -5 and -10 years" CreationDate="2010-10-18T16:42:34.627" UserId="59" />
  <row Id="5146" PostId="3640" Text="no, I don't want to fuss with $\mu$ = 0; these variables are generally treated as independent, and covariance data is rarely available. Since C is fairly constant, independence is a reasonable assumption." CreationDate="2010-10-18T16:47:19.380" UserId="1381" />
  <row Id="5147" PostId="3709" Score="3" Text="@PeterR In my opinion, your response may have been better as a comment. " CreationDate="2010-10-18T17:06:40.400" />
  <row Id="5148" PostId="3705" Score="2" Text="Another point in favor of the second layout is that it is consistent with good database practices. The data normalization advantages of databases and a knowledge of SQL is very useful if used appropriately." CreationDate="2010-10-18T17:10:43.347" />
  <row Id="5150" PostId="3658" Score="1" Text="Thanks for the response. Don't feel obligated to update anything (as it is not your job to teach me mathematics), but I do not have any clue how you &quot;integrate over the prior distributions&quot;. Does it involve just calculating all possible combinations since my distributions are uniform?" CreationDate="2010-10-18T18:10:49.120" UserId="1036" />
  <row Id="5151" PostId="3705" Text="@Srikant: Thanks, I've updated my post to include." CreationDate="2010-10-18T18:20:47.620" UserId="485" />
  <row Id="5152" PostId="3711" Text="@Srikant: Yes, there's always opportunity for multiple interpretations.  I took the OP's careful investigation, including the standardization and graphing, as an indication of a deeper and more sophisticated perception of the phenomenon than just being something &quot;getting to zero&quot; or converging to a delta distribution.  The ultimate arbiter of the question's intent is its proposer, of course." CreationDate="2010-10-18T18:25:21.273" UserId="919" />
  <row Id="5154" PostId="3658" Text="Thank you again, that certainly answers my question." CreationDate="2010-10-18T18:44:12.003" UserId="1036" />
  <row Id="5155" PostId="3715" Score="1" Text="As I understand the [Agresti-Coull](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Agresti-Coull_Interval) procedure is used to generate approximate confidence intervals for the binomial proportion confidence interval. I am not sure I see the relationship to Bayes theorem. Could you provide some context and your goals within the context to understand how the bayes arises in the context of the Agresti-Coull method?" CreationDate="2010-10-18T18:55:43.290" />
  <row Id="5157" PostId="3717" Score="2" Text="@Andy Perhaps, spatial dependencies exist because employers tend to co-locate in certain geographical areas which has spill-over effects to neighboring areas? For example, during the IT bust of 2000 I am sure several regions around the Silicon valley would have had high unemployment rates but perhaps not in Detroit (which is dominated by the auto industry)." CreationDate="2010-10-18T18:58:43.033" />
  <row Id="5159" PostId="3717" Text="@Srikant you are right. I was thinking about it some more and spatial externalities are much more likely to occur in economic data than in crime data I am used to working with. Although I would still be skeptical with a lag of a year, it deserves more attention than my answer would suggest." CreationDate="2010-10-18T19:02:11.980" UserId="1036" />
  <row Id="5161" PostId="3717" Text="The effects that Srikant has highlighted cannot be ignored to my mind, these need to be taken into account..." CreationDate="2010-10-18T19:10:58.563" UserId="1443" />
  <row Id="5163" PostId="3717" Text="@Srilant:&gt; what you see is correct, and J. Hamilton has done some work on economic contagion between US states. But these sorts of model need more than 9 observations per cross section to be estimated. For this types of situation, i recommend the A/B estimator (see my answer) since it has been designed, *precisely* for these types of situation (small **T** large **N**)." CreationDate="2010-10-18T19:21:28.273" UserId="603" />
  <row Id="5164" PostId="3715" Text="@cool-RR: How exactly are you estimating those three probabilities?" CreationDate="2010-10-18T19:50:26.933" UserId="919" />
  <row Id="5165" PostId="3692" Score="1" Text="Any reason for the downvote? It would be fair and constructive to point the author and the readers to potential mistakes, forgetfulness, or whatever." CreationDate="2010-10-18T20:56:34.100" UserId="930" />
  <row Id="5166" PostId="2214" Text="My understanding is that you need a finite mean to for a population L-*moment* to be defined, but not for the estimands corresponding to all other L-estimators. For example, the sample median is an L-estimator, though it isn't an L-moment." CreationDate="2010-10-18T20:59:37.130" UserId="449" />
  <row Id="5167" PostId="3715" Text="@whuber: For $P(A)$ I check how many of the messages in my finite sample have $A$, and divide that by the size of the sample. For $P(B|A)$ I check how many of the messages that have $A$ also have $B$, and divide that by the number of messages that have $A$. etc." CreationDate="2010-10-18T21:01:36.793" UserId="1523" />
  <row Id="5168" PostId="3715" Score="1" Text="@cool-RR: why don't you just estimate P(A|B) directly, then, as the fraction of B samples that are also A?" CreationDate="2010-10-18T21:03:13.997" UserId="919" />
  <row Id="5170" PostId="3715" Text="@Srikant: Agresti-Coull lets you draw conclusion about the true proportion from a small finite sample, and gives you a confidence interval with an answer. I have a similar situation: I have a small finite sample and I want to get the value of $P(A|B)$ along with a confidence interval. To give more context: I'm developing something like a Bayesian spam filter, and this is the probability that a message is spam given that it has a certain word." CreationDate="2010-10-18T21:05:01.823" UserId="1523" />
  <row Id="5171" PostId="3725" Score="8" Text="VAR is unbiased for any distribution with a finite variance, not just for the normal distribution." CreationDate="2010-10-18T21:08:53.843" UserId="159" />
  <row Id="5173" PostId="3717" Score="4" Text="I wouldn't use a univariate time series method with 9 observations." CreationDate="2010-10-18T21:12:58.030" UserId="159" />
  <row Id="5174" PostId="3673" Text="(+1) Thanks for the precision. It's easier to edit your original question and ask for dedicated references, otherwise some of us may be confused; without any upvote we cannot know whether responses were useful or not, nor improve them. Good luck anyway!" CreationDate="2010-10-18T21:19:10.800" UserId="930" />
  <row Id="5175" PostId="3725" Score="1" Text="Thanks Rob.  That condition was indeed unnecessary." CreationDate="2010-10-18T21:41:27.963" UserId="919" />
  <row Id="5176" PostId="3720" Text="Given that the unemployment rate must be strictly positive, I guess, for simulation purposes, I could model the log returns for example. Is this reasonable?" CreationDate="2010-10-18T21:49:28.387" UserId="1443" />
  <row Id="5177" PostId="3724" Text="@onestop The &quot;statistical-bias&quot; tag is somewhat confusing here." CreationDate="2010-10-18T21:57:23.987" UserId="930" />
  <row Id="5179" PostId="3715" Text="Wow @whuber, you blew my mind. I have no idea why I've been doing this. I got the impression this is what people do on Bayesian spam filters, so that's what I've been doing. ($P(A|B)$ is the probability a message is spam given that it contains a certain word, for example 'cheese'.) Now I tried to simply calculate $P(A|B)$ directly, and it seems to work, so I don't even need the answer to this question. Unless I'm missing something. Maybe you have an idea why people use Bayes' Theorem in Bayesian spam filtering when filtering by a single word like this? " CreationDate="2010-10-18T22:04:44.817" UserId="1523" />
  <row Id="5180" PostId="3712" Text="Sorry, I misread your question. Add treat + treatXquin2 for the treatment impact for those in quintile 2. I have updated my answer." CreationDate="2010-10-18T22:05:24.367" UserId="401" />
  <row Id="5181" PostId="3724" Text="Perhaps you're right. The guidelines suggest updating tags in light of the the answers as well as the original question, so I was wondering if there was a tag for 'unbiased estimator' or 'unbiasedness', but there wasn't, so i used 'statistical bias' on the grounds that separate tags for flip sides of the same coin seem a bit unecessary, and the guidelines favor reuse of existing tags over creation of new ones. I'm about to fall sleep but I'll take another look at this in the morning." CreationDate="2010-10-18T22:13:36.427" UserId="449" />
  <row Id="5182" PostId="3706" Text="Sorry, I downvoted this answer to indicate I disagreed, but that was an inapproriate and lazy way of doing so, for which I apologise. I'm afraid it's now too late to undo my vote, however - the software won't allow it, for reasons I don't entirely understand. I've just upvoted your question in an attempt to balance things out, though I realise two wrongs don't make a right." CreationDate="2010-10-18T22:21:23.707" UserId="449" />
  <row Id="5183" PostId="3725" Text="Right, whatever variance mean when the distribution is not close to normal ;-)" CreationDate="2010-10-18T22:26:47.270" UserId="88" />
  <row Id="5184" PostId="3724" Text="I changed statistical-bias to unbiased-estimator. Does that work?" CreationDate="2010-10-19T00:14:47.540" />
  <row Id="5185" PostId="3731" Score="1" Text="No. Pearson's correlation does NOT assume normality. It is an estimate of the correlation between any two continuous random variables and is a consistent estimator under relatively general conditions. Even tests based on Pearson's correlation do not require normality if the samples are large enough because of the CLT." CreationDate="2010-10-19T01:46:20.777" UserId="159" />
  <row Id="5186" PostId="3731" Score="2" Text="I am under the impression that Pearson is defined as long as the underlying distributions have finite variances and covariances. So, normality is *not* required. If the underlying distributions are not normal then the test-statistic may have a different distribution but that is a secondary issue and not relevant to the question at hand. Is that not so?  " CreationDate="2010-10-19T01:47:13.190" />
  <row Id="5187" PostId="3731" Text="@Rob, @Srikant: True, I was thinking of significance testing.  " CreationDate="2010-10-19T01:59:28.437" UserId="251" />
  <row Id="5188" PostId="2214" Text="Somehow my thumb bumped this dad-blamed touchpad and I voted down on this question?  Now it won't let me un-downvote unless the question is edited?  Would somebody please consider changing a comma somewhere so I can fix it?" CreationDate="2010-10-19T02:07:40.147" UserId="1108" />
  <row Id="5189" PostId="3734" Score="3" Text="Good question, although my immediate reaction, backed up by my limited experience of teaching this, is that the CLT isn't initially at all intuitive to most people. If anything, it's counter-intuitive!" CreationDate="2010-10-19T02:39:52.323" UserId="449" />
  <row Id="5190" PostId="3008" Text="Great code but there is a problem. I am not as smart as you are. I need it broken down stepwise. I take it is runs the number of simulations? What is nn? Is it the number of subjects in the study? Then I see you created a distribution of covariates and made them determine a yes or a no depending on a threshold. " CreationDate="2010-10-19T03:03:51.900" UserId="104" />
  <row Id="5191" PostId="3643" Text="@rob + @david  if the variables in the ratio are *asymptotically* normal [with a non-zero mean in the denominator] and also consistent, the ratio is also asymptotically normal - as the usual delta method shows. perhaps that will render moot the discussion about non-existence of moments." CreationDate="2010-10-19T03:15:10.907" UserId="1112" />
  <row Id="5192" PostId="3643" Text="@ronaf. Actually, that doesn't help. It is possible to have asymptotic normality as well as having non-existent moments. Asymptotics with probability distributions can have weird properties." CreationDate="2010-10-19T03:16:36.970" UserId="159" />
  <row Id="5193" PostId="3734" Text="@onestop AMEN! staring at the binomial distribution with *p* = 1/2 as n increases does show the CLT is lurking - but the intuition for it has always escaped me. " CreationDate="2010-10-19T03:18:40.243" UserId="1112" />
  <row Id="5194" PostId="3725" Score="2" Text="@mbq to some, the variance is a moment of inertia. there is also chebyshev's inequality, that applies to any distribution. it also has an interpretation similar to that for the normal for any location/scale family with a finite second moment - like the logistic, for example. [the pdfs need not be symmetric, tho.] " CreationDate="2010-10-19T03:34:14.603" UserId="1112" />
  <row Id="5195" PostId="3643" Text="@rob - i have in mind the ratio of two sample means. even tho the actual moments may be infinite or undefined, there is an asymptotic mean and asymptotic variance that go with the limiting normal distribution [again assuming the denominator is not consistently estimating zero]. it is often the parameters of the asymptotic distribution that are relevant for analyses of the data [judging by the way in which many statistical analyses are carried out these days]. in that case, the non-existence of actual moments is a side issue [or a quibble?]." CreationDate="2010-10-19T03:44:15.610" UserId="1112" />
  <row Id="5196" PostId="3725" Text="altho, as whuber points out, there are two excel functions for variance, if my recollection is correct, there is only one for covariance - and it divides by n, not n-1. [i haven't checked the latest version of excel, tho. can someone tell us if that is still the case?]" CreationDate="2010-10-19T03:56:00.090" UserId="1112" />
  <row Id="5197" PostId="3643" Text="@ronaf. I don't really follow your last comment. The asymptotic normal distribution will have a well-defined mean and variance. But they are not the same as the true asymptotic mean and variance which are undefined whenever the denominator has a non-zero density at 0. In practice, this may not matter. Imagine, for example, if the denominator is N(100,1). Then the sample values of the ratio will behave nicely with very high probability." CreationDate="2010-10-19T04:26:18.347" UserId="159" />
  <row Id="5198" PostId="3720" Text="@kwak, could you provide a link to the Hamilton work you talked about in a comment to my answer. Also do you know of any work that used the A/B estimator and included spatial effects in the models?" CreationDate="2010-10-19T04:40:45.767" UserId="1036" />
  <row Id="5201" PostId="3731" Text="@Srikant: I'm not sure it's a &quot;secondary issue&quot;. You can compute anything after all -- it's the inference that matters. @Rob: your &quot;if&quot; qualifier is key here -- it seems to me that's central to this question. We can justify a whole lot with asymptotic hand waving; exceptions matter." CreationDate="2010-10-19T05:54:39.277" UserId="251" />
  <row Id="5202" PostId="3718" Score="2" Text="Excel's RAND() the worst pseudo RNG I have ever measured.  (I applied the DieHard tests to it about ten years ago.)  It's not even equidistributed!  Nevertheless, it's OK for this kind of investigation." CreationDate="2010-10-19T05:55:05.900" UserId="919" />
  <row Id="5203" PostId="3709" Text="@PeterR: The plots are histograms of simulations (with fairly small numbers of iterations) and they have been standardized to zero mean and unit variance.  The graphic's title says all of this!  The presentation really is quite clear IMHO." CreationDate="2010-10-19T06:03:21.337" UserId="919" />
  <row Id="5204" PostId="3739" Text="the distribution is given by a few parameter already with your definition ? what is the difference for the application you havde in mind, between sampling things directly and using a transfomation after sampling ? " CreationDate="2010-10-19T06:20:05.567" UserId="223" />
  <row Id="5205" PostId="3713" Score="1" Text="Can you try to give more specific description of what you want to cluster? or is it just a state of the art in clustering that you need? " CreationDate="2010-10-19T06:23:19.697" UserId="223" />
  <row Id="5206" PostId="3702" Text="I don't see the question here. It seems to be &quot;how to load that type of data&quot; as suggested by Dirk, but the data seems already here. I vote for close. " CreationDate="2010-10-19T06:28:35.707" UserId="223" />
  <row Id="5207" PostId="3734" Text="Similar question with some nice ideas: http://stats.stackexchange.com/questions/643/how-do-you-convey-the-beauty-of-the-central-limit-theorem-to-a-non-statistician" CreationDate="2010-10-19T06:42:56.980" UserId="88" />
  <row Id="5208" PostId="3742" Text="This seems more like a topic for math.stackexchange.com ." CreationDate="2010-10-19T06:46:18.593" UserId="88" />
  <row Id="5209" PostId="832" Text="Looks like it could be very nice, but I just tried installing and running it in Stata 11.1 (i.e. the latest version) and it keeps giving me an r(3000) error upon clicking &quot;Done&quot; in the dialog, even if i type -version 6: clt-." CreationDate="2010-10-19T06:49:38.010" UserId="449" />
  <row Id="5211" PostId="3720" Text="@Andy W:see edited answer above." CreationDate="2010-10-19T07:02:19.660" UserId="603" />
  <row Id="5212" PostId="3720" Text="@Teucer: certainly. There are some facilities in the plm package to do these transformations (i.e. log, difference, dlog,...) in a single command-line: look for the **dynformula()** function.&#xA;" CreationDate="2010-10-19T07:04:29.857" UserId="603" />
  <row Id="5213" PostId="3720" Text="@kwak Actually what you propose is very close to what I have in mind: I was using the package lme4 to model the log returns. However, if I well understood, the difference lies in the estimator: lme4 uses REML whereas with plm you can use GMM. Is that correct? If so, why GMM is in this case superior to REML? Btw do you think that one can use the package heavy which is fitting with heavy tailed distributions (e.g. Student t)?" CreationDate="2010-10-19T07:08:34.377" UserId="1443" />
  <row Id="5214" PostId="3745" Text="@onestop Unless I'm misleading, the second moment are not exactly identical between the two." CreationDate="2010-10-19T07:18:07.400" UserId="930" />
  <row Id="5216" PostId="3720" Text="@Teucer. The difference between the two approaches is discussed in the vignette to the plm package. I would think that gmm estimation would be more efficient in your case, because one can assume that the innovations to the unemployment rate have a well behaved Gaussian distribution (i.e. as aggregation of a large number of small shocks). This assumption also explain why i don't understand the need to use the **heavy** package." CreationDate="2010-10-19T07:20:34.490" UserId="603" />
  <row Id="5217" PostId="3724" Text="Fine with me. I've just added this new tag to a few other old posts - what do you think? Also does the &quot;statistical bias&quot; tag still have a place? If so, when would you use it? Is this what the &quot;tag wiki&quot; system is meant to be for? I'm afraid I haven't worked out if or how i can edit that. Maybe we should take this to meta..." CreationDate="2010-10-19T07:24:00.550" UserId="449" />
  <row Id="5220" PostId="3745" Text="@chi - I meant the usual parameterisation of the logit-normal is in terms of the mean and SD of the related normal dist, not that the mean and SD of the logit-normal are the same as that of the related normal. Sorry if that was, or still is, unclear - i'm editing in a hurry as i should really have left for work by now!" CreationDate="2010-10-19T07:29:31.357" UserId="449" />
  <row Id="5222" PostId="3733" Score="1" Text="@Rob: Pearson's $\rho$ does not assume normality, but is only an exhaustive measure of association if the joint distribution is multivariate normal. Given the confusion this distinction elicits, you might want to add it to your answer.  " CreationDate="2010-10-19T07:42:26.000" UserId="603" />
  <row Id="5223" PostId="3744" Text="I'm also a big fan of Kendall's tau. Pearson is far too sensitive to influential points/outliers for my taste, and while Spearman doesn't suffer from this problem, I personally find Kendall easier to understand, interpret and explain than Spearman. Of course, your mileage may vary." CreationDate="2010-10-19T07:44:51.847" UserId="1352" />
  <row Id="5224" PostId="3733" Text="@kwak. Good point. I'll update the answer." CreationDate="2010-10-19T07:45:38.827" UserId="159" />
  <row Id="5226" PostId="3717" Score="1" Text="Yes, simple methods usually work better than complex ones (sometimes to a surprising extent). However, in this case univariate methods would throw away 99% of the data - whenever we forecast for one geography, we disregard all the other geographies. And I would definitely expect some kind of panel data model to be better than a univariate one. Not so much because employers co-locate, but because regulatory, tax, central bank and other external factors will be common and largely similar drivers of unemployment for *all* geographies." CreationDate="2010-10-19T07:49:56.843" UserId="1352" />
  <row Id="5227" PostId="3740" Text="@mbq Try the plots with a N(0,10^2) (i.e., 10 is standard deviation). It will not appear to be normal anymore. " CreationDate="2010-10-19T08:21:24.067" />
  <row Id="5228" PostId="3747" Text="Thanks @Stephan Kolassa, this will already help towards finding a solution." CreationDate="2010-10-19T08:39:35.277" UserId="1623" />
  <row Id="5229" PostId="3720" Text="@kwak On using heavy: I believe that the estimates are  more robust with heavy tailed innovations (my measurements are imperfect), but I might be wrong. Another reason was that I did a correlation clustering and looked at the distribution of log returns per year for each cluster, a student t distribution fits not too bad. But maybe it is irrelevant here. I have some questions: now assume I have more points, let's say about 25 points per region, would you still use GMM or REML for estimation? Where is the threshold? For a small sample is the asymptotic efficiency relevant?  " CreationDate="2010-10-19T09:12:09.160" UserId="1443" />
  <row Id="5230" PostId="3740" Text="@Srikant I agree, this is an answer to second point of the question. " CreationDate="2010-10-19T09:14:24.320" UserId="88" />
  <row Id="5231" PostId="3720" Text="@Teucer:&gt; *Another reason was that I did a correlation clustering and looked at the distribution of log returns per year for each cluster, a student t distribution fits not too bad.* this does not in itself justifies using a $t$ distribution: mixes of Gaussian distribution with varying variances, for instance, will also converge to a fat tailed distribution. " CreationDate="2010-10-19T09:32:10.207" UserId="603" />
  <row Id="5232" PostId="3720" Text="@Teucer:&gt; * For a small sample is the asymptotic efficiency relevant?* I think there is a slight misunderstanding here. If your assumptions on the distribution of the residuals are correct, then asymptotic efficiency is a measure of mean accuracy of your estimates (the 'asymptotic' here refers to the relative average precision over a large number of estimation instances)." CreationDate="2010-10-19T09:35:33.387" UserId="603" />
  <row Id="5233" PostId="3720" Text="@Teucer:&gt; *now assume I have more points, let's say about 25 points per region, would you still use GMM or REML for estimation? Where is the threshold?* At some point, the balance indeed tilts. And for larger sample sizes the gains in efficiency do not outweighs the costs in (statistical **AND** computational) complexity of the GMM approach over the REML one. The exact point depends also to the extend to which your data conforms to the working assumptions underlying each model (REML does not,imho, have less requirements, just different ones)...." CreationDate="2010-10-19T09:39:28.890" UserId="603" />
  <row Id="5234" PostId="3697" Score="1" Text="See http://stackoverflow.com/questions/2492947/boxplot-in-r-showing-the-mean for solutions using R" CreationDate="2010-10-19T09:40:03.723" UserId="229" />
  <row Id="5235" PostId="3720" Text="@Teucer:&gt; It's certainly a good sign if the two approaches do not lead to wildly different results (in the hypothetical that this would not be the case, one should explain why) " CreationDate="2010-10-19T09:40:57.370" UserId="603" />
  <row Id="5237" PostId="3582" Text="@Rob:&gt; given that three days have elapsed, i think for the benefit of the wider community, you should consider posting this as a separate question. " CreationDate="2010-10-19T10:55:46.730" UserId="603" />
  <row Id="5238" PostId="3724" Text="@Skrikant In fact, I didn't think another tag was needed at all, but yours looks better." CreationDate="2010-10-19T11:22:42.143" UserId="930" />
  <row Id="5239" PostId="3750" Score="1" Text="Although this is not a real answer; Excel and R both have their places on my desktop. I'm a firm believer that Excel has it's place in the workforce, especially when it comes to &quot;let me whip something in 5 minutes - rule of thumb type analysis&quot;. The same argument goes to Latex vs. word" CreationDate="2010-10-19T11:34:08.523" UserId="59" />
  <row Id="5240" PostId="3582" Text="@kwak. OK. I've posted something at http://stats.stackexchange.com/questions/3752/" CreationDate="2010-10-19T11:35:49.087" UserId="159" />
  <row Id="5241" PostId="3732" Text="@Andy (+1) Thanks for the link! You may be interested in this related question, http://stats.stackexchange.com/questions/3200/is-adjusting-p-values-in-a-multiple-regression-for-multiple-comparisons-a-good-id. Feel free to contribute." CreationDate="2010-10-19T11:37:43.700" UserId="930" />
  <row Id="5242" PostId="3720" Score="2" Text="@kwak, the spatial effect you suggest is what is referred to as local Geary's C, Global formula, http://en.wikipedia.org/wiki/Geary's_C, (or here is a link for the local version  http://www.passagesoftware.net/webhelp/Introduction.htm#Local_Geary_s_c.htm ) you could also consider local Moran's I, http://en.wikipedia.org/wiki/Indicators_of_spatial_association" CreationDate="2010-10-19T11:41:37.173" UserId="1036" />
  <row Id="5243" PostId="3731" Text="@ars,@Srikant. Even with small samples, you can still do inference on correlations, but not using the asymptotic normality result. " CreationDate="2010-10-19T11:41:55.763" UserId="159" />
  <row Id="5244" PostId="3746" Score="1" Text="This answer is similar enough to help you with the averaging of the points, http://stats.stackexchange.com/questions/2493/managing-error-with-gps-routes-theoretical-framework/2497#2497, it is simple to incorporate weights in that framework. I would think you would be able to use some simple heuristics to identify outliers, but that doesn't preclude you from taking a more empirical approach like Stephan suggested." CreationDate="2010-10-19T11:52:04.783" UserId="1036" />
  <row Id="5245" PostId="3752" Score="1" Text="You wrote &quot;On the other hand, variable kernels are usually thought to lead to poor estimators in kernel density estimation&quot;, what is the part of the paper you mention that makes you believe that ? I have plenty of references that go in the other derection, see for example the references mentioned in this paper: http://arxiv.org/PS_cache/arxiv/pdf/1009/1009.1016v1.pdf" CreationDate="2010-10-19T12:37:01.253" UserId="223" />
  <row Id="5246" PostId="3752" Score="1" Text="The abstract of Terrell and Scott summarises it nicely: &quot;Nearest neighbor estimators in all versions perform poorly in one and two dimensions&quot;. They only seem to find much advantage in multivariate density estimation." CreationDate="2010-10-19T12:50:45.813" UserId="159" />
  <row Id="5247" PostId="3720" Text="also to note how you define k is only limited to your imagination. There is currently no consensus on what is proper or improper, although many people suggest you try to optimize it like you suggested." CreationDate="2010-10-19T13:05:20.750" UserId="1036" />
  <row Id="5248" PostId="3755" Text="I'm running Windows 7 64-bit on Quad CPU 2.67 GHz, 4GB RAM" CreationDate="2010-10-19T13:07:04.933" UserId="315" />
  <row Id="5249" PostId="3732" Text="@chl, I don't think I can add anything to the already excellent answers for that question. I actual think Brendan's response is very poignant because I suspect the original poster is really interested in causal inference not solely prediction based on the context of the question." CreationDate="2010-10-19T13:12:58.473" UserId="1036" />
  <row Id="5250" PostId="3720" Text="@kwak @Andy thx for all the explanations. Just another question: now let's assume that I can further aggregate my regions on some broader regions (synthetic, using correlation clustering, or economic) would it make sens to compute the average  $\bar{u}_{−it}$ on these regions?" CreationDate="2010-10-19T13:13:00.600" UserId="1443" />
  <row Id="5251" PostId="3752" Score="1" Text="&quot;Nearest neighbor&quot; is not the only variable kernel. The papers I mention use other tool such as Lepskii's algorithm. I'll read the AOS paper but as the performences of nearest neighbor should decrease with the dimension, I found it strange that increasing the dimension gives advantages to a &quot;very non-parametric&quot; estimator (If we admit constant bandwidth is less non parametric than varying bandwith). In this type of situation, the evaluation case that is used often determine the results ... " CreationDate="2010-10-19T13:13:58.207" UserId="223" />
  <row Id="5252" PostId="3720" Text="@teucer, your asking if it would make sense to use that spatial neighborhood average as a predictor right? That is close to what local Moran's I does, it makes no difference if you choose neighbors based on theoretical reasons or empirical ones. I can probably guess why kwak initially suggested using the spatial differences as opposed to the average, but I will let kwak clarify." CreationDate="2010-10-19T13:22:16.067" UserId="1036" />
  <row Id="5253" PostId="3699" Score="1" Text="@Roger Such responses, ideally, should be comments to your own question. You are not really providing an answer to your own question. I would suggest deleting this 'answer' so that future readers are not confused by this response." CreationDate="2010-10-19T13:57:34.123" />
  <row Id="5254" PostId="3522" Score="1" Text="I would also suggest deleting this answer as it does not address the question that you actually asked in this thread." CreationDate="2010-10-19T13:58:12.983" />
  <row Id="5255" PostId="500" Score="1" Text="You may want to read this answer to a seperate question and see why adjusting p-values in such a manner may not be the best solution, http://stats.stackexchange.com/questions/3200/is-adjusting-p-values-in-a-multiple-regression-for-multiple-comparisons-a-good-id/3317#3317" CreationDate="2010-10-19T14:02:05.563" UserId="1036" />
  <row Id="5256" PostId="3731" Text="@Rob: Sure, but it seems this is where one should advocate Spearman's method over Pearson's.  For example suppose small samples where X is normal but Y isn't -- you can compare the two on even terms with ranking methods such as Spearman's.  Using Pearson's requires more work, for example, finding an appropriate transformation.  " CreationDate="2010-10-19T14:18:59.750" UserId="251" />
  <row Id="5257" PostId="3759" Text="Hey @Jonathan; machine learning is on-topic here (in fact, the machine learning proposal was merged with this one): http://meta.stats.stackexchange.com/questions/492/are-the-machine-learning-questions-on-topic" CreationDate="2010-10-19T14:32:58.203" UserId="5" />
  <row Id="5258" PostId="3759" Text="thanks Shane, wasn't clear on that" CreationDate="2010-10-19T14:38:52.667" UserId="1127" />
  <row Id="5259" PostId="3720" Text="@kwak, I also think the random component to the error term is a very good idea. There are many logical situations in which you wouldn't expect Beta(s) to be the same in different regions. I remember an example that New York City probably influences its neighbors, but the neighbors of NYC are less likely to influence it." CreationDate="2010-10-19T14:40:56.860" UserId="1036" />
  <row Id="5260" PostId="3720" Text="@Teucer:&gt; As Andy W said, any distance measure from which you can obtain a matrix of pairwise distances (geographic, economic as well as those coming from a clustering algorithm and there variations) are certainly to be tried. @Andy W: i don't advise Teucer to directly use the local average as a predictor because of the risk of correlation between the component of the residuals accounting for heterogeneity in the $\beta_s$'s and the local average. Is this what you had in mind ?" CreationDate="2010-10-19T14:57:52.290" UserId="603" />
  <row Id="5261" PostId="3720" Text="@Andy W:&gt; that (NYC) is a very good intuitive example. Worth using as an illustration." CreationDate="2010-10-19T14:58:55.777" UserId="603" />
  <row Id="5262" PostId="3713" Text="I don't have an immediate application in mind.  I'm just interested in a general approach to choosing a clustering method and measure of similarity." CreationDate="2010-10-19T15:02:42.813" UserId="485" />
  <row Id="5264" PostId="3720" Score="1" Text="@kwak, that was actually not the reason I imagined (I was thinking more along the lines of stationary estimates of Beta(s)). Local Moran's I does scale the average (ie it Z scores the average of the neighbors based on global mean and variance), but I think your concern is still legitimate. Is that a big deal though if your only interested in prediction?" CreationDate="2010-10-19T15:16:44.573" UserId="1036" />
  <row Id="5265" PostId="3732" Text="@Andy Yes, I was thinking of his answer. I have initiated a reflexion on data dredging issue (not exactly about model/variable selection issues or causal inference), but so far receive few responses. If you like to add your own ideas, it would be interesting: http://stats.stackexchange.com/questions/3252/how-to-cope-with-exploratory-data-analysis-and-data-dredging-in-small-sample-stud" CreationDate="2010-10-19T15:38:37.703" UserId="930" />
  <row Id="5266" PostId="3760" Text="+1 for linking to the full essay rather than one of the all-too-common oversimplifications - it's not long and is still relevant today and well worth reading in full. Hill's criteria for causation are about much more than just pyramid / piling up of evidence though." CreationDate="2010-10-19T15:39:25.823" UserId="449" />
  <row Id="5267" PostId="3762" Score="2" Text="@Shadi How many discrete values do you have? Are they really to be considered to be ordinal, if they lie between 0 and 1?" CreationDate="2010-10-19T15:42:12.453" UserId="930" />
  <row Id="5268" PostId="3754" Score="4" Text="Why do you insist on crossposting here and on SO?" CreationDate="2010-10-19T15:46:11.870" UserId="334" />
  <row Id="5269" PostId="3762" Text="They are continuous, every float value between 0 and 1. Is there any incoherence between ordinal data and [0 1] interval? Thanks." CreationDate="2010-10-19T15:49:16.650" UserId="1564" />
  <row Id="5270" PostId="3758" Text="Can you add the name of a function/package you use?" CreationDate="2010-10-19T15:51:59.797" UserId="88" />
  <row Id="5271" PostId="3762" Score="1" Text="For me this would rather be considered an interval scale: http://en.wikipedia.org/wiki/Interval_scale#Interval_scale" CreationDate="2010-10-19T15:54:27.337" UserId="442" />
  <row Id="5272" PostId="65" Text="As far as I remember, I was taught the 'sample' calculation in GCSE maths and science (age 14-16) and the distinction between populations and samples and their associated variance measures was covered (though not in depth) at A-level (age 16-18).  So I'm not sure this is a simple UK/US difference." CreationDate="2010-10-19T15:55:26.023" UserId="266" />
  <row Id="5273" PostId="3720" Text="@Andy W:&gt; *(ie it Z scores the average of the neighbors based on global mean and variance)* As you said, i don't think this alleviate my concern. The main issue for the use of RE is that we need $E(x_{it})\approx E(x_{jt})$ for any $i \neq j$. this most likely holds true if x_{it} is $u_{it}-\bar{u}_{-it}$ (or as Teucer suggested, $\log(u_{it})-\log(u_{-it})$) but it most certainly does not hold when x_{it} is $\bar{u}_{it}$ because it could be that $\bar{u}_{jt}\neq\bar{u}_{it}$. At any rate, this hypothesis has to be tested (Haussman test)." CreationDate="2010-10-19T15:57:22.300" UserId="603" />
  <row Id="5274" PostId="3762" Text="Yeah, I believe you are right. So you mean I cannot use ordinal measures for interval data, right? I corrected my question. I am looking forward to your guidances. Thanks." CreationDate="2010-10-19T15:59:20.330" UserId="1564" />
  <row Id="5275" PostId="3758" Text="Thanks @mbq! I have tried to figure out which one provides F-test, but no success..." CreationDate="2010-10-19T16:08:22.737" UserId="930" />
  <row Id="5276" PostId="3762" Text="When you stay between 0 and 1, does that include the endpoints 0 and 1 themselves or exclude them?" CreationDate="2010-10-19T16:10:34.000" UserId="449" />
  <row Id="5277" PostId="3720" Text="@Andy W:&gt; *Is that a big deal though if your only interested in prediction?* Can you post this as a separate question on the main board? It is a very important point, one which should interest many future readers. " CreationDate="2010-10-19T16:11:11.910" UserId="603" />
  <row Id="5278" PostId="3762" Text="the interval includes 0 and 1." CreationDate="2010-10-19T16:14:19.157" UserId="1564" />
  <row Id="5279" PostId="3752" Text="@Robin Girard:&gt; * found it strange that increasing the dimension gives advantages to a &quot;very non-parametric&quot; estimator (If we admit constant bandwidth is more non parametric than varying bandwith)* is there a typo in this sentence ? Otherwise you would seem to agree with the authors, at least on an intuitive level. Thanks to confirm/correct." CreationDate="2010-10-19T16:18:05.960" UserId="603" />
  <row Id="5280" PostId="3740" Text="@mbq You will run into trouble when you use any normal distribution with substantially nonzero mean.  The logistic transformation is almost linear for values near zero but becomes strongly nonlinear for more extreme values.  The resulting distribution will be strongly skewed; a normal approximation will be poor.  This explains why your answer is effective for the particular distribution proposed by the OP (so I upvoted it and hope others do too) but also puts constraints on its generalization to similar looking problems." CreationDate="2010-10-19T16:21:24.807" UserId="919" />
  <row Id="5281" PostId="3739" Text="To sample y, just sample x from a Normal(0, 0.2) distribution and compute y = e^x/(e^x+1).  You can use (0, 0.2) as the parameters: by means of this formula they completely determine the distribution." CreationDate="2010-10-19T16:25:44.677" UserId="919" />
  <row Id="5282" PostId="3762" Text="@Shadi So, I suggested you change your title accordingly." CreationDate="2010-10-19T16:26:15.380" UserId="930" />
  <row Id="5283" PostId="3762" Text="You can use any ordinal measures on interval data. But, you should not do so. You should use measures for interval data as they use more information are more powerful, etc as measures for ordinal data. Measurements are itself ordinally scaled (from low to high): nominal, ordinal, interval (and ratio). It is generally a good idea to use measures of that level of measurement that one has as these are the most powerful/informative ones." CreationDate="2010-10-19T16:29:20.030" UserId="442" />
  <row Id="5284" PostId="3767" Score="2" Text="@Henrik No you can't use correlation to assess the reliability of the measurements or the inter-rater agreement, even for two series of measurement. The correlation computed from two raters will remain the same even if you add some arbitrary value to the 2nd rater's assessments, while the agreement ICC will decrease and correctly reflects that there is a rater-effect." CreationDate="2010-10-19T16:39:48.670" UserId="930" />
  <row Id="5285" PostId="3740" Text="@whuber I agree, but I won't run into trouble, rather see that in such case it is not normal. This answer is just to promote trying as a technique, not to imply that resulting distribution will be always normal." CreationDate="2010-10-19T16:40:21.500" UserId="88" />
  <row Id="5286" PostId="3767" Text="No, it is for more than two measures." CreationDate="2010-10-19T16:40:42.120" UserId="1564" />
  <row Id="5288" PostId="3005" Text="@kwak Thanks for the fruitful exchange, I think I'm going to work throughout the LARS algorithm to try to connect the two approaches." CreationDate="2010-10-19T16:42:48.327" UserId="930" />
  <row Id="5289" PostId="3767" Score="1" Text="@Shadi Yes, I understand your design; I just take as an illustration @Henrik's POV with 2 series; the same line of reasoning applies with $k$ series of measurement." CreationDate="2010-10-19T16:44:22.613" UserId="930" />
  <row Id="5290" PostId="3582" Text="Thanks @Rob @Kwak -- I got hung up noodling KDE variants. (Is 3 days a statisticians' limit :)" CreationDate="2010-10-19T16:47:15.913" UserId="557" />
  <row Id="5291" PostId="3769" Text="@ars (+1) Many thanks! I missed this package... Always better to look at the code directly, I appreciate." CreationDate="2010-10-19T16:48:22.863" UserId="930" />
  <row Id="5292" PostId="3740" Score="1" Text="@mbq Thank you for sharing your philosophy.  I agree: it's good to help people learn to answer their own questions.  Empirical investigation (&quot;trying&quot;), though, is usually best when informed by theoretical considerations to indicate the limits of its applicability.  In this case you're safe with the Q-Q plot, because it is so sensitive to deviations from normality and you used a largish sample size of about a thousand, but the histogram alone can be deceiving.  (Almost *any* bell-shaped histogram &quot;looks&quot; normal!)" CreationDate="2010-10-19T16:50:36.193" UserId="919" />
  <row Id="5293" PostId="3766" Text="If I want to explain the data in each set, it can be: 0.98, 0.01, 0.5, ... which shows 'sound1' and 'sound2' are very similar (0.98), 'sound1' and 'sound3' are much different (0.01) and so on. In this case, do you believe I can use ICC or Pearson is better?" CreationDate="2010-10-19T16:59:40.067" UserId="1564" />
  <row Id="5294" PostId="3766" Text="@Shadi Did you read the thread I pointed to? You cannot use correlation-based criteria. So, I would better advice you to rely on the ICC unless someone has a better idea to cope with bounded values. For me it's not a problem as you're likely to end-up with similar results than with any other most complicated method. Still I agree with others than it makes sense not to use ANOVA or mixed-effects models with inappropriate link function in certain cases." CreationDate="2010-10-19T17:00:42.773" UserId="930" />
  <row Id="5295" PostId="3770" Text="That seems to be it. I will go away and think about it so more. Many thanks all." CreationDate="2010-10-19T17:51:08.960" UserId="1614" />
  <row Id="5297" PostId="3720" Text="@Teucer:&gt; I'm sorry i do not understand your last message. But you can certainly edit your question." CreationDate="2010-10-19T17:56:29.920" UserId="603" />
  <row Id="5298" PostId="3766" Text="Yes, I have studied them. Thank you. Actually I am completely new to this concept. I did not understand almost 80% of the thread. I wanted to use ICC in Matlab, but at first I should know the meaning of 'type', 'alpha' and 'r0' to know what value is the best for my purpose. Do you know any quick way to get some information? Thanks for your guidance." CreationDate="2010-10-19T18:17:20.853" UserId="1564" />
  <row Id="5299" PostId="3720" Text="@kwak I think it will help me a lot if you can edit your answer with the model formulas and the R code: I have read the vignette of plm several times, but I do not understand the model specification! Thanks in advance for the effort..." CreationDate="2010-10-19T18:20:25.880" UserId="1443" />
  <row Id="5300" PostId="3773" Score="2" Text="@onestop Fisher's exact test and Pearson's chi-squares test the null hypothesis that all 5 methods are equally effective against the alternative that at least 1 is better than the others. The p-values tell me that the null is rejected. So, if I want to find out which methods is actually better than the others won't I have to do 10 pairwise comparisons?" CreationDate="2010-10-19T18:26:53.513" UserId="1558" />
  <row Id="5301" PostId="3720" Text="I would have used the following specification with lme4: fm &lt;- lmer(lu~lu1+ls+(1|R),data=df) where $lu=log(u)$, $lu1=lag(log(u),1)$, $ls=log(\bar{u})$ and R is the region factor. Is this the model you have in mind? How can I specify with pgmm the following model lu~lu1+R+(lu1|R) for example?" CreationDate="2010-10-19T18:29:07.223" UserId="1443" />
  <row Id="5302" PostId="3720" Text="@Teucer:&gt; i will do this, but not tonight ;&lt;" CreationDate="2010-10-19T18:43:19.127" UserId="603" />
  <row Id="5303" PostId="3775" Text="That's a much better answer than mine! I failed to read the question properly I'm afraid (Step 3 in particular). I thought of deleting my answer, but I stand by the greater interpretability of a Bayesian approach is it's really the ranking that's of interest." CreationDate="2010-10-19T19:09:55.940" UserId="449" />
  <row Id="5305" PostId="3775" Text="@whuber Just to make sure I understand correctly- The indicator that tracks the relative difference between method 4 and 5 will be updated whenever we see a difference that is greater than 0.21. " CreationDate="2010-10-19T19:21:08.197" UserId="1558" />
  <row Id="5307" PostId="3769" Text="@chl: I was looking at the code and about to look up the formula, when I saw your answer come in. Looks like we started from different ends and met in the middle -- making a nice complement of answers. :)  (Already upvoted you.)" CreationDate="2010-10-19T19:23:28.847" UserId="251" />
  <row Id="5308" PostId="3769" Text="@ars I have to turn back to Zar's textbook because I didn't find the function though I've heard of the alternative statistic :)" CreationDate="2010-10-19T19:25:16.003" UserId="930" />
  <row Id="5309" PostId="3775" Text="@sxv Yes, that's right.  (Well, I actually used greater than or equal.  Ties do happen.  I think including equality among the significant results is the correct thing to do, because we're evaluating the probability that differences *this large or larger* can occur by chance.)" CreationDate="2010-10-19T19:26:17.230" UserId="919" />
  <row Id="5310" PostId="3752" Text="@kwak thanks to notice that! this is a typo: I wanted to say constant bandwidth is less NP ... I can't modify my comment :( sorry about that. " CreationDate="2010-10-19T19:27:55.387" UserId="223" />
  <row Id="5311" PostId="3766" Text="@Shadi Oups, sorry, I didn't think of language issue. I've updated my response. HTH" CreationDate="2010-10-19T19:42:50.790" UserId="930" />
  <row Id="5313" PostId="3721" Text="I need just one number for all of the judges, not a matrix that shows the pairwise correlations of judges. for example: pearson(a,b,c)=0.53     Is it possible?   Thanks." CreationDate="2010-10-19T21:39:52.147" UserId="1564" />
  <row Id="5315" PostId="3680" Text="Is that the &quot;Riemann manifold Langevin&quot; paper? Do integrate Fisher information at some point?" CreationDate="2010-10-19T21:51:45.250" UserId="511" />
  <row Id="5316" PostId="3721" Text="@shadi Correlation is a bivariate relationship in the sense that it tells you the relationship between two sets of variables. Thus, calculating pearson(a,b,c) where I presume a,b,c are different judges is not possible." CreationDate="2010-10-19T21:52:18.690" />
  <row Id="5319" PostId="3731" Text="@ars. You can just use Monte Carlo methods or a bootstrap. Not much work in that, just computation." CreationDate="2010-10-19T22:24:18.203" UserId="159" />
  <row Id="5320" PostId="3752" Text="@robin. I've edited your comment" CreationDate="2010-10-19T22:25:56.670" UserId="159" />
  <row Id="5321" PostId="3766" Text="Dear chl, Thanks for your great guides. I'm not sure if I explained my data well or not. when you asked : &quot;how many discrete data do you have&quot;, did you mean something like &quot;partner1&quot; and &quot;partner2&quot; in (uvm.edu/~dhowell/StatPages/More_Stuff/icc/icc.html)? If so, I should say just 1. I try to explain my data better. I have, for example, 100 different sounds. I want to know how similar each two sounds are. I used a program to do that for me. It generated some (4851) numbers. I also used 5 other programs to do the same thing. Now, I want to know how close the results of the 6 programs are" CreationDate="2010-10-19T22:43:28.310" UserId="1564" />
  <row Id="5322" PostId="3731" Text="@Rob: Yes, we can always come up with workarounds to make things work out roughly the same.  Simply to avoid Spearman's method -- which most non-statisticians can handle with a standard command.  I guess my advice remains to use Spearman's method for small samples where normality is questionable.  Not sure if that's in dispute here or not." CreationDate="2010-10-19T23:43:16.250" UserId="251" />
  <row Id="5323" PostId="3731" Text="@ars. I would use Spearman's if I was interested in monotonic rather than linear association, or if there were outliers or high levels of skewness. I would use Pearson's for linear relationships provided there are no outliers. I don't think the sample size is relevant in making the choice." CreationDate="2010-10-20T00:32:52.033" UserId="159" />
  <row Id="5324" PostId="3779" Text="Should it not be 'pick k tiles *without* replacement'? Very interesting question." CreationDate="2010-10-20T00:39:34.920" />
  <row Id="5325" PostId="3769" Text="sorry for the late post, but yes, i am using agricolae package.  friedman.test doesn't work on my data as it is returning an error message &quot;your data is not an unreplicated design&quot; to that effect.       but why are there 2 p-value f's?  am i right that the first is for the the treatment and the second for the block? " CreationDate="2010-10-20T00:59:09.607" UserId="1627" />
  <row Id="5326" PostId="3731" Text="@Rob: OK, thanks for the discussion.  I agree with the first part, but doubt the last, and would include that size only plays a role because normal asymptotics don't apply.  For example, Kowalski 1972 has a pretty good survey of the history around this, and concludes that the Pearson's correlation is not as robust as thought.  See: http://www.jstor.org/pss/2346598" CreationDate="2010-10-20T01:00:48.653" UserId="251" />
  <row Id="5327" PostId="3769" Text="@kathy: see chl's answer for an explanation of what the two mean -- the first value you see is $F_r$ and the &quot;value f&quot; is $F_{obs}$ in his answer.  Also, it's possible you're passing the arguments in the wrong order to the friedman.test method; it should be (data, group, block)." CreationDate="2010-10-20T01:08:07.203" UserId="251" />
  <row Id="5328" PostId="3769" Text="aw, thanks ars, now i get it, its the adjusted value...mmmm that's why i'm wondering why its so close to the chisqr value" CreationDate="2010-10-20T01:16:45.097" UserId="1627" />
  <row Id="5329" PostId="3779" Text="oops. indeed it should." CreationDate="2010-10-20T01:58:11.113" UserId="795" />
  <row Id="5330" PostId="1332" Text="I think it's a great one (I think it goes well with - &quot;no models are true, some  are useful&quot;)" CreationDate="2010-10-20T02:27:35.627" UserId="253" />
  <row Id="5331" PostId="3783" Text="great!  thank you so much" CreationDate="2010-10-20T02:57:18.263" UserId="1627" />
  <row Id="5332" PostId="3780" Text="The quick and dirty approach may not be so quick! The dictionary may contain 100,000 words, and the search for a match of the given tiles could be a coding disaster." CreationDate="2010-10-20T04:26:51.890" UserId="795" />
  <row Id="5333" PostId="189" Score="1" Text="This would be the approach when the CDF is only approximated empirically. It gives lousy estimates of the PDF, though." CreationDate="2010-10-20T05:13:19.947" UserId="795" />
  <row Id="5334" PostId="3766" Text="@Shadi So you have 6 &quot;raters&quot; or &quot;methods&quot; that are assessing 100 objects similarity. How do you explain that there are 4851 measurements: Are there replicate measurements for each object (sound)? The coding (ordered or discrete, continuous) is for the measurements. In your case, I consider it as reflecting a continuous scale of similarity on [0,1]." CreationDate="2010-10-20T06:30:14.833" UserId="930" />
  <row Id="5335" PostId="3780" Text="@shabbychef This is something well done to suit spell checkers.  See for instance http://www.n3labs.com/pdf/lexicon-squeeze.pdf" CreationDate="2010-10-20T07:06:49.607" UserId="88" />
  <row Id="5336" PostId="3781" Text="*How* have you simulation the datasets?" CreationDate="2010-10-20T07:07:33.130" UserId="449" />
  <row Id="5337" PostId="3767" Text="@chl You are right! I will keep this post to remind everyone that this is wrong." CreationDate="2010-10-20T08:21:23.680" UserId="442" />
  <row Id="5338" PostId="3788" Text="@K-1, IMHO lag auto-correlation makes sense only if you have an &quot;time-seris like&quot; ordering otherwise it doesn't make a lot of sense. Also, independence =&gt; correlation = 0, but not the other way.   " CreationDate="2010-10-20T08:46:25.650" UserId="1307" />
  <row Id="5339" PostId="3790" Text="I am sorry, I am just  a first year math student. Could you please provide/recommend a link/book/paper that describes how the relationship was derived?" CreationDate="2010-10-20T08:52:00.580" UserId="1636" />
  <row Id="5340" PostId="3788" Text="Why do you want to know if the data are independent? If you are fitting a model the model might assume the **errors** are IID, not the data. Can you provide more context - either as a comment or (better) by editing your Question (see the edit link beneath the tags)?" CreationDate="2010-10-20T08:57:34.153" UserId="1390" />
  <row Id="5341" PostId="3780" Text="@shabbychef Reg monte-carlo- if the dictionary is sorted a match should be fairly quick no? In any case, the direct approach that I outlined earlier was flawed. I fixed it. The problem in my earlier solution was that the same word can be formed multiple ways (e.g., 'bat', 'b*t' etc)." CreationDate="2010-10-20T09:14:00.123" />
  <row Id="5342" PostId="3792" Text="@kwak Your first point remind me of the *run test* (at least for simple design)." CreationDate="2010-10-20T09:31:13.887" UserId="930" />
  <row Id="5343" PostId="3790" Score="2" Text="@Sara I think it dates back to Karl Pearson, which uses this empirical relationship for his &quot;Pearson mode skewness&quot;. Aside from this, you may find interesting this online article, http://j.mp/aWymCv." CreationDate="2010-10-20T09:42:06.293" UserId="930" />
  <row Id="5344" PostId="3780" Score="1" Text="@shabbychef On further reflection, I agree with you that the monte carlo approach will not work. One issue is that you need to figure out which words you can actually form with the k tiles and the second one is that you can form multiple words with the k tiles. Calculating these combinations from k tiles is probably not that easy." CreationDate="2010-10-20T09:49:17.853" />
  <row Id="5346" PostId="3795" Text="Out of interest, what did you find was the problem with GraphViz? Not flexible enough? In what way(s)?" CreationDate="2010-10-20T10:50:18.420" UserId="449" />
  <row Id="5347" PostId="3792" Text="G. Jay Kerns:&gt; Thanks, i've edited my post to adress your comment." CreationDate="2010-10-20T11:02:06.410" UserId="603" />
  <row Id="5348" PostId="3790" Text="Thank you chl and kwak for the link and answer you have provided. I will study them." CreationDate="2010-10-20T11:13:39.383" UserId="1636" />
  <row Id="5350" PostId="3794" Text="So, in excel, if I wanted to look at a time frame of 365 days...then I'd add 365 columns along side the ID table?" CreationDate="2010-10-20T11:35:00.903" UserId="1641" />
  <row Id="5351" PostId="3792" Text="OK, I've deleted my earlier comment." CreationDate="2010-10-20T11:42:30.720" UserId="1108" />
  <row Id="5353" PostId="2732" Text="+1, I second all your recommendations and have only one *very* minor quibble: there is no &quot;c&quot; in Frank Harrell's name... He's American and not French, after all ;-) I'll wait for another 317 rep and edit your answer..." CreationDate="2010-10-20T11:50:29.223" UserId="1352" />
  <row Id="5354" PostId="2732" Text="Oups... thanks Stephan :)" CreationDate="2010-10-20T11:56:24.037" UserId="930" />
  <row Id="5356" PostId="3799" Text="Given my limited experience, I cannot give an exact answer. However, I believe that you can use a panel data (because you consider in your example variations within individuals and between individuals) approach with logit. Maybe others can elaborate on this...  " CreationDate="2010-10-20T12:14:46.110" UserId="1443" />
  <row Id="5357" PostId="3799" Text="Your small example is very useful, but I assume your real dataset is larger. How much larger, i.e. (roughly) how big are your real *N* and *k*?" CreationDate="2010-10-20T12:25:25.513" UserId="449" />
  <row Id="5358" PostId="3794" Text="@Alex Yes. I suppose you'd need one of the more recent versions of Excel that overcomes the old column limit. And the algorithm would be more elegant in something like R." CreationDate="2010-10-20T12:29:56.867" UserId="183" />
  <row Id="5359" PostId="3588" Text="@Denis:&gt; i would say it makes the problem worst (i.e. NN kernel should work better on less clumpy data). I have an intuitive explanation but it won't fit here, plus you may want to asks this out on the main board as a separate question (linking to this one) to have additional opinions." CreationDate="2010-10-20T13:37:36.417" UserId="603" />
  <row Id="5360" PostId="3712" Text="Thanks Charlie - Is it possible to get the estimates separately for treated and untreated? The lincom command only gives me the difference between treated and untreated. " CreationDate="2010-10-20T13:52:24.473" UserId="834" />
  <row Id="5361" PostId="3798" Text="@chl: thanks!  I remember plspm being announced on the semnet list -- for some reason PLS isn't as big on this side of the Atlantic, not sure why.  plotSEMM looks really interesting, can't wait to play with it." CreationDate="2010-10-20T13:57:35.317" UserId="251" />
  <row Id="5362" PostId="3802" Text="@ars (+1) Thanks for linking with OpenMx! Really great package that has replaced Mx on my Mac now." CreationDate="2010-10-20T14:09:25.497" UserId="930" />
  <row Id="5363" PostId="3690" Text="Thanks Srikant. Since the group-level effects to which I'm referring often unique, complex, and erratic, I'm not sure I'll be able to quantify them with interaction terms. However, your answer has led me to approach my problem from a different angle and refine my question. I'll restate in more formal terms once I've more clearly defined the problem. " CreationDate="2010-10-20T14:30:45.113" UserId="1611" />
  <row Id="5364" PostId="3766" Text="yes, as you said I have 100 objects and if I want to know the similarity between each two sound, it will be (99*98)/2. (s1,s2)(s1,s3)(s1,s4)...(s1,s100)(s2,s3)(s2,s4)...(s2,s100)...(s99,s100).  and I know that (s2,s1)=(s1,s2) so I do not calculate that. so, I will have 4851 measurements for each method(each method calculates the similarity of all the sounds pairwise).and totally I have 6 methods. So, I have 6*4851 data in general. It is a continuous scale of similarity on [0 1]. Thanks." CreationDate="2010-10-20T14:46:21.733" UserId="1564" />
  <row Id="5365" PostId="1251" Score="1" Text="That's a good answer. In addition, I'd suggest horizontally jittering the points, so they don't overlap, especially if you have more points per group than this. In ggplot2, the geom_jitter() will do that. " CreationDate="2010-10-20T15:00:21.693" UserId="6" />
  <row Id="5366" PostId="3805" Text="How many percentiles are included in your real data? I hope it's more than in your example!" CreationDate="2010-10-20T15:04:43.220" UserId="449" />
  <row Id="5367" PostId="3793" Score="2" Text="What's the reason for the vote to close?  How does this problem not relate to statistical analysis?" CreationDate="2010-10-20T15:31:37.067" UserId="919" />
  <row Id="5368" PostId="3794" Text="@Jeromy Your idea is a good one but the implementation can be greatly improved :-).  I posted an explanation." CreationDate="2010-10-20T15:33:30.733" UserId="919" />
  <row Id="5369" PostId="3803" Text="@whuber The tree is a neat idea (upvote for that idea) but would it not require lot of memory? I guess it depends on how diverse the dictionary is but I am guessing a reasonably diverse dictionary would require many trees For example, the 'b' tree would start with the letter 'b' instead of 'a' for all those words which do not have 'a' in them. Similarly, the 'c' tree would start with the letter 'c' for those words which do not have 'a' and 'b' but have 'c'. My proposed direct approach seems simpler as it requires a one-time traversal of all the words in the dictionary, no?" CreationDate="2010-10-20T15:49:09.823" />
  <row Id="5370" PostId="3803" Score="1" Text="@Srikant: The tree would likely require far less RAM than caching the entire dictionary to begin with.  Are you really concerned about a few megabytes of RAM, anyway?  BTW, there's only one tree, not many: they are all rooted at the empty word.  Your approach, as I have understood it, requires multiple searches of the dictionary (up to 7! of them) on *every iteration*, making it impracticable as @shabbychef fears.  It would help if you could elaborate on the algorithm you have in mind where you write &quot;see if you can form a word&quot;: that hides a lot of important details!" CreationDate="2010-10-20T16:09:27.643" UserId="919" />
  <row Id="5371" PostId="3803" Text="@whuber: I realized the fact that there is only one tree after I posted my comment. Reg my approach- I agree that my monte carlo proposal is fuzzy and your answer fleshes out how one can actually implement monte carlo in this setting. I actually meant that the *direct approach* (see my answer) may actually be simpler as that approach requires a one-time operation on the dictionary unlike a monte carlo which requires several thousands of iterations on the tree. Just wondering on the relative merits of the approaches." CreationDate="2010-10-20T16:15:11.737" />
  <row Id="5372" PostId="3803" Text="@Srikant I refrained from commenting on your direct approach because it I suspect it gets the wrong answers.  It does not appear to account for the dictionary structure: that is, the subset relationships among words.  For instance, would your formula get the correct answer of zero for all dictionaries that contain all the possible one-letter words?" CreationDate="2010-10-20T16:23:00.457" UserId="919" />
  <row Id="5373" PostId="3780" Text="@Srikant I'm confused.  What exactly do you mean by &quot;the sth word from the alphabet&quot;?  Alphabets contain only letters, not words." CreationDate="2010-10-20T16:23:50.700" UserId="919" />
  <row Id="5374" PostId="3780" Text="@whuber Sorry. I meant that m_a counts the number of 'a's needed by the s^th word. I corrected the text." CreationDate="2010-10-20T16:26:40.023" />
  <row Id="5375" PostId="3803" Text="@whuber hmmm good point. Perhaps, I am answering the wrong question!" CreationDate="2010-10-20T16:30:03.380" />
  <row Id="5376" PostId="3803" Text="@whuber re your bet that I could conduct this study in seconds: you overestimate my programming skills! My intuition about MC methods is they should be easy to implement: &quot;let the computer do the work.&quot; If MC is going to take a lot of work, maybe the exact approach is better. In fact, it seems like once the dictionary is in tree form (+1, BTW), you are a long way towards counting the subsets of letters from the bag that form legitimate words..." CreationDate="2010-10-20T16:34:59.353" UserId="795" />
  <row Id="5377" PostId="3780" Score="1" Text="@Srikant Thanks.  Your formula seems to assume you have to use all k letters to form the word, but I don't think that's what the OP is asking.  (That's not how Scrabble is played, anyway.)  With that implicit assumption, you're on the right track but you need to modify the algorithm: you mustn't repeat the calculation for words in the dictionary that are permutations of each other.  For example, you mustn't subtract both t_{stop} and t_{post} in your formula.  (This is an easy modification to implement.)" CreationDate="2010-10-20T16:35:04.107" UserId="919" />
  <row Id="5378" PostId="3769" Text="@kathy, you're welcome; glad to help. :)  BTW, you should accept @chl's answer since he noted the adjustment (click the check mark next to his answer)." CreationDate="2010-10-20T16:38:15.747" UserId="251" />
  <row Id="5379" PostId="3803" Text="@shabbychef That's an excellent point.  I suspect the computation is not easy.  The leaves of my tree have the property that (a) each can be permuted into at least one word but (b) no proper subset can be permuted into a word.  We can readily count the number of k-letter sets that contain a given leaf, but we're going to double- (and triple- and quadruple-...) count things that way.  E.g., in English &quot;a&quot; is a leaf and so is &quot;ept&quot;, but &quot;aaeptxz&quot; contains *both*.  We could apply PIE (principle of inclusion-exclusion) but it would ferociously complicated to do, I think." CreationDate="2010-10-20T16:45:10.487" UserId="919" />
  <row Id="5380" PostId="3805" Text="It depends on the factor I'm using.  For body weight, I'm actually using more percentiles.  But for something like dust ingestion rates, which has less observed measurements, I am using as few as four percentiles." CreationDate="2010-10-20T16:45:33.467" UserId="1645" />
  <row Id="5381" PostId="3810" Text="What do you mean by *power*? If you wish to bring them to a common scale then you can use a linear transformation to convert all scales to a common scale (say 5 point). Likert scales are interval scales and hence linear transformations do not 'destroy' the properties of the scale. Is this what you want/meant?" CreationDate="2010-10-20T16:46:47.490" />
  <row Id="5382" PostId="3798" Text="@chl: btw, I meant to add that it's shame PLS isn't more noted here, since there seems to be a lot of exciting stuff happening around it, especially with tools being developed (e.g. SmartPLS in addition to plspm).  I read some of Wold's work a while back and some of his ideas are only just being realized (e.g. &quot;having a conversation with your data&quot;).  I really need to set aside some time to explore it more." CreationDate="2010-10-20T16:47:53.263" UserId="251" />
  <row Id="5383" PostId="3810" Score="1" Text="The usual way to accomplish this is to do the PCA on the correlation matrix instead of the covariance matrix." CreationDate="2010-10-20T16:52:19.620" UserId="919" />
  <row Id="5384" PostId="3766" Text="Could you please tell me if you still believe that the best one is Intra-class correlation? Thanks a lot." CreationDate="2010-10-20T16:54:31.087" UserId="1564" />
  <row Id="5385" PostId="3766" Text="@Shadi Huh, that makes a difference; because here you might also be interested in assessing whether your 6 similarity matrix (and not six series of measurement) present some form a variance attributable to the raters. Provided you consider a stacked version of your pairwise similarity (a long vector), the ICC remain applicable; otherwise there exist methods to assess the comparability of (dis)similarity matrix but I feel this should be clarified either in your question or best, in a new question (and mods could close this one) so that others may contribute. Let me ask the mods first." CreationDate="2010-10-20T17:02:52.613" UserId="930" />
  <row Id="5386" PostId="3804" Text="Hello Harlan, can your details be translated in mathematical notation? It would be very helpful to understand the details (for me)." CreationDate="2010-10-20T17:07:40.033" UserId="1307" />
  <row Id="5387" PostId="3798" Text="@ars Do you want a list of recommend readings? I also worked with Arthur Tenenhaus who submitted a nice paper with his father (yes, Michel Tenenhaus) to Psychometrika: They are unifying all two-block methods (PCA, CCA, PLS, inter-battery, etc.) thanks to a very neat rewrite of the argmax constraint. I've been playing myself with penalized PLS/CCA (L1/L2) in genomics, but I feel it will bring more interesting on my biomedical data." CreationDate="2010-10-20T17:14:12.933" UserId="930" />
  <row Id="5388" PostId="3801" Text="@mbq:&gt; you could add a link to Efron's bootstrap paper**s**. It's as good a place as any to start." CreationDate="2010-10-20T17:22:40.057" UserId="603" />
  <row Id="5389" PostId="3780" Text="@whuber you are correct; I will edit the question: the words are of any length up to $k$. This is how Scrabble is played, but the question should be self contained." CreationDate="2010-10-20T17:24:30.677" UserId="795" />
  <row Id="5390" PostId="3762" Score="2" Text="@Shadi Following my latest comment, I asked for closing this question so that you can reformulate a new one by adding precision on your design, especially the fact that you actually have 6 similarity matrices instead of 6 series of measurement. This way, others may provide useful insights into this question. You can still link to this question, but I really feel it call for a new thread with your added clarifications so that everyone can contribute." CreationDate="2010-10-20T17:34:06.517" UserId="930" />
  <row Id="5392" PostId="3809" Text="But that's two parameters to set for one dependent variable! I want to keep the underlying beta-binomial structure in $prior_1$, and just update it, perhaps by shifting the mean without changing the variance, to give $prior_2$. And I want to do it in a principled way, as I only 20% trust that scalar anyway..." CreationDate="2010-10-20T17:57:19.297" UserId="6" />
  <row Id="5393" PostId="3806" Text="@whuber (+1) Very nice answer." CreationDate="2010-10-20T18:00:50.703" UserId="930" />
  <row Id="5394" PostId="3801" Text="@kwak Ok, yet it is easier to put a link to Wikipedia http://en.wikipedia.org/wiki/Bootstrapping_(statistics)#References " CreationDate="2010-10-20T18:06:27.470" UserId="88" />
  <row Id="5395" PostId="3804" Text="added some notation, hope it helps clarify!" CreationDate="2010-10-20T18:16:37.147" UserId="6" />
  <row Id="5396" PostId="3799" Text="N and k can be huge, but computational power is not a problem." CreationDate="2010-10-20T18:37:28.580" UserId="1643" />
  <row Id="5397" PostId="3809" Text="@harlan See edited answer." CreationDate="2010-10-20T18:42:22.690" />
  <row Id="5398" PostId="3539" Text="@shadi Any reason for removing the &quot;reliability&quot; tag that I added? It is used elsewhere (together with the more general &quot;psychometrics&quot; one) for related questions." CreationDate="2010-10-20T18:48:23.487" UserId="930" />
  <row Id="5399" PostId="3812" Text="Thanks for the pointers to optimal scaling." CreationDate="2010-10-20T18:53:35.647" UserId="919" />
  <row Id="5400" PostId="3761" Text="Thanks, you are right that FNN falls into the category of neuro-fuzzy systems.  The links you gave are helpful.  I think I will contact the author of the paper to see if there are closer implementations." CreationDate="2010-10-20T18:58:56.757" UserId="1127" />
  <row Id="5401" PostId="1251" Text="@Harlan: I agree. Although if I had many more points I would probably use a boxplot." CreationDate="2010-10-20T19:00:39.260" UserId="8" />
  <row Id="5402" PostId="3813" Text="@shabbychef or *biclustering* (+1)." CreationDate="2010-10-20T19:09:37.063" UserId="930" />
  <row Id="5405" PostId="3814" Text="@csgillespie Does it extend to justifications received in response to an initial review (where minor and/or major revisions were asked)?" CreationDate="2010-10-20T19:14:34.573" UserId="930" />
  <row Id="5406" PostId="3814" Text="@chl: Yes, why not." CreationDate="2010-10-20T19:19:20.240" UserId="8" />
  <row Id="5410" PostId="3813" Text="That is what I was looking for. Name of the problem helps a lot. Thanks" CreationDate="2010-10-20T19:35:59.990" UserId="1643" />
  <row Id="5411" PostId="3816" Text="+1 for me. This frustrates me, especially when they cite the wrong thing and I've provided the relevant details on how to cite the packages" CreationDate="2010-10-20T19:36:52.963" UserId="1390" />
  <row Id="5412" PostId="3818" Score="7" Text="@Neelam Huh... You should add a little bit of contextual information here, e.g. at least specify your working hypotheses." CreationDate="2010-10-20T19:39:55.223" UserId="930" />
  <row Id="5414" PostId="3818" Score="3" Text="Definitely needs more information.  " CreationDate="2010-10-20T19:41:23.753" UserId="5" />
  <row Id="5415" PostId="3800" Text="Thanks for info about Netflix, shame on me that I haven't heard about it earlier :/" CreationDate="2010-10-20T19:44:24.590" UserId="1643" />
  <row Id="5416" PostId="3821" Score="1" Text="s.d. is shift invariant. " CreationDate="2010-10-20T19:53:44.827" UserId="603" />
  <row Id="5417" PostId="3821" Text="Thanks Srikant.  Do you have an idea on how to define the transformation I gave in my example?" CreationDate="2010-10-20T19:54:01.493" UserId="253" />
  <row Id="5418" PostId="3821" Score="1" Text="@Tal Y = X + 1?" CreationDate="2010-10-20T19:55:00.057" />
  <row Id="5419" PostId="3817" Score="2" Text="I'm a little curious about the stepwise regression bullet. What makes stepwise regression so bad? Is it the data dredging and multiple comparisons issue?" CreationDate="2010-10-20T19:56:12.030" UserId="1118" />
  <row Id="5420" PostId="3813" Text="+1 Wow, it has a name... " CreationDate="2010-10-20T20:02:05.283" UserId="88" />
  <row Id="5421" PostId="3817" Score="4" Text="The problem is that stepwise procedures completely invalidate all the assumptions and preconditions for &quot;normal&quot; inferential statistics based on p values, which are then badly biased (downwards towards being &quot;more significant&quot;). So basically, the answer is &quot;yes&quot;, with the caveat that one could in principle correct for all these multiple comparisons (but which I have never seen done). I believe strongly that this is the single most important reason why I see so much research in psychology that cannot be replicated - which in turn leads to a huge waste of resources." CreationDate="2010-10-20T20:04:48.097" UserId="1352" />
  <row Id="5422" PostId="3821" Score="2" Text="You missed a=-1 ;-)" CreationDate="2010-10-20T20:07:16.167" UserId="88" />
  <row Id="5423" PostId="3820" Text="Actually, you cannot change *one* variable and still keep the same standard deviation. In your example, after all, you are not changing *one* variable but *three*." CreationDate="2010-10-20T20:08:41.380" UserId="1352" />
  <row Id="5424" PostId="3822" Score="2" Text="@shadi Still asking for a Matlab solution? In this case, I let you add this tag. And feel free to add or delete the edits I made if they don't exactly reflect your design." CreationDate="2010-10-20T20:10:20.080" UserId="930" />
  <row Id="5425" PostId="3821" Score="2" Text="The standard deviation will not change if you simply reorder your data, but this does not seem to be too helpful here..." CreationDate="2010-10-20T20:10:42.700" UserId="1352" />
  <row Id="5426" PostId="3821" Text="@mbq Good point!" CreationDate="2010-10-20T20:13:28.093" />
  <row Id="5427" PostId="3821" Score="2" Text="@Srikant: if we expand our search from linear transformations to analytic transformations (i.e., transformations that can be represented by their Taylor series), I think one could prove that all higher order terms need to vanish for the standard deviation to be unchanged. That is, the only analytic transformation that preserves sd would be adding or subtracting a constant to all values. Might be a nice exercise for statistics undergrads ;-)" CreationDate="2010-10-20T20:14:54.813" UserId="1352" />
  <row Id="5428" PostId="3821" Text="@stephan I was actually thinking of the same idea but it seemed too much tex typing so I abandoned the idea." CreationDate="2010-10-20T20:16:31.300" />
  <row Id="5429" PostId="3818" Score="1" Text="And a question mark... fixed." CreationDate="2010-10-20T20:18:51.703" UserId="88" />
  <row Id="5430" PostId="3822" Text="thanks for editing." CreationDate="2010-10-20T20:20:16.450" UserId="1564" />
  <row Id="5431" PostId="3817" Score="4" Text="@Stephan: I agree, stepwise is a bad idea.  Though, while they may have not made it to psych methods yet, but there are a variety of selection procedures that adjust for bias related to overfitting by adjusting estimates and standard errors.  This is not typically thought of as an issue of multiple comparisons. They are known as shrinkage methods.  See my response in this thread &lt;http://stats.stackexchange.com/questions/499/when-can-you-use-data-based-criteria-to-specify-a-regression-model&gt; and Harrell's &quot;Regression Modeling Strategies&quot; or Tibshirani on the lasso." CreationDate="2010-10-20T20:34:42.767" UserId="485" />
  <row Id="5432" PostId="3820" Text="@Stephan I understand the question as referring to an univariate distribution, hence one variable with multiple observations. Am I missing something?" CreationDate="2010-10-20T20:34:50.210" UserId="930" />
  <row Id="5433" PostId="3820" Score="1" Text="@chl - I'd say it's more probable that *I* am missing something. Tal asks about &quot;changing one *value*&quot;, and I should have written about &quot;values&quot;, not &quot;variables&quot; in my comment, where &quot;value&quot; to me sounds much like &quot;observation&quot;." CreationDate="2010-10-20T20:38:47.067" UserId="1352" />
  <row Id="5434" PostId="3817" Score="1" Text="@Brett Magill: +1 on that, and yes, I know about shrinkage and the lasso. Now all I need is some way to convince psychologists that these make sense... but people have been fighting with very limited success just to get psychologists to report confidence intervals, so I'm not too optimistic about psychologists' accepting shrinkage in the next twenty years." CreationDate="2010-10-20T20:42:34.620" UserId="1352" />
  <row Id="5435" PostId="3820" Score="1" Text="@Stephen In the example he changed 2 to 5: only one value was altered." CreationDate="2010-10-20T20:47:54.913" UserId="919" />
  <row Id="5436" PostId="3821" Text="@Stephen Not true for analytic transformations or even the most general transformations.  Apply *any* transformation, such as a Box-Cox transformation.  Determine the new variance.  If it exists and is nonzero, rescale the result to match the original variance." CreationDate="2010-10-20T20:50:20.383" UserId="919" />
  <row Id="5437" PostId="3820" Score="1" Text="@Tal I changed &quot;roles&quot; to &quot;rules&quot; in your question and hope that was what you intended." CreationDate="2010-10-20T20:57:13.517" UserId="919" />
  <row Id="5438" PostId="3820" Text="@chl It's univariate all right but I take it that he is asking about the *observations,* not the variable." CreationDate="2010-10-20T20:58:03.950" UserId="919" />
  <row Id="5439" PostId="3820" Text="@whuber I understood it in the same way (or I think so): we have a series of observations (= observed numerical values for a given variable), is there a way to keep the same SD by altering one or more of these values?" CreationDate="2010-10-20T21:08:38.413" UserId="930" />
  <row Id="5440" PostId="3820" Text="@chl We take it in the same way.  Although it is based on a different interpretation, Srikant's answer is nevertheless an interesting response.  As the comments afterwards show, though, it doesn't lead to very interesting solutions: there are too many ways one can change a random variable while preserving its variance." CreationDate="2010-10-20T21:13:51.623" UserId="919" />
  <row Id="5441" PostId="3823" Score="7" Text="Reminds me that in my early days as a referee i spent *far* too long reviewing a statistical paper that was eventually rejected by that particular journal, but the other referees and I suggested a more useful application for the method, and I also sketched an algebraic proof to replace an unsatisfactory simulation study in the manuscript. The authors have since got two published papers out of it. I'm not *annoyed* by that, but an acknowledgement such as &quot;we thank referees of an earlier version of the paper for helpful comments&quot; would have been good manners." CreationDate="2010-10-20T21:23:38.027" UserId="449" />
  <row Id="5442" PostId="3820" Text="@whuber: 'Interesting' is subjective. I believe that both interpretations are equally valid, interesting in their own right." CreationDate="2010-10-20T21:44:50.087" />
  <row Id="5443" PostId="3823" Text="@onestop Yes, I can imagine how disappointing such a situation might be..." CreationDate="2010-10-20T21:56:10.877" UserId="930" />
  <row Id="5444" PostId="3539" Text="I did not remove that tag ???!!!" CreationDate="2010-10-20T22:06:01.650" UserId="1564" />
  <row Id="5445" PostId="3812" Text="Thanks to all of you for your very good comments and suggestions." CreationDate="2010-10-20T22:08:02.857" UserId="1647" />
  <row Id="5446" PostId="3712" Text="Estimates of what? Using a version of the lincom statement above, you get the treatment effect for each quintile (the estimate of the treatment effect for the first quintile is just the coefficient on treatment)." CreationDate="2010-10-20T22:08:24.037" UserId="401" />
  <row Id="5447" PostId="3813" Text="@mbq don't feel bad, I only learned the name a week ago! " CreationDate="2010-10-20T22:40:01.307" UserId="795" />
  <row Id="5449" PostId="3792" Text="My background is civil/hydraulic engineering, i try to understand statistical discussions :)" CreationDate="2010-10-21T00:13:10.170" UserId="1637" />
  <row Id="5450" PostId="3798" Text="@chl: wow, you keep good company; very cool!  I would really appreciate some references, thanks!" CreationDate="2010-10-21T01:02:09.673" UserId="251" />
  <row Id="5451" PostId="3831" Text="the velocity components (3D) belong to a turbulent flow. Although the trend follows a sinusoidal pattern, there are high-frequency variations. in my case the trend follows 1/12hr frequency but measurement frequency is 1 Hz, i.e. if at time t speed is 20cm/s, speed at t+1 the speed might have any value saying 20+-5 cm/s. Hence, essentially i cannot predict the next value in a turbulent field.  " CreationDate="2010-10-21T01:11:11.963" UserId="1637" />
  <row Id="5452" PostId="3831" Text="could you please explain more about checking ID condition?" CreationDate="2010-10-21T01:11:48.547" UserId="1637" />
  <row Id="5453" PostId="3830" Text="residuals? you mean i should model the time series, then estimate residuals, ...." CreationDate="2010-10-21T01:14:24.110" UserId="1637" />
  <row Id="5454" PostId="3831" Text="Well, I do not know enough about your context to give a sensible comment but you have to model the velocity vector as a time series of some sort with an additive error term which you assume is iid. You estimate the model and test if the residuals are iid. Perhaps, you could ask another question with some context of the data and your goals are as far as data analysis is concerned. Perhaps, the issue of iid or not is not that relevant given what you want to achieve?" CreationDate="2010-10-21T01:39:33.467" />
  <row Id="5455" PostId="3828" Score="1" Text="+1 Log-linear sounds good given the plentiful requirements! :)  I'd also recommend Agresti's text on categorical data analysis." CreationDate="2010-10-21T03:24:56.497" UserId="251" />
  <row Id="5456" PostId="3830" Text="I don't think this answers K1's question. You are describing here how to check model assumptions in a usual regression analysis. " CreationDate="2010-10-21T03:49:37.703" UserId="1307" />
  <row Id="5457" PostId="3823" Text="@onestop, I hear you brother :-)" CreationDate="2010-10-21T03:51:03.220" UserId="1307" />
  <row Id="5458" PostId="3809" Text="@Srikant, a (hypothetical) Bayesian will have strong disagreements with your answer. She would have done something like this: prior  $\propto f(\alpha_1,\beta_1|-) \alpha + f(\alpha_2,\beta_2|-) (1-\alpha)$ and then put prior on $\alpha$. However, your answer will be a little less flexible than the Bayesian's answer." CreationDate="2010-10-21T04:16:41.607" UserId="1307" />
  <row Id="5459" PostId="3823" Score="7" Text="A few weeks ago I was given a paper to review and found that 85% of it had been published in another journal...by the same authors.  That, too, is still considered plagiarism.  For the last several years I have routinely submitted chunks of papers--especially abstracts, introductions, and conclusions--to Web search engines *before* doing any review.  I want to be sure the work is original before I invest any time in reading it." CreationDate="2010-10-21T04:40:42.427" UserId="919" />
  <row Id="5460" PostId="3794" Text="@whuber I kept my answer pretty software-neutral. It's good to have the Excel implementation. " CreationDate="2010-10-21T05:03:27.023" UserId="183" />
  <row Id="5461" PostId="3794" Score="1" Text="@Jeromy I see what you mean.  I was responding to your comment in answer to Alex's question.  Of greater interest to me is the idea that Excel can be used as a prototype for commands in R, Mathematica, (and even APL if anyone remembers it), provided you use its array-oriented procedures.  Excel itself is pretty bad as a statistical tool but good for rapid prototyping because it's easy to see and correct one's mistakes." CreationDate="2010-10-21T05:09:15.183" UserId="919" />
  <row Id="5462" PostId="3539" Text="@shadi No probs. An unfortunate rollback. I have added the tag again. Is it ok with you?" CreationDate="2010-10-21T06:05:58.893" UserId="930" />
  <row Id="5463" PostId="3817" Score="3" Text="I'd also argue that in psychology maximising prediction is not typically the theoretical aim, yet stepwise regression is all about maximising prediction, albeit in a quasi-parsimonious way. Thus, there is typically a disconnect between procedure and question." CreationDate="2010-10-21T06:28:49.813" UserId="183" />
  <row Id="5464" PostId="3794" Text="@whuber good point" CreationDate="2010-10-21T06:47:18.210" UserId="183" />
  <row Id="5465" PostId="3779" Text="As far as I remember Scrabble does not allow one letter words, so at least that part of the problem is solved ;)" CreationDate="2010-10-21T07:07:57.473" UserId="582" />
  <row Id="5466" PostId="3821" Text="To Whuber - &quot;rescale the result to match the original variance&quot; - here's a thought, thanks!" CreationDate="2010-10-21T07:11:07.757" UserId="253" />
  <row Id="5467" PostId="3820" Text="Thanks everyone for the replies.  I meant &quot;observations&quot; indeed.  I'm honored to be able to converse with all of you through here." CreationDate="2010-10-21T07:13:41.027" UserId="253" />
  <row Id="5468" PostId="3825" Text="Amazingly detailed answer Whuber - thank you very much!  This obviously leaves open the questions of non additive transformations - but for my curiosity needs - I am satisfied.  Thanks again! " CreationDate="2010-10-21T07:16:47.340" UserId="253" />
  <row Id="5469" PostId="3833" Text="@Jeromy I like this. I would add that (1) the OP is likely to be interested in studying asymmetrical relationships between all three variables, so that a SEM approach would make sense; proceeding this way would also (2) allow to account for specific measurement error at the level of the scales. " CreationDate="2010-10-21T07:27:34.010" UserId="930" />
  <row Id="5470" PostId="3818" Text="@Neelam (1) presumably you have three groups of variables and within each group, you have multiple variables (e.g., perhaps you have the Big 5 factors of personality within the personality group); (2) calling your variables categorical makes people think of 'unordered categorical' when the variables are probably best treated as numeric. Thus, in addition to providing more information, you may wish to change the title of your question." CreationDate="2010-10-21T08:14:28.030" UserId="183" />
  <row Id="5472" PostId="3827" Text="+1 esp as you gave a link, and hence traceability and attribution!" CreationDate="2010-10-21T08:43:51.230" UserId="449" />
  <row Id="5473" PostId="3838" Score="2" Text="I think that this question may be a bit to far off topic - even for me. But, why not try the [Biostar SE](http://biostar.stackexchange.com/)?" CreationDate="2010-10-21T09:24:54.273" UserId="8" />
  <row Id="5474" PostId="3839" Text="hehe great!  Thanks a lot ars" CreationDate="2010-10-21T09:49:51.213" UserId="1709" />
  <row Id="5475" PostId="3827" Text="@onestop Any idea for the difference between the two series of results?" CreationDate="2010-10-21T09:58:02.370" UserId="930" />
  <row Id="5476" PostId="3838" Text="Thanks, and done:  http://biostar.stackexchange.com/questions/3016/whats-the-difference-between-18s-rrna-and-18s-rdna" CreationDate="2010-10-21T10:02:09.780" UserId="386" />
  <row Id="5477" PostId="3840" Score="1" Text="I would suggest you to work on clarity of the question; this helps in getting answers." CreationDate="2010-10-21T10:56:34.243" UserId="88" />
  <row Id="5478" PostId="3840" Text="@mbq, i have updated to my post" CreationDate="2010-10-21T11:15:01.460" UserId="1655" />
  <row Id="5479" PostId="3795" Text="@onestop, with graphviz I was not able to draw an arrow to the center of a line without &quot;cracking&quot; the arrow that is pointed to. See my question at stackoverflow http://stackoverflow.com/questions/3718025/graphviz-dot-how-to-insert-arrows-from-a-node-to-center-of-an-arrow" CreationDate="2010-10-21T11:33:50.717" UserId="767" />
  <row Id="5480" PostId="3847" Text="@teucer Thanks for the ggplotized version :)" CreationDate="2010-10-21T13:02:54.367" UserId="930" />
  <row Id="5481" PostId="3689" Text="One of the best ways to improve a forecast is to average different forecasts (from different models, or from different experts or both...). You may want to look around on forecastingprinciples.com or look through past issues of the International Journal of Forecasting." CreationDate="2010-10-21T13:02:54.803" UserId="1352" />
  <row Id="5482" PostId="3488" Text="Other posters have noted that stationarity is important here. If both series have a linear upwards trend (one kind of nonstationarity), they will be correlated - but all the correlation may be due to the common trend, which may or may not be what we are interested in." CreationDate="2010-10-21T13:06:39.010" UserId="1352" />
  <row Id="5483" PostId="3841" Text="I am dealing with a similar problem, I guess you can use `pgmm` from **plm** package but as your response variable is binary I don't know exactly how to do it. Maybe others can elaborate... (And yes you are right: my understanding is whenever you have an endogenous variable, in this case the lagged value, you can't use REML to estimate because it is biased, so you need to use GMM.)   " CreationDate="2010-10-21T13:08:39.447" UserId="1443" />
  <row Id="5484" PostId="3846" Text="I'm sorry, it does not. I don't know how 2(k-1)sum(vi(di-kvi)) can become sum(di-kvi)^2 after k was estimated. Obviously, I have failed to grasp the significance of the constraint imposed. Please explain the significance of the constraint's role in solving the equation...Thank you." CreationDate="2010-10-21T13:21:25.237" UserId="1636" />
  <row Id="5485" PostId="3818" Score="2" Text="I'm inclined to close this question if further information isn't provided. At present it's impossible to see what is being asked." CreationDate="2010-10-21T13:43:57.840" UserId="8" />
  <row Id="5486" PostId="3791" Text="hi S.  yes we did transformed the data, then we standardize against the largest value, then transformed the standardized value" CreationDate="2010-10-21T13:49:28.767" UserId="1627" />
  <row Id="5487" PostId="1966" Text="your experience -- has run on 1k / 100k points, in 1d / 2d / 3d -- would also be useful." CreationDate="2010-10-21T14:16:47.237" UserId="557" />
  <row Id="5489" PostId="3791" Text="but was your design a Randomized Complete Block Design?" CreationDate="2010-10-21T14:32:09.557" UserId="1307" />
  <row Id="5490" PostId="3845" Text="I haven't used this package extensively myself. I think individual effect is indeed what you want. Two step will yield smaller standard errors (see the paper i linked to in my original answer). The number of instruments should be smaller than the number of years-1 you have. Try 2:9 (i think you have 9 years per region). You could also start by using the gretl implementation (which is simpler to use) and come back to R whence you have mastered the model a bit." CreationDate="2010-10-21T14:33:34.897" UserId="603" />
  <row Id="5493" PostId="3825" Text="@Tal I hope you're satisfied, because I can't do better than this: *every* transformation is additive!  Consider replacing x_i by arbitrary z_i; just define y_i = z_i - x_i." CreationDate="2010-10-21T14:53:51.797" UserId="919" />
  <row Id="5494" PostId="3807" Text="Thanks @whuber!  That's really helpful and I'm going to try it out." CreationDate="2010-10-21T14:54:07.553" UserId="1641" />
  <row Id="5496" PostId="3793" Text="@whuber, i don't understand the comment just above this one...it sounds like its in reply to something (that I don't see.)    (I also couldn't figure out how to contact you directly to ask that...)" CreationDate="2010-10-21T14:59:33.407" UserId="1641" />
  <row Id="5497" PostId="3793" Text="@Alex Members with sufficiently high reputation have the option to &quot;close&quot; a question.  This is typically done when a question is vague or off-topic.  It takes five votes to close, but before all five votes are in they are anonymous.  In this case somebody cast a vote shortly after your question appeared.  It's considered polite to explain such votes and to provide constructive suggestions for making the question acceptable, but that courtesy was not offered in this case." CreationDate="2010-10-21T15:04:07.123" UserId="919" />
  <row Id="5498" PostId="3845" Text="@kwak but the model specification is correct, isn't it? Btw, why the numbers of instruments should be smaller than years-1? If you look the example of **plm** vignette with EmplUK data set (p.23) they use 2:99 (I don't really understand why)?  I came across the following formula for number of instruments (T-1)*(T-2)/2+(T-3)...I'll have a look at gretl, thx!" CreationDate="2010-10-21T15:08:19.040" UserId="1443" />
  <row Id="5500" PostId="3849" Score="3" Text="@radek You will likely get excellent and authoritative answers to this question on the stata list (http://www.stata.com/statalist/ ).  It is THE place to go with issues about how Stata works." CreationDate="2010-10-21T15:11:22.223" UserId="919" />
  <row Id="5501" PostId="3849" Score="1" Text="i would agree with @whuber but no harm in asking here either. The question is on-topic as far as this site is concerned." CreationDate="2010-10-21T15:15:20.060" />
  <row Id="5502" PostId="3809" Text="@suncoolsu Sure you can do that as well. However, if you choose the prior for $\alpha$ to be very tight around 0.8 then your suggestion essentially collapses to mine. However, I agree imposing a prior on $\alpha$ is a bit more flexible than assuming that it is 0.8." CreationDate="2010-10-21T15:29:53.990" />
  <row Id="5503" PostId="3725" Text="@ronaf Allegedly Excel 2010 now has both, COVARIANCE.P and COVARIANCE.S.  However, the documentation is so dismal one cannot really be sure!  (In the older versions if you know about covariance at all you're probably knowledgeable enough to be able to multiply the result by n/(n-1) in the formula.)  For more info visit http://office.microsoft.com/en-us/excel-help/statistical-functions-reference-HP010342920.aspx" CreationDate="2010-10-21T16:36:14.737" UserId="919" />
  <row Id="5504" PostId="3779" Text="@nico good point, but I think this is only for mid-game. 1 letter words either don't require one to play a letter, or would allow one to place a single letter anywhere on the board, both clearly unacceptable. However, I was thinking of the opening move. In fact, the question can be compactly stated, for those familiar with Scrabble, as &quot;what is the probability that the first player will have to pass?&quot;" CreationDate="2010-10-21T16:43:55.547" UserId="795" />
  <row Id="5505" PostId="3857" Text="Although you will probably get some good info from posters here, I would also suggest you check out StackOverflow. I wouldn't be surpised if they already have similar questions and answers already posted." CreationDate="2010-10-21T17:14:22.893" UserId="1036" />
  <row Id="5506" PostId="3856" Text="what type of data are you working with? From your examples I assume you are dealing with time series? " CreationDate="2010-10-21T17:57:52.243" UserId="582" />
  <row Id="5507" PostId="3827" Score="1" Text="@chi Oh dear, I think -sizefx- contains a coding error! It's ignoring the values of one variable for which the other variable is missing (the problem is with the use of -marksample-). I'll contact the author. I've never used it before myself (i'm relieved to say...)" CreationDate="2010-10-21T19:06:52.657" UserId="449" />
  <row Id="5508" PostId="3827" Text="You also make a good point that it would be better if -sizefx- supported a &quot;by()&quot; option. I'm also not sure why it's referring to Cohen's d as Hedges' g. The latter really involves a more complex bias correction. All in all, i'd advise against using -sizefx- for the time being!" CreationDate="2010-10-21T19:21:36.567" UserId="449" />
  <row Id="5509" PostId="3864" Text="Agreed that things need to be user friendly.  Since people get very protective over their working practices, any changes must make people's lives easier or they'll fail. " CreationDate="2010-10-21T20:13:18.803" UserId="478" />
  <row Id="5510" PostId="3863" Score="1" Text="Absolutely agreed on the version control.  I use it; as do a substantial proportion of the developers and statisticians.  (I'd like to see 100% adoption but that's another pipe-dream for now.)  The hard bit is getting non-techies to use it.  Any ideas appreciated." CreationDate="2010-10-21T20:18:11.890" UserId="478" />
  <row Id="5511" PostId="3793" Text="@Alex Your question makes sense, from a statistical POV. I don't understand the vote to close, given that there are some very basic questions for which browsing online help would have suffice (and is educative)... and those questions were never voted down or voted to be closed. Anyway, that's community life. However, may I also remind you that you can also upvote the question you accepted as a the correct one. " CreationDate="2010-10-21T20:48:56.443" UserId="930" />
  <row Id="5512" PostId="3827" Text="@onestop (+1) Good to hear that you spotted the problem (not enough Stata experience myself). Anyway, the distinction between Cohen's d and Hedges's g is rather confusing, even on Wikipedia (I admit this is not really a reference, though). In addition to not supporting a `by()` option, no CIs are computed (we'd need to rely on a non-central $t$ distribution, or use some kind of bootstrap approach)." CreationDate="2010-10-21T21:02:58.003" UserId="930" />
  <row Id="5513" PostId="3862" Score="1" Text="Excellent links.  I think two important messages for me to pass on are: we need more automated data checking and I need to start explaining about separating data entry and data presentation." CreationDate="2010-10-21T21:06:04.643" UserId="478" />
  <row Id="5514" PostId="3853" Score="1" Text="@Skarab I can understand that reading online help might seem a tiresome activity at first sight (still, it's very educative and often help to capitalize knowledge on a particular software), but could you at least accept answer(s) you find helpful for your ongoing R activities?" CreationDate="2010-10-21T21:09:55.990" UserId="930" />
  <row Id="5515" PostId="3863" Text="@Richie Cotton: I don't know why, but version control seems to be a difficult concept for non-techies to grasp. People continue to just make some changes to a file, rename it and send it via email. How I hate those &quot;PaperDraftCorrectedByJohnRevision3RewroteByLeslie-NewVersion3.doc&quot; files..." CreationDate="2010-10-21T21:14:05.317" UserId="582" />
  <row Id="5516" PostId="3838" Text="Can we delete this question?" CreationDate="2010-10-21T21:45:52.423" UserId="386" />
  <row Id="5517" PostId="3868" Score="1" Text="Have you tried dropping that zero from the analysis?  I don't think zero is a part of the gamma distribution.  Also: are you really sure it's zero, and not just very, very small?" CreationDate="2010-10-21T23:57:17.420" UserId="71" />
  <row Id="5518" PostId="3846" Text="&quot;The idea is the following insight: Suppose that we choose k such that the above constraint is satisfied then it immediately follows that:&#xA;                                          ∑i(di−lvi)2=∑i(d−kvi)2+(k−l)2∑iv2i                                                      &quot;                                                                Could you please show the intermediate steps? I really cannot see how that equation follows immediately from satisfying the constraint. Sorry for the inconvenience caused." CreationDate="2010-10-22T00:36:17.620" UserId="1636" />
  <row Id="5519" PostId="3252" Text="I'm not quite sure why the size of your sample matters. Can you offer anymore specific reasoning as to why you think it is different for small n than it is for big n?" CreationDate="2010-10-22T03:39:10.140" UserId="1036" />
  <row Id="5520" PostId="1116" Text="Although this is the only Paul Allison text I have read, I would like to say various other texts of his have come highly recommended for me to read. Even if you don't use SAS (I have in the past so I was familiar with the code, although when I read this book I had completed migrated to other software) it is a really excellent book on survival analysis." CreationDate="2010-10-22T03:47:03.580" UserId="1036" />
  <row Id="5521" PostId="3650" Text="Both of the answers by Srikant and whuber were equally informative in helping me answer this question. I wish I could give both an accepted checkmark, but I chose whubers response simply for the added context in helping to demonstrate how I would integrate given my prior distributions." CreationDate="2010-10-22T05:00:40.453" UserId="1036" />
  <row Id="5522" PostId="3651" Text="@Srikant, I assume it would be inappropriate to use the observed 100 realizations for my prior distributions in this context? Hence the need for &quot;priors&quot; to begin with." CreationDate="2010-10-22T05:02:33.790" UserId="1036" />
  <row Id="5523" PostId="3252" Score="1" Text="@Andy Because then it becomes very difficult to consider an holdout sample and/or class imbalance with very limited sample size ($13&lt;n&lt;25$) generally yields larger classification error rate when applying CV; some individuals might be considered as outliers when studying bivariate distributions; and measures gathered on instruments with their own measurement error are less reliable (small $n$, large $\sigma$). In a certain sense, it is sometimes difficult to disentangle an unexpected relationship from an artifact." CreationDate="2010-10-22T05:29:38.510" UserId="930" />
  <row Id="5524" PostId="3876" Score="1" Text="Should one set lmer's REML argument to FALSE when generating those models since they'll eventually be compared using the anova() function?" CreationDate="2010-10-22T05:47:06.053" UserId="364" />
  <row Id="5525" PostId="3875" Score="1" Text="IMHO t-test may not be a good idea here" CreationDate="2010-10-22T06:52:40.040" UserId="1307" />
  <row Id="5526" PostId="3876" Score="3" Text="When comparing models using likelihood-ratio tests, you can compare different *random effects* structures using REML (restricted/residual maximum likelihood, as above), but you must use ML (maximum likelihood) to compare different *fixed effect* models." CreationDate="2010-10-22T07:36:08.553" UserId="449" />
  <row Id="5527" PostId="3872" Text="How are you quantifying the results? Numbers of events, or continuous measurements (on a scale with a true zero, i hope) or...?" CreationDate="2010-10-22T07:39:48.163" UserId="449" />
  <row Id="5528" PostId="3877" Text="@onestop (+1) Nice follow-up. My understanding of the question was &quot;how to calculate an SMD in the case of a logistic regression?&quot;. Maybe worth clarifying that this is indeed the case." CreationDate="2010-10-22T07:57:54.167" UserId="930" />
  <row Id="5529" PostId="3871" Text="Thanks Rob and Srikant for your explanation. I understand independent events now. Just out of curiosity, if I were to draw the Venn Diagram of aforementioned events, are there any characteristics that I can look for to reliably identify independent events?" CreationDate="2010-10-22T08:15:53.343" UserId="1636" />
  <row Id="5530" PostId="3848" Text="@Shane Thanks for your input. Will test this approach and will update. Also, do you think, i can use Kolmogorov-Smirnov Test in my case. Please see the plot, http://img254.imageshack.us/img254/9645/25971376.gif" CreationDate="2010-10-22T08:40:08.237" UserId="1655" />
  <row Id="5531" PostId="3846" Text="Thank you very much, Srikant." CreationDate="2010-10-22T09:07:18.913" UserId="1636" />
  <row Id="5532" PostId="3871" Text="@Sara No, You cannot. Venn diagrams will only help you write down the formulas for the various possible relationships between the events (e.g., $P(A \cup B) = P(A) + P(B) - P(A \cap B)$. But, the diagram itself will not tell you anything about the probabilities of the events under consideration." CreationDate="2010-10-22T09:34:38.537" />
  <row Id="5533" PostId="3871" Text="@Srikant: Noted. Thank you so much for your assistance." CreationDate="2010-10-22T09:44:16.360" UserId="1636" />
  <row Id="5534" PostId="3847" Text="@teucer: I hope you don't mind, but I added the output of your commands." CreationDate="2010-10-22T10:16:36.597" UserId="8" />
  <row Id="5535" PostId="3847" Text="@csgillespie no problem :)" CreationDate="2010-10-22T10:52:37.140" UserId="1443" />
  <row Id="5536" PostId="3252" Text="I think I can understand that sentiment if what your interested in is solely classification. I think for causal inference the problems with data snooping are the same (i.e. the problems aren't solved by increased power to identify relationships). I'll try to formulate this opinion into an answer. I may ask a question on the main forum in the meantime about the use of cross-validation for causal inference, as I have not come across any work in my field that does this." CreationDate="2010-10-22T12:48:13.493" UserId="1036" />
  <row Id="5537" PostId="3876" Text="@onestop: Ah! Good to know, thanks!" CreationDate="2010-10-22T12:56:16.673" UserId="364" />
  <row Id="5538" PostId="3833" Text="Thank You Jeromy for clearing the concept. I am really grateful. Can you please explain PCA." CreationDate="2010-10-22T13:12:10.107" UserId="1649" />
  <row Id="5540" PostId="3252" Score="1" Text="@Andy Thanks. Hopefully, your question will receive a lot of interesting answers." CreationDate="2010-10-22T13:40:25.370" UserId="930" />
  <row Id="5541" PostId="3865" Text="Thanks @Andy W, looks relevant but Wiley want $$. &quot;Nature of my data&quot;: messing about with Kdtree variants / KNN; as I said, looking for an overview." CreationDate="2010-10-22T13:45:08.010" UserId="557" />
  <row Id="5542" PostId="3891" Text="depends on which package list ? http://www.stat.ucl.ac.be/ISdidactique/Rhelp/doc/html/packages.html" CreationDate="2010-10-22T14:15:05.597" UserId="603" />
  <row Id="5544" PostId="3891" Text="@kwak Your link must be to an outdated mirror." CreationDate="2010-10-22T14:17:41.510" UserId="5" />
  <row Id="5546" PostId="3877" Text="Thanks, these answers are incredibly helpful.  To clarify (hopefully), I intend to calculate the Cohen's d statistic for the difference in mean values of the predictions generated by a logit model.  I suspect that there are simpler ways to estimate the effect size of a logit model, but I want to compare results to a meta-analysis reporting Cohen's d.  Thanks again." CreationDate="2010-10-22T14:28:42.293" />
  <row Id="5547" PostId="3891" Text="@Shane:&gt; Ran/build on latest R (here on ubuntu). Thanks." CreationDate="2010-10-22T14:30:59.200" UserId="603" />
  <row Id="5548" PostId="3865" Text="@Denis, I have added in a link to the pdf. Often times if you search for the publication on Google Scholar a version is posted free to the public. The supplementary material I posted is free as well, and the R code would be insightful even without access to the paper. The description &quot;messing about with Kdtree variants / KNN&quot; is not specific enough to be helpful. Based on that I imagine what I quoted is not that useful unless you are explicitly interested in geographic data." CreationDate="2010-10-22T14:31:23.613" UserId="1036" />
  <row Id="5549" PostId="3886" Text="@gd047:&gt; Can you add a link (paper, explanation, hypothesis,...) to the denominator used in the first one ? (i.e. your $\hat{\sigma}_1$) - thanks." CreationDate="2010-10-22T14:34:13.370" UserId="603" />
  <row Id="5550" PostId="3720" Text="@kwak, When I get the time I will post the question on the role of regression model assumptions when one is solely interested in prediction. Feel free to post it yourself in the meantime though if you would like. " CreationDate="2010-10-22T14:44:15.703" UserId="1036" />
  <row Id="5551" PostId="3884" Text="Some very interesting points here.  Convincing people to simplify and standardise their spreadsheets is likely to be more successful than getting them to abandon them.  Also I had no idea that version control could integrate with Excel.  Nice to know." CreationDate="2010-10-22T14:46:21.737" UserId="478" />
  <row Id="5552" PostId="3885" Score="2" Text="Don't these become unbiased only in conjunction with additional distributional assumptions?  For example, the relationship between the range and the SD *has* to depend on the shape of the distribution.  In that context, it's worth observing that many more unbiased estimators of SD have been studied, such as those based on linear combinations of quantiles and ranks." CreationDate="2010-10-22T15:06:59.537" UserId="919" />
  <row Id="5553" PostId="3872" Text="@pom I hope you can clarify this query.  In addition to the issues raised by @onestop, your modification of both &quot;samples&quot; and &quot;models&quot; by the word &quot;independent&quot; makes one wonder about your precise meaning and your synonymous use of &quot;ratio&quot; and &quot;difference&quot; in the same sentence raises questions about what you mean by those words.  One possible interpretation is that each sample consists of a set of ratios; another interpretation is that you are estimating some kind of statistic in each sample, taking the ratio of those two numbers, and want to compare it to some standard value (such as 1.0)." CreationDate="2010-10-22T15:13:55.853" UserId="919" />
  <row Id="5554" PostId="3871" Score="1" Text="@Sara, if you look at a probability density function http://en.wikipedia.org/wiki/Probability_density_function of the two events fxy then you will get a rectangular distribution if they are independent. " CreationDate="2010-10-22T15:18:51.173" UserId="1673" />
  <row Id="5555" PostId="3877" Score="1" Text="Thanks Fred. I've looked up some old refs and an alternative way to convert a log-odds ratio (i.e. the coef for a binary variable in a logistic regression model) to an effect size comparable to Cohen's d is to multiply the logOR by √3/π. See:&#xA;&#xA;Hasselblad V, Hedges LV.  Meta-analysis of screening and diagnostic tests.  Psychological Bulletin 1995; 117:167-178. http://content.apa.org/journals/bul/117/1/167&#xA;&#xA;Chinn S.  A simple method for converting an odds ratio to effect size for use in meta-analysis.  Stat Med 2000;19:3127-3131. http://www3.interscience.wiley.com/cgi-bin/abstract/75500445" CreationDate="2010-10-22T15:22:06.093" UserId="449" />
  <row Id="5556" PostId="3894" Score="1" Text="That wouldn't measure dispersion (no dimensionless value possibly could): it's a measure of *shape* for a non-negative variable.  The range itself is a (poor) measure of dispersion." CreationDate="2010-10-22T15:23:43.033" UserId="919" />
  <row Id="5557" PostId="3894" Text="@whuber: range is a difference between max and min values." CreationDate="2010-10-22T15:25:08.837" UserId="219" />
  <row Id="5558" PostId="3894" Score="2" Text="@SilentGhost I think everyone knows what the range is, but there are two possible areas in which it can be applied: to a *sample* or to a *distribution*.  In the latter case the range is not often used because the range is often infinite.  Thus I assumed you were asking about a sample.  Regardless, in either case the ratio of the range to a central statistic cannot measure dispersion." CreationDate="2010-10-22T15:30:30.230" UserId="919" />
  <row Id="5559" PostId="3896" Score="1" Text="That's a reasonable guess.  It still does not measure dispersion,though!" CreationDate="2010-10-22T15:31:17.513" UserId="919" />
  <row Id="5560" PostId="3868" Score="1" Text="@Matt Dropping a zero is usually not a good idea unless the dataset is so large that losing an extreme value will make no difference in the analysis.  Your implicit suggestion of treating it as very small is good, but then the result can be sensitive to the value chosen.  One could check for that in various ways.  A more formal approach is to treat the zero as a left-censored value (censored perhaps at the second lowest value) and use methods that handle censored or interval-valued data (which includes ML)." CreationDate="2010-10-22T15:35:35.303" UserId="919" />
  <row Id="5561" PostId="3871" Text="@Kortuk: Thank you for the link. I will study it." CreationDate="2010-10-22T15:37:27.807" UserId="1636" />
  <row Id="5562" PostId="3539" Text="Yes, Thank you." CreationDate="2010-10-22T15:56:54.887" UserId="1564" />
  <row Id="5563" PostId="3070" Text="another oddity of Matlab: `std(randn(1))` returns `0`, not `nan`. This makes utterly no sense. " CreationDate="2010-10-22T16:08:29.657" UserId="795" />
  <row Id="5564" PostId="3803" Text="@whuber I really want to see the PIE solution, but it looks like MC would have to suffice. " CreationDate="2010-10-22T16:10:59.347" UserId="795" />
  <row Id="5565" PostId="3865" Text="Thanks again @Andy W. Does my wish for *both* analysis and synthesis make sense ?" CreationDate="2010-10-22T16:12:44.760" UserId="557" />
  <row Id="5566" PostId="3871" Score="1" Text="@Sara, the wiki is not perfect for the multivariable case. There are better sources. but a prob density function of 2 variable can be very interesting." CreationDate="2010-10-22T16:15:21.077" UserId="1673" />
  <row Id="5567" PostId="3779" Text="@nico Thank you for that clarification.  Theoretically a similar issue pertains in dictionaries containing all possible two-letter combinations as words: when that's the case, any hand of 2 or more letters automatically contains a word.  @shabbychef's comment about mid-game shows how irrelevant the original question is to most of Scrabble, because in mid-game you have available an array of word parts (prefixes, suffixes, and even middle sections) in addition to the 7 letters in your hand.  This greatly increases the chances of being able to make a word." CreationDate="2010-10-22T16:29:30.057" UserId="919" />
  <row Id="5568" PostId="3897" Score="1" Text="With one df the range is not even an approximation--it's directly proportional to the SD :-)." CreationDate="2010-10-22T16:31:05.583" UserId="919" />
  <row Id="5569" PostId="3865" Text="@Denis, yes your distinction between analysis and synthesis make sense given your description in the question (although I would use the word &quot;simulate&quot; instead of synthesize). I'm not sure if reducing clustering to a single coefficient for simulation will be all that insightful, and the geographic coefficients for clustering will likely be harder to implement in higher dimensional space and would probably take custom coding. The more refined a question the more applicable answers you will receive are. " CreationDate="2010-10-22T16:33:14.127" UserId="1036" />
  <row Id="5570" PostId="3873" Text="This is such a nice summary!  I am grateful for your effort in sharing it with us.  I can't help upvoting it despite your appeals not to :-).  (I don't think it will change your rep, anyway, and there is a lot of merit to having it appear close to the original question if it does get substantially upvoted.)" CreationDate="2010-10-22T16:38:54.270" UserId="919" />
  <row Id="5571" PostId="3887" Text="Python isn't widespread in our organisation but it looks like an interesting project.  I'll see if I can pinch some ideas on how things should be done from their documentation." CreationDate="2010-10-22T17:02:17.420" UserId="478" />
  <row Id="5572" PostId="3829" Text="Agree -- struggling to understand what the author(s) meant before even evaluating the scientific content is really annoying. " CreationDate="2010-10-22T17:04:53.940" UserId="1355" />
  <row Id="5573" PostId="3880" Text="@csgillespie I don't really understand why this option was added to `hist()` (`labels=T`) and not `barplot()`." CreationDate="2010-10-22T18:07:58.037" UserId="930" />
  <row Id="5574" PostId="3883" Text="@gd047 (+1) This is the hard way :) I like it." CreationDate="2010-10-22T18:08:58.240" UserId="930" />
  <row Id="5575" PostId="3877" Text="@Fred Still, it is not clear (sorry for that!) if your model includes a single predictor or more. In the latter case, SMDs are good summary statistics but they are not adjusted for other covariates of interest. @onestop I think there's a typo and you meant &quot;the coef relating a predictor to the binary outcome&quot; instead of &quot;the coef for a binary variable...&quot;." CreationDate="2010-10-22T18:18:45.753" UserId="930" />
  <row Id="5576" PostId="3689" Text="Thanks Stephan, I'll check it out." CreationDate="2010-10-22T19:33:57.950" UserId="1611" />
  <row Id="5577" PostId="3902" Score="1" Text="The easy way to find out is to feed it some simple datasets having ties!" CreationDate="2010-10-22T21:12:33.137" UserId="919" />
  <row Id="5578" PostId="3891" Score="1" Text="That mirror was last updated in October 2003. It's better to use http://cran.r-project.org/web/packages/" CreationDate="2010-10-23T00:10:01.837" UserId="159" />
  <row Id="5579" PostId="3904" Text="@whuber +1 It will take me some time to digest your answer. I admit that asking for an intuition for the CLT within the constraints I imposed may be nearly impossible. " CreationDate="2010-10-23T01:42:43.967" />
  <row Id="5580" PostId="3895" Text="&quot;Orthogonal regression&quot; is another thing one might encounter when looking for methods to deal with contaminated abscissas and ordinates." CreationDate="2010-10-23T02:31:14.540" UserId="830" />
  <row Id="5581" PostId="3833" Text="@Neelam no problems. PCA = Principal Components Analysis, a data reduction technique: see this excellent tutorial by Lindsay I Smith: http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf" CreationDate="2010-10-23T02:40:22.643" UserId="183" />
  <row Id="5582" PostId="3907" Text="I think the answers to this question will be helpful to you. http://stats.stackexchange.com/q/3611/1036" CreationDate="2010-10-23T02:43:04.073" UserId="1036" />
  <row Id="5583" PostId="3907" Score="1" Text="Have you seen the August issue of The American Statistician?  The article &quot;Two Pitfalls in Survival Analyses of Time-Dependent Exposure&quot; covers exactly this issue.  Abstract at  http://pubs.amstat.org/doi/abs/10.1198/tast.2010.08259" CreationDate="2010-10-23T02:44:43.033" UserId="919" />
  <row Id="5585" PostId="2700" Text="Good call on the Lindsay I Smith manuscript - just read it today; very helpful." CreationDate="2010-10-23T05:58:21.330" UserId="901" />
  <row Id="5586" PostId="3885" Text="Yes, indeed. Under the normality assumption." CreationDate="2010-10-23T07:21:27.607" UserId="339" />
  <row Id="5587" PostId="3907" Text="@whuber Sorry, I can't see why that AmStat article is relevant. Where is the time-*dependent* exposure in the question?" CreationDate="2010-10-23T07:25:01.637" UserId="449" />
  <row Id="5588" PostId="3877" Score="1" Text="@chl Yes, thanks for the correction. Was a bit more than a typo to be honest, more of a thinko. This conversion requires some assumptions, of course - i think they amount to requiring that the predictor should have (approx) the same variance in those with and w/o the outcome after a transformation, if required, to make the distribution of the predictor approximately normal/logistic (strictly logistic, but the two are hard to distinguish without a huge dataset)." CreationDate="2010-10-23T07:38:39.607" UserId="449" />
  <row Id="5590" PostId="3886" Text="1. Quality Control and Industrial Statistics - Duncan&#xA;2. Statistical Quality Control - Grant, Leavenworth&#xA;3. Introduction to Statistical Quality Control - Montgomery" CreationDate="2010-10-23T08:51:34.830" UserId="339" />
  <row Id="5591" PostId="3912" Text="Why do you say hypothesis tests are often meaningless for odds ratios, any more than any other effect estimate? I'd stress instead that confidence intervals are more useful than standard errors for odds ratios and other estimates with asymmetric sampling distributions in finite samples." CreationDate="2010-10-23T08:53:17.850" UserId="449" />
  <row Id="5592" PostId="3914" Text="Thanks. But what about frequentist confidence intervals specifically? Are there any circumstances at all where they would be relevant?" CreationDate="2010-10-23T09:33:52.977" UserId="1393" />
  <row Id="5593" PostId="3895" Text="+1 This is yet a niche only in statistics; more complex least squares methods (not only adding X variability but also different penalties for points based on error approximations) are common in experimental physics; ROOT framework has dozens of such." CreationDate="2010-10-23T11:38:36.590" UserId="88" />
  <row Id="5594" PostId="1280" Score="2" Text="http://imgur.com/0dsVC.gif" CreationDate="2010-10-23T11:59:06.837" UserId="830" />
  <row Id="5595" PostId="3912" Text="@onestop Well, I was partly thinking of what you say about &quot;asymmetric sampling distributions...&quot; (and it seems I was not so clear), but also of the fact that in epidemiological studies we are generally most interested in CIs (that is, how precise is our estimate) than HT." CreationDate="2010-10-23T12:33:33.110" UserId="930" />
  <row Id="5596" PostId="3918" Score="1" Text="@Srikant. No! This is how classical CIs bite. Let's assume for simplicity that the amount of cereal filled in a box is normal with mean $\mu$ and variance $\sigma^2$. The confidence interval of $\mu$ is based on its *sampling* distribution which is different. A particular CI may be way off due to sampling errors and then it will have no relation to how the machine performs. If you were to repeatedly sample and repeatedly form CIs then 95% of them would be right, but that is no consolation. " CreationDate="2010-10-23T13:44:59.757" UserId="1393" />
  <row Id="5597" PostId="3920" Text="@richie (+1) `psych::alpha` does a better job than `score.items`, thanks for adding that. I never remember the one I should use!" CreationDate="2010-10-23T13:53:50.730" UserId="930" />
  <row Id="5598" PostId="3918" Score="1" Text="@Jyotirmoy Of course, a specific CI may be way-off. In other words, there is a 5% chance that the CI does not contain the true value. Nevertheless, the interpretation I gave is consistent with how CIs are actually constructed. We imagine using the method repeatedly and construct the CI such that the probability that the observed CI contains the true value is 0.95. Notice that my answer does not say anything about the probability of where the true value actually lies as that is a statement that can only be made with credible intervals and not confidence intervals." CreationDate="2010-10-23T14:03:22.307" />
  <row Id="5599" PostId="3918" Score="1" Text="@Jyotirmoy Lower/Upper bounds for a $(100-\alpha)$% CI of an observed mean are constructed under $H_0$, where the sampling distribution of a mean (or a difference of means) is the one you assumed depending on your sample ($t$ or $z$ distribution). I found Srikant's answer correct, and his interpretation doesn't seem to go beyond the experiment that was framed. CIs are random variables." CreationDate="2010-10-23T14:09:23.740" UserId="930" />
  <row Id="5600" PostId="3918" Text="@Srikant. I perhaps misunderstood &quot;method=machine&quot; in the answer. I thought you were saying that 95% of all boxes coming out of the assembly line would have weights within the 95% confidence interval derived from a particular sample of the boxes. " CreationDate="2010-10-23T14:43:15.860" UserId="1393" />
  <row Id="5601" PostId="3914" Text="@onestop I believe having different priors is a non issue (at least from the objective Bayesian point of view), if it happens that you have different knowledge about the situation at hand. We meed to see the priors as a way of casting our a priori information. I know that it is not simple...  " CreationDate="2010-10-23T14:57:15.250" UserId="1443" />
  <row Id="5602" PostId="3914" Text="@Jyotirmoy About bayesian vs. frequentist approaches, interesting points were made here: http://stats.stackexchange.com/questions/1611/do-working-statisticians-care-about-the-difference-between-frequentist-and-bayesi/3370#3370" CreationDate="2010-10-23T15:23:13.577" UserId="930" />
  <row Id="5603" PostId="3925" Score="1" Text="@Tal (+1) This remind me of the `pairs.panels` function in the `psych` package by W Revelle." CreationDate="2010-10-23T20:03:15.373" UserId="930" />
  <row Id="5604" PostId="3904" Text="@Srikant: How did you fix the line numbering?  (Thank you!)  I looked at the source but couldn't detect any changes." CreationDate="2010-10-23T20:53:58.943" UserId="919" />
  <row Id="5606" PostId="3904" Text="@whuber You need to type a space before you start the next paragraph so that it is indented. So, all I did was to add a space before &quot;The first generalization of the CLT adds,..&quot; and &quot;The next generalization, and perhaps the ..&quot;." CreationDate="2010-10-23T21:02:19.203" />
  <row Id="5607" PostId="3924" Text="I'm missing something: you have your dataset so you compute its mean *m* and standard deviation *s*.  The value at *z* standard deviations from the mean therefore equals m + z*s for any *z* you like.  Where is the problem?" CreationDate="2010-10-23T21:19:14.023" UserId="919" />
  <row Id="5608" PostId="3904" Text="@Srikant Thank you again.  That was too subtle for me to see, but I'll remember it now." CreationDate="2010-10-23T21:20:17.790" UserId="919" />
  <row Id="5609" PostId="3927" Text="I've been looking at my data, and I misrepresented the question a little bit." CreationDate="2010-10-23T21:47:35.743" />
  <row Id="5610" PostId="3927" Text="@Jane If the added context makes my answer irrelevant I will delete my answer or perhaps edit to fit your context better." CreationDate="2010-10-23T21:49:08.173" />
  <row Id="5611" PostId="3924" Text="@whuber Perhaps, the problem is that we cannot assume that the distribution is normal? It seems to me that the values are bounded from below (at 0) and from above (at 255). The normal may not make sense unless the standard deviation is very small and the mean far from the the boundary points. @Dan Could you please clarify?" CreationDate="2010-10-23T21:51:59.663" />
  <row Id="5612" PostId="3925" Text="Interesting. I did come across that code, but never knew it also existed in the psych package.  I am sure it inspired me in someway when I wrote that post (I should add this to the credits on the post...)" CreationDate="2010-10-23T22:11:25.700" UserId="253" />
  <row Id="5613" PostId="3924" Score="1" Text="@Srikant That's a reasonable guess.  But nothing in the current statement requires assuming any kind of distribution at all.  Perhaps the question really is about how to *interpret* values of the form m + z*s?  Regardless, although the question makes sense, it requires some amplification in order to elicit an answer that would be genuinely helpful." CreationDate="2010-10-23T22:26:49.670" UserId="919" />
  <row Id="5614" PostId="3929" Score="1" Text="@Jane Please add the above information to your question using the 'edit' link below the tags and delete this 'answer' as it really does not answer your question." CreationDate="2010-10-23T22:26:57.187" />
  <row Id="5615" PostId="3914" Text="@chl. Thank you!" CreationDate="2010-10-24T05:13:12.027" UserId="1393" />
  <row Id="5616" PostId="3909" Text="So you don't think repeatability in effect estimates can be useful? Although you are not alone in your conception of what proof of causality is , I think it is quite narrow. We will never be able to indefinately prove a causal relationship, even with an experiment, absent all the evidence in the universe. Hence in my opinion the goal is to proffer evidence that whatever relationship we estimate is as close to the truth given the information we do know. Given that don't you think repeatability in prediction from a training set to a hold out sample could be a useful check on inferences made?" CreationDate="2010-10-24T05:48:17.460" UserId="1036" />
  <row Id="5617" PostId="3909" Text="I appreciate your comments as well, and I completely agree that inferences are heavily dependent on logic and the research design." CreationDate="2010-10-24T05:50:03.480" UserId="1036" />
  <row Id="5618" PostId="3917" Text="Thank you for the reference. So say you are not concerned about model selection, could cross validating the effect estimates of the training data set to the hold out dataset be useful? " CreationDate="2010-10-24T05:54:54.043" UserId="1036" />
  <row Id="5619" PostId="3934" Text="Hi Michael, I feel that some combination of your answer and Whuber one is good, removing the second moment part :)  ,  Thanks)" CreationDate="2010-10-24T07:13:20.963" UserId="253" />
  <row Id="5620" PostId="3932" Text="Thank you Whuber.  I have to teach the students with the n-1 correction, so dividing in n alone is not an option.  As written before me, to mention the connection to the second moment is not an option.  Although to mention how the mean was already estimated thereby leaving us with less &quot;data&quot; for the sd - that's important.  Regarding the bias of the sd - I remembered encountering it - thanks for driving that point home.  Best, Tal" CreationDate="2010-10-24T07:15:34.927" UserId="253" />
  <row Id="5621" PostId="3922" Score="1" Text="BTW, A question about *Graphing Likert scale responses* just came across Andrew Gelman's weblog yesterday :) http://j.mp/aBm8mZ" CreationDate="2010-10-24T08:46:37.780" UserId="930" />
  <row Id="5622" PostId="3937" Text="@user1689 As much as I'd like to be able to ask questions along the lines of &quot;what is the best free psychological test to measure X?&quot;, this seems to me to be outside the scope of this site." CreationDate="2010-10-24T09:01:35.437" UserId="183" />
  <row Id="5623" PostId="3924" Text="In most cases the values will be nearer the high range (ie. nearer 255). The SD can also vary quite a lot unfortunately. As I mentioned in my post, I'm completely new to stats, so the formula you gave whuber might be what I want. I'll need to play around with various datasets to see if it works or not." CreationDate="2010-10-24T09:37:32.653" UserId="1683" />
  <row Id="5624" PostId="3937" Text="This question belongs on meta; since migration is currently not working, please repost it there." CreationDate="2010-10-24T10:25:12.223" UserId="88" />
  <row Id="5625" PostId="3941" Text="That's a nice answer :)" CreationDate="2010-10-24T11:19:21.560" UserId="253" />
  <row Id="5626" PostId="3942" Text="Thanks onestop - I didn't think there would be an answer from that direction.  Any way to sum-up the intuition, or is that not likely to be possible?  Cheers, Tal" CreationDate="2010-10-24T11:20:45.130" UserId="253" />
  <row Id="5627" PostId="3942" Text="I couldn't do so myself, but a book reviewer summarised the approach in a paragraph in Amer. Stat. in 1993: http://www.jstor.org/stable/2684984. &#xA;&#xA;I'm not sure it's really practical to use this approach with your students unless you adopt it for the entire course though." CreationDate="2010-10-24T12:10:18.830" UserId="449" />
  <row Id="5629" PostId="3909" Text="Andy, I've edited my post to address your comments.  Also, I don't mean to suggest that causal inference cannot be done outside of the context of a designed experiment.  Nonetheless, it is more difficult and less certain in observational studies and we shouldn't look to model building procedures to help us with that problem.  Rather, we should try to better understand the problems for which we are attempting to understand causal relationships." CreationDate="2010-10-24T14:53:16.550" UserId="485" />
  <row Id="5630" PostId="3934" Score="1" Text="@Michael Let's not confuse &quot;intuitive&quot; with &quot;nontechnical&quot;." CreationDate="2010-10-24T15:38:31.633" UserId="919" />
  <row Id="5631" PostId="3932" Score="1" Text="@Tal I was writing in your language, not that of your students, because I am confident you are fully capable of translating it into whatever you know will reach them.  In other words, I interpreted &quot;intuitive&quot; in your question to mean intuitive to *you*." CreationDate="2010-10-24T15:40:16.843" UserId="919" />
  <row Id="5632" PostId="3929" Text="@Srikan Vadali For some reason the site wont let me edit my own question, but it does give me an edit button for the question, so there seems to be something strange going on... Sorry about that." CreationDate="2010-10-24T15:40:58.207" />
  <row Id="5633" PostId="3943" Text="mbq - ok, I get the point, I'll stop using question marks with spaces before them!  Thank you for the gentle corrections :)" CreationDate="2010-10-24T15:55:39.740" UserId="253" />
  <row Id="5634" PostId="3210" Text="I knew of no freeware GWR implementation either until I came across this website recently, http://www.ecoevol.ufg.br/sam/#Graphics . I wouldn't be surprised if others exist though." CreationDate="2010-10-24T16:03:18.187" UserId="1036" />
  <row Id="5635" PostId="3952" Text="This should be community wiki." CreationDate="2010-10-24T17:46:06.990" UserId="5" />
  <row Id="5636" PostId="3952" Text="Are you looking to just take some classes, do a master's degree, or get a certificate?" CreationDate="2010-10-24T17:46:39.033" UserId="5" />
  <row Id="5637" PostId="3954" Text="@mbq (+1) Great that you add some pictures to illustrate the whole thing!" CreationDate="2010-10-24T17:54:28.340" UserId="930" />
  <row Id="5638" PostId="3952" Text="@Shane - I don't think I've got enough rep to make this a comm wiki Q." CreationDate="2010-10-24T19:14:48.163" UserId="276" />
  <row Id="5639" PostId="3952" Text="looking to just take some classes, but open to a masters or certificate" CreationDate="2010-10-24T19:15:26.870" UserId="276" />
  <row Id="5640" PostId="3937" Text="I can't post to Meta, I don't have the reputation points..." CreationDate="2010-10-24T19:16:42.100" UserId="1689" />
  <row Id="5641" PostId="3937" Text="Don't think of it as the best psychological test test, but what is the most statistically valid test of X...  That's a real stats question." CreationDate="2010-10-24T19:18:37.527" UserId="1689" />
  <row Id="5642" PostId="3952" Score="2" Text="I think this question is too local and will be have to be closed - unless it's changed. For example, can I ask the same question but instead ask about classes in &quot;Yorkshire and Northumbria&quot;? Sorry to be picky." CreationDate="2010-10-24T20:02:24.547" UserId="8" />
  <row Id="5643" PostId="3952" Text="I think the same question, but for &quot;Yorkshire and Northumbria&quot; would be totally valid." CreationDate="2010-10-24T20:17:36.447" UserId="276" />
  <row Id="5644" PostId="3952" Text="@Scott-weinstein: I've opened a [meta question](http://meta.stats.stackexchange.com/questions/543/is-this-question-to-local) about this issue." CreationDate="2010-10-24T20:55:47.160" UserId="8" />
  <row Id="5645" PostId="3958" Text="@Indolering: Your question is great, unfortunately it's very similar to a previous (very highly rated) [question](http://stats.stackexchange.com/questions/3). Perhaps you could change your question to just ask about websites?" CreationDate="2010-10-24T21:04:13.887" UserId="8" />
  <row Id="5646" PostId="3210" Text="@Andy Thank you!  That's a real find.  This software looks like it reproduces GeoDa's capabilities with the addition of logistic regression and GWR.  I look forward to exploring it." CreationDate="2010-10-24T21:09:43.460" UserId="919" />
  <row Id="5647" PostId="3798" Score="1" Text="@ars So, I'd like to suggest the following papers from Father &amp; Son: http://j.mp/dvEDgb, http://j.mp/csD1Yf, http://j.mp/dkEHq5." CreationDate="2010-10-24T21:11:56.823" UserId="930" />
  <row Id="5648" PostId="3932" Text="Hi Whuber.  Thank you for the vote of confidence :).  The loose of the degree of freedom for the estimation of the expectancy is one that I was thinking of using in class.  The problem is that the concept of &quot;degrees of freedom&quot; by itself is one that needs knowledge/intuition.  But combining it with some of the other answers given in this thread will be useful (to me, and I hope others in the future).  Best, Tal" CreationDate="2010-10-24T21:12:58.773" UserId="253" />
  <row Id="5649" PostId="3958" Text="@csgillespie Oh yes, I forgot about this one, sorry!" CreationDate="2010-10-24T21:18:38.333" UserId="930" />
  <row Id="5650" PostId="3940" Score="1" Text="Thanks. I think it would be handy to have built-in functions (with min and max paraemters ala the unif family). It's a bit ugly to have to add function definitions into scripts just to use the discrete uniform distributions the way you would use other standard distributions. The builtin functions also deal with error handling (for example - if parameters aren't integers) and are optimized for speed. " CreationDate="2010-10-25T00:47:13.803" />
  <row Id="5651" PostId="3917" Text="It could be, but I'd say that you're basically doing bootstrapping (or some variation thereof) at that point." CreationDate="2010-10-25T01:46:59.787" UserId="303" />
  <row Id="5652" PostId="3940" Text="+1 for the implementation of the PDF and CDF! :)" CreationDate="2010-10-25T06:46:56.877" UserId="582" />
  <row Id="5655" PostId="3939" Text="This isn't correct for the edge cases. To see this, try running the following command: `table(round(runif(10000, min=0, max=2)))` It's clearly not discrete uniform." CreationDate="2010-10-25T08:30:36.550" UserId="8" />
  <row Id="5656" PostId="3939" Text="@csgillespie: nicely spotted, I updated my answer :)" CreationDate="2010-10-25T08:32:44.180" UserId="582" />
  <row Id="5657" PostId="3937" Text="@Indolering perhaps there needs to be a behavioural or social sciences stackexcange: http://area51.stackexchange.com/" CreationDate="2010-10-25T09:01:43.313" UserId="183" />
  <row Id="5659" PostId="3948" Text="+1 (although it deserves +3 at least) - very thorough and comprehensive. Made the whole issue much clearer for me. Thanks!" CreationDate="2010-10-25T11:20:54.977" UserId="22" />
  <row Id="5660" PostId="3953" Text="Very useful. Thanks!" CreationDate="2010-10-25T11:26:25.430" UserId="22" />
  <row Id="5662" PostId="3951" Text="Thanks for the input. Using poly() to compare different polynomial transforms of grade was also suggested to me this weekend on the R-SIG-mixed-models list, and I think this solves things. I've developed code that automates the process that will be included in the next version of my R package &quot;ez&quot;." CreationDate="2010-10-25T12:05:46.773" UserId="364" />
  <row Id="5663" PostId="3967" Score="3" Text="@Michael Just to be sure, are you talking about ST as found in clinical trials, e.g. http://en.wikipedia.org/wiki/Sequential_analysis?" CreationDate="2010-10-25T14:07:41.080" UserId="930" />
  <row Id="5664" PostId="3931" Text="I'd like to quote this zinger from the book *Numerical Recipes*: &quot;...if the difference between $n$ and $n-1$ ever matters to you, then you are probably up to no good anyway - e.g., trying to substantiate a questionable hypothesis with marginal data.&quot;" CreationDate="2010-10-25T14:09:30.380" UserId="830" />
  <row Id="5665" PostId="3888" Text="@Dave I don't think a &quot;ggplot&quot; tag will be very useful here (maybe mods would have a different opinion)." CreationDate="2010-10-25T14:16:02.897" UserId="930" />
  <row Id="5667" PostId="3917" Text="I agree, I and think there are other things regularly done that reflect this same sort of logic (such as subset specificity tests or non-equivalent dependent variables). I simply posed the question because I imagined more formal treatments existed." CreationDate="2010-10-25T15:06:40.330" UserId="1036" />
  <row Id="5668" PostId="3909" Text="I agree with pretty much everything you say, except that issues of accuracy and repeatability are essential to make correct inferences in the face of doubt. I can give experts the benefit of the doubt that they are building logical models. Where I am concerned is repeatability of the findings in many observational contexts. Although I agree repeatability does not necessarily account for confounding influences that are best dealt with in experimental settings." CreationDate="2010-10-25T15:11:45.533" UserId="1036" />
  <row Id="5669" PostId="3970" Text="The reason I posed the question is because we don't have to think of cross validation as solely a way to evaluate a models predictive ability. It is not uncommon to be concerned that a models results (and hence inferences made) are artifacts for many potential reasons. Hence we want to examine the robustness of the findings, and I figured cross validation could be a useful context to examine the robustness of the results." CreationDate="2010-10-25T15:18:49.283" UserId="1036" />
  <row Id="5670" PostId="3940" Text="Nice answer.  And for the quantiles we can do something like qdu &lt;- function(p, k) ifelse(p &lt;= 0 | p &gt; 1, return(&quot;undefined&quot;), ceiling(p*k))" CreationDate="2010-10-25T15:24:32.287" UserId="1108" />
  <row Id="5671" PostId="3965" Score="2" Text="Although these statements make no sense as given, they make perfect sense--and lead to an intriguing question--provided &quot;probability distribution&quot; is replaced by &quot;mean&quot; throughout.  Is this perhaps what you intend to ask?  Also, is this &quot;certain numerical range&quot; specified in advance of the analysis or are you asking how to construct an interval that has a given chance of enclosing at least one of the [means]?" CreationDate="2010-10-25T15:35:36.407" UserId="919" />
  <row Id="5673" PostId="3924" Score="1" Text="@Dan It would also help if you could tell us why you are doing this. As an aside: doing something like m + z*s may very well end with a upper bound of greater than 255 (depending on your values for m, z and s). Some sense of what you want to achieve (i.e., your goals) would help." CreationDate="2010-10-25T15:53:13.623" />
  <row Id="5674" PostId="3884" Text="Concerning the advice not to store redundant variables: this is appropriate for RDBMSes but I would like to suggest that the opposite should be encouraged for spreadsheets.  The latter are so error-prone that mechanisms to detect and correct errors are invaluable.  One of the best consists of redundant information, such as computed fields and statistical summaries.  For example, if column C is the ratio of columns A and B, then an error in a single column in any given row can be detected and usually fixed." CreationDate="2010-10-25T15:55:25.510" UserId="919" />
  <row Id="5676" PostId="3971" Text="I had suspected there was a 'trivial' algorithm, which generated the identity permutation with probability $r$, and was uniform on all permutations with probability $1-r$, for the case $r &gt; 0$. It looks like that is not exactly correct, and your &quot;uninteresting&quot; solution fixes my error." CreationDate="2010-10-25T16:29:06.120" UserId="795" />
  <row Id="5677" PostId="3884" Text="@whuber : that is what we check in the data control step. You can use that extra column to check fast, but you shouldn't keep it in the final sheet. Formulas in spreadsheets are horror, and the larger the spreadsheet, the more difficult to get the data out of it. Plus, in the case of Excel you'll be fighting the differences between .xls and .xlsx anyway. Be sure that a decision of a manager to update Microsoft Office can break tons of code if you rely heavily on excel files. So: save as csv, and keep these csv files as small as possible." CreationDate="2010-10-25T16:33:03.637" UserId="1124" />
  <row Id="5678" PostId="3974" Score="2" Text="What do you mean by 'trivariate'? It would also help if you could to the extent possible use equations to illustrate your situation. Do you mean that you observe $Y$ where $Y = X + Z$, $X \sim Exp(\lambda)$ and $Z \sim N(\mu,\sigma^2)$?" CreationDate="2010-10-25T16:47:16.130" />
  <row Id="5679" PostId="3884" Text="@Joris After spending a significant part of the last 24 years of my career coping with data transmitted in spreadsheets and managing substantial databases, I must respectfully disagree.  There is no such thing as &quot;control&quot; over spreadsheets (whether .xls, .xlsx, .wks, .wb*, etc) or even csv files.  The presence of redundant information in such files--even when they're available only in printed form--has many times resurrected some fairly large databases (100k+ records).  Every time this happens I (and my clients) have been grateful for the redundancies." CreationDate="2010-10-25T17:14:21.963" UserId="919" />
  <row Id="5680" PostId="3975" Score="1" Text="@xan (+1) Interesting! Which software do you use? Just a remark: There's no indication about absolute values for % or counts, so this seems to allow only a relative interpretation." CreationDate="2010-10-25T17:45:40.450" UserId="930" />
  <row Id="5682" PostId="3975" Text="@xan Sorry, I didn't read your last sentence (the x-axis is invisible). I'll try another remark: Any chance to get the NA counts visible in the centered view (i.e. distinguish them from neutral)?" CreationDate="2010-10-25T18:11:11.580" UserId="930" />
  <row Id="5683" PostId="3971" Text="@shabbychef You have a fine intuition: picking a permutation uniformly at random gives an expected correlation of zero. To achieve an expectation of r &gt; 0, then, select the identity with probability r and otherwise select any uniformly random permutation (*including* the identity) with probability 1-r. A similar approach (using the reversal of ranks) will yield any expectation down to the minimum possible value for the data (which depends on the data, not just on how many there are). These solutions don't yield the most uniform distributions overall, though: one permutation clearly is favored." CreationDate="2010-10-25T18:30:48.727" UserId="919" />
  <row Id="5684" PostId="3974" Text="Sorry to use a non-standard term.  The data are multivariate with three variables: R,G,B.  I have the joint J(R,G,B) and I can compute the marginals.  The form of the (univariate) marginals is the Y you defined." CreationDate="2010-10-25T18:31:11.657" UserId="1704" />
  <row Id="5685" PostId="3976" Text="I know that copulas capture advanced correlation structures; I don't know much else about them.  Unfortunately, I don't think they would be useful to me (for understanding the relationships in the data), nor to scientific colleagues (who would have to have some intuition/experience with copulas to understand my conclusions)." CreationDate="2010-10-25T18:32:23.373" UserId="1704" />
  <row Id="5686" PostId="3974" Score="1" Text="It would also help if you clarify a bit more what you mean by &quot;understand this data&quot;. Is there a specific question(s) you are trying to answer or is this just exploratory data analysis? I would suggest editing the question to incorporate your comment and make the question as precise as possible so that useful answers can be given." CreationDate="2010-10-25T18:36:45.130" />
  <row Id="5687" PostId="3970" Text="sorry for the misinterpretation." CreationDate="2010-10-25T18:52:46.330" UserId="1307" />
  <row Id="5688" PostId="3924" Text="Unfortunately it's not that easy to explain why I want it without writing a huge article worth of text. The upper bound going greater than 255 isn't a problem. It's the lower bound I need.&#xA;I think the m+z*s pretty much answers my question. Thankyou for that. If you post it as an answer, I'll mark it as the answer. Unfortunately I don't have the rep to upvote." CreationDate="2010-10-25T19:02:44.190" UserId="1683" />
  <row Id="5689" PostId="3967" Text="Yes. There are quite a few variants of sequential testing, including sequential t-tests, but none are used in basic research. I don't see any impediment to their use." CreationDate="2010-10-25T19:19:09.187" UserId="1679" />
  <row Id="5690" PostId="3971" Text="@whuber yes, I had mistakenly thought that I could also achieve any negative $r$ by this trivial method, by 'reversing' the indices with probability $|r|$. I guess I was tacitly assuming the vector $X$ was symmetric about $0$. The trivial algorithm is unsatisfying, though, as you suspected." CreationDate="2010-10-25T19:25:01.373" UserId="795" />
  <row Id="5691" PostId="3974" Text="I will try improve the question.  I need to give it a bit of thought.  -If- the data were multivariate normal (and, as follows, normal in the marginals), I would compute r and be done with it.  So, maybe I have two options:  (1) transform the data so they are normal or (2) compute r (via resampling) to assess its validity.  As far as &quot;understanding the data&quot;:  ideally, I would want to characterize the conditional distributions (not necessarily by closed form equations) ... that is, I'd like to know how the variables affect each other." CreationDate="2010-10-25T19:29:34.650" UserId="1704" />
  <row Id="5692" PostId="3971" Text="@shabbychef I believe these permutation methods tend to be used more with Spearman (rank) correlation coefficients, where all values between -1 and 1 are attainable." CreationDate="2010-10-25T20:15:36.690" UserId="919" />
  <row Id="5694" PostId="3961" Text="@kwak Concerning your deleted response: I see vaguely that there is some kind of limited permuting going on until somehow a correlation coefficient below 0.5 (e.g.) is attained, but I cannot quite connect that with the conditions of the original question.  Are you saying you just keep applying &quot;small&quot; random permutations until you obtain a correlation within a targeted range of correlations?  If that's so, I think it's a really creative idea but some analysis is needed to show it can actually work.  Regardless, thanks for sharing it (however briefly!)." CreationDate="2010-10-25T20:25:53.773" UserId="919" />
  <row Id="5695" PostId="3977" Score="3" Text="I don't think that this is on-topic, but I will wait for others to vote to close it.  " CreationDate="2010-10-25T20:29:24.767" UserId="5" />
  <row Id="5696" PostId="3977" Score="2" Text="Plenty of answers are at http://www.google.com/search?q=abbreviate+millions ." CreationDate="2010-10-25T20:40:08.123" UserId="919" />
  <row Id="5697" PostId="3884" Text="@whuber : We do data control with extra scripts, looking for the impossible values/outliers/odd cases. That's what I mean with the data control step. This is industry standard btw in companies like SGS and others doing analyses of clinical trials etc. The redundant information that is needed is kept in seperate databases. If one of them fails, the other is needed for the resurrection. In case you don't have a decent backup system, that is..." CreationDate="2010-10-25T20:56:58.570" UserId="1124" />
  <row Id="5698" PostId="3971" Text="@whuber: you read my mind! A solution to this problem could be used to generate a bootstrap sample from a vector with a fixed level of Spearman correlation to any other vector of the same length (modulo degeneracies in that other vector.) In that case, it suffices to think of $X_i = i$ up to location and scaling shifts." CreationDate="2010-10-25T21:02:29.437" UserId="795" />
  <row Id="5700" PostId="3979" Text="I cannot think of any situation in which molar can be confused with &quot;millions&quot;. Millions is not generally used by itself, you usually write, for instance, 2M€, 2Mb or 2MΩ. Moreover, you use Molar next to a chemical name, es. CaCl2 0.5M, so there is no ambiguity." CreationDate="2010-10-25T21:44:36.983" UserId="582" />
  <row Id="5701" PostId="3951" Text="@Matt (+1) and thanks for the reproducible example." CreationDate="2010-10-25T21:49:27.047" UserId="930" />
  <row Id="5702" PostId="3975" Text="(+1) - very nice." CreationDate="2010-10-25T22:13:36.790" UserId="253" />
  <row Id="5703" PostId="3975" Text="@chl Thanks. I use JMP, which I get paid to work on. The first one is a stacked bar chart with positive and negative values, which should be possible in lots of tools. NA counts could be done different ways (at one end, split over both ends, in the middle, separate column) and none seems obviously better for most situations." CreationDate="2010-10-25T23:55:45.120" UserId="1191" />
  <row Id="5704" PostId="3970" Text="No need for apologies. I'm the one suggesting something apparently fringe, and cross validation is apparently always used in the context you suggest." CreationDate="2010-10-26T00:07:07.783" UserId="1036" />
  <row Id="5705" PostId="3955" Score="1" Text="I'm surprised you have not received any answers yet (although it is an obviously popular question). Since each of your four bullets are worth a question in and of themselves, how about limiting this question to one of them and post the others in separate questions. They are all really excellent questions that take distinct approaches, and all four are worthwhile on their own." CreationDate="2010-10-26T00:11:17.587" UserId="1036" />
  <row Id="5706" PostId="3924" Text="@whuber See Dan's comment. Perhaps, you should post your suggestion as an answer for him to accept it. FYI." CreationDate="2010-10-26T05:02:12.963" />
  <row Id="5707" PostId="3982" Score="1" Text="Might this be more likely to be find an answer at http://stackoverflow.com/?" CreationDate="2010-10-26T06:26:20.397" UserId="449" />
  <row Id="5708" PostId="2032" Text="This Q has reappeared on the 'active questions' page as it was 'poked' by the 'Community' background process, one of whose tasks is &quot;Randomly poke old unanswered questions every hour so they get some attention&quot;. I don't think this question *deserves* any more attention. It's community wiki, it received only one up-vote, got a reasonable answer on Aug 26 but the user who asked it was last seen on Aug 24, so it seems unlikely that user is ever going to accept an answer. I suggest it should be closed by a moderator." CreationDate="2010-10-26T06:40:40.083" UserId="449" />
  <row Id="5710" PostId="3876" Text="Shouldn't time be a random effect since the heart rate measurements are samples taken during the surgery? &#xA;&#xA;If this is the case, would the following fit make sense (since I'm still reading up on the lmer function and have not quite understood the syntax)?&#xA;&#xA;lmer(heart.rate~treatment+(1|id)+(1+time),data=heart)" CreationDate="2010-10-26T08:10:34.803" UserId="1663" />
  <row Id="5712" PostId="3975" Text="@xan Thanks. I was hesitating between JMP and Systat." CreationDate="2010-10-26T08:20:41.420" UserId="930" />
  <row Id="5713" PostId="3371" Text="Just to say that my answer may actually look a little bit off-topic since this question has been merged with another one, http://stats.stackexchange.com/questions/3369/difference-between-fa-and-pca (I initially answer to the latter)." CreationDate="2010-10-26T09:16:23.327" UserId="930" />
  <row Id="5715" PostId="3986" Text="Hi chl,Thanksfor the reply..I admit that I got some information from your answer..Let me put my understanding and then you can comment please. (1) I get a hint that P values can go down if your sample size is big...--Is that so ?? To my understanding p values can only show whether or not ur null hypothesis is rejected. (2) I understand now that I need to see difference in AIC values with intercept only and with covariates. I suppose when we say that we want lower AIC we mean for the same dataset. I am getting character character left in my comment so will comment again once you answer please," CreationDate="2010-10-26T09:48:39.483" UserId="1763" />
  <row Id="5716" PostId="3986" Score="1" Text="@ayush (1) the test statistics (e.g. Wald) depend on the sample size (the standard error decrease with increasing sample size, and you're likely to get lower p-values with a larger sample). (2) yes, although AIC may be used to compare non-nested models, here I was thinking of it as a way to compare different models of increasing complexity." CreationDate="2010-10-26T10:35:32.587" UserId="930" />
  <row Id="5717" PostId="3965" Text="first of all thanks for pointing out that my question is imprecise. i still need to to a lot practising for that. i modified the question accordingly. bear in mind what i want to do evolves along with what i probably need :)" CreationDate="2010-10-26T10:53:03.480" UserId="1516" />
  <row Id="5718" PostId="3986" Text="@chl,thanks again..I get the essence of the p value now. Some 5mins back I ran a model which is giving me p values below .05 for all the variables but AIC of 28238.407 with intercept only and with covariates 21507.933. I also have a case in which AIC is 16035.xy with intercept only and with covariates 4234.xy. What is your opinion comparing two cases ? Please note that the second model had different variables 25 var while first had 20. so second though had more variables ( 25 comparison to 20) had lower AIC. Though p values werent .05 for all. Please suggest..more to ask after this..Thanks.&#xA;" CreationDate="2010-10-26T11:33:26.513" UserId="1763" />
  <row Id="5719" PostId="3980" Text="I think we need greater clarity with respect to the timing of interventions and the measurements. If I am reading your question correctly the timing is: A, B, {k1 successes in n1 trials}, B, C, D, {k2 successes in n2 trials}, {k3 successes in n3 trials} etc. I also assume that there are no other treatments beyond the 5 treatments listed. Is the above correct?" CreationDate="2010-10-26T12:43:51.017" />
  <row Id="5720" PostId="3955" Text="Hi Andy, thanks a lot for your answer. I'll try to rephrase it and focus on a single issue at a time. My initial thought was making it as broad as possible (and avoid repeating the same question afterwards) but given the lack of answers, it seems it is too general for someone to answer." CreationDate="2010-10-26T12:56:23.727" UserId="1694" />
  <row Id="5721" PostId="3981" Text="(It was Fred, I just like dumping my reputation into bounties every time I break 500, apparently...)" CreationDate="2010-10-26T13:22:06.643" UserId="71" />
  <row Id="5722" PostId="3986" Text="@ayush It's difficult to answer about model quality without knowing how variables were selected. The gap in AIC between a model including only an intercept and some covariates gives you an indication about the &quot;explanatory power&quot; of those predictors (the residual deviance seems to decrease by a larger extent in the 2nd case you showed, and AIC penalizes for the # parameters as I said in my response). It's by no means a full answer about the relevance of these predictors. I'd suggest you to ask for a more specific question (IMO), e.g. about *variable selection in GLMs* for your specific study." CreationDate="2010-10-26T13:29:07.270" UserId="930" />
  <row Id="5723" PostId="3982" Text="I agree, this has nothing to do with statistical analysis as stated." CreationDate="2010-10-26T13:29:23.953" UserId="1036" />
  <row Id="5724" PostId="3993" Text="I will add a more detailed comment later, but yes what I am suggesting is a method of cross validation that is concerned entirely with internal validity." CreationDate="2010-10-26T13:44:57.863" UserId="1036" />
  <row Id="5725" PostId="3876" Score="1" Text="The term '(time|id)' on the random effects side tells the function to fit different (linear) slopes for each person. So you can have time as both a fixed effect and a random effect, but they mean different things. Have a look at the sleepstudy example in Douglas Bates' book: http://lme4.r-forge.r-project.org/book/Ch4.pdf" CreationDate="2010-10-26T13:47:09.617" UserId="966" />
  <row Id="5726" PostId="3982" Score="3" Text="IMO, it does have to do with visualization (which is on-topic), so I'm open to allowing it.  That being said, I'm sure that you will have a hard time getting an answer either here or SO for this question: your best bet is the protovis mailing list." CreationDate="2010-10-26T13:50:17.320" UserId="5" />
  <row Id="5727" PostId="3982" Score="2" Text="@Saneef It would be helpful if you could provide a data sample so this is reproducible.  Either make geoPopList available, or else use a smaller data same that can be posted here." CreationDate="2010-10-26T13:51:44.213" UserId="5" />
  <row Id="5728" PostId="560" Text="The link to the paper doesn't work for me. Should it be http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.1675 ?" CreationDate="2010-10-26T14:46:11.670" UserId="461" />
  <row Id="5729" PostId="3982" Score="1" Text="@AndyW Thanks for a flag, but I agree that it is on topic. " CreationDate="2010-10-26T14:52:52.680" UserId="88" />
  <row Id="5730" PostId="3955" Text="I edited the question to expose the two main issues I'm mostly interested in. I chose to put them in the same question given their tight relationship." CreationDate="2010-10-26T15:03:35.240" UserId="1694" />
  <row Id="5731" PostId="3993" Score="1" Text="@chl: Can you explain your statement in the first paragraph on internal v. external validity? In the tradition that I am familiar with: **internal validity** refers to the ability to assert cause and effect relationships among the variables within the particular sample; **external validity** is about the ability to generalize from a sample to other persons, places, and times.  Traditionally, cross-validation is about the latter and thus by the above definition about external validity, whereas you state that it is about internal validity. Did I misunderstand your statement?" CreationDate="2010-10-26T15:08:34.677" UserId="485" />
  <row Id="5732" PostId="3907" Text="Is the case that the gene change took place for all participants *before* they enrolled in the registry? The other possibility is that gene changes took place at some random time before/after enrollment but you simply do not know when it took place. Can you clarify this issue?" CreationDate="2010-10-26T15:12:07.830" />
  <row Id="5733" PostId="3993" Score="1" Text="@Brett I was thinking of CV as a statistical technique to avoid overfitting or to provide a measure of prediction accuracy on the working sample (hence not necessiraly as a dedicated tool to demonstrate internal validity). I was not very clear, thanks or pointing that. I agree that this is then used to generalize over the sample at hand, but there I think it has nothing to do with causal inference (CV doesn't prove anything about causal links as modelled on the working sample). I share your view on external validity, but to demonstrate it we need other samples, no?" CreationDate="2010-10-26T15:21:36.573" UserId="930" />
  <row Id="5734" PostId="3993" Score="1" Text="You might clarify that first paragraph.  I think you're trying to say that CV doesn't do internal validity.  That's a matter for other processes.  But, if we've got good internal validity for other reasons, whatever that might be, CV will help estimate that effect more accurately across persons, places, and times--i.e. improve external validity. I still cannot think of any way that CV would help us to make causal claims about relationships between variables--the internal validity question itself--only to help generalize an established causal relationship." CreationDate="2010-10-26T15:28:48.147" UserId="485" />
  <row Id="5735" PostId="3924" Text="@Srikant I am happy Dan has found something useful in these comments (and that you have helped him in that).  However, I am reluctant to post an answer to a question I just don't understand!  Remember, too--I think I'm recalling this correctly--it is this site's policy not to go to extraordinary efforts to help someone formulate their question, so I'm even more reluctant to restate what I think the question might be asking and then answer the restated question.  Whom would that serve?  Others should feel free to jump in and do this if they want some easy points." CreationDate="2010-10-26T15:53:42.103" UserId="919" />
  <row Id="5736" PostId="3999" Text="As a hypothetical example, suppose the categories are named &quot;A&quot; through &quot;E&quot;.  Could an object simultaneously have values of 10 for A.Low, 20 for B.High, 50 for B.Medium, 15 for B.Low, and 5 for C.High (and zeros in all other subcategories)?  Or are you treating all five categories independently so that, e.g., the object could have 100 in A.Medium, 90 in B.High and 10 in B.Medium, 50 in C.Low, 10 in E.High (and zeros in the other 10 subcategories)?  In the former case you would seek a single index; in the latter, five index values (one for each of A, B, ..., E)." CreationDate="2010-10-26T15:58:40.693" UserId="919" />
  <row Id="5737" PostId="3971" Score="1" Text="@shabbychef Then you might want to give some thought to achieving a desired *variance* in the correlation coefficients that will be generated.  You can include that within the constraints and, provided the desired variance is neither too great nor too small, you should still have a nonempty feasible set.  For this application it might be wise to achieve as uniform a distribution as possible.  Alternatively, you can impose a desired variance by means of a penalty term in the objective function, seeking a balance between uniformity and achieving the target values of variance and expectation." CreationDate="2010-10-26T16:16:45.210" UserId="919" />
  <row Id="5738" PostId="3980" Text="Sorry, I think I did not communicate clearly.  Case 1 had k_11 successes in n_11 trials in one month, then A and B were applied to Case 1, and then it had k_12 successes and n_12 trials in the following month.  " CreationDate="2010-10-26T16:47:19.320" />
  <row Id="5739" PostId="3980" Text="Followup - Case 2 had k_21 successes in n_21 trials in one month, then B,C and D were applied to Case 2, and then it had k_22 successes and n_22 trials in the following month." CreationDate="2010-10-26T16:48:31.143" />
  <row Id="5740" PostId="3993" Score="1" Text="@Brett I think your comments to this question are very pertinent and sum up some of the issues very nicely. I doubt it will help any of the confusion between internal and external validity at this point, but chl's genetic epidemiology example is actually a problem of internal validity not external validity (except for between dataset heterogeneity (or population substructure), but that IMO is of less concern than internal validity in these examples)." CreationDate="2010-10-26T16:52:18.270" UserId="1036" />
  <row Id="5741" PostId="3881" Text="You must first figure out that barplots are made by barplot function... this is not that easy when you don't know that." CreationDate="2010-10-26T17:24:39.350" UserId="88" />
  <row Id="5742" PostId="3694" Text="+1 for the J example." CreationDate="2010-10-26T17:26:34.803" UserId="919" />
  <row Id="5743" PostId="3907" Text="@Srikant Vadali They where born with these genes. There was not change in genes :-) This is the root of the problem/question. I have not seen this spelled out well but I assume the starting time could be birth. But it cannot be unrelated to the outcome event? But what if age is part counted is used as an exogenous variable, I not clear on how this effect the estimation. Does this help. If so I will edit the original question to include this." CreationDate="2010-10-26T18:06:14.620" UserId="1189" />
  <row Id="5744" PostId="3981" Text="@Srikant, how did u manage to get 1., 2., 3. numbering working. I, for the life of me, could not figure it out!" CreationDate="2010-10-26T18:17:25.953" UserId="1307" />
  <row Id="5745" PostId="3981" Score="1" Text="@suncoolsu Magic! The secret is: Type a space at the start of the following paras: &quot;Apart from...&quot; and &quot;There is a R package...&quot;." CreationDate="2010-10-26T18:20:39.650" />
  <row Id="5746" PostId="3907" Text="I do not see any issues in using 'age' as an independent variable while using birth time as the starting time. Think of the following analogy: Suppose that you are investigating the variation in salary increases and you have a hypothesis that the higher someone's income is the lower is that person's salary. In such a situation, the 'income variable' will appear on both sides of your model: In 'Percent salary increase' income will appear as part of the dependent variable and income will again appear as an independent variable in the model. " CreationDate="2010-10-26T18:25:59.123" />
  <row Id="5747" PostId="3981" Text="@Srikant! Thanks! ... " CreationDate="2010-10-26T18:52:41.813" UserId="1307" />
  <row Id="5748" PostId="3999" Text="Thanks for you answer, yes an object could have A.low = 20, A.medium = 40, B.High = 40 and C.High = 5. All others would be zero. So yes, I am seeking for a single index. " CreationDate="2010-10-26T19:07:43.500" UserId="791" />
  <row Id="5749" PostId="3999" Score="1" Text="OK, just to be clear, because this is an important detail: is there a natural hierarchy to the categories and subcategories?  Could one assume that the ordering is A.High A.Medium A.Low B.High ... E.Low (or something comparable)? (If so, is there anything at all to prevent us from thinking of these 15 classifications as forming a single *ordinal* variable?) Also, it would help to clarify what the numerical values are intended to mean. *E.g.*, should they be interpreted as fuzzy degrees of membership, as probability of membership, as combined results of multiple measurements, or something else?" CreationDate="2010-10-26T19:24:23.930" UserId="919" />
  <row Id="5750" PostId="3976" Score="1" Text="@Mark In light of the revised question, I agree that looking at a copula-based description is a good suggestion.  There's nothing terribly difficult about copulas: because you have the marginals, you can apply the probability integral transform to make the three variables (R, G, B) uniform on [0,1].  A copula merely describes their multivariate distribution in these terms.  In effect, it eliminates the parameters from the marginals to allow you to focus on describing their multivariate dependencies.  It affords the non-parametric approach you're asking for." CreationDate="2010-10-26T19:31:53.173" UserId="919" />
  <row Id="5751" PostId="3987" Text="Basic biomedical researchers do interim analyses all the time, they just don't declare them because they don't even know that it matters! I have surveyed researchers at a national congress and found that more than 50% did not know that the control of error rates from Student's t-test is dependent on a pre-determined fixed sample size. You can see evidence of that in the sometimes erratically varying sample sizes used." CreationDate="2010-10-26T19:44:16.463" UserId="1679" />
  <row Id="5752" PostId="3999" Text="For example: if I am interested in the overall urbanization of a state, my five categories would be: roads, residential areas, industrial areas, airports and intense agriculture. Each of the categories is divided into 3 subcategories (low, mid, high) with the percentage of area falling into these categories. Now I would like to express the degree of urbanization in one number without loosing to much detail. Maybe I should get rid of the subcategories? Thanks a lot for your effort. " CreationDate="2010-10-26T19:44:48.650" UserId="791" />
  <row Id="5753" PostId="3987" Text="Some of the disadvantages that come from the complexities of sequential designs come specifically in the design of the analyses rather than in their implementation. Perhaps we could have a set of pre-canned designs for small sample basic experiments." CreationDate="2010-10-26T19:49:06.993" UserId="1679" />
  <row Id="5754" PostId="3993" Score="2" Text="Brett's definition between internal and external validity is accurate, but for our purposes it will help to define it in different terms. External validity is only concerned with the sample and how that sample relates to other populations. Internal validity is concerned with various aspects about the effects estimated and the constructs used to estimate those effects." CreationDate="2010-10-26T20:06:13.893" UserId="1036" />
  <row Id="5755" PostId="4002" Text="&quot;However, if you use the starting time as the time at which they enter the registry....bias your conclusions.&quot; Thanks for your answer. I agree with your answer I am trying to convince others that using time of enrollment is wrong as it is not related to outcome or genetics. Age could be the endog which is equivalent to starting the survival interval at birth.  Do you have a reference that would indicate that using time of enrollment is wrong." CreationDate="2010-10-26T20:06:26.323" UserId="1189" />
  <row Id="5756" PostId="3998" Text="Thanks Shane, I have not had any esperience analysing timeseries i tried briefly dxts = as.xts(as.POSIXct(ae$date_time)) but had no luck any pointers?" CreationDate="2010-10-26T20:14:48.017" UserId="1716" />
  <row Id="5757" PostId="3993" Text="@Andy @Brett I fully agree with Brett's definitions. My point was that CV won't help confirming an hypothesized conceptual model, this one contributing to internal validity. But I must admit that it wasn't clear in my original answer. I'll update it when I get back to my computer." CreationDate="2010-10-26T20:31:10.860" UserId="930" />
  <row Id="5758" PostId="3976" Text="Thanks for sticking with it through the reformulation.  I will look into copulas.  Will copulas allow me to compare marginals and conditionals from different Js:  $J_1$, $J_2$, ...  I might also wish to compare $R_1|G_1$ (from $J_1$) and $R_2|G_2$ in some way.  The different joints are measurements on individuals in experimental conditions.  Some of the measurements are repeated measurements of the same individual)." CreationDate="2010-10-26T20:31:47.003" UserId="1704" />
  <row Id="5760" PostId="3976" Text="@Mark Copulas will be useful for identifying, creating, and comparing models of joint distribution, but I don't think they will help in formal tests of hypothesis concerning those models.  Consider asking another question in which you reveal more about the nature of the data and how they are collected, because that ought to play an important role in the modeling and choice of analytical techniques." CreationDate="2010-10-26T21:00:40.597" UserId="919" />
  <row Id="5761" PostId="4003" Text="Many thanks for this detailed answer, it makes a lot of things clearer and I think I will follow your suggestion regarding PCA." CreationDate="2010-10-26T21:06:12.213" UserId="791" />
  <row Id="5763" PostId="4002" Score="2" Text="@Vincent At risk of being repetitive, I would refer you again to the TAS article (&quot;Two Pitfalls in Survival Analyses...&quot;) because the authors address *precisely* this issue.  They use &quot;multistate modeling&quot; to examine &quot;specific types of survival bias such as *length bias* and *time-dependent bias*.&quot;  You should be particularly concerned about length bias, which is a form of sample selection bias.  (What about people who died before they had a chance to ever make it into your study?)  This article, as is typical for TAS, has a good bibliography for further research." CreationDate="2010-10-26T21:33:35.970" UserId="919" />
  <row Id="5764" PostId="3907" Text="@onestop See my comment to Srikant's answer." CreationDate="2010-10-26T21:34:12.897" UserId="919" />
  <row Id="5765" PostId="3976" Text="Will do.  It's always difficult to get the right abstraction of a problem.  These two questions also seem to get at distribution free, multivariate analysis:  http://stats.stackexchange.com/questions/4/assessing-the-significance-of-differences-in-distributions and http://stats.stackexchange.com/questions/1927/is-there-an-accepted-definition-for-the-median-of-a-sample-on-the-plane-or-highe" CreationDate="2010-10-26T21:35:25.313" UserId="1704" />
  <row Id="5766" PostId="3976" Text="I'm reading over some of the terminology regarding copulas.  I'm missing a &quot;process&quot;.  For example:  with an empirical joint and empirical marginals and a distribution for the marginals, what do I do?  It seems that I need a transformation function to take my marginals to uniform distributions and then a &quot;copula function&quot; (term?) to combine those to for a distribution which (hopefully?) models the joint.  It may turn out that I'm more interested in the pairwise marginals (RG, RB, GB) than I am in the full joint." CreationDate="2010-10-26T21:39:59.270" UserId="1704" />
  <row Id="5767" PostId="3876" Text="Dear Matt, I just came across the sleepstudy example yesterday. I will read it up. Thanks!" CreationDate="2010-10-27T00:42:06.167" UserId="1663" />
  <row Id="5768" PostId="4002" Text="@Vincent No, I do not know of any reference. Perhaps, the reference pointed out by whuber may help. Alternatively, you can perform a simulation study and demonstrate what the right choice is. By the way, if people enroll roughly at the same age then the issue is not relevant as enrollment time and age will be synonymous. The issue of incorrect estimates becomes more prominent as enrollment times vary a lot vis-a-vis age. Or, at least that is my intuition." CreationDate="2010-10-27T00:55:49.127" />
  <row Id="5769" PostId="3643" Text="@rob - if i catch your drift, i think we are on the same page regarding actual vs asymptotic parameters. we seem to agree that the latter are what really matter." CreationDate="2010-10-27T03:01:12.623" UserId="1112" />
  <row Id="5770" PostId="4010" Text="You say this sequence 'may represent the correlation between a couple of measurements'. Do you have the data for the measurements? If there are only a couple of measurements, how can you derive a sequence of *N* correlations from them?" CreationDate="2010-10-27T06:48:08.893" UserId="449" />
  <row Id="5771" PostId="3992" Text="I'm using Mathematica for developing the codes I need, how can I use what I downloaded from robustbase website?" CreationDate="2010-10-27T08:24:29.117" UserId="1637" />
  <row Id="5772" PostId="4006" Text="(+1) Thanks for reiterating the need to read the R vignette. Nothing's missing there. And for a more thorough understanding of it, all preprint/draft papers that sit on Jan's website, http://gifi.stat.ucla.edu/" CreationDate="2010-10-27T09:28:46.370" UserId="930" />
  <row Id="5774" PostId="4006" Text="@chl Yeach, infamous Cuddy Valley..." CreationDate="2010-10-27T09:48:26.767" UserId="88" />
  <row Id="5775" PostId="3992" Text="You can find the source of the function in this thread: http://stats.stackexchange.com/questions/4017/translate-with-rcpp and try to translate from there to mathemtica." CreationDate="2010-10-27T10:02:57.763" UserId="603" />
  <row Id="5776" PostId="4017" Score="3" Text="CW? No, it certainly has a potential single best answer." CreationDate="2010-10-27T10:28:57.323" UserId="88" />
  <row Id="5777" PostId="4011" Text="@tosonb Which comment of mine confused you? By the way, I am not the OP. Just wanted to clarify as it feels as if your response including the last line is addressed to me." CreationDate="2010-10-27T12:07:59.080" />
  <row Id="5779" PostId="2697" Text="Okay: the Eigenvalue associated with each principal component tells you how much variation in the data set it explains (in my example, how clearly it separates your bottles into groups).  They are usually expressed as a percentage of the total variation in the data set.  As for the Eigenvectors, well, that's where as claws said I follow the output of an analysis like a machine ;)  In my head, they are related to how you rotate Vince's mobile to its 'best' orientation, but this might not be the right way to think of them." CreationDate="2010-10-27T13:07:34.637" UserId="266" />
  <row Id="5780" PostId="4018" Text="@Dirk:&gt; a) there are plenty of implementation of median() (say 'pull' in package pcaPP) so it's fair game. b) you mean one won't notice a sizable increase in running times ? c) okay, but i think the issue with this code is not really the translation to C++, rather the idea of calling some R functions [pnorm, dnorm,...] in C++ (of course i can be really wrong) d) can you provide the link to your mailing list ? " CreationDate="2010-10-27T13:21:12.227" UserId="603" />
  <row Id="5781" PostId="4017" Text="how do i make it cw ?" CreationDate="2010-10-27T13:21:55.953" UserId="603" />
  <row Id="5782" PostId="4018" Text="Can we please split the sub-questions off one by one?  A) you can call R function from C++ -- for convenience but not necessarily speed.  See the examples/ in Rcpp.  B)  I said no such thing.   C)  That is a all easy since Rcpp 0.8.7, see the 'Rcpp sugar' docs, posts on Rcpp-devel and our recent presentations.  D) It hangs off the R-forge page; just google for 'rcpp-devel'." CreationDate="2010-10-27T13:25:05.713" UserId="334" />
  <row Id="5783" PostId="4017" Score="2" Text="There is no reason for CW. " CreationDate="2010-10-27T13:25:30.427" UserId="334" />
  <row Id="5786" PostId="4018" Text="@Dirk to A)B)C)D-&gt;thanks i'll read more. Would you know of a directory/listings of examples of simple R functions together with there translation in C++ (i know there are C++ funciton in any library but i would prefer modern examples, that is, ones that use as much as possible the features of Rcpp). " CreationDate="2010-10-27T13:36:23.407" UserId="603" />
  <row Id="5788" PostId="4018" Text="Did you see examples/ within the Rcpp package?  Also, some of the HPC tutorials are by now 'too old' for current Rcpp practice. Make sure you lick a recent one -- say from useR this summer." CreationDate="2010-10-27T13:41:16.710" UserId="334" />
  <row Id="5789" PostId="4018" Text="The HPC tutorial i saw was over a year ago (btw thanks for these and the rest of your contribs), so maybe there is more there. I saw the examples in '/Rcpp/inst/examples/functionCallback/' but that's really not much. I realize there may be more spread around on your mailing list (and on your HPC slides) but that's too sparse. Would you have learned regression if it were taught that way (couple of contrived examples on slides and ML)? " CreationDate="2010-10-27T13:56:10.700" UserId="603" />
  <row Id="5790" PostId="4018" Score="1" Text="1) Start at http://dirk.eddelbuettel.com/presentations.html and work your way down.  2) There are six subdirectories to examples/ so I am unsure why you focus on one.  3)  There are 770+ unit tests that double as examples if you care to look closely enough.  4) There are eight (8) vignettes in the Rcpp package.  5) We authored a few other packages that use Rcpp, you could look at those too.  6) Lastly, CRAN lists fifteen packages depending upon Rcpp -- these all are examples too.  " CreationDate="2010-10-27T14:00:36.777" UserId="334" />
  <row Id="5792" PostId="4020" Text="Thank you for your answer!&#xA;&#xA;But I have some doubts. The problem is that A always outperforms B if initial params and learning/validation/testing sets are the same; but it doesn't neccessarily hold&#xA;if they differ.&#xA;See question update." CreationDate="2010-10-27T14:13:14.413" UserId="1725" />
  <row Id="5794" PostId="4019" Text="It would be helpful if you could give a bit more detail about the nature of the machine learning algorithms, in particular what is meant by &quot;randomly generated initial approximation (parameters)&quot; - does this mean something like a random set of initial weights for a neural network, or do you mean somplething like a random choice of kernel parameters for an SVM?  The correct evaluation protocol may depend on the nature of these &quot;parameters&quot;.&#xA;" CreationDate="2010-10-27T14:32:21.077" UserId="887" />
  <row Id="5796" PostId="4018" Score="1" Text="Dude: There is a mailing list for the project you are interested in. *All* our documentation suggests to ask on the mailing list. So why-oh-why do you keep piling on here?  Can we **please** stop that now.  Lastly, your 'too superficial' would require some backing up.  I will gladly review patches, just don't post them *here*. Ok?" CreationDate="2010-10-27T14:38:48.417" UserId="334" />
  <row Id="5797" PostId="4017" Score="1" Text="I *strongly disagree* with your edited title and added/edited question.  You are simply mistaken if you consider Rcpp to be a code compiler, or when asking us to rewrite code for you.  " CreationDate="2010-10-27T14:41:37.197" UserId="334" />
  <row Id="5798" PostId="4023" Text="The Table did not come out right. Every odd number is a count from group 1 and every even number is the respective count from group 2" CreationDate="2010-10-27T14:42:40.753" UserId="1727" />
  <row Id="5800" PostId="4018" Score="2" Text="@kwak: Responding to &quot;It's something that should be outsourced to the community&quot;: I look forward to seeing your contributions as you work through these examples yourself." CreationDate="2010-10-27T14:45:56.143" UserId="1657" />
  <row Id="5801" PostId="4019" Text="This initial approximation is a couple of matrixes filled in with random values drawn uniformly &amp; independently from [0;0.02]. (In fact, I'm comparing matrix factorizations via SGD with different loss functions; this matrixes are initial approximation of factors). So, it's closer to weights for NN" CreationDate="2010-10-27T14:46:21.747" UserId="1725" />
  <row Id="5802" PostId="4017" Text="@Dirk:&gt; sure, what would a suggested title be (the older one?). I don't really care about this particular function I'm interested in learning ways to make my codes run faster. If you have another example, please post it. I'll happily close this one.  " CreationDate="2010-10-27T14:49:33.550" UserId="603" />
  <row Id="5803" PostId="4018" Text="@Joshua Ultrich:&gt; Sure: my question was my initial contribution to the process. Let's see if we can keep this ball rolling. " CreationDate="2010-10-27T14:52:40.613" UserId="603" />
  <row Id="5804" PostId="4023" Text="@Membran: I've reformatted your question. Is the table now correct?" CreationDate="2010-10-27T15:04:31.373" UserId="8" />
  <row Id="5805" PostId="4019" Score="1" Text="In that case, I'd probably use the bootstrap, but generate a different initial matrix for each iteration, so that the resampling procedure averages over the variability due to the partitioning of the dataset and due to the initialisation, but I would use the same initialisation in each iteration for both methods.  I'd then use the Wilcoxon signed rank test for performance evaluation (fewer assumptions).  " CreationDate="2010-10-27T15:28:30.713" UserId="887" />
  <row Id="5806" PostId="4019" Text="Thank you! Wilcoxon signed rank test looks promising." CreationDate="2010-10-27T15:41:48.190" UserId="1725" />
  <row Id="5807" PostId="4025" Score="1" Text="Crossposting between here and StackOverflow is discouraged. " CreationDate="2010-10-27T15:47:25.007" UserId="334" />
  <row Id="5808" PostId="4025" Score="2" Text="I would suggest that this would be the more appropriate place for this question." CreationDate="2010-10-27T15:48:40.180" UserId="8" />
  <row Id="5809" PostId="4023" Text="yes, many thanks!" CreationDate="2010-10-27T15:50:08.177" UserId="1727" />
  <row Id="5810" PostId="4025" Text="sorry for crossposting.i didn't know here when i posted the question to stackoverflow.somebody suggested here and i posted here then. thanks." CreationDate="2010-10-27T15:50:50.197" />
  <row Id="5811" PostId="4024" Text="Thank you for suggestions. Logically, it is not quite possible to merge phenotypes as each of them is a unique combination of three recorded parameters. Since each of these parameters can go &quot;up&quot;, &quot;down&quot; or stay &quot;unchanged&quot; as a result of a mutation, so there can be 3^3=27 distinct phenotypes. In the example above I removed those phenotypes for which both groups scored &quot;0&quot;, so there were only 21 of them. I do see the prevalence of certain phenotypes but i would like to have some statistical proof that distributions of such phenotypes in various groups of mutants is similar (or not). Thank you!" CreationDate="2010-10-27T16:03:56.497" UserId="1727" />
  <row Id="5812" PostId="4019" Score="2" Text="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.3325" CreationDate="2010-10-27T16:26:57.720" UserId="511" />
  <row Id="5813" PostId="4019" Text="Yaroslav, thanks for the link!" CreationDate="2010-10-27T16:31:38.800" UserId="1725" />
  <row Id="5814" PostId="4020" Score="2" Text="Note that if you adjust parameters of an algorithm after seeing its performance on the test set, test set error is no longer guaranteed to be an unbiased estimate of true error. " CreationDate="2010-10-27T16:41:42.403" UserId="511" />
  <row Id="5815" PostId="4020" Text="Yaroslav! Actually, I've got learn,validation and test sets. I use validation set to tune some params like regularization term weight and optimization step (via simplex [meta]optimization), while initial weights (the parameters I was talking about before) are not adjusted at all... surely, the test set is used only to calculate quality metrics" CreationDate="2010-10-27T16:51:31.513" UserId="1725" />
  <row Id="5816" PostId="4026" Text="unfortunately, i have no background in statistics so i don't know what to expect from a &quot;MLE&quot; output. sorry for that. so, when i print model$table, i get something like below. is this MLE?                                                          &#xA;            Sepal.Length                                               &#xA;Y             [,1]      [,2]                                           &#xA;  setosa     5.006 0.3524897                                           &#xA;  versicolor 5.936 0.5161711                                           &#xA;  virginica  6.588 0.6358796 " CreationDate="2010-10-27T17:18:24.923" />
  <row Id="5817" PostId="4027" Score="4" Text="Do you *really need* more than 3 digits? I would guess most people would report p-values to 2 digits for which an output of 3 digits is more than enough. Could you clarify *why* you need more digits?" CreationDate="2010-10-27T17:25:52.480" />
  <row Id="5818" PostId="4029" Score="1" Text="@onestop Interesting, Fisher's test crashed on R. " CreationDate="2010-10-27T18:51:38.150" UserId="930" />
  <row Id="5819" PostId="4030" Text="@Jonathan Hope you don't mind but I added a link to &quot;VECM&quot;. Feel free to correct." CreationDate="2010-10-27T18:57:52.653" UserId="930" />
  <row Id="5820" PostId="3982" Text="Is there a reason why the code you link to on the example page doesn't work? From my reading of the pv.Geo.scale source it appears that it supports setting the domain like pv.linear.scale does... What happens when you try that approach and attempt to zoom with the mousewheel?&#xA;&#xA;Please update your question with a simple reproducible example like Shane asked for as well; if you can do that I'd be happy to take a look." CreationDate="2010-10-27T19:26:32.613" UserId="1730" />
  <row Id="5822" PostId="4026" Score="1" Text="@Yuan I don't know, because simple &quot;MLE&quot; can mean different things... Try reading ?naivebayes, it is quite detail description what is what in table, maybe you could find something familiar there. Or try to specify what you need in more detailed way -- what is your desired use, for what you need it? " CreationDate="2010-10-27T20:06:18.430" UserId="88" />
  <row Id="5823" PostId="4020" Text="mbq, I'm not sure if I was clear enough =| I do not want to nail this weight " CreationDate="2010-10-27T20:07:09.530" UserId="1725" />
  <row Id="5824" PostId="4020" Text="@bijey It is only that you will get a stronger result showing that A&gt;B in unpaired test. " CreationDate="2010-10-27T20:12:31.387" UserId="88" />
  <row Id="5825" PostId="4020" Text="...I do understand that it's better to show that A&gt;B in unpaired test. But I'm afraid, it is very unlikely to show since (informaly) [quality(A) - quality(B)] &lt;&lt; variance(quality(A))" CreationDate="2010-10-27T20:17:24.543" UserId="1725" />
  <row Id="5826" PostId="3955" Score="1" Text="I'm still failing to understand why having one estimate with a small error makes an approach 'unusable'." CreationDate="2010-10-27T20:43:00.453" UserId="449" />
  <row Id="5827" PostId="3993" Score="1" Text="@Brett I made some edits; I hope it helps clarifying my initial thoughts, but let me know. I could also remove any reference to CV wrt. internal/external validity if this happens to be unclear, although I think you were more concerned with CV and external validity (i.e. generalizing on new samples) while I was envisioning CV as part of model validation (hence, on the working sample). " CreationDate="2010-10-27T20:44:16.027" UserId="930" />
  <row Id="5828" PostId="3909" Text="@Brett (+1) My apologies. It seems I also forgot to upvote your very nice answer. Already voted up your helpful comments." CreationDate="2010-10-27T20:56:04.923" UserId="930" />
  <row Id="5829" PostId="4029" Text="@onestop Cannot upvote more, sorry. It seems I hadn't increase the wksp enough :)" CreationDate="2010-10-27T20:59:27.243" UserId="930" />
  <row Id="5830" PostId="4027" Score="4" Text="Bonferroni corrections with large models (hundreds of variables) is one reason." CreationDate="2010-10-27T21:06:15.073" UserId="919" />
  <row Id="5831" PostId="4024" Score="1" Text="@Membran Aggregation doesn't have to be meaningful: you're free to combine bins any way you please.  A subtle problem, though, is that *post-facto* aggregation casts the p-values in doubt; the aggregation ought to be independent of the data." CreationDate="2010-10-27T21:16:24.403" UserId="919" />
  <row Id="5832" PostId="4020" Score="1" Text="I thought so; then stick to paired test, yet don't hide this fact. This is because in a sense you can say that they are equivalent -- B can be better than A when started with luckily picked seed." CreationDate="2010-10-27T22:02:35.957" UserId="88" />
  <row Id="5833" PostId="4027" Text="@whuber That is a very good point. " CreationDate="2010-10-27T22:08:41.637" />
  <row Id="5834" PostId="4029" Text="Isn't it that Fisher's &quot;exact&quot; test actually addresses slightly different question: &quot;...it is used to examine the significance of the association (contingency) between the two kinds of classification&quot; (wiki page). In my case I sought to confirm (or refute) the hypothesis that distributions of phenotypes between 2 groups are similar (equal). When I found that online test (see the first post) named &quot;Chi-square test for equality of distributions&quot; I thought it was precisely for my problem..." CreationDate="2010-10-27T22:38:24.300" UserId="1727" />
  <row Id="5835" PostId="2909" Text="That would require about a million simulated random walks..." CreationDate="2010-10-27T22:40:44.953" UserId="919" />
  <row Id="5836" PostId="4029" Text="Also, if you think that mentioned version of Fisher's test is fine for comparing two distributions, can it also be used for checking uniformity of distribution (i.e. to say that phenotypes within one group were distributed non-uniformly between a finite number of possible phenotypes)? One can do this even in Excel using CHITEST function, but what if I have a distribution similar to the ones above, with lots of phenotypes observed less than 5 times?" CreationDate="2010-10-27T22:42:57.687" UserId="1727" />
  <row Id="5838" PostId="4010" Text="The measurements may be many time series, then we can derive the correlation between each two of them." CreationDate="2010-10-28T02:56:43.877" />
  <row Id="5839" PostId="4027" Score="1" Text="@suncoolsu I suspect Gunter's comment was made in a different context, probably one where someone was trying to overinterpret something like a p-value of 10^-100 or thereabouts.  The fact is that interpreting and using moderately low p-values (10^-4 to 10^-6) is a significant issue: it has been the subject of national regulations and of federal litigation in the US, for instance, so it's not something to be dismissed with a flippant comment.  I thank you for making clear that you offer the quotation at least partially in jest!" CreationDate="2010-10-28T03:13:47.283" UserId="919" />
  <row Id="5840" PostId="4025" Score="1" Text="At the very least, you should provide the link to the other posting." CreationDate="2010-10-28T03:20:28.063" UserId="183" />
  <row Id="5841" PostId="4027" Text="@whuber .. My sincere apologies, the comment was fully in jest. I removed it cuz it doesn't belong to a serious discussion like this. I actually didn't know that p-value that low is a big issue in US. Thanks for letting me know. I wonder what do they gain gain from trusting p-value that low apart from gaining the knowledge &quot;something is wrong&quot;." CreationDate="2010-10-28T03:25:18.063" UserId="1307" />
  <row Id="5842" PostId="4029" Text="@Membran # 1: It is a *slightly* different question as Fisher's exact test conditions on both sets of marginal totals. This seems something of an academic statistical nicety to me though, and I'm a statistician in academia. (BTW could you clarify to *which* wiki you refer?)&#xA;&#xA;&#xA;&#xA;@Membran #2: I would not call the conditional exact test &quot;Fisher's exact test&quot; in the case of a one-way table, but such a test should be possible.and  I would have thought more straightforward for one-way tables, but I can't currently find software to assist and I don't have time to perform the calculation without." CreationDate="2010-10-28T04:07:33.330" UserId="449" />
  <row Id="5843" PostId="4011" Text="There is no change in the genes, they are born with them. Therefore entering the registry does not increase risk. I don't expect a change in behavior but it is possible. There care does not change it is just tracked in the registry." CreationDate="2010-10-28T04:16:02.493" UserId="1189" />
  <row Id="5846" PostId="3992" Text="Ok, I installed the software, I used the scan to read my data: mydat&lt;-scan(&quot;C:\\mydat.txt&quot;) then Qn(mydat) but an error message appears telling me: Qn function is not found or for scaleTau2(mydat) scale2Tau2 function was not found. I guess I need to upload a package first, but which one?" CreationDate="2010-10-28T07:16:56.320" UserId="1637" />
  <row Id="5847" PostId="4032" Text="Good idea. I'd add that -est tab- is an abbreviation (understood by Stata) for the built-in command -estimates table-. That was introduced with the release of Stata 8.0 in 2003." CreationDate="2010-10-28T07:20:27.613" UserId="449" />
  <row Id="5848" PostId="3723" Text="@ars: The uncertainties package does apply error propagation theory (it was created just for that), *and* can take covariances as an input (http://packages.python.org/uncertainties/user_guide.html#correlated-variables). :)" CreationDate="2010-10-28T08:03:33.787" UserId="1735" />
  <row Id="5849" PostId="3715" Text="@cool-RR: Just a small technical remark, to make things precise: the factory *function* `ufloat()` creates a `UFloat` *object*.  Case does matter, here (the two names were chosen to be close to each other on purpose).  :)" CreationDate="2010-10-28T08:18:53.523" UserId="1735" />
  <row Id="5850" PostId="3874" Text="Before I update the answer, is treatment a repeated factor? i.e., does each subject receive both treatment &quot;a&quot; and treatment &quot;b&quot; or is this a between-subjects factor?" CreationDate="2010-10-28T08:46:23.030" UserId="966" />
  <row Id="5851" PostId="4041" Text="@Tal (+1) Nice `plyr` package, isn't it? :)" CreationDate="2010-10-28T08:47:11.533" UserId="930" />
  <row Id="5852" PostId="3874" Text="Treatment is a between-subjects factor. Each subject only receive 1 kind of treatment. I've coded the two treatments as 1 and 0 and set treatment as a factor variable." CreationDate="2010-10-28T09:08:00.190" UserId="1663" />
  <row Id="5853" PostId="3955" Text="Thanks for the comment. I'm not saying the aproach (trend calculation) was unusable, but its error calculation could have a potential problem. Given that I got no answer so far, I'll remove that item from the question hoping it gets easier to answer." CreationDate="2010-10-28T10:05:16.033" UserId="1694" />
  <row Id="5854" PostId="4044" Text="Hmm, it seems that Mutual Information may be what I'm looking for." CreationDate="2010-10-28T13:13:13.083" UserId="1737" />
  <row Id="5855" PostId="4044" Text="Post it as an answer, you'll gather reputation." CreationDate="2010-10-28T13:37:27.610" UserId="88" />
  <row Id="5856" PostId="4046" Text="-- Thank you! -- You're welcome." CreationDate="2010-10-28T13:49:27.627" UserId="1737" />
  <row Id="5857" PostId="4027" Text="@suncoolsu It shows up in regulations governing the monitoring of the environment.  Monitoring programs can result in conducting tens of thousands of tests simultaneously, creating a need to avoid false positive results." CreationDate="2010-10-28T14:06:40.273" UserId="919" />
  <row Id="5858" PostId="4046" Score="4" Text="Are you enjoying your own personal Q &amp; A? ;-)" CreationDate="2010-10-28T14:19:34.593" UserId="1390" />
  <row Id="5859" PostId="4041" Text="This works great. Thanks for pointing out the plyr package! Could you please explain the &quot;.(group)&quot; syntax?" CreationDate="2010-10-28T14:31:51.303" UserId="439" />
  <row Id="5860" PostId="4042" Text="Nice, thanks! I've been experimenting with `by`, but couldn't figure out how to transform the result into a data frame." CreationDate="2010-10-28T14:38:13.680" UserId="439" />
  <row Id="5861" PostId="4045" Text="Thank you dirk. I suspected `zoo` would have something on that line, I'll have a look and see if it fits my needs. I assume it would work also for &quot;relative&quot; times? Each of my time series arbitrarily starts at 0s and ends at ~3500s, so I don't have precise dates." CreationDate="2010-10-28T14:42:29.247" UserId="582" />
  <row Id="5862" PostId="4048" Score="6" Text="This is probably the wrong site for this question - I suspect you'll get much better answers at the GIS StackExchange site: http://gis.stackexchange.com/" CreationDate="2010-10-28T14:48:38.557" UserId="71" />
  <row Id="5863" PostId="4045" Text="I think so for as long as you can compute some indexing -- something the to-be-averaged items have in common." CreationDate="2010-10-28T14:55:50.613" UserId="334" />
  <row Id="5864" PostId="4048" Score="3" Text="@user1738 &quot;Reverse geocoding&quot; is not a good description of this.  (RG would return a full postal address for a lat-lon.)  This question is one of computational geometry (find nearest polygon to a point) with a specific GIS application in mind.  You will get plenty of answers at the GIS site Matt Parker refers to." CreationDate="2010-10-28T15:14:25.910" UserId="919" />
  <row Id="5865" PostId="3056" Text="+1 That's a pretty direct answer, considering these functions implement the formulas in the paper!" CreationDate="2010-10-28T15:43:07.510" UserId="919" />
  <row Id="5866" PostId="4013" Text="Not because it answers your question but because they are connected: http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675#675 (I really like the answer of Mark/Peter)  and http://stats.stackexchange.com/questions/2641/what-is-the-difference-between-likelihood-and-probability" CreationDate="2010-10-28T17:13:07.430" UserId="223" />
  <row Id="5867" PostId="4027" Text="@whuber Thanks you very much ! .. I really value the knowledge that I gain from this forum!" CreationDate="2010-10-28T18:20:07.457" UserId="1307" />
  <row Id="5868" PostId="2287" Text="+100 if I could. That example was amazing." CreationDate="2010-10-28T20:04:32.707" UserId="1106" />
  <row Id="5869" PostId="4054" Text="@Srikant Could you explain how you derive the &quot;trivial answer&quot;?  (There must be some unstated assumptions behind it.)  And to what theorem do you refer when you conclude that &quot;eventually everyone will be connected&quot;?  That's not at all obvious!" CreationDate="2010-10-28T20:26:45.380" UserId="919" />
  <row Id="5870" PostId="4054" Text="@whuber I am assuming that the network size is fixed. The question states: A person picks another person at random to make a connection and presumably this is an ongoing process. Thus, as time goes to infinity everyone should be connected. No theorem, just intuition. Perhaps, I am using imprecise language." CreationDate="2010-10-28T20:29:30.320" />
  <row Id="5871" PostId="4055" Text="Note that I do not interpret the question to state that everyone chooses *exactly one* other person to connect to--that would lead to a sterile theory because the number of connections would be determined, not random.  Instead I have interpreted it to state that everyone when they enter the network chooses connections randomly among the n others, winding up with anywhere from 0 through n connections total.  The assumption on the variances is assured when there is a limit on the number of connections any newcomer will make *and* that number has some &quot;minimal&quot; randomness." CreationDate="2010-10-28T20:30:02.363" UserId="919" />
  <row Id="5872" PostId="4053" Score="1" Text="Clarification: Is the question about the &quot;total number of connections for the whole group&quot; or &quot;the total number of connections for one person&quot;? My answer implicitly assumes the latter." CreationDate="2010-10-28T20:35:26.960" />
  <row Id="5873" PostId="4054" Text="@Srikant I am still confused, because after a long time, &quot;Prob(No of connections=n)&quot; equals 1 when n = 3 and otherwise is always zero.  After all, when &quot;everyone should be connected&quot; the number of connections equals n(n-1)/2.  I suspect you may have several different random processes in mind at the same time.  It could help to disclose the assumptions you are making and be a little more precise." CreationDate="2010-10-28T20:43:52.877" UserId="919" />
  <row Id="5874" PostId="3056" Text="It seems like this package computes the _expected_ max drawdown based on the paper, but does not compute the CDF. The paper gives 'shortcut' results, IIRC, to compute that expectation. " CreationDate="2010-10-28T22:39:41.790" UserId="795" />
  <row Id="5875" PostId="4053" Score="1" Text="*Riley* distribution? That's a new one on me. Do you have a reference or link?" CreationDate="2010-10-28T22:48:45.217" UserId="449" />
  <row Id="5876" PostId="4055" Text="I'm a bit confused about $X_{i}$ and the variance. Does this suggest people have an intrinsic variance?" CreationDate="2010-10-28T23:56:26.017" UserId="1036" />
  <row Id="5877" PostId="4011" Text="sorry for addressing the wrong person. I understand that the gene doesn't change in their life time my English is failing me here I thought you used that  term as gene mutation. But regardless of that thanks for clarifying that you think that being in the registry doesn't change their risk." CreationDate="2010-10-29T00:10:45.597" UserId="694" />
  <row Id="5878" PostId="4053" Score="3" Text="&quot;Rayleigh&quot; maybe?" CreationDate="2010-10-29T01:11:44.343" UserId="919" />
  <row Id="5879" PostId="4055" Score="1" Text="@Andy Not people: the number of connections made.  The important thing is that there should be a good chance that the number of connections made by individuals actually varies and doesn't settle down to a constant.  When that happens, the limiting distribution (of the number of connections) is determined by the finite number of initial connections that do vary, so it's not possible to approach a Normal distribution asymptotically." CreationDate="2010-10-29T01:14:14.667" UserId="919" />
  <row Id="5880" PostId="3056" Text="@shabbychef Sorry, I missed that nicety.  I see how obtaining the entire CDF can be more useful than just knowing the expectation. (Financial risk is much more than just expected losses...)  But now I feel a little bit better about the work I did to approximate the CDF!" CreationDate="2010-10-29T01:17:31.153" UserId="919" />
  <row Id="5881" PostId="4033" Text="@whuber, this is really great, and should go a long way towards the CDF. Above and Beyond badge material. " CreationDate="2010-10-29T02:21:25.783" UserId="795" />
  <row Id="5882" PostId="4015" Text="Thank you Henrik - the distinction between the definitions (and your thoughts about them) was helpful." CreationDate="2010-10-29T06:48:33.720" UserId="253" />
  <row Id="5883" PostId="4050" Score="1" Text="Thank you C.p, you make an interesting point.  Although, from the perspective of Henrik's answer above, you are still in the realm of inductive, since the statistical reasoning you describe is one that involves uncertainty." CreationDate="2010-10-29T06:51:03.607" UserId="253" />
  <row Id="5884" PostId="4041" Score="1" Text="aix - sure.  It means &quot;split the data by the variable between .(), and on each subset perform the function&quot;.  In order to have it include more variables, you should simply use this syntax:  .(var1, var2, var3) .  Which is like cutting your data by each combination of levels of var1, var2 and var3.  And on each cut to perform your function.  This package is maintained by Hadley (also th author of ggplot2), so I trust it will keep developing. " CreationDate="2010-10-29T06:54:01.873" UserId="253" />
  <row Id="5885" PostId="4041" Score="1" Text="Oh, and BTW, you could also use plyr with a parallel computing on several cores (almost automatically), see: http://www.r-statistics.com/2010/09/using-the-plyr-1-2-package-parallel-processing-backend-with-windows/          " CreationDate="2010-10-29T06:55:37.453" UserId="253" />
  <row Id="5886" PostId="4041" Text="@chl - indeed :)" CreationDate="2010-10-29T06:57:21.570" UserId="253" />
  <row Id="5887" PostId="3989" Text="OK, the answer for the guys who will read this later: if you just want to calculate a robust scale estimator for a piece of data 1-install latest version of R 2-install the robustbase package 3-ready to go! but if you are developing a code outside this environment you need to use weighted high medians to minimize required calculations for Sn or Qn. " CreationDate="2010-10-29T07:28:11.610" UserId="1637" />
  <row Id="5888" PostId="3992" Text="thanks kwak, after all i could do test my code with R results. :)" CreationDate="2010-10-29T07:32:38.830" UserId="1637" />
  <row Id="5889" PostId="2998" Text="Hi Jeromy.  Q: what does it mean to take the dist of abs(cor(...)) ?  Isn't a cor matrix is one where a larger number indicate more of a relation, while in a distance matrix the opposite is true?  (I suspect I am missing something, could you please explain?)" CreationDate="2010-10-29T07:47:46.923" UserId="253" />
  <row Id="5890" PostId="4047" Text="Yup, I'm using that, thanks :)" CreationDate="2010-10-29T08:40:11.303" UserId="1737" />
  <row Id="5891" PostId="4059" Text="Seems like only yesterday someone else asked virtually the exact same question... oh look! it was yesterday: http://stats.stackexchange.com/q/4025/449" CreationDate="2010-10-29T08:40:34.423" UserId="449" />
  <row Id="5893" PostId="4064" Text="@mbq (+1) I looked for this one, but cannot found it in the `ape` package!" CreationDate="2010-10-29T09:56:03.663" UserId="930" />
  <row Id="5894" PostId="4062" Score="1" Text="What is the advantage of a polar representation (aside from saving space)? It looks to me like it is trickier to look at. " CreationDate="2010-10-29T09:56:08.057" UserId="582" />
  <row Id="5895" PostId="4062" Score="1" Text="@nico It is more cool (-;" CreationDate="2010-10-29T10:08:04.147" UserId="88" />
  <row Id="5896" PostId="4062" Score="1" Text="It is also useful when you don't have one stem..." CreationDate="2010-10-29T10:29:09.613" UserId="253" />
  <row Id="5897" PostId="4064" Text="Bingo.  That's what I was looking for.  I wonder if there is something similar in ggplot2..." CreationDate="2010-10-29T10:33:18.453" UserId="253" />
  <row Id="5898" PostId="2998" Text="@Tal My understanding is that the dist function takes the euclidean distance of the matrix passed to it, which in this case was an absolute correlation matrix. Perhaps, a more literal conversion from the correlation matrix would be the as.dist() function which takes an existing distance matrix: e.g., as.dist(1-abs(cor(na.omit(x)))). see ?dist" CreationDate="2010-10-29T11:33:17.520" UserId="183" />
  <row Id="5899" PostId="2998" Text="using a 1-abs(cor...) would make more sense to me :)" CreationDate="2010-10-29T12:06:56.137" UserId="253" />
  <row Id="5900" PostId="3992" Text="@K-1:&gt; oh, i was going to respond...then i somehow forgot. Sorry about that. So it worked ?" CreationDate="2010-10-29T12:10:56.723" UserId="603" />
  <row Id="5901" PostId="4063" Text="Good point. I've only come across the converse: assuming an effect is linear when it isn't can lead to spurious statistical evidence for multiplicative interaction terms." CreationDate="2010-10-29T12:23:44.897" UserId="449" />
  <row Id="5902" PostId="4045" Text="Dirk, unfortunately aggregate did not solve my problem, as I want to resample at essentially the same number of points of the original trace. However I found the `reglin` function in `pastecs` that does exactly what I need! http://rgm2.lab.nig.ac.jp/RGM2/R_man-2.9.0/library/pastecs/man/reglin.html" CreationDate="2010-10-29T12:30:08.070" UserId="582" />
  <row Id="5903" PostId="3205" Text="@whuber (+1) because it helped me a lot to find what I actually want to know. Thank you ! Maybe this question can be marked as community wiki, because I dont think there is a definite answer. " CreationDate="2010-10-29T12:43:51.130" UserId="264" />
  <row Id="5904" PostId="229" Text="@drknexus Your intuition is correct.  However, extremes (and near-extremes) of sampling distributions are somewhat special: their values are constrained by the mass of data on one side of them, whereas--for parent distributions with infinite tails--there is no constraint at all on their values on the other side.  Thus, for instance, the distribution of a maximum (from a distribution with no upper bound) is positively skewed.  This *increases* its expectation relative to the corresponding percentile." CreationDate="2010-10-29T13:32:30.693" UserId="919" />
  <row Id="5906" PostId="4059" Score="1" Text="i don't need a programming solution. i need it's formula." CreationDate="2010-10-29T14:08:26.640" />
  <row Id="5907" PostId="3997" Score="1" Text="Your code doesn't work because the components of a POSIXlt object are numeric/integer, not character.  But as Shane said, you really should be using a time-series class..." CreationDate="2010-10-29T15:29:53.203" UserId="1657" />
  <row Id="5908" PostId="4059" Text="I disagree that this is a duplicate for the reason Enso mentioned in the comment. The other question is a closely related but not a duplicate. I have voted to re-open the question." CreationDate="2010-10-29T15:43:15.257" />
  <row Id="5909" PostId="4067" Text="Elegant solution Joshue.  Do you think there are cases when one solution is better then another?" CreationDate="2010-10-29T16:05:36.667" UserId="253" />
  <row Id="5910" PostId="4067" Score="2" Text="I think it's a matter of preference.  My example is essentially what `plyr` does but it gives you finer control, though it's not nearly as clean.  My opinion would change if one solution had a better time/memory profile.  I haven't compared them though." CreationDate="2010-10-29T16:08:37.660" UserId="1657" />
  <row Id="5911" PostId="4056" Text="@whuber great answer!" CreationDate="2010-10-29T18:18:25.327" UserId="1307" />
  <row Id="5912" PostId="4069" Text="aha, I have not worked with R enough to spot the `NA` vs `NaN` distinction. Nice that it has both..." CreationDate="2010-10-29T18:22:36.100" UserId="795" />
  <row Id="5913" PostId="4069" Text="@Conjugate Of course a single number has a variance!  It's the mean square deviation from its mean, namely zero.  Perhaps you meant to say a dataset with one number cannot have a *sample* variance?" CreationDate="2010-10-29T18:23:55.827" UserId="919" />
  <row Id="5914" PostId="4068" Text="Did you mean to ask about the population variance definition or about the *sample* variance definition?  There's no issue with the former." CreationDate="2010-10-29T18:27:17.260" UserId="919" />
  <row Id="5915" PostId="4070" Score="3" Text="Actually, a single observation of a random variable often provides information about that variable's variance.  It depends on the possibilities you allow.  For instance, a single observation from a Uniform, Poisson, or Exponential distribution lets you estimate its single parameter, which gives you an estimate of the variance." CreationDate="2010-10-29T18:34:01.917" UserId="919" />
  <row Id="5916" PostId="4068" Text="@whuber I think what Matlab is doing is the following. When it encounters a scalar it reports the population variance by default. When it encounters a vector it reports the sample variance by default unless you ask it to report population variance. In either context they use same function: `var()`." CreationDate="2010-10-29T18:39:11.193" />
  <row Id="5917" PostId="4059" Text="An answer to the &quot;possible duplicate&quot; question requires a reference to procedures in R.  An answer to the present question requires a generic formula, without necessarily any reference to R, such as the one under &quot;Constructing a classifier from the probability model&quot; at http://en.wikipedia.org/wiki/Naive_Bayes_classifier .  I therefore added my vote to Srikant's for reopening." CreationDate="2010-10-29T18:44:55.897" UserId="919" />
  <row Id="5918" PostId="4068" Text="@Srikant That may be, but it does not address my request for clarification.  I suspect there may be a typographical error in the last line.  The var() function appears to be returning the correct values of population variance but not the correct values for *sample* variance." CreationDate="2010-10-29T18:47:47.610" UserId="919" />
  <row Id="5919" PostId="4071" Text="You have restated the problem without offering a resolution.  The concern is that this convention can lead to wrong answers when the variance estimate is used later.  It certainly delays the stage at which a failure is recognized.  (Think about what happens if you try to use this &quot;variance&quot; in a t-test, for instance.)" CreationDate="2010-10-29T18:50:13.783" UserId="919" />
  <row Id="5920" PostId="4068" Text="@whuber Actually, `var(randn(1),0)` weights the variance by 0 and hence the output is 0. The second element is used to do a weighted average if it is different from 1. See: [Mathworks help for `var`](http://www.mathworks.nl/help/techdoc/ref/var.html)" CreationDate="2010-10-29T18:52:11.457" />
  <row Id="5921" PostId="4071" Text="@whuber Not exactly. The OP is thinking that `var` returns the sample variance for a scalar whereas it returns the 'population variance'. Whether it is a problem or not depends on what type of error handling routines they have internally. However, I agree that their convention is susceptible to errors." CreationDate="2010-10-29T18:57:37.223" />
  <row Id="5922" PostId="4068" Text="@srikant, the 'weight' is a red herring: mathWorks wanted to add weighted variance without boogering the API, so they added this odd option. At least, that is how I see it." CreationDate="2010-10-29T19:07:06.697" UserId="795" />
  <row Id="5923" PostId="4071" Text="@Srikant In order to determine who is more confused, you or I, I have learned that we both are!  Wikipedia terms the formula with 1/(n-1) the &quot;sample variance&quot; (at http://en.wikipedia.org/wiki/Variance ) whereas Mathworld clearly uses 1/n at http://mathworld.wolfram.com/SampleVariance.html .  However, the OP uses &quot;sample variance&quot; in the first sense when remarking that &quot;the sample variance is not dividing by 0=n−1&quot;.  In this sense *there is no such thing* as &quot;sample variance&quot; for a dataset of one value (a &quot;scalar&quot;), so your characterization of what the OP is &quot;thinking&quot; looks incorrect." CreationDate="2010-10-29T19:10:53.510" UserId="919" />
  <row Id="5924" PostId="4068" Text="@Srikant Excellent!  What you have found is that the OP has mischaracterized the meaning of the second parameter.  *That* is a fine resolution of the question.  Why don't you point that out in your answer?  (Regardless, I think Matlab is using a terrible convention, even though they clearly document it.)" CreationDate="2010-10-29T19:14:07.200" UserId="919" />
  <row Id="5925" PostId="4068" Score="2" Text="@shabbychef: The page Srikant has referenced clearly documents the behavior you have noted.  In particular, it does not maintain that var() returns the 1/(n-1) version of the variance when n=1.  So the issue is not that Matlab is *wrong*, per se.  (From a software design point of view this implementation of var() is terrible because it tries to do too much in a single function and consequently risks misunderstanding and erroneous results by unvigilant users.)" CreationDate="2010-10-29T19:17:27.917" UserId="919" />
  <row Id="5926" PostId="4073" Text="Can anyone figure out why link #2 is not working correctly?  Maybe there's a problem with using square brackets elsewhere in the text?" CreationDate="2010-10-29T19:46:02.623" UserId="919" />
  <row Id="5927" PostId="4073" Text="Looks like my edit overlapped with shabbychef. Hope we did not break anything." CreationDate="2010-10-29T19:58:35.743" />
  <row Id="5928" PostId="4073" Text="there was a space in the URL. fixed." CreationDate="2010-10-29T19:58:41.797" UserId="795" />
  <row Id="5929" PostId="4073" Text="Thanks guys.  You are not only smart and sharp-eyed but helpful, too." CreationDate="2010-10-29T20:01:53.320" UserId="919" />
  <row Id="5930" PostId="4072" Score="1" Text="Fred, a couple of questions: (1) when you say 'commitment' here, is that meaning 'the trait of sincere and steadfast fixity of purpose' or 'the official act of consigning a person to confinement'? (2) If the latter, are you really saying people will be confined or not based on the predictions from your model? If so I think I might wish to seek legal advice before posting an answer!&#xA;&#xA;By the way, you might like to take a quick look at the latter paragraphs of the section 'How do I ask questions here?' in the site FAQ:http://stats.stackexchange.com/faq" CreationDate="2010-10-29T21:28:56.190" UserId="449" />
  <row Id="5931" PostId="4056" Text="@suncoolsu It's a shame we cannot vote up the update..." CreationDate="2010-10-29T21:47:18.193" UserId="930" />
  <row Id="5932" PostId="3987" Text="@Michael About &quot;fake&quot; interim analyses (looking at p-values while study is still in an evolving stage): it looks like it is a misappropriate use of statistics, no more." CreationDate="2010-10-29T22:24:19.280" UserId="930" />
  <row Id="5933" PostId="3992" Text="yes, it worked. It's pretty faster than my code in mathematica. Now i should look for a way to connect mathematica and R kernels together. thanks for the helpful comments." CreationDate="2010-10-30T00:42:18.697" UserId="1637" />
  <row Id="5934" PostId="3723" Text="@EOL: oh! good to know, thanks!" CreationDate="2010-10-30T02:32:05.623" UserId="251" />
  <row Id="5935" PostId="3798" Text="@chl: many thanks!  Fun reading for this weekend.  :)  (BTW, sorry, I think I should have just asked this as a question. :-/ )" CreationDate="2010-10-30T02:33:41.957" UserId="251" />
  <row Id="5936" PostId="4039" Text="+1 thanks for pointing out how to do this with the uncertainties package." CreationDate="2010-10-30T02:48:40.293" UserId="251" />
  <row Id="5937" PostId="2848" Text="+1, thanks for the plos paper link." CreationDate="2010-10-30T02:59:50.343" UserId="251" />
  <row Id="5938" PostId="3904" Text="+1 that was a great read, thank you.   " CreationDate="2010-10-30T03:03:33.557" UserId="251" />
  <row Id="5939" PostId="3912" Text="+1.  This reminds me that I've been using your scripts to learn asymptote by jumping in and changing stuff around, trying different things.  Thanks again for that, *very* helpful to get started." CreationDate="2010-10-30T03:07:47.273" UserId="251" />
  <row Id="5940" PostId="4068" Text="Yes, the behaviour is documented in the help for `var`, but the API for this function is annoying at best, as you point out. My question is _why_ do they do this? You simply cannot ask the Matlab `var` command to 'weight by zero'/give sample variance for  a scalar, it refuses to answer the question, pretending the user has asked a different question. What is the sensible answer? " CreationDate="2010-10-30T03:10:10.533" UserId="795" />
  <row Id="5941" PostId="2998" Text="@Tal Yes. I agree. I did a quick look. In this case the vector of distances (euclidean of abs cor with 1-abs cor) correlate around .96, so it doesn't make much difference." CreationDate="2010-10-30T05:13:12.943" UserId="183" />
  <row Id="5942" PostId="4056" Text="@chl .. I agree, I would have would like to give (+2) :-), actually I never thought so deeply about this question until this update." CreationDate="2010-10-30T07:38:38.533" UserId="1307" />
  <row Id="5943" PostId="4062" Score="3" Text="@mbq: you missed a &quot;good&quot; pun there... you could have said &quot;it's more *fan*&quot; :)" CreationDate="2010-10-30T08:22:48.633" UserId="582" />
  <row Id="5944" PostId="3912" Text="@ars Actually, I seem to remember that this picture was made with PStricks. Anyway, a good starting point for Asymptote is http://www.piprime.fr/asymptote/." CreationDate="2010-10-30T08:59:36.117" UserId="930" />
  <row Id="5945" PostId="4069" Text="You'll notice that I said it can't have a *population variance*, not that it cannot have a variance.  Clearly it has a sample variance, which is indeed trivially zero." CreationDate="2010-10-30T09:07:41.183" UserId="1739" />
  <row Id="5946" PostId="319" Score="1" Text="I agree wholeheartedly about the use of simulation for explaining this.  But just a small note on the example at the end: I find that people (not just students) do find it difficult to distinguish for any particular distributional assumption, e.g. the poisson, between being *marginally* poisson distributed and being *conditionally* poisson distributed.  Since only the latter matters for a regression model, a bunch of dependent variable values that aren't poisson need not necessarily be any cause for concern." CreationDate="2010-10-30T09:19:39.623" UserId="1739" />
  <row Id="5947" PostId="2998" Text="wait a sec, are we talking about building a dist on the cor matrix?  dist(1-abs(cor(...)))?  Or about looking at the cor matrix AS a dist matrix? as.dist(1-abs(cor(...)))   The second option makes sense to me.  The first one, I am not sure what it means - if you do, I'd be glad to know :)" CreationDate="2010-10-30T09:44:15.177" UserId="253" />
  <row Id="5948" PostId="4077" Text="Guys,thanks for the answer. I have a data to analyse. How do i attach data to this. Lots to ask you people..Thanks in anticipation. Expecting a prompt reply.	" CreationDate="2010-10-30T10:18:06.417" UserId="1763" />
  <row Id="5949" PostId="4057" Text="Personally, just to be sure, I'd polish off an initial approximation derived from CF with Newton-Raphson. Even then, based on some experiments I've done, I'm not convinced of the virtue of carrying more than three or so terms of an expansion." CreationDate="2010-10-30T10:18:49.237" UserId="830" />
  <row Id="5950" PostId="4077" Score="2" Text="Huh? You said in the question you'd already computed the t-statistic, and chl has provided sample R code. What more do you want? By the way, I'm not sure you have any right to expect or request a prompt reply; we don't get paid for this you know." CreationDate="2010-10-30T10:45:04.057" UserId="449" />
  <row Id="5951" PostId="3916" Score="1" Text="@chl: Thanks for this! I'm reading your code and it's helping me figure out the R syntax a lot.  One thing that's giving me trouble is the line `freq.resp &lt;- raw.resp/apply(raw.resp, 1, sum, na.rm=T)`.  The error is that &quot;dim(X) must have positive length&quot; while dim(raw.resp) is NULL.  Could it be that since my data doesn't have all options with positive frequencies, my tables aren't all the same length?  How can I fill in the zeroes in my `table` invocation?" CreationDate="2010-10-30T12:03:28.360" UserId="1681" />
  <row Id="5952" PostId="3916" Text="@Matthew Yes, it is very likely that this is indeed the source of the problem (well, you're learning R quickly, I appreciate). So, if one response category is not observed for one or more items, then the length of the table will be &lt; 4, and `freq.resp` will throw an error. I will fix the code (by tomorrow), this will also show you how to concatenate irregular tables with little code." CreationDate="2010-10-30T12:47:56.597" UserId="930" />
  <row Id="5953" PostId="4064" Text="@Tal No official support for tree structures in ggplot2. Look at this Google group thread, http://j.mp/c85l5l (but it's definitively not circular)." CreationDate="2010-10-30T12:52:51.447" UserId="930" />
  <row Id="5954" PostId="4077" Score="1" Text="@ayush For your preceding question, I provide a complete answer to your question (IMHO) -- then I gave some follow-up to your comments before stopping when I thought you were asking for another question which is not the purpose of comment option here. So, I would suggest that either you clearly state if your question relates to theoretical consideration or applied data analysis (in the latter case, give us a reproducible example) or separate your questions. BTW, you still have the option to accept answers that you find useful (again, wrt. your original question, not the comments that follow)." CreationDate="2010-10-30T13:01:11.667" UserId="930" />
  <row Id="5957" PostId="4050" Text="Please see the (hopefully understandable) update to my answer, where I try to address the issue brought up here." CreationDate="2010-10-30T14:17:08.973" UserId="442" />
  <row Id="5958" PostId="4050" Text="@Henrik  That is clearer (to me at least).  Just a little niggle:  It's not quite the case that &quot;the same reasoning applies for probabilities in classical p-value and Bayesian statistics&quot;.  The latter *will* give you single event probabilities, e.g. the probability that the true mean is between some value and some other value (although your other caveats all apply) while 'classical' frequentist methods such as confidence intervals will not even do that, despite the fond and widespread hope that they do.  Their interpretation is indeed as you describe it." CreationDate="2010-10-30T14:29:56.523" UserId="1739" />
  <row Id="5959" PostId="4015" Text="Your update was clear and to the point.  If I could have given you another (+1), I would." CreationDate="2010-10-30T14:44:53.323" UserId="253" />
  <row Id="5960" PostId="4078" Text="+1 for the creative (and insightful) interpretation and explaining the distinction." CreationDate="2010-10-30T15:11:23.307" UserId="919" />
  <row Id="5961" PostId="4072" Text="@onestop, there actually is a venue through which a person can be &quot;civilly committed&quot; based on future dangerousness to themselves or others. Here's a website that has a good description of what civil commitment is, http://www.oregoncounseling.org/LawsRights/CivilCommit.htm . It also is often used against sexual offenders, see http://www.nytimes.com/2010/05/18/us/politics/18offenders.html . Although I'm not familiar with any &quot;legal&quot; standards determined through predictive modeling. See my comment to Fred directly." CreationDate="2010-10-30T15:17:38.187" UserId="1036" />
  <row Id="5962" PostId="4072" Text="@Fred, although you've provided a description well enough to be able to answer your question, given onestop's and mine question over what exactly you are modelling I think it would be nice if you could clarify your design a bit more precisely. In your question do you really mean &quot;legal standards&quot; of commitment are determined by some type of logistic model (and if you do I'd like to know what State (or State entity) you are referring to? Or is this model simply evidence of whether someone should be committed." CreationDate="2010-10-30T15:23:16.963" UserId="1036" />
  <row Id="5963" PostId="4072" Text="@Fred, Also could you clarify if you mean civilly committed? I am aware of no circumstances where a legal statute has a calculated measure of risk in guiding the laws decision. Although it could be used as evidence in a parole hearing it wouldn't be used in any standardized way that I am aware." CreationDate="2010-10-30T15:25:43.013" UserId="1036" />
  <row Id="5964" PostId="4077" Score="2" Text="@ayush Ah, and I just realize that you **never** vote up any of the answers that were provided to you (though you have enough rep now)." CreationDate="2010-10-30T15:31:02.927" UserId="930" />
  <row Id="5965" PostId="3912" Text="Neat.  That's a great reference site, thanks." CreationDate="2010-10-30T16:22:13.973" UserId="251" />
  <row Id="5966" PostId="4072" Text="@Andy W #1: Seems the term 'civil commitment' is a rather wonderful example of our two nations' separation by a common language! e.g. http://www.guthriecastle.com/terms-and-conditions/" CreationDate="2010-10-30T17:18:12.633" UserId="449" />
  <row Id="5967" PostId="4072" Text="@onestop, yes I am speaking entirely in the context of USA law. I hope marriage is not considered so undesirable that one refers to it as recidivism!" CreationDate="2010-10-30T17:45:44.783" UserId="1036" />
  <row Id="5968" PostId="319" Text="I have to confess that I didn't know that.  I've really appreciated your comments around this site over the past few days of your membership - I hope you'll stick around." CreationDate="2010-10-30T18:29:32.153" UserId="71" />
  <row Id="5969" PostId="4059" Text="i hope someone reopens this discussion. " CreationDate="2010-10-30T19:31:34.510" />
  <row Id="5970" PostId="4079" Text="+1 for bringing out the key ideas: (1) we can guarantee the means will differ when the datasets are this large and (2) some other analysis is likely to be more appropriate and useful.  But because we don't know about the purpose of the analysis, we should be cautious about making specific recommendations." CreationDate="2010-10-30T20:45:54.120" UserId="919" />
  <row Id="5971" PostId="4069" Text="@Conjugate I have come to realize that our conversation derives from two different understandings of &quot;population&quot; and &quot;sample&quot; variance. The terms are confused on the Internet, so I think we're both blameless and that we may be in agreement. I was attempting to use the convention adopted by the OP, whose terminology indicates the &quot;sample&quot; variance divides by n-1 while the &quot;population&quot; variance divides by n.  Incidentally, your comment seems (correctly) to allow for *three* variances: that of a random variable, the unbiased estimator thereof, and that of a set of independent realizations of it." CreationDate="2010-10-30T20:55:35.840" UserId="919" />
  <row Id="5972" PostId="1171" Text="@Onestop May I ask what your intention is in offering a bounty on someone else's question when you have already provided a good answer?" CreationDate="2010-10-30T21:02:13.290" UserId="919" />
  <row Id="5974" PostId="4070" Text="@whuber thanks for the pointers. Actually, I was thinking more on the lines of sample variance. What I meant to say was, estimating sample variance may not be appropriate with one sample. It stands corrected now." CreationDate="2010-10-30T22:30:04.667" UserId="1307" />
  <row Id="5975" PostId="4064" Text="Hello chl, thank you for the link.  I'll respond there with reference to this code as well..." CreationDate="2010-10-30T23:16:08.900" UserId="253" />
  <row Id="5976" PostId="3987" Text="@Chi On one level, yes, undeclared and uncorrected interim analyses are inappropriate (but it is done in ignorance, an ignorance that I believe points to inadequacies in the methods of teaching statistics to basic biomedical researchers...). However, if we consider it at a meta level, then it is possible to find some partial justifications. Many experiments involve such small samples that an increased false positive error rate may be a reasonable tradeoff for more power. Convention precludes a declared level of alpha higher than 0.05." CreationDate="2010-10-30T23:25:03.780" UserId="1679" />
  <row Id="5977" PostId="3987" Text="I note in this context that basic biomedica researchers do not work in an exclusively Neyman-Pearson approach, even if statements that &quot;results where P&lt;0.05 were considered significant&quot; might suggest otherwise. If we stay within the confines of Fisher's significance testing wherein considerations other than the achieved P value can be incorporated into decisions of how to deal with the test results, perhaps interim analyses might not be so bad. However, it is certain that a designed sequential test would be superior to an undesigned one." CreationDate="2010-10-30T23:29:49.427" UserId="1679" />
  <row Id="5978" PostId="2998" Score="1" Text="@Tal the dist option: each variable has a vector of absolute correlations. Variables with more similar vectors would have smaller euclidean distances: http://en.wikipedia.org/wiki/Euclidean_distance" CreationDate="2010-10-31T02:21:04.380" UserId="183" />
  <row Id="5979" PostId="3916" Text="@Matthew I've added a fix. Should work with dichotomous items too. Let me know if this is now ok. http://gist.github.com/642219" CreationDate="2010-10-31T09:04:18.033" UserId="930" />
  <row Id="5980" PostId="1171" Text="I was taking an experimental approach to a question i've just discovered is answered at http://meta.stackoverflow.com/questions/16065/how-does-the-bounty-system-work, namely 'Can I award a bounty to my own answer?'" CreationDate="2010-10-31T09:06:11.303" UserId="449" />
  <row Id="5981" PostId="4082" Text="@whuber (+1) A good paper, thank you. I also found this article interesting, Conditions for factor (in)determinacy in factor analysis (Krijnen et al.), http://j.mp/dwo7c8." CreationDate="2010-10-31T09:39:04.550" UserId="930" />
  <row Id="5982" PostId="4077" Score="1" Text="@chl Very generous of you, thanks." CreationDate="2010-10-31T09:42:53.753" UserId="449" />
  <row Id="5983" PostId="3987" Text="@Michael Yes, you made a good point. This reminds me of Rothman arguing against a strict correction for multiple testing at the expense of Power considerations. However, planned *vs.* unplanned comparisons, as well as pitfalls associated to what I called &quot;fake&quot; interim analysis, should be part of every courses on DoE or RCTs. " CreationDate="2010-10-31T10:39:57.157" UserId="930" />
  <row Id="5984" PostId="3992" Text="@K-1:&gt; if you look at the code/formula of the Tau estimator (type &quot;scale2tau2&quot; in R), it's basically just a weighted mean (with weights set by the median/MAD  and cut of to 0 below a threshold). This should run fast on anything." CreationDate="2010-10-31T11:38:45.060" UserId="603" />
  <row Id="5985" PostId="3946" Text="Thanks chl for the answer, I accepted it for the sheer scope of it.  Best, Tal" CreationDate="2010-10-31T13:34:32.593" UserId="253" />
  <row Id="5986" PostId="3945" Text="Great answer onestop - thank you." CreationDate="2010-10-31T13:34:54.183" UserId="253" />
  <row Id="5987" PostId="3945" Text="I think the thanks are due to Roger Newson, as I'm just quoting from his article." CreationDate="2010-10-31T15:52:25.477" UserId="449" />
  <row Id="5988" PostId="4011" Text="@Vincent, even if their care does not change, becoming part of a register will probably lead to changes in how they represent their health to themselves, and thus cause changes in their behaviour /end(somewhat irrelevant psychology tangent}" CreationDate="2010-10-31T15:56:31.287" UserId="656" />
  <row Id="5989" PostId="1171" Text=".. but i forgot i'd have to wait 24 hours before i could award the bounty, during which time the question would be 'featured'. Sorry about that! Now i've learnt of the existence of the Community FAQ for the original trinity (or triumvirate, or whatever it's called) of SE sites i won't need to experiment again. http://meta.stackoverflow.com/questions/7931/" CreationDate="2010-10-31T16:02:04.720" UserId="449" />
  <row Id="5990" PostId="4086" Score="2" Text="Are you sure you want to limit yourself so severely?  If this problem has any practical application, then by the time you know the count at time x you also know all previous counts.  Why not use them to help with the prediction?" CreationDate="2010-10-31T16:21:39.610" UserId="919" />
  <row Id="5991" PostId="4086" Text="Indeed. You're correct. Thanks for pointing that out." CreationDate="2010-10-31T16:30:38.363" UserId="485" />
  <row Id="5992" PostId="2629" Text="@ars Just came across this post from A Gelman, *How does statistical analysis differ when analyzing the entire population rather than a sample?*, http://j.mp/cZ1WSI. A good starting point about diverging opinions on the concept of a &quot;super-population&quot;." CreationDate="2010-10-31T17:20:06.970" UserId="930" />
  <row Id="5993" PostId="2629" Score="1" Text="@chl: interesting -- reminds me that Gelman had a discussion of finite/super population inference being comparable to fixed-/random-effects in his paper on ANOVA [ http://www.stat.columbia.edu/~gelman/research/published/econanova3.pdf ]." CreationDate="2010-10-31T17:53:26.807" UserId="251" />
  <row Id="5994" PostId="4074" Text="Thanks for the comments and answer. First on the comments: I do mean civil commitment (ie, confinement).  I am not proposing a model to be used, but rather using a model to illustrate shortcomings in current practice.  The statistics are generally used as evidence, not alone determinative.  Some jurisdictions do define sufficient future dangerousness quantitatively (eg, more likely than not)." CreationDate="2010-10-31T18:12:41.967" />
  <row Id="5995" PostId="4074" Text="Based on the answer and my own research, I've written new code that generates plausible results. With a little trepidation (I read the FAQs and still think the question is appropriate, but my apologies if I'm missing something), here it is (feedback welcome) (0.03... is the constant from the logit model): .predict sr2, xb&#xA;.gen byte sr2C1R1=0 if sr2!=.&#xA;.gen sr2w = (0.0310417 + sr2 - 0.67448975*stdsr1)&#xA;.replace sr2C1R1=1 if 1/(1 + exp(-sr2w)) &gt; .75 &amp; sr2!=. &amp; sexrecS!=.&#xA;.sum sr2C1R1 if sr2!=. &amp; sexrecS!=." CreationDate="2010-10-31T18:20:39.413" />
  <row Id="5996" PostId="3877" Text="Thanks again for the helpful comments.  @chl The model has several independent variables.  Does that raise problems for my proposed method of calculating the Cohen's d on the difference in mean values of the logit predictions for &quot;hits&quot; and &quot;misses&quot;?" CreationDate="2010-10-31T18:30:15.720" />
  <row Id="5998" PostId="4070" Text="@whuber But this is not the case here; this function is documented to return variance/(n-1) and is returning 0 for 0/0 symbol. " CreationDate="2010-10-31T19:51:27.377" UserId="88" />
  <row Id="5999" PostId="4071" Text="+1 As I know MATLAB, I think your hypothesis is (sadly) true. " CreationDate="2010-10-31T19:53:59.873" UserId="88" />
  <row Id="6000" PostId="4070" Text="@mbq You missed the disclaimer in the documentation: &quot;For N=1, V is normalized by N.&quot;  That's an elliptical way of saying &quot;For N=1, a value of 0 is returned.&quot;  The point I was making with my comment actually is about something else: it addresses @suncoolsu's assertion that &quot;you are trying to extract information that is *NOT* there&quot; in the case N=1.  On the contrary, a single observation from a random variable indeed *does* provide information about its variance in many practical applications.  (However, I am *not* saying that the variance is best estimated by some kind of variance formula!)" CreationDate="2010-10-31T20:23:53.770" UserId="919" />
  <row Id="6001" PostId="4074" Text="@Fred Questions specifically about how to code such-and-such are of marginal interest, but you actually have presented a statistically interesting question: how does one compute a prediction interval for logistic regression?" CreationDate="2010-10-31T20:30:08.270" UserId="919" />
  <row Id="6002" PostId="4074" Text="@Fred Your calculations still look incorrect.  You seem to be mixing up predicted probabilities and standard errors of their logits." CreationDate="2010-10-31T20:32:25.303" UserId="919" />
  <row Id="6003" PostId="4070" Text="@whuber Fair enough; yet this does not justify this behavior." CreationDate="2010-10-31T20:42:12.873" UserId="88" />
  <row Id="6004" PostId="4070" Text="@mbq I agree with you completely.  Sadly, even Mathematica has troubles with singleton datasets (in its &quot;HypothesisTesting&quot; package: it issues error messages but actually returns a confidence interval of [0,0] when asked for a CI for the variance of a single observation).  It seems that if you want to handle statistical estimators correctly you really need to use a true statistics package and not just some wannabe add-on to Matlab, Mathematica, Excel, or whatever." CreationDate="2010-10-31T20:48:11.540" UserId="919" />
  <row Id="6005" PostId="4090" Text="thanks a lot for the answer! i really appreciate it!" CreationDate="2010-10-31T21:20:30.407" />
  <row Id="6006" PostId="2629" Text="@ars Thanks! Forgot about this one." CreationDate="2010-10-31T22:19:02.283" UserId="930" />
  <row Id="6007" PostId="4088" Text="@Conjugate (+1) Really neat R code. Of note, it will not work if $n$ is odd." CreationDate="2010-10-31T22:24:02.550" UserId="930" />
  <row Id="6008" PostId="3877" Text="@Fred No. Just to say that in this case, you show crude estimates of effect size since they are not adjusted for other covariates. Odds-ratio are better in this respect, since they are adjusted." CreationDate="2010-10-31T22:34:54.283" UserId="930" />
  <row Id="6010" PostId="2629" Text="+1 I just came back to this again (through google).  I think that your answer is spot on." CreationDate="2010-10-31T23:37:58.750" UserId="5" />
  <row Id="6011" PostId="3982" Text="I don't see this as even being on topic. Sure, it's for a visualization app, but the question has to do with a mechanical detail of a particular programming environment.  It is unlikely to lead to any answer or comments of any relevance to data visualization." CreationDate="2010-11-01T02:36:45.707" UserId="919" />
  <row Id="6012" PostId="4098" Score="1" Text="Great answer! Thanks!" CreationDate="2010-11-01T04:12:04.243" UserId="190" />
  <row Id="6013" PostId="4082" Text="my google search for 'Heywood case definition' had been rather unsatisfactory. I am glad to see that such a google search now links to this question. " CreationDate="2010-11-01T04:31:00.260" UserId="795" />
  <row Id="6014" PostId="4088" Text="@chl Thanks!  But I think it will work. The task was to give samples of size 10 from a set of datapoints. Assume n = length(datapoints). The code gives the maximum number (n %/% 10) of such samples. The first corner case is n&lt;10 (anyway ruled out in the problem statement by describing the dataset as 'large', i.e. n&gt;10). In that case you get the datapoints back and a warning (not an error). The second corner case is if there are dangling elements (when n %% 10 != 0). Then you get as many samples as possible and a warning (not an error). Odd n situations are subsumed in one of these two cases." CreationDate="2010-11-01T08:44:35.887" UserId="1739" />
  <row Id="6015" PostId="4088" Text="@Conjugate It seems the first element of the list is of length 11, not 10, and `sum(unlist(lapply(sample, length)))` return the length of `datapoints` (which I set to 1001)." CreationDate="2010-11-01T09:05:27.143" UserId="930" />
  <row Id="6016" PostId="4088" Text="@chl Damn! You're quite right." CreationDate="2010-11-01T09:37:07.880" UserId="1739" />
  <row Id="6018" PostId="4077" Text="@chl-- yeah..even I realize this fault of mine and shall rectify this for sure in posts to come..Thanks for pointing this out..Consider me for some days a naive amateur.." CreationDate="2010-11-01T10:27:41.067" UserId="1763" />
  <row Id="6019" PostId="4077" Text="@onestop..I realize that you people are not paid for teaching me statistics..It seems like my comment has offended you ( though I can tell you for sure I didnt mean to )..Never mind..here for a common cause called &quot;stati-tarian&quot; analogous to humanitarian." CreationDate="2010-11-01T10:30:20.063" UserId="1763" />
  <row Id="6020" PostId="4079" Text="Thanks Gaetan..got you..I think what I take away from this is that standard deviation is a better measure when you have large samples like mine..please let me know if I missed anything." CreationDate="2010-11-01T10:43:36.723" UserId="1763" />
  <row Id="6021" PostId="4100" Text="@Conjugate +1 but Parametric IRT models make strong assumptions about unidimensionality (not obvious in the question) and local independence. Anyway, in this particular case, I would favor some kind of explanatory IRT model, like the LRRM, as explained in de Boeck &amp; Wilson's book: person-specific covariates can be directly tested within such a model, no need to consider weights of any kind." CreationDate="2010-11-01T10:56:14.490" UserId="930" />
  <row Id="6026" PostId="4100" Text="@chl True those *are* strong unidimensionality assumptions. That's why I suggested the Mokken procedures to see how reasonable they were.  I should have put that part first not second.  And it's quite true Joris doesn't say anything about a scale - I'm probably overgeneralizing from the survey problems I see there...  Finally, with more than a handful of respectably informative items, the explanatory IRT models I've seen (i.e covariates on theta) tend to generate the same coefficients as the regression part of a two step irt + regression procedure. Gotta ref for that someplace if you want." CreationDate="2010-11-01T12:00:17.657" UserId="1739" />
  <row Id="6027" PostId="3916" Text="@chl: Thanks so much for holding my hand.  Your function gives me an error: &quot;Error in raw.resp[i, names(tmp)] &lt;- tmp : subscript out of bounds&quot;, but oddly enough, I could run each line of the function without problems.  So I have a working report." CreationDate="2010-11-01T12:29:15.047" UserId="1681" />
  <row Id="6028" PostId="4104" Text="@Max It seems like a good question. I'm not sure about the &quot;data-mining&quot; tag, though. I'd suspect that there should be some kind of appropriate statistical (parametric) model for the data and question at hand." CreationDate="2010-11-01T13:05:19.473" UserId="930" />
  <row Id="6030" PostId="2998" Text="@Tal I'll second Jeromy on that point. Correlations (or $1-r$) are useful metric for clustering or PCA, and sometimes $1-|r|$ are preferred simply because a negative correlation may be considered as informative as a positive one. Taking the complementary value just amount to work with dissimilarities rather than similarities (the same consideration applies in MDS)." CreationDate="2010-11-01T13:22:34.797" UserId="930" />
  <row Id="6032" PostId="4104" Text="@chl I tagged it data-mining because I was thinking about finding some rules and patterns in the data. Wouldn't that make it a data mining technique? The workout is just an example. In general I would like to know if there exist some algorithm that can deal with temporal cause-and-effect data. But I also would like to know if there's some specific statistical model for this particular example." CreationDate="2010-11-01T14:10:50.897" UserId="255" />
  <row Id="6033" PostId="4032" Text="OP here. Thanks for your answer, works perfectly! :)" CreationDate="2010-11-01T14:35:18.790" UserId="1729" />
  <row Id="6034" PostId="4028" Text="Thanks for your answer! :)" CreationDate="2010-11-01T14:36:18.400" UserId="1729" />
  <row Id="6035" PostId="4027" Text="Thanks everyone for the input! " CreationDate="2010-11-01T14:37:12.610" UserId="1729" />
  <row Id="6036" PostId="4104" Text="@Max A simple initial approach is to posit an exponential decay in effectiveness over time.  The decay rate becomes a parameter in a model relating cumulative workout effect (as weighted by the decay) to outcome measures.  If the decay is not huge and not zero and you have sufficient data you can move on to exploring other forms of decay.  This is really in the spirit of exploratory data analysis." CreationDate="2010-11-01T14:45:37.920" UserId="919" />
  <row Id="6037" PostId="4104" Text="@Max How large is your dataset?  If it's relatively small, what you're doing might be better characterized as &quot;exploratory data analysis&quot; rather than &quot;data mining.&quot;" CreationDate="2010-11-01T14:46:38.200" UserId="919" />
  <row Id="6039" PostId="4104" Text="@whuber Depending on how the data is presented, the dataset is from several tens to about a hundred records." CreationDate="2010-11-01T15:59:06.237" UserId="255" />
  <row Id="6040" PostId="4079" Score="1" Text="ayush...  You are right.  That is basically it.  And, this is because your standard error will become so small (due to the large sample size).  This in turn overstates the statistical distance between your test and control groups.  And, causes you to ultimately run into a Type I Error (uncover a difference that is so small as to be immaterial).  This is a common problem in hypothesis testing with large samples.  " CreationDate="2010-11-01T16:17:15.013" UserId="1329" />
  <row Id="6041" PostId="4082" Text="So it looks like this phrase is _not_ used for the case of numerical issues (loss of precision) causing a negative variance estimate, but is not ambiguous." CreationDate="2010-11-01T16:39:03.887" UserId="795" />
  <row Id="6042" PostId="4082" Text="@shabbychef The trick with searches is to focus them with appropriate keywords.  &quot;Case&quot; and, to some extent, &quot;definition&quot; don't accomplish much.  Including &quot;negative&quot; and &quot;variance&quot; encouraged Google to cough up more relevant material ;-)." CreationDate="2010-11-01T17:05:21.740" UserId="919" />
  <row Id="6043" PostId="4104" Text="@Max Thanks: that indicates this is EDA.  One implication is that you shouldn't hope for too much; a pattern will be difficult to detect unless it is extremely strong.  Another implication is that graphical and robust techniques (key elements of EDA) are likely to be helpful." CreationDate="2010-11-01T17:07:18.337" UserId="919" />
  <row Id="6044" PostId="4109" Text="I see one problem with this model. It is assumed that the effect of a workout is positive and descreases with the time. In fact at first the effect is negative, then it moves to a positive range and then decreases to zero. Other than that I believe that this procedure can generate some interesting results. Thanks." CreationDate="2010-11-01T19:46:16.700" UserId="255" />
  <row Id="6045" PostId="4109" Text="@Max It is assumed only that the effect, whether positive or negative, decays at a constant rate over time.  If you are sure that the change in effect over time is more complex than that, you can incorporate that assumption.  (E.g., Y could be the sum of two terms [presumably of opposite signs] having two different decay rates.)  This will introduce additional parameters that play the role of \theta and it could complicate the model sufficiently that you might have to rely on ML methods.  I think you'll find you need lots of data!" CreationDate="2010-11-01T19:56:30.417" UserId="919" />
  <row Id="6046" PostId="4114" Text="+1 Great question.  I was wondering the same thing." CreationDate="2010-11-01T20:32:45.233" UserId="5" />
  <row Id="6047" PostId="4114" Text="must be something in the air. " CreationDate="2010-11-01T20:33:56.933" UserId="29" />
  <row Id="6048" PostId="4111" Score="4" Text="You could also use other anecdotal evidence (e.g. amount of garbage generated compared to normal)." CreationDate="2010-11-01T20:56:01.410" UserId="5" />
  <row Id="6049" PostId="4115" Score="2" Text="@Dirk For illustration, here are some additional papers: http://j.mp/cBJ0hI, http://j.mp/deMrA8, http://j.mp/dhwcrv, http://j.mp/cyVx0m." CreationDate="2010-11-01T21:03:12.483" UserId="930" />
  <row Id="6050" PostId="4115" Text="Thanks for those links!" CreationDate="2010-11-01T21:09:15.457" UserId="334" />
  <row Id="6051" PostId="4111" Text="But Stewart specifically asked to leave the place cleaner then they got it :)" CreationDate="2010-11-01T21:13:08.620" UserId="253" />
  <row Id="6052" PostId="4111" Score="2" Text="DC metro ridership might factor in.  Also overflow from the mall, when they stopped letting people in, might place a minimum -- assuming one has reasonable historical estimates of how many fit in the mall." CreationDate="2010-11-01T21:23:51.023" UserId="251" />
  <row Id="6053" PostId="4111" Text="@Shane @ars yes, you are right.  Now that you mention it, there should be other correlates running around, too, which may do a pretty good job of predicting crowd size." CreationDate="2010-11-01T22:26:51.300" UserId="1108" />
  <row Id="6054" PostId="4116" Text="@Tim Thanks.  This was essentially just what I was looking for.  (I should have googled crowd size instead of population size, yours is there with a bunch of others).  " CreationDate="2010-11-01T22:31:05.553" UserId="1108" />
  <row Id="6055" PostId="4117" Text="@Owe  Thanks, I had wondered how police make their guesstimates and if you ever run across that information I would appreciate your posting it back here, for the record." CreationDate="2010-11-01T22:32:59.280" UserId="1108" />
  <row Id="6056" PostId="4118" Text="@Wayne Yes, I agree that it is a harder problem than many would like to admit.  In fact, it's not even trivial to define what it means to &quot;attend&quot;; do we mean just stopping by for a second, staying all day long?, etc.  I think the actual problem is too hard to solve exactly, and I am happy to hear about first order approximations.  :-)  " CreationDate="2010-11-01T22:37:04.113" UserId="1108" />
  <row Id="6057" PostId="4119" Text="@Uri, thanks, hadn't thought about that.  " CreationDate="2010-11-01T22:37:59.010" UserId="1108" />
  <row Id="6058" PostId="4122" Text="I know that E(XY) is 1 (this is the correlation), so E(X^2)+E(Y^2) must equal 2 but I'm having trouble showing this." CreationDate="2010-11-01T23:29:57.937" UserId="1395" />
  <row Id="6059" PostId="4122" Score="2" Text="Can you connect E(X^2) to the variance formula?" CreationDate="2010-11-01T23:31:03.977" />
  <row Id="6060" PostId="4122" Text="Ahh so E(X^2) = Variance(X) when the mean is 0, and so variance is 1." CreationDate="2010-11-01T23:33:34.940" UserId="1395" />
  <row Id="6061" PostId="4122" Text="Thanks Srikant!" CreationDate="2010-11-01T23:34:43.330" UserId="1395" />
  <row Id="6062" PostId="4120" Text="Does the Y column give the means?" CreationDate="2010-11-02T00:12:38.103" UserId="159" />
  <row Id="6063" PostId="4122" Score="2" Text="+1 Good answer. Just enough help given. " CreationDate="2010-11-02T00:35:42.110" UserId="8" />
  <row Id="6064" PostId="4125" Text="What have you done so far?" CreationDate="2010-11-02T01:30:33.687" />
  <row Id="6065" PostId="4125" Text="I don't know where to start." CreationDate="2010-11-02T01:38:40.100" UserId="1395" />
  <row Id="6066" PostId="4125" Text="I know the simple regression line is y=B0 + B1x" CreationDate="2010-11-02T01:39:03.763" UserId="1395" />
  <row Id="6067" PostId="4125" Score="1" Text="And I guess by the average x and y is E(x) and E(y), not sure how to link that to the regression line though." CreationDate="2010-11-02T01:40:20.457" UserId="1395" />
  <row Id="6068" PostId="4125" Text="The regression line is the line that minimizes the sum of squared errors. Knowing that, and a basic knowledge of calculus, find the values of B0 and B1 that minimize that sum of squared errors. The rest requires a little bit of high school level algebra." CreationDate="2010-11-02T02:06:22.780" UserId="1118" />
  <row Id="6069" PostId="4127" Text="Can somebody with more reputation points edit my question to remove the stars?" CreationDate="2010-11-02T02:59:19.063" UserId="1709" />
  <row Id="6070" PostId="4127" Text="I don't know if this is appropriate for the forum. I will let senior members handle this one." CreationDate="2010-11-02T03:17:16.153" UserId="1307" />
  <row Id="6071" PostId="4125" Text="Starting with the definition is good.  You can carry out the demonstration without calculus, though: if the regression line does not go through the point of averages, you can raise or lower it by some amount to make it pass through that point.  In so doing, you will lower the sum of squares of residuals by *n* times the square of the amount of raising or lowering (which is a one-line algebraic calculation).  If the line was a least-squares line the sum of squares cannot be lowered, demonstrating the shift must have been zero." CreationDate="2010-11-02T05:52:22.713" UserId="919" />
  <row Id="6072" PostId="4125" Text="@Justin: Are you doing regression or correlation?  In the former, the x's are fixed (not random variables) so one usually does not refer to E[x].  Moreover, y is presumed to depend on x, so E[y|x] makes sense, but E[y] would not usually be considered." CreationDate="2010-11-02T05:54:56.173" UserId="919" />
  <row Id="6075" PostId="4127" Score="4" Text="Dear Fred, your question is very general.  You will get people saying &quot;yes i intend&quot; , &quot;no I don't&quot;, but with regards to the feedback - what is it you expect to read?" CreationDate="2010-11-02T07:43:55.523" UserId="253" />
  <row Id="6076" PostId="4131" Text="Quick clarification: Do you want to assume that exactly one functional form should fit all of the participants, or do some participants have one form and some another?  If one of the functional form fits them all, do the parameter values get to vary across participants, or not? " CreationDate="2010-11-02T09:01:55.597" UserId="1739" />
  <row Id="6077" PostId="4131" Text="@Conjugate prior: No. It is quite clear from the data that the best functional form varies across participants. I am running separate nonlinear regressions." CreationDate="2010-11-02T09:07:25.803" UserId="183" />
  <row Id="6078" PostId="3259" Text="I am not an expert but I found the sentence &quot;controlling for associated subscores when devising cut-off scores on each dimension of a measurement scale&quot; a bit esoteric. Can you give me one more line of explanation (otherwise I found it difficult to understand the question) ?" CreationDate="2010-11-02T12:38:29.587" UserId="223" />
  <row Id="6079" PostId="4065" Text="@steffen wrote &quot;the overall precision of the scores should be e.g. 0.1&quot; do you mean you want the score to be a random variable with mean 0.1 ? why don't you generate a classification problem (say with gaussian variables) and use something fast and simple (like LDA) to solve it ? can you tell us more on your motivations ? " CreationDate="2010-11-02T12:45:55.800" UserId="223" />
  <row Id="6080" PostId="4134" Text="+1 Amazing that no one mentioned Matlab before..." CreationDate="2010-11-02T13:05:46.190" UserId="5" />
  <row Id="6082" PostId="1654" Text="Cool - I didn't know about the KFAS package.  There's also the dlm and dse for state space-ish approaches, and an general overview for R users here: http://cran.r-project.org/web/views/TimeSeries.html" CreationDate="2010-11-02T13:19:29.190" UserId="1739" />
  <row Id="6083" PostId="4127" Text="As Tal mentioned, your question could be answered by *Yes* or *No*. Therefore, I'm closing this question." CreationDate="2010-11-02T13:49:50.757" UserId="8" />
  <row Id="6084" PostId="2825" Score="5" Text="If you're going to use R, I'd recommend embedding your R code in an Sweave document that produces your report.  That way the R code stays with the report." CreationDate="2010-11-02T14:00:05.077" UserId="319" />
  <row Id="6085" PostId="4139" Text="But, your answer gives the probability that $P(X_i &gt;= X_j)$ for a specific value of $j$. It does not take into account the possibility that $X_i &lt; X_k$ for some $k \ne j$. My intuition suggests that the answer is $\frac{1}{p}$ where $p$ is the dimension of the random vector." CreationDate="2010-11-02T15:30:21.590" />
  <row Id="6086" PostId="4139" Text="My intuition is flawed! It cannot be $\frac{1}{p}$. A simple simulation shows otherwise." CreationDate="2010-11-02T15:52:05.510" />
  <row Id="6087" PostId="4139" Text="@Srikant oups ... I have updated the answer... but I won't go any further than n=2 (the idea is here but there is too much calculation for me :) ) " CreationDate="2010-11-02T16:08:33.713" UserId="223" />
  <row Id="6088" PostId="4142" Score="2" Text="Instead of controlling type I error while minimizing type II you can minimize the sum of type and type II error (or any combination)... in this case you don't use the likelihood ratio like in Neymann pearson but it's as easy (looks like bayes rule with uniform prior). Anyway, recall that under the null your p-value is uniform (can be close to 0.05) also, I don't really like your procedure. " CreationDate="2010-11-02T17:12:35.520" UserId="223" />
  <row Id="6090" PostId="4143" Text="@suncoolsu: Thanks for your comment. The reason why I want to be using the result from both the test is that, I have some evidence that by choosing a particular null hypothesis, the test result is actually getting biased towards what the null hypothesis tests. Thanks for the paper link though - will go through it." CreationDate="2010-11-02T17:26:37.520" UserId="528" />
  <row Id="6091" PostId="4143" Text="@Samik in a way, when you do a hypothesis test using p-values, you are assuming the null to true and thus the bias for the null. Its just a suggestion: try using Bayes Factor, may be that will remove the &quot;biasness&quot; you observe (although, I must remark, its not a guarantee that this will work)" CreationDate="2010-11-02T17:38:35.823" UserId="1307" />
  <row Id="6092" PostId="4139" Score="2" Text="This is a nice try but unfortunately Part 2 is incorrect (even after interpreting \Phi as the Gaussian probability in the *upper* tail): the right answer has to involve an *integral* of the Gaussian CDF, not the product of two of them (as the iterated expectation indicates).  Alternatively, consider what happens when both variances are 1, the correlation coefficient approaches 1, and u1=u2: the joint probability that X1&gt;u1 and X2&gt;u2 should converge to Pr[X1&gt;u1] (due to near-perfect correlation), but on the contrary the first factor in the right hand side goes to zero." CreationDate="2010-11-02T18:06:06.373" UserId="919" />
  <row Id="6093" PostId="4142" Score="2" Text="@Samik Could you clarify what it means for null hypotheses to be &quot;reversed&quot;?  E.g., if one null is that two means are equal, do you take the other null to be that they are unequal?  (This particular example leads to trouble...)" CreationDate="2010-11-02T18:08:37.463" UserId="919" />
  <row Id="6094" PostId="4143" Text="@suncoolsu: thanks, will read up on that. I have also added the actual context of the question in the OP." CreationDate="2010-11-02T18:19:51.390" UserId="528" />
  <row Id="6096" PostId="4139" Text="@whuber I should not try to write anything else today, to many mistakes in a few hours ;) ... anyway I have updated the answer but it is not friendly anymore... maybe there is a way to make the calculation in the particular case of the question!" CreationDate="2010-11-02T18:40:35.253" UserId="223" />
  <row Id="6097" PostId="4144" Score="1" Text="I didn't have time for more than a pithy answer this morning. I was planning to expand on it this evening but you've done the the job very nicely, whuber." CreationDate="2010-11-02T18:49:31.560" UserId="449" />
  <row Id="6098" PostId="4074" Text="@whuber.  Thanks again.  I agree that something isn't right.  I'll keep working on this, but I would appreciate any guidance you (or anyone else) could provide." CreationDate="2010-11-02T18:58:27.473" />
  <row Id="6099" PostId="4139" Text="@robin Thank you!  This is a difficult calculation on which you have made excellent progress.  (I already upvoted it so unfortunately I can't do so again...)" CreationDate="2010-11-02T19:34:18.637" UserId="919" />
  <row Id="6100" PostId="4074" Score="1" Text="@Fred Ignore the probabilities: work entirely in logits.  That will bring you closer to familiar OLS and ML calculations and lets you avoid all that stuff with logarithms and exponentials.  All that's required is to convert your threshold of 0.75 into its logit in order to make comparisons." CreationDate="2010-11-02T19:40:20.593" UserId="919" />
  <row Id="6102" PostId="4140" Text="@David. Probably I didn't understand something, but can you please clarify how can you simulate &quot;real&quot; CDFs? (I am assuming you are doing a real life data analysis, as you mention the analysis over a genome) " CreationDate="2010-11-02T20:01:26.807" UserId="1307" />
  <row Id="6103" PostId="4147" Text="Thank you for sharing this!  (I like your style of making all parameters variables and clearly commenting them.)  I'm all out of votes today :-( so I'll have to wait to upvote it." CreationDate="2010-11-02T20:03:32.343" UserId="919" />
  <row Id="6104" PostId="4140" Text="@suncoolsu: I guess it was a bad choice of words. I don't simulate real data, I simulate the experiment under the assumption the genome has nothing special in it." CreationDate="2010-11-02T20:12:01.943" UserId="634" />
  <row Id="6105" PostId="4147" Text="@Srikant Just a few words about R code (per your request): you can use `replicate()` whenever you want to call a given function like `rbinom()` $k$ independent times; the `{*}apply()` family functions are very useful to work with matrix, data.frame or list and make a call to the same function by column or row; it is recommended to avoid the underscore for variable naming (and use a dot instead), but it is subject to debate. Otherwise, I also like simple and explicite R code like yours because it will be understandable to everybody. I gave my +1." CreationDate="2010-11-02T21:51:42.987" UserId="930" />
  <row Id="6106" PostId="4147" Text="@chl Thanks. I am using these questions to learn R! So, feedback such as yours is valuable." CreationDate="2010-11-02T21:56:05.153" />
  <row Id="6107" PostId="4147" Text="@Skrikant LOL I was just adding: &quot;BTW I like the way you learn R!&quot;" CreationDate="2010-11-02T21:59:07.403" UserId="930" />
  <row Id="6108" PostId="3259" Text="@robin Yes, basically I meant: we have $j=4$ scores (e.g. anxiety, impulsivity, neuroticism, sensation seeking) and we want to find a cut-off value $t_j$ (i.e. &quot;positive case&quot; if $x_j&gt;t_j$, &quot;negative case&quot; otherwise) for each of them. We usually adjust for other risk factors like gender or age when devising such cut-off (using ROC curve analysis). Now, what about adjusting impulsivity (IMP) on gender, age, *and* sensation seeking (SS) since SS is known to correlate with IMP? In other words, we would have a cut-off value for IMP where effect of age, gender and anxiety level are removed." CreationDate="2010-11-02T22:54:09.403" UserId="930" />
  <row Id="6109" PostId="4149" Text="Further to this, you might want to check out [What Every Computer Scientist Should Know About Floating-Point Arithmetic, by David Goldberg](http://docs.sun.com/source/806-3568/ncg_goldberg.html) for advice on how to deal with issues of numerical representation." CreationDate="2010-11-02T22:58:45.020" UserId="179" />
  <row Id="6110" PostId="4148" Text="@Srikant: Thanks for the link to the notes. I am actually trying to detect (D) [I have separate tests for (B) and (C) which I run before (D)] and from that perspective it does seem like one test is reverse of other. Other than that, do you also have a comment about the procedure itself that I was thinking of using?" CreationDate="2010-11-02T23:19:18.407" UserId="528" />
  <row Id="6111" PostId="4149" Text="@Rob: Thanks for the answer. I guess you are then implying that it is alright to scale the series before doing the analysis." CreationDate="2010-11-02T23:27:14.213" UserId="528" />
  <row Id="6112" PostId="4149" Text="@fmark: Thanks for the comment - I am pretty familiar about that material actually." CreationDate="2010-11-02T23:27:38.393" UserId="528" />
  <row Id="6113" PostId="4149" Text="@Samik: For linear models such as Gaussian ARIMA processes, yes." CreationDate="2010-11-02T23:46:35.830" UserId="159" />
  <row Id="6114" PostId="4149" Score="3" Text="Within vast limits, scaling makes no difference whatsoever for floating point calculations: it simply amounts to a shift of exponent with no loss of precision.  Where scaling can help is where a calculation involves sets of data that are themselves on different scales.  My guess is that the time series formulae are using some time measurement (milliseconds?  years? just integral steps?) that may have a hugely different range than the range of the data.  Good stats SW will internally scale its matrices to avoid loss of precision; this could explain the differences between FP and Minitab." CreationDate="2010-11-02T23:49:32.487" UserId="919" />
  <row Id="6115" PostId="4152" Text="Thanks suncoolsu; I looked into fitdistr but I've found that a mixture model fits my data better than a univariate distribution.  I'm not sure how to use fitdistr for a multivariate distribution while constraining the sum of the weights to 1 - any tips?  I think my problem is primarily in how to optimize a multivariate function subject to constraints.  Thanks, Robert" CreationDate="2010-11-03T00:29:18.067" />
  <row Id="6116" PostId="4150" Score="2" Text="@robert1 If you could disclose the purpose of this fit you will increase your chances of getting really effective responses.  Why do you need a parametrization instead of, say, the empirical distribution function (as revealed by the sample quantiles)?  Do you have any theoretical expectations concerning the shapes of the mixture components (or concerning why one might expect this to a be a mixture)?  What might be the consequences of uncertainty in the parameters?  (That uncertainty will be *huge* unless your distribution is sharply multimodal.)" CreationDate="2010-11-03T00:45:20.233" UserId="919" />
  <row Id="6117" PostId="188" Text="+1 for the last paragraph.  With a minimum of technicalities it conveys the idea well." CreationDate="2010-11-03T00:52:31.300" UserId="919" />
  <row Id="6118" PostId="4152" Score="1" Text="@robert1 Sorry for the previous mistake. I thought you wanted to fit _a_ parametric distribution and _not_ mixtures. OK, before I suggest anything, can you please tell what your data set looks like - I mean how do you get your data, do you have any guesses about the number of mixtures there will be ... and other similar aspects." CreationDate="2010-11-03T01:32:00.713" UserId="1307" />
  <row Id="6119" PostId="3912" Text="@chl, this may be off-topic, but can you please tell me if you made these graphs in R?" CreationDate="2010-11-03T01:41:43.713" UserId="1307" />
  <row Id="6120" PostId="4150" Text="I'm modeling liability - in a personal auto context (let's temporarily assume there is no insurance limit); the amount of liability is almost certainly a mixture - you have a &quot;fender bender&quot; process, a &quot;third party severely injured&quot; process; a catastrophic process (&quot;DWI + injury involving school bus w/ multiple youth injuries&quot;).  I'd like to have a parametrized distribution for use with simulating future loss and certain other applications.  My sample data is small relative to the population, but I know this type of process is long tailed so I am thinking logn/pareto/gamma type distributions. " CreationDate="2010-11-03T01:45:34.103" />
  <row Id="6121" PostId="4154" Text="The EM algorithm looks promising, thank you.  I'll have to become better acquainted with it - I'd be interested in exploring any R packages for this. " CreationDate="2010-11-03T01:56:14.497" />
  <row Id="6122" PostId="4155" Score="1" Text="Do you mean ARIMA rather than X-12-ARIMA? The latter is a decomposition method not a forecasting method." CreationDate="2010-11-03T02:56:47.853" UserId="159" />
  <row Id="6123" PostId="4130" Text="I'm marking this as correct as it is short and sweet.  However, see whuber's lovely longer explication as to WHY it is correct.  And, indeed, in fooling around with the problem, weighted by n produces the correct parameter estimates and relatively similar SE and p values to working with simulated data." CreationDate="2010-11-03T04:48:30.147" UserId="101" />
  <row Id="6126" PostId="4157" Score="3" Text="Re your e.g.: It's a common misconception that a mixture of two Gaussians is necessarily bimodal. A mixture of two Gaussians with equal standard deviations is bimodal only if the difference in their means is at least twice their standard deviation. See: Schilling, Mark F.; Watkins, Ann E.; Watkins, William (2002). &quot;Is Human Height Bimodal?&quot;. *The American Statistician* 56(3): 223–229. http://dx.doi.org/10.1198/00031300265" CreationDate="2010-11-03T07:15:11.680" UserId="449" />
  <row Id="6129" PostId="4154" Text="@Skrikant The [mclust](http://cran.r-project.org/web/packages/mclust/index.html) package should work here. To reproduce the example on Wikipedia (two-cluster solution), you will have to fix parameter `modelNames` (shape of covariance matrices) or `G` (No. clusters). Otherwise, the returned solution (i.e. minimizes BIC) is a three-cluster solution. Maybe you could add it to your original answer, for illustration purpose." CreationDate="2010-11-03T08:26:04.860" UserId="930" />
  <row Id="6130" PostId="3912" Text="@suncoolsu No, it is LateX + PSTricks. Here is the code if you'd like to play with it, https://gist.github.com/660876 (it's old and pretty ugly for sure). In R, it should not be very difficult to reproduce, though." CreationDate="2010-11-03T08:32:37.077" UserId="930" />
  <row Id="6132" PostId="4154" Text="@Skrikant After browsing the site, I also came across @csgillespie's example with [mixtools](http://cran.r-project.org/web/packages/mixtools/index.html), http://stats.stackexchange.com/questions/899/separating-two-populations-from-the-sample/1010#1010." CreationDate="2010-11-03T09:09:57.727" UserId="930" />
  <row Id="6135" PostId="854" Text="Why not?  Because Fisher's Exact Test it assumes that data generating process has both sets of marginal totals fixed but actually occurring processes seldom do.   As Hadley points out, we need to know about that generating process." CreationDate="2010-11-03T12:59:32.180" UserId="1739" />
  <row Id="6136" PostId="4169" Score="2" Text="That's nice, Mike, but *what does the paper say?*" CreationDate="2010-11-03T14:17:32.643" UserId="919" />
  <row Id="6137" PostId="4117" Text="Well, I asked, but he didn't remember. On a lighter note, Rob Hyndman has a story about the way headline numbers are arived at. Scroll down a bit on http://robjhyndman.com/researchtips/statistical-jokes/" CreationDate="2010-11-03T15:09:04.370" UserId="1766" />
  <row Id="6139" PostId="4177" Text="Stochastic processes are also very useful in the industry (think Wall Street, financial industry)." CreationDate="2010-11-03T15:37:20.187" />
  <row Id="6140" PostId="4177" Score="1" Text="@Srikant-vadali: Good point. I suppose I should add that I did a PhD in Stoc Proc and have found it very helpful in my new field of systems biology." CreationDate="2010-11-03T15:45:15.260" UserId="8" />
  <row Id="6141" PostId="4065" Text="@Robin: No, I am talking about &quot;precision&quot; as measure of the quality of a binary classification model (see e.g. http://en.wikipedia.org/wiki/Sensitivity_and_specificity here). Regarding your second question: I require the scores for a simulation where multiple binary classification models are combined (more in the area of data processing than Data Mining). It is hard to explain the task in details, but I am pretty sure I need these scores :D " CreationDate="2010-11-03T16:02:34.667" UserId="264" />
  <row Id="6142" PostId="4178" Text="I'd appreciate if anyone with more skills in latex could edit the calculation in 4. so that it is more readable." CreationDate="2010-11-03T16:04:53.260" UserId="264" />
  <row Id="6143" PostId="4176" Text="Surely p^ is 7/100 (or however many trails were used in the original test)?" CreationDate="2010-11-03T16:09:03.253" UserId="229" />
  <row Id="6144" PostId="4174" Text="The iteration numbers are surely just identifiers, and should not be used as numbers in computations. How many iterations did you do in total in your original trial?" CreationDate="2010-11-03T16:11:19.397" UserId="229" />
  <row Id="6145" PostId="4176" Text="No I don't think so. I took the question to mean that he ran device until he reached a failure. So it failed on 100 iterations, 22 iterations, ..." CreationDate="2010-11-03T16:14:51.210" UserId="8" />
  <row Id="6147" PostId="4174" Text="@James I could be wrong but if the iterations are simply identifiers then most people will naturally report them in an increasing order like so: 22 24 36 44 ... The fact that the numbers are reported in an apparently random sequence suggests that each number represents the failed iteration number of 7 separate tests." CreationDate="2010-11-03T16:26:56.213" />
  <row Id="6149" PostId="3912" Text="@chl thanks! much appreciated" CreationDate="2010-11-03T16:42:40.093" UserId="1307" />
  <row Id="6150" PostId="4175" Score="1" Text="What's the purpose of the experiment?  The answer to that determines the answers to all your questions." CreationDate="2010-11-03T16:43:43.987" UserId="919" />
  <row Id="6151" PostId="4074" Text="@whuber.  Thanks once more.  I think I've got it." CreationDate="2010-11-03T16:44:05.503" />
  <row Id="6152" PostId="4175" Text="The purpose of the &quot;experiment&quot; was to test the claim that subjects can score above chance. Hence H1." CreationDate="2010-11-03T16:47:18.287" UserId="1614" />
  <row Id="6153" PostId="4180" Text="@whuber: +1 very nice answer." CreationDate="2010-11-03T16:49:56.203" UserId="8" />
  <row Id="6154" PostId="4176" Text="Then shouldn't each test have its own failure probability?" CreationDate="2010-11-03T16:56:13.533" UserId="229" />
  <row Id="6155" PostId="4174" Text="I take your point, but I'm still a little unsure of the value of the mean of these numbers." CreationDate="2010-11-03T16:57:26.433" UserId="229" />
  <row Id="6156" PostId="4176" Text="Each test does have it's own failure probability - it's 1. We've got the data to prove it ;) Anyway I'm assuming that each test is a replicate from the Geometric distribution." CreationDate="2010-11-03T16:59:33.463" UserId="8" />
  <row Id="6159" PostId="4180" Text="@whuber: thank you so much for this thoughtful answer. I have included an **Update** section in my question that attempts to address your concern as well as ask an additional question." CreationDate="2010-11-03T17:54:52.130" UserId="1786" />
  <row Id="6161" PostId="4174" Text="@Srikant: Yes, your interpretation is spot on.   To James' question, I think it would be fair to say that with this given device and software test bench, the device fails *on average* at the 55th iteration.  Does that not sit will with you?" CreationDate="2010-11-03T18:01:03.233" UserId="1786" />
  <row Id="6162" PostId="4176" Text="@csgillespie: Thank you for taking the time to write this answer.  I have added an **Update** section to my original answer.  One more question I have for you is if there is an underlying reason why we believe failures rates to be exponentially distributed or are we just back fitting the empirical data we see from known failures to a distribution that fits?" CreationDate="2010-11-03T18:21:01.953" UserId="1786" />
  <row Id="6163" PostId="4179" Text="What is this doing exactly? Im not familiar with R-code but if I had to guess it is using the original 7 figures to extrapolate out the exponential series with 100,000 data points?" CreationDate="2010-11-03T18:26:52.417" UserId="1786" />
  <row Id="6164" PostId="4181" Score="2" Text="+1 Very nice!  This problem *had* to require numerical integration (except for special cases such as all means being equal), so the availability of this package and the reduction of one in dimensionality are both useful contributions.  By the way, as a check you can compute P(Y&gt;X and Y&gt;Z) = 0.3200386 and P(Z&gt;X and Z&gt;Y) = 0.5353126; the three results sum to 0.9999999, which is reasonably close to 1." CreationDate="2010-11-03T18:28:49.287" UserId="919" />
  <row Id="6165" PostId="4179" Text="I edited the code with comments (which in R are after a #)" CreationDate="2010-11-03T18:59:55.947" UserId="1766" />
  <row Id="6166" PostId="4176" Text="@SiegeX: no, we're not playing fishing games here.  When the probability of failure is constant over time the distribution of failure times will be exponential.  Alternative theories of failure probabilities lead to other predictions of the failure time distribution.  You have to be prepared to be wrong about your assumptions!  (That's one reason to check them using techniques like probability plots.)" CreationDate="2010-11-03T19:05:23.890" UserId="919" />
  <row Id="6167" PostId="4176" Score="3" Text="Assuming a geometric distribution is assuming that the failures are independent. In other words, on any trial the device fails with probability p, independent of the other trials (like tossing a coin). This is a reasonable starting point, but if the failure is due to something like a memory leak, then the longer the device runs, the more likely it is to fail, and the assumption of independent failures would be invalid." CreationDate="2010-11-03T19:06:01.297" UserId="247" />
  <row Id="6168" PostId="4182" Text="That's a clever approach, but it seems biased to me, for two reasons.  The first is that the occurrence of a failure in any experiment precludes the possibility of any more successes, calling into question the independence of failure and success which is assumed by Fisher's Exact Test.  In other words, how can you model these results as if they were like drawing balls out of an urn?  The second is the censoring of the last value at 223 has to be handled a little more carefully.  (This problem can be surmounted.)" CreationDate="2010-11-03T19:11:08.750" UserId="919" />
  <row Id="6170" PostId="4181" Text="@whuber Thanks.  Thanks also for following through with P(Y&gt;X and Y&gt;Z) and P(Z&gt;X and Z&gt;Y); that didn't occur to me." CreationDate="2010-11-03T19:14:24.200" UserId="1108" />
  <row Id="6171" PostId="4179" Text="PS: Of course, this is not really the worst case (which would be a draw of 7x100), but then you wouldn't have to use bootstrap, because you would ignore the other observations. " CreationDate="2010-11-03T19:15:42.907" UserId="1766" />
  <row Id="6172" PostId="4164" Score="1" Text="@Simon  I think the question isn't fully specified and so it is hard to tell exactly what the OP meant without asking him directly.  Maybe instead of drafting a solution to a hypothetical problem I should have asked for clarification.  Why not do that now?  :-)" CreationDate="2010-11-03T19:16:37.240" UserId="1108" />
  <row Id="6173" PostId="4138" Text="@gregor What is &quot;l&quot; in this problem?  Is it fixed (but arbitrary), or something else?" CreationDate="2010-11-03T19:17:28.097" UserId="1108" />
  <row Id="6174" PostId="4175" Text="Just to be clear (since I have no idea what &quot;H1&quot; means): are you testing the claim that the *average score* in a *population* is better than 1/5 or that there exist *particular subjects* who have a tendency to score better than 1/5?" CreationDate="2010-11-03T19:32:23.887" UserId="919" />
  <row Id="6175" PostId="4174" Text="@SiegeX Your last approach has the right flavor, but please don't use the formulae on the Wikipedia page: they are only for normally distributed data.  It is rare for failure time data to be normal.  They typically are positively skewed.  This implies the normal theory upper prediction limits can be (way) too low.  BTW, a lower prediction limit is irrelevant here--although the fact that it is hugely negative is a clear indicator of how bad the normal theory methods are for these data.  To adjust the equation, see the reference I provided in my answer." CreationDate="2010-11-03T19:35:41.480" UserId="919" />
  <row Id="6176" PostId="4175" Text="The experimental hypothesis (H1) is that subjects in the study will score above chance. The null hypothesis (H0) assumes a binomial distribution for each subject. I tried to make that clear in the question." CreationDate="2010-11-03T19:38:25.937" UserId="1614" />
  <row Id="6177" PostId="539" Score="4" Text="This question and its responses remind us of how crude and limited this antiquated division of variables into categorical-ordinal-interval-ratio really is. It can guide the statistically naive, but for the thoughtful or experienced analyst it's a hindrance, an obstacle in the way of *expressing variables in ways that are appropriate for the data and the decisions to be made with them.* Someone working from this latter point of view will freely move between categorical and &quot;continuous&quot; data representations; for them, this question cannot even arise! Instead, we should ask: how does it help?" CreationDate="2010-11-03T20:29:39.853" UserId="919" />
  <row Id="6178" PostId="4175" Text="@RSoul. As whuber as has already said, is your $H_o$: mean score = 1/5 and $H_1$: is mean score $\ge$ 1/5 ? " CreationDate="2010-11-03T20:43:17.630" UserId="1307" />
  <row Id="6179" PostId="4175" Text="Okay. That's a good question. I'll just put down my copy of Feller." CreationDate="2010-11-03T20:48:20.530" UserId="1614" />
  <row Id="6180" PostId="4175" Text="There was a stray zero in the question which may have confused things. I am testing the claim that subjects will score significantly above mu. " CreationDate="2010-11-03T20:55:43.533" UserId="1614" />
  <row Id="6181" PostId="539" Text="@whuber (+1) At the very least, it seems difficult to optimize measurement reliability and diagnostic accuracy at the same time." CreationDate="2010-11-03T21:02:30.223" UserId="930" />
  <row Id="6182" PostId="4175" Text="@RSoul Sorry to keep asking questions, but I do have a lot.  Is this really a die or is it a pseudorandom number generator?  (Charles Tart famously &quot;demonstrated&quot; ESP by virtue of a flawed RNG!)  If it's a die, is it just one or are there several copies of it?  (With a physical die, it's doubtful all outcomes have probability 1/5, so your experiment needs to pay attention to learning effects and nonuniformity of outcomes.)  What is your justification for looking solely at the one-sided alternative?  How are the subjects selected for testing?  (Answers to these can guide resampling methods.)" CreationDate="2010-11-03T21:10:05.683" UserId="919" />
  <row Id="6183" PostId="4175" Text="The real experiment is not a dice experiment. It is computer based, but tbh I didn't want to go into the details. The information given should really be more than enough. I was essentially testing chance and finding chance. I realise it seems an odd thing to do." CreationDate="2010-11-03T21:19:35.093" UserId="1614" />
  <row Id="6184" PostId="4182" Text="@mcdowella: Your website's abstract seems spot on to the problem I'm facing here.  Thank you for taking the time to put together this page.  Prior to the 223 continuous run, I also had two additional continuous runs of 102 and 60 which I stopped early for time consideration.  There has been no failure between the 102, 60 and 223 runs so I think I can add those up for the figure in your &quot;New Version&quot; Successes cell." CreationDate="2010-11-03T21:34:06.980" UserId="1786" />
  <row Id="6185" PostId="4182" Text="@whuber: Actually, I think your first concern doesn't apply in this situation because once the device fails there can no longer be any more successes *for that test* because the device hard locks and requires me to cycle the power to bring it back to a normal state.  There is still the concern about me censoring the data, however." CreationDate="2010-11-03T21:38:10.410" UserId="1786" />
  <row Id="6186" PostId="4175" Text="BTW I was trying to keep it simple, not obfuscate." CreationDate="2010-11-03T21:53:00.433" UserId="1614" />
  <row Id="6187" PostId="4179" Text="Thank you for the comments, I understand it much better now.  However, could you put into layman's term what &quot;quantile(p.vec,probs=0.01) #calculates the 1%-quantile of the probs&quot; is essentially telling us? I see that we get a value of 0.0122 for the 1% quantile. How do I interpret that?" CreationDate="2010-11-03T22:46:44.700" UserId="1786" />
  <row Id="6188" PostId="4149" Text="Thanks for the discussion." CreationDate="2010-11-03T23:11:06.873" UserId="528" />
  <row Id="6190" PostId="4180" Text="@whuber: I've added an **UPDATE 2** section that attempts to use the equations in the *Normal Based Methods in a Gamma Distribution* paper by Krishnamoorthy, Mathew, and Mukherjee.  Do you agree with my results?  If so, I intend to get more failure data by removing the fix then testing the new data set against an exponential probability plot to ensure they fit the exponential distribution.  I'll then update my R code with the new dataset to get my new upper limits." CreationDate="2010-11-04T01:51:41.727" UserId="1786" />
  <row Id="6191" PostId="4180" Text="@SiegeX Good job!  Using equation (2) of the paper with your data and your confidence levels, but using a different computing platform as a check, I obtain UPLs of 42.5763,50.1065,58.4814,71.5132,83.4132,121.271,143.914,220.331.  That looks very close to what you have plotted." CreationDate="2010-11-04T02:13:49.787" UserId="919" />
  <row Id="6192" PostId="4186" Score="1" Text="You are correct: the formula is an elementary plane geometry exercise.  The lines create three triangles at the left, top, and right.  The left and right ones are similar, whence the estimated mode partitions its bin in the same proportion as the bases of these triangles.  In the notation of the formula, one base is *fm-f1* and the other is *fm-f2*, so the fraction of the way to go across the bin equals *(fm-f1)* / (sum of the two bases).  The interpolated value is obtained by starting at the left endpoint *L* and adding this fraction of the bin width *h*." CreationDate="2010-11-04T02:32:37.267" UserId="919" />
  <row Id="6193" PostId="4187" Text="@george Please verify that the parentheses I inserted into the shrinkage formula are where you intended them to be." CreationDate="2010-11-04T04:10:29.143" UserId="919" />
  <row Id="6194" PostId="4165" Score="4" Text="It's a little discouraging that any teacher would be so frankly ignorant of the applications of their field." CreationDate="2010-11-04T04:27:56.093" UserId="919" />
  <row Id="6195" PostId="4193" Text="@gabgoh (+1) .. good question. The Bayesian Central Limit Theorem is called the Laplace Approximation ie the posterior behaves &quot;more or less&quot; like a normal distribution. (formally posterior converges in distribution to a normal distribution)" CreationDate="2010-11-04T06:17:32.527" UserId="1307" />
  <row Id="6196" PostId="2635" Text="@Alone I attempted to make it a better title; feel free to change it to something more appropriate" CreationDate="2010-11-04T06:25:39.737" UserId="183" />
  <row Id="6197" PostId="4182" Text="I agree that my approach is not completely correct for the data you actually have. I suspect that the error is small, especially in comparison to worries about non-independence of successive trials if the hardware is e.g. failing partly due to the build up of heat. I don't know this for sure, though. I personally would be happier if I could run enough trials to come up with a tail probability of 1E-6 or smaller, or if I could run experiments to find a way to provoke a hardware failure 100% of the time and then file a bug report the hardware guys could work on." CreationDate="2010-11-04T06:29:48.843" UserId="1789" />
  <row Id="6198" PostId="4194" Text="To add to this answer, you need not marginalize the other variables as originally stated in the question. All you need to do is to 're-organize' $Pr(X_i|X_{i-1},X_{i+1})$ such that you recognize the result as a known pdf and you are done. As long as you are able to re-organize the above everything else (i.e., all other constants, the integral in the denominator etc) will equal the appropriate constant for the pdf to integrate to 1. " CreationDate="2010-11-04T09:15:29.453" />
  <row Id="6199" PostId="4179" Score="1" Text="The quantile is a rank-based value. One you probably know is the median: You sort your observations according to size, and the median is the observation in the middle (say, you have the income of 100 people, then the median income is the 50th-highest income). Now generalizing, the 1%-quantile is the probability at 1% of the observations. The advantage of a quantile is that it is non-parametric, you don't have to make assumptions about the underlying distribution, and that it is less biased than the quantile-function of a symmetrical distribution when you have skewed data." CreationDate="2010-11-04T09:16:26.810" UserId="1766" />
  <row Id="6200" PostId="4186" Text="@Srikant: Thanks. I know it is only an estimate but is there any reason why the mode must be at the intersection of the two diagonal lines and not somewhere else?                                                           @whuber: Thanks. Your comments was very helpful in understanding the answer." CreationDate="2010-11-04T09:55:56.953" UserId="1636" />
  <row Id="6201" PostId="4187" Text="@george Shouldn't the 2nd part of your question (with a binary variable) be a separate question?" CreationDate="2010-11-04T11:45:06.007" UserId="930" />
  <row Id="6202" PostId="4138" Text="@gregor are you interested in some particular value for $\Sigma$ ? do you need to calculate a limit when the dimension is large ? do you need to simulate things? " CreationDate="2010-11-04T12:15:48.267" UserId="223" />
  <row Id="6204" PostId="4169" Text="@whuber: possibly you missed the link to the paper posted in my response. If not, possibly you didn't bother to look at it, in which case the gist is that I find that the EM algorithm performs well in some cases (lots of data, reasonable proportion of reasonably concentrated Von Mises data), not so well in others (little data, low proportion of Von Mises, Von Mises with low concentration)." CreationDate="2010-11-04T12:30:32.380" UserId="364" />
  <row Id="6205" PostId="2495" Text="I don't thing consistency can have any sense in a goodness of fit test (such as KS test).  You said &quot;with asymptotic power the alternatives are changing&quot; do you have an idea of how they change (different way of changing them will lead to different conclusions)? &quot;Under some regularity conditions&quot; is it easy to meet these regularity conditions in the case of Kolmogorov Smirnov test ? Did you find something about that in Lehmann's book ? " CreationDate="2010-11-04T12:33:44.750" UserId="223" />
  <row Id="6206" PostId="2495" Text="1) Sure, it can.  Let H0: X ~ N(0,1), suppose X1,...Xn ~ N(1,1), and let n go to infinity.  That's consistency for the alternative $N(1,1)$.  The KS-test is nonparametric so I bet it would be lots of fun to show consistency for all possible alternatives.  2) Yes, they change on the order of $\sqrt n$, say.  3) I'm not quite clear on the question(s): are you wanting to show the asymptotic power of the KS-test is 1?  The first step would be to find a definition of asymp power that makes sense for a nonparametric test.  I don't happen to know of one, in Lehmann's book or elsewhere." CreationDate="2010-11-04T13:30:35.943" UserId="1108" />
  <row Id="6207" PostId="4169" Score="3" Text="@Mike This issue has come up on other SE sites, too. The point is that answers should be stand-alone, for many reasons. The intent is to make our threads a source of information, not just some gateway of links to possibly evanescent material. The response &quot;an answer is at such-and-such a place&quot; is appropriate when the answer is simple, standard, and easily reachable from a search (e.g., has a good Wikipedia article).  Otherwise, *especially* when the answer reflects a significant investigation like yours, please at least provide a summary. There's no need to attack me for making this reminder." CreationDate="2010-11-04T14:05:16.430" UserId="919" />
  <row Id="6211" PostId="4196" Text="@bitbox What kind of inference are you making?  E.g., do you view the discrete values X(t1), X(t2), ... as a sample from a continuous set of values about which you would like to draw conclusions, or do you view this time series as a sample from many possible such series for a particular combination of car/driver/conditions/location, or do you perhaps view it as a sample from possible time series for many cars and drivers?" CreationDate="2010-11-04T14:21:02.353" UserId="919" />
  <row Id="6213" PostId="4178" Text="since this is exactly what I wanted, I will mark it as &quot; the answer&quot;. However, I am still open for suggestions and ideas." CreationDate="2010-11-04T14:54:33.637" UserId="264" />
  <row Id="6214" PostId="4206" Text="@onestop: Thanks for the links! From what my understanding is worth though, these links discuss the distribution of the fit parameters (in my clumsy notation, p), not the distribution of the fit function itself at a fixed value of the argument. Or do I miss something simple?" CreationDate="2010-11-04T15:22:59.560" UserId="1197" />
  <row Id="6215" PostId="4186" Text="@Sara The mode in general depends on how the data are grouped, so the best you can hope for is some reasonable estimate. I suspect this formula may originally have been derived by assuming numbers were drawn independently from a unimodal distribution and then grouped somehow. On the average the interpolated value ought to depart less from the mode of the underlying distribution than some other estimate (such as the midpoint or one of the endpoints of the largest group). When you know more about the distribution--e.g., if it's Normal--then this interpolator is not necessarily best on average." CreationDate="2010-11-04T15:30:03.110" UserId="919" />
  <row Id="6216" PostId="4156" Text="@Rob But doesn't the cap indicate a possibly strong heteroscedasticity (and skewness) in the residuals?  That would suggest using a non-linear re-expression of the customer counts." CreationDate="2010-11-04T15:50:10.277" UserId="919" />
  <row Id="6218" PostId="4174" Text="@SiegeX OK, that does make more sense, seems like your data could be drawn from an overdispersed poisson process" CreationDate="2010-11-04T16:45:02.050" UserId="229" />
  <row Id="6220" PostId="4216" Text="Can you clarify how this is related to data analysis? If not, perhaps this is a question more appropriate for http://math.stackexchange.com" CreationDate="2010-11-04T17:25:04.447" />
  <row Id="6221" PostId="1476" Text="What I find interesting about this paper is that the *perspective* is Bayesian, but the hierarchical modeling approach offered to replace corrections for multiple comparisons does *not* require you to be Bayesian. " CreationDate="2010-11-04T17:35:17.963" UserId="1739" />
  <row Id="6222" PostId="4216" Text="and clarify why this isn't just f[a]" CreationDate="2010-11-04T18:07:07.733" UserId="247" />
  <row Id="6223" PostId="4216" Score="3" Text="@tarrasch Mathematica computes limits with Limit[].  When you use it you will find the limiting value is zero, as it must be.  If instead you mean you want the of the integral *divided by a-b*, then for functions continuous at b the fundamental theorem of calculus tells you the answer equals f[b]." CreationDate="2010-11-04T18:08:02.960" UserId="919" />
  <row Id="6224" PostId="4206" Text="You're right, I didn't read your question properly. Once you've got an estimate and standard error for p from asympototic theory, however, an estimate and standard error for any function of p follows straightforwardly using the delta method. &#xA;http://en.wikipedia.org/wiki/Delta_method" CreationDate="2010-11-04T18:43:55.740" UserId="449" />
  <row Id="6226" PostId="4206" Text="Let me make sure I got it right. So, basically, once I have the best-fit p0 and its variance dp, I take f(x,p0+dp), Taylor expand it, and identify the deviation from f(x,p0) as a variance of f(x,p0) itself, is this right?" CreationDate="2010-11-04T19:40:37.640" UserId="1197" />
  <row Id="6227" PostId="4213" Score="1" Text="+1 this is neat, I'd forgotten about the vcdExtra package." CreationDate="2010-11-04T20:01:12.337" UserId="251" />
  <row Id="6228" PostId="4212" Score="1" Text="I should have pointed out that this ordering is not necessarily transitive!  E.g., on a nine-point scale (0,2,0,2,0,0,0,0,2) beats (2,0,0,0,0,2,0,2,0) beats (0,0,2,0,2,0,2,0,0) which beats the first one!  See http://en.wikipedia.org/wiki/Nontransitive_dice ." CreationDate="2010-11-04T20:17:35.957" UserId="919" />
  <row Id="6229" PostId="4156" Text="@whuber. Yes. But non-linear re-expression of X is really a different problem. The &quot;right&quot; way to do this is to assume a latent process Y that is partially observed and try to model that using a non-linear state space model. But I figured that was more complicated than the OP would want. My suggestion will work ok provided the cap is not hit very often. Otherwise it will underestimate the forecast variance." CreationDate="2010-11-04T21:06:31.107" UserId="159" />
  <row Id="6230" PostId="4196" Text="@whuber: in this case, it's really a single car making measurements so my hunch is that it's not independent : am I right ? I will, at one point, be in a &quot;crowdsourcing&quot; case where a lot of independent cars will be driving and I will be looking at what's happening in the same spots : in that case, can I consider them independent even though they should be measuring the same thing ? I think so but I'd like to be extra sure." CreationDate="2010-11-04T21:12:40.163" UserId="1784" />
  <row Id="6231" PostId="4176" Text="@PeterR: Does this assumption of independence in failures not apply to a normal/Gaussian distribution?" CreationDate="2010-11-04T21:19:33.873" UserId="1786" />
  <row Id="6232" PostId="4196" Text="@bitbox &quot;Independence&quot; makes sense only after you have adopted a chance model for the data.  Several good models are appropriate.  Which one(s) to use depend on the decisions you plan to make.  So: it seems you wish to estimate the proportion of time spent moving, stopped, or off.  Will you use these proportions to characterize (a) the history of a single ride, (b) conditions along a route traveled by many cars at the same time, (c) a route to be traveled extensively in the future, or (d) some other generalization of the data?" CreationDate="2010-11-04T21:55:50.983" UserId="919" />
  <row Id="6233" PostId="4214" Score="1" Text="@Tal What would be the purpose of (or what would you conclude from) a meta-analysis on studies involving the same outcome but different predictors or risk factors?" CreationDate="2010-11-04T22:19:46.987" UserId="930" />
  <row Id="6234" PostId="4218" Score="2" Text="(This is rather marginally related to data analysis.)  Were you not able to obtain answers with a Web search? Googling &quot;social gaming metric churn arpu&quot; turns up good results at http://blog.kontagent.com/2009/02/ , http://www.quora.com/What-are-the-most-relevant-metrics-for-social-games-and-how-are-they-calculated/answers/143000 , http://www.wavedash.net/2010/04/the-secret-glossary-of-social-games-analytics/ , http://playthisthing.com/social-network-games-arpu-over-design , and many other sites." CreationDate="2010-11-04T22:22:20.293" UserId="919" />
  <row Id="6235" PostId="3332" Text="@Andre Yes, there's a [prim](http://cran.r-project.org/web/packages/prim/) package, with an R vignette: [Using prim for bump hunting](http://cran.r-project.org/web/packages/prim/vignettes/prim-2d.pdf). It's not obvious to me how it will work in this case, though." CreationDate="2010-11-04T22:36:05.603" UserId="930" />
  <row Id="6237" PostId="4212" Text="@whuber I've already +1, but I wonder (maybe a naive question) how it would generalize to more than two items? (btw, shouldn't &quot;integral&quot; (3rd §) actually read &quot;integer&quot;?)" CreationDate="2010-11-04T22:52:48.680" UserId="930" />
  <row Id="6239" PostId="4214" Text="Hi chl,  If I where to send 100 researchers to study different risk factors for the same disease, and 5 of them came back with a P value bellow 5% (but not much less).  Would it be relevant to somehow take all the studies into account? (I believe on the practical level, the answer is no.  But on a more hypothetical level - I am not sure).  From the little I know of meta analysis, it would be relevant only if there was an overlap in risk factors, otherwise, it doesn't offer much (at least from the little I know).  Is that what you where aiming at?" CreationDate="2010-11-04T23:34:41.670" UserId="253" />
  <row Id="6240" PostId="4213" Text="Very nice list chl - thank you!" CreationDate="2010-11-04T23:40:03.240" UserId="253" />
  <row Id="6241" PostId="4155" Text="Yes, I mean ARIMA, though I've found using the ARIMA within the X12-ARIMA package to be more convenient than the various ARIMA packages in R. (The diagnostics, the ease of adding variables like AO2004.Jan, etc. The actual workflow is a bit more awkward, but it seems to be a win to me.) So, to your point, there's X12-ARIMA the package, and X12-ARIMA the method, and I didn't think to differentiate." CreationDate="2010-11-04T23:46:27.397" UserId="1764" />
  <row Id="6242" PostId="4218" Text="Thanks a lot for the links they are really usefull to me" CreationDate="2010-11-04T23:52:58.590" UserId="1808" />
  <row Id="6243" PostId="4156" Text="@Rob You're right, whuber's suggestion is currently beyond me. Unfortunately. Though, I am trying to understand State Space models as we speak and figure that's eventually my answer. (To account for a capacity cap and to provide for trend prediction.) Somehow, the SS concept seems simple but figuring out what the actual matrices accomplish... I've started watching a great SS class from Stanford on Youtube, and there are two or three R packages to choose from." CreationDate="2010-11-04T23:53:27.513" UserId="1764" />
  <row Id="6244" PostId="4156" Text="@Rob In terms of X=min(Y, 1000), that's a good first-cut. I've seen a &quot;resistance&quot; when nearing capacity, including something like bounces when the cap is hit, so perhaps something that adjusts the point forecast down as the upper prediction interval exceeds the cap could be a second-cut?" CreationDate="2010-11-04T23:58:40.350" UserId="1764" />
  <row Id="6245" PostId="4212" Score="1" Text="@chl (&quot;Integral&quot; is the adjectival form of &quot;integer&quot;.) With more than two items you can of course still rank them in pairs this way.  If you have intransitivity you can resolve it in some *ad hoc* ways, such as computing the percentage of time one distribution is the &quot;winner&quot; among a group of three or more distributions.  As you know, the subject of comparing distributions gets complex because it includes most of the paradoxes of voting theory.  But I like this approach because it forces us to confront such inherent difficulties rather than letting us stay in ignorance of them." CreationDate="2010-11-05T01:18:08.733" UserId="919" />
  <row Id="6246" PostId="4194" Score="1" Text="They don't need to be found analytically.  All full conditionals are proportional to the joint distribution, for instance. And that's all that's needed for Metropolis-Hastings." CreationDate="2010-11-05T02:05:50.703" UserId="493" />
  <row Id="6247" PostId="4217" Text="Hello Srikant, thank you for thinking about this.  Here is a counter example:  Let's say we have done a research (which will be published) on which of a 100 risk factors effects a disease.  In our article we have received 100 P values, 5 of which where around 5% (&quot;significant&quot;).  The Null hypothesis would be different (&quot;risk factor i is not effecting the disease&quot;).  But still if we won't correct for multiplicity, we will get E(x)=5 wrong &quot;discoveries&quot;.  So unless I'm missing something, relying on the null hypothesis, IMHO, won't be enough." CreationDate="2010-11-05T04:25:51.417" UserId="253" />
  <row Id="6248" PostId="4224" Text="Dear Charlotte, this is very helpful, thank you!" CreationDate="2010-11-05T04:36:24.103" UserId="253" />
  <row Id="6249" PostId="4194" Score="1" Text="@Tristan of course. I am, however, talking about gibbs sampling." CreationDate="2010-11-05T06:22:08.597" UserId="1760" />
  <row Id="6250" PostId="4209" Text="for some reason, I don't find your answer entirely convincing. I do agree that the distribution needs to be &quot;stretched&quot;, and that it's not correct to say it converges to X before it converges to a normal. That would be a mistake on my part. Still I think there should exist some way to scale the distribution such that only the fourth order and above &quot;moments&quot; go towards zero. I need to think a little harder as to what exactly that scaling factor thing would look like, if such a thing were to exist" CreationDate="2010-11-05T06:27:53.097" UserId="1760" />
  <row Id="6251" PostId="4206" Text="I think you're right though I'm having trouble understanding your notation - dp for variance looks odd and adding a variance to an estimate is wrong on purely dimensional grounds." CreationDate="2010-11-05T07:20:15.263" UserId="449" />
  <row Id="6252" PostId="4214" Text="@Tal Well, I was rather thinking of meta-analysis as a tool to pool measures of association (or ESs) across several studies, in a *retrospective* way, which means working with at least a common risk factor (most of the time adjusted for potential confounders). So your use of &quot;meta-analysis&quot; encompasses a much broader way of combining results from different independent studies (IMO)." CreationDate="2010-11-05T08:36:06.087" UserId="930" />
  <row Id="6253" PostId="4079" Text="@gaeton..thanks" CreationDate="2010-11-05T08:40:52.733" UserId="1763" />
  <row Id="6254" PostId="4212" Text="@whuber (so confused, should have search myself!) Thanks for that." CreationDate="2010-11-05T08:45:32.907" UserId="930" />
  <row Id="6255" PostId="4228" Text="@mariana Would you agree that a linear combination of some variables can still be interpreted as reflecting some kind of a weighted contribution of each of them to the factor axis?" CreationDate="2010-11-05T08:48:28.820" UserId="930" />
  <row Id="6256" PostId="4228" Text="Yes, that is exactly it." CreationDate="2010-11-05T09:39:35.777" UserId="1808" />
  <row Id="6257" PostId="4206" Text="Indeed, it's not variation, is it standard deviation? [apologies, I'm not accustomed to this terminology]. I mean, if p is gaussian, then the probability of having p within [p0-dp,p0+dp] is 66%. Does it make it right then?" CreationDate="2010-11-05T10:07:54.203" UserId="1197" />
  <row Id="6258" PostId="4228" Text="@mariana So, why preventing from giving it a name? Variables are just considered as manifest variables, and in some cases it makes sense to consider their weighted combination as reflecting a latent (unobserved) factor." CreationDate="2010-11-05T10:16:38.080" UserId="930" />
  <row Id="6259" PostId="4233" Score="1" Text="Does it predict new data with 85% accuracy, or part of the original data set that you held back for validation?" CreationDate="2010-11-05T11:08:07.723" UserId="656" />
  <row Id="6260" PostId="4233" Text="@richie..that rings an alarm..I didnt test it for any data. I ran a decision tree and it gave me say x false positives and y true positives than x/x+y is .156 which is misclassification..am i right then in saying that it has 84.6% accuracy ?" CreationDate="2010-11-05T11:24:03.837" UserId="1763" />
  <row Id="6261" PostId="4168" Text="Just to add to what owe said I think the real essence of random variable would only come with such courses. The concepts like expected value, correlation have deep implications in statistics. As some body also said it makes you more mature to deal with statistical processes." CreationDate="2010-11-05T11:28:14.690" UserId="1763" />
  <row Id="6262" PostId="4234" Text="@richiemorrisroe Yes, you're right in saying that no model of errors is incorporated in PCA, as opposed to FA. I've +1 for that particular point. Note that I said &quot;it makes sense to consider&quot;, not that principal components extracted from PCA are true LVs. Unless you're interested in assessing scale reliability or measurement models, it makes little difference whether you use PCA or FA, though. Now, data analysis is often concerned with explaining correlation between variables or finding groups of subjects, hence the idea of *interpreting* one or more dimensions of the factorial space. (...)" CreationDate="2010-11-05T12:06:16.677" UserId="930" />
  <row Id="6263" PostId="4234" Text="@richiemorrisroe (...) The [FactoMineR](http://cran.r-project.org/web/packages/FactoMineR/index.html) includes a data set about wines, and many factor methods can be used to play with it (PCA, MFA), and even PLS or CCA as has been done by Michel Tenenhaus." CreationDate="2010-11-05T12:07:57.503" UserId="930" />
  <row Id="6264" PostId="4233" Score="1" Text="@ayush Which $R^2$ measure are your referring to? If I understand you question correctly, this is logistic regression, and you may find useful information on this related question, http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox-s." CreationDate="2010-11-05T12:14:12.800" UserId="930" />
  <row Id="6265" PostId="4233" Score="1" Text="For future: when you don't know which tag to use, add for-retag." CreationDate="2010-11-05T12:16:36.703" UserId="88" />
  <row Id="6266" PostId="4176" Text="For any mean and variance a Gaussian has a non-zero probability of a negative value, and also a non-zero value of a non-integer value. Both these are not appropriate in this situation (what would it mean for the -2.7th trial to fail?)." CreationDate="2010-11-05T12:22:31.580" UserId="247" />
  <row Id="6267" PostId="4231" Score="1" Text="*&quot;we select variables which are most statistically significant...&quot;* No we don't! That's really not a good way to do model selection." CreationDate="2010-11-05T13:06:21.483" UserId="449" />
  <row Id="6268" PostId="4196" Text="@whuber: thanks for the socratic approach :) Right now, I am working on proving that a new process is equivalent to an old one (see another post I made) so (a) is what I'm interested in. Am I correct in thinking that had I been in case (b), I could consider the time series to be independent ? I'm curious to know what (c) is about ? Prediction based on my current measurements ?" CreationDate="2010-11-05T13:13:14.183" UserId="1784" />
  <row Id="6269" PostId="4172" Text="chl's additiion of the reliability tag helped me find a similar question here : http://stats.stackexchange.com/questions/859/anova-with-non-independent-observations" CreationDate="2010-11-05T13:26:28.667" UserId="1784" />
  <row Id="6270" PostId="4233" Score="4" Text="Isn't it $R^2$ rather than $r^2$ ?" CreationDate="2010-11-05T13:41:18.783" UserId="334" />
  <row Id="6271" PostId="4225" Text="Could you provide a link to the Welch paradox?" CreationDate="2010-11-05T14:05:35.487" />
  <row Id="6272" PostId="4217" Text="@Tal But, your null hypothesis is &quot;None of the risk factors impact disease&quot;. This is evident when you talk about wrong discoveries. Thus, you are using the same null hypothesis when you are performing these multiple comparisons and hence the need for control. If on the other hand your null was: &quot;Age has no impact on disease&quot; then you would not be interested in a study that did not have age as a covariate and hence your multiple comparisons would be restricted to the studies where age was used as a covariate. Therefore, it seems to me that my original point is still valid in your example." CreationDate="2010-11-05T14:37:37.020" />
  <row Id="6273" PostId="4234" Text="@chl,Thanks for the hint as to the package, i'll check that out. On PCA vs FA i agree up to a point. I prefer FA for most applications, as i fund the communalities (the common variance) estimates to be very useful in assessing the worth of a particular factor structure. That may just be a personal preference, however. " CreationDate="2010-11-05T14:54:51.577" UserId="656" />
  <row Id="6274" PostId="4243" Text="@richiemorrisroe (+1) That sounds good. Thanks for sharing your experience with IRT modeling and FA. Apart from the graphics functionalities, the conditional approach in eRm is more in line with the initial thinking of theta by Rasch (as a fixed parameter)." CreationDate="2010-11-05T15:03:27.423" UserId="930" />
  <row Id="6275" PostId="4246" Text="Wow, that would be really useless then :D Can't people see that the means are at the edges already?" CreationDate="2010-11-05T15:03:39.640" UserId="1320" />
  <row Id="6276" PostId="4246" Score="2" Text="@JCL Perhaps, they can if they see the table. But, what if there is no table? Then a table can be re-constructed to some extent using the text which refers to such things as 'grand means', 'marginal means' etc. I am guessing the phrase 'marginal mean' is a literary convention to signal what we are talking about. In a similar vein you could argue that why use the term 'sample average' when 'average' is sufficient. Qualifying words such as 'sample', 'marginal' add some context." CreationDate="2010-11-05T15:07:31.960" />
  <row Id="6277" PostId="4234" Text="@richiemorrisroe You're totally right (I already upvoted your earlier response because it was made very clear). It's just that (unrotated) PCA has its own history in data analysis (esp. the French school), together with CA, MFA, MCA. On the other hand, Paul Kline has two very nice books on the use of FA in Personality research. And the upcoming book of William Revelle should rock for R users :) Well, in any case, I think we agree that these are useful tools to analyse the structure of a correlation matrix." CreationDate="2010-11-05T15:12:53.823" UserId="930" />
  <row Id="6278" PostId="4246" Score="1" Text="@Skrikant (+1) I think the term now extends beyond contingency table since we also speak of a marginal (vs. conditional or subject-specific) approach, e.g. in mixed-effects model. So I think, the distinction has to be made between computing a marginal statistic or working with conditional values (this applies for an ANOVA table as for a contingency table)." CreationDate="2010-11-05T15:18:44.163" UserId="930" />
  <row Id="6279" PostId="4221" Score="1" Text="The NIST is usually authoritative, but here it is technically incorrect (and ungrammatical to boot): having a probability defined at &quot;an infinite number of points&quot; does not imply the &quot;probability at a single point is always zero.&quot;  Of course they're just dodging a distraction about infinite cardinalities, but the reasoning here is misleading.  It would be better for them just to omit the first sentence in the quotation." CreationDate="2010-11-05T15:20:00.553" UserId="919" />
  <row Id="6280" PostId="4247" Text="There is a lot of vagueness here.  Are you saying (a) there is a set of proteins present in both samples and (b) each protein yields a ratio equal to its abundance in A divided by its abundance in B?  If so, where do you obtain the value for &quot;mean_population&quot;?  Is it a measurement?  If so, how is it made?  In fact, what do you really mean by &quot;H0&quot;?  Are you looking for *outliers* in the log ratios or are you truly trying to compare the mean log ratio, averaged over all the proteins, to some known constant (the &quot;mean_population&quot;)?" CreationDate="2010-11-05T16:27:26.070" UserId="919" />
  <row Id="6281" PostId="4247" Text="There is some confusion here. Your hypothesis states 'means' whereas your text refers to 'ratios'. What do you mean by 'population of protein ratios'? Are these some sort of theoretical ratios that you expect to see? Can you clarify?" CreationDate="2010-11-05T16:29:16.363" />
  <row Id="6282" PostId="2723" Score="2" Text="I learnt this one the hard way. Twice." CreationDate="2010-11-05T16:38:57.007" UserId="449" />
  <row Id="6283" PostId="4249" Text="I agree. (2) is a reason for conditioning on an ancilliary statistic." CreationDate="2010-11-05T16:50:41.907" UserId="449" />
  <row Id="6284" PostId="4225" Text="What's paradoxical about the Welch example?  There's no contradiction in knowing \theta for sure.  The classical statistician's explanation in this case might go, &quot;I generated an interval using a method having *at least* a 99% chance of  covering the true parameter (*a priori*).  In this case, we *know* (*ex post facto*) that it indeed covers the parameter.  Be happy!&quot;" CreationDate="2010-11-05T16:56:05.740" UserId="919" />
  <row Id="6285" PostId="3997" Text="yeah, if you notice I already posted the solution below a few days ago." CreationDate="2010-11-05T17:16:50.320" UserId="1716" />
  <row Id="6286" PostId="4254" Text="You rule, thanks!" CreationDate="2010-11-05T18:00:05.410" UserId="1833" />
  <row Id="6287" PostId="4225" Text="@whuber The paradox is about the CI only answering questions about repetitions, and not answering any questions for an observed data set. In the given data set ($X_1=0 and X_2=1$), we know with 100% prob that the $\theta$ lies between them. &#xA;&#xA;I agree with your comment. I guess I am not doing a great job of explaining, please see this link for what I am trying to say: http://books.google.com/books?id=LhiroD1D72oC&amp;pg=PA37&amp;lpg=PA37&amp;dq=welch's+paradox,+pratt+1961&amp;source=bl&amp;ots=Un-jc8ucEC&amp;sig=EQxlfFL2uh_N61-cSXc61RKrRbU&amp;hl=en&amp;ei=WEbUTLb3McL7lwf3qNGiBQ&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=2&amp;sqi=2&amp;ve" CreationDate="2010-11-05T18:09:16.280" UserId="1307" />
  <row Id="6288" PostId="4254" Text="You're welcome." CreationDate="2010-11-05T18:22:37.413" UserId="69" />
  <row Id="6289" PostId="78" Text="&quot;Using the covariance matrix standardises the data&quot; - I think you mean correlation matrix here...cheers" CreationDate="2010-11-05T19:05:45.593" UserId="74" />
  <row Id="6290" PostId="4256" Text="Those matrices are not equal, here's the covariance matrix http://yaroslavvb.com/upload/multinomial-covariance-matrix.png" CreationDate="2010-11-05T19:31:36.890" UserId="511" />
  <row Id="6291" PostId="3912" Text="Could the downvoters leave a comment to help improve or correct anything wrong in others' response?" CreationDate="2010-11-05T19:37:42.133" UserId="930" />
  <row Id="6292" PostId="4256" Text="Yes, this is indeed the covariance matrix.  Dropping any ith column and row results in the same normalization term for the Gaussian was my point.  Perhaps I'm missing something obvious?" CreationDate="2010-11-05T19:41:48.793" UserId="1835" />
  <row Id="6293" PostId="4249" Text="@onestop and @whuber I also agree about conditioning on ancillary statistic. But please let me know the fallacy in my line of thinking: A classical statistician will think that a datum (from the sample space) shouldn't trouble her (even though it doesn't look that it's wrong) as the CI is for repetitions and not for that particular datum. &#xA;&#xA;As whuber points out, a practical statistician would choose better CIs, but the point is more philosophical in my case." CreationDate="2010-11-05T19:42:41.003" UserId="1307" />
  <row Id="6294" PostId="4251" Text="@VitalStatistix thanks a lot for the comment and for the paper." CreationDate="2010-11-05T19:45:14.997" UserId="1307" />
  <row Id="6295" PostId="4249" Score="2" Text="@suncoolsu It might not look like it, but I did give an answer in a philosophical spirit!  The point is that there is no problem or paradox with someone noting that the confidence actually is 100% *in this particular case* when they used a procedure designed to assure *at least* 95% confidence in *any possible situation.*  In fact, something like this happens all the time with discrete data: in selecting a 95% CI procedure it may happen that the actual confidence is, say, *always* 97% or greater.  A careful statistician would recognize and report that fact." CreationDate="2010-11-05T19:55:45.717" UserId="919" />
  <row Id="6296" PostId="4254" Text="@Mehper In the last example, which three observations would you remove?  Obviously the 2 and the 121, but what else?  To remain unbiased, it seems you must remove *half* of the 4 and *half* of the 105 for a trimmed mean of (4/2 + 6 + 7 + 11 + 21 + 81 + 90 + 105/2)/7 = 34.64" CreationDate="2010-11-05T19:59:27.507" UserId="919" />
  <row Id="6297" PostId="4256" Text="Ah...didn't notice the determinant sign. Hm...they do seem to be equal on some examples I tried, is there a simple proof of this? Eigenvalues are not equal however. The motivation for question was to find out whether central limit theorem gives you the same approximation error for finite $n$ regardless of which multinomial dist. component you drop" CreationDate="2010-11-05T20:06:16.977" UserId="511" />
  <row Id="6298" PostId="4249" Text="@whuber Thanks again. " CreationDate="2010-11-05T20:09:18.407" UserId="1307" />
  <row Id="6299" PostId="4250" Text="Funny, tex doesn't render on my browser... Chrome on ubuntu." CreationDate="2010-11-05T20:17:49.523" UserId="1320" />
  <row Id="6300" PostId="2413" Text="ya, i wish people would sign up there faster - it will be awesome!" CreationDate="2010-11-05T20:23:36.150" UserId="74" />
  <row Id="6301" PostId="162" Text="You're right. What I described is ortogonal regression, as the spring would end up being perpendicular to the stick at the resting point. For linear regression, you'd have to force the springs to point &quot;vertically&quot;, but I think it's still a useful way to visualize regression. " CreationDate="2010-11-05T20:29:24.473" UserId="74" />
  <row Id="6302" PostId="4256" Text="Probably the easiest way to convince yourself is that $p_i=1-\sum_{j\ne i}p_j$ and plug that in for $p_i$ in $S$." CreationDate="2010-11-05T20:33:44.140" UserId="1835" />
  <row Id="6303" PostId="4254" Text="@Mehper: just FYI, you can format maths by writing TeX expression in between $ signs. E.g. `$X_i$`" CreationDate="2010-11-05T20:33:47.623" UserId="582" />
  <row Id="6304" PostId="4256" Text="BTW, I like your application of this idea--hence my interest in responding." CreationDate="2010-11-05T20:34:47.243" UserId="1835" />
  <row Id="6305" PostId="4254" Text="@whuber: Thanks for your comment, I've added your comment to the answer; @nico: Thanks for letting me know about TeX formatting. I tried to update the answer using TeX format but I couldn't manage it well. Could you please give me a link which explains how to use TeX style in posts? I have no experience in TeX." CreationDate="2010-11-05T21:14:29.383" UserId="69" />
  <row Id="6306" PostId="4140" Text="@David B: It makes sense.  I wonder whether it would help to consider the magnitudes of the local minima along with their values (where the &quot;magnitude&quot; is the amount by which its value differs from neighboring local maxima).  Regardless, start by reading about resampling statistics on Wikipedia: http://en.wikipedia.org/wiki/Resampling_%28statistics%29 ." CreationDate="2010-11-05T21:23:38.243" UserId="919" />
  <row Id="6307" PostId="4254" Score="1" Text="@Mehper: Google &quot;TeX Manual&quot; and take your pick.  I like the &quot;gentle introduction&quot; because it contains useful, readable tables: http://www.tex.ac.uk/tex-archive/info/gentle/gentle.pdf" CreationDate="2010-11-05T21:25:37.273" UserId="919" />
  <row Id="6308" PostId="4254" Score="1" Text="@Mepher: sure, here you go! http://www.mathjax.org/help/user/ (note that if you right click on any math formula you'll have a context menu linking to that page). You can also use MathML instead of TeX (if you're very brave :P)." CreationDate="2010-11-05T21:32:29.333" UserId="582" />
  <row Id="6309" PostId="4254" Text="@whuber, @nico: Great links. Thanks a lot." CreationDate="2010-11-05T21:33:25.723" UserId="69" />
  <row Id="6311" PostId="4261" Text="I could suggest &quot;Randomization, Bootstrap and Monte Carlo Methods in Biology&quot; by Manly. http://www.amazon.com/Randomization-Bootstrap-Methods-Biology-Statistical/dp/1584885416/ref=sr_1_1?ie=UTF8&amp;qid=1288995733&amp;sr=8-1" CreationDate="2010-11-05T22:29:01.110" UserId="582" />
  <row Id="6312" PostId="4259" Score="3" Text="Hi Nick - welcome to CV.  Your question is very broad; you might have better luck getting good answers if you broke it up into smaller questions (and once you do, you may find that some of them have already been answered here).  At a minimum, though, you ought to mark your question as &quot;community wiki&quot;.  That basically means that instead of the usual competing-answers format here, all of the answers as a whole will be considered The Answer." CreationDate="2010-11-05T22:49:57.433" UserId="71" />
  <row Id="6313" PostId="4252" Score="1" Text="Should this be tagged trimmed instead of truncated?" CreationDate="2010-11-05T23:57:39.017" />
  <row Id="6314" PostId="4259" Score="1" Text="@Matt The CW check-box no longer appears for a question. A mod will need to mark a question as CW as needed." CreationDate="2010-11-06T00:06:22.780" />
  <row Id="6315" PostId="4099" Text="@ayush Please clarify the question. In particular, these were some comments from before: By @chl - &quot;you should consider writing clear questions (they are interesting on their own), with one definitive issue, and reserve comments for additional infos relevant to your original question, not follow-up&quot;. By @shane - &quot;Regarding this current question: it could also be improved because it's asked many different questions without a clear common thread. Are you interested in multicollinearity in general? Or are you interested in VIF? It would be better to break these out for clarity.&quot;&#xA;&#xA;" CreationDate="2010-11-06T00:11:08.707" />
  <row Id="6316" PostId="4265" Text="RRDTool can be found here: http://www.mrtg.org/rrdtool/index.en.html" CreationDate="2010-11-06T00:37:37.940" UserId="118" />
  <row Id="6317" PostId="4266" Text="Technically, the problem is that singular covariance matrix means that some subset of variables is perfectly correlated, so probability density should be exactly 0 in some areas, but that's not possible with a Gaussian. One solution is to instead look at conditional density, conditioned on the fact that random variable lies in a feasible region. This looks like what they are doing in the link. Never heard the term &quot;G- inverse&quot;, I'm guessing it's Penrose-Moore pseudo-inverse?" CreationDate="2010-11-06T01:42:19.313" UserId="511" />
  <row Id="6318" PostId="4250" Text="A good quote from a UMich document: &quot;The marginal probability (of A) is obtained by summing all the joint probabilities. Marginal probability can be used whether the events are dependent or independent. If the events are independent then the marginal probability is simplified to simply the probability. The following example will clarify this computation.&quot; " CreationDate="2010-11-06T01:43:38.307" UserId="1764" />
  <row Id="6319" PostId="4266" Text="While it's true that a conventional d-dimensional Gaussian has support on all of $\Re^d$, the singular Gaussian does not.  G-inverse is generalized inverse, and yes, I believe that the Penrose-Moore definition works here.  I think that there's a CLT for singular covariances, stating as expected, convergence in distribution to the singular CLT, although I can't find a ref right now." CreationDate="2010-11-06T02:30:37.003" UserId="1860" />
  <row Id="6320" PostId="1830" Text="@srikant..the word overfitting as a word suggests me that something is &quot;overly&quot; done meaning ..more than required..so whats overly done ? Is it no of variables ? I am sorry I have been hearing the term overfitting many times and would take this oppurtunity to cleare this with you." CreationDate="2010-11-06T05:35:42.380" UserId="1763" />
  <row Id="6321" PostId="4259" Text="@Nick..I am new as well. I think a general thing and the foremost thing that one needs to keep in thing is how do you want to describe your output variable..is it continious, is it binary ? Because at the end of day you want to observe/model an output variable. Next thing I would think is what are the ways possible to model the required variable..things that would come then is that if the variable is dichotomous the procedure is logit model..The next consideration would be then data, its nitty gritty and the various issues one encounters..Hope this makes sense." CreationDate="2010-11-06T06:16:17.643" UserId="1763" />
  <row Id="6322" PostId="4267" Score="2" Text="Sounds like a candidate for a community wiki." CreationDate="2010-11-06T06:51:28.503" UserId="159" />
  <row Id="6323" PostId="4250" Text="@JCL i admit i was experimenting with some more slightly advanced (but mathematically correct) TeX. it renders fine for me in Firefox on Windows XP, and in Internet Explorer on Windows XP after a refresh. It doesn't render in IE on Windows Mobile 6, but then no TeX on this site does. I have Ubuntu on a USB drive so i'll be back after a quick reboot..." CreationDate="2010-11-06T08:55:51.780" UserId="449" />
  <row Id="6324" PostId="4247" Text="Thanks for your answers. whuber: I do say (a) and (b). I changed the definition of what I want to test to make it clearer. I want to get the significant outliers of the population of ratios at hand." CreationDate="2010-11-06T09:06:23.920" UserId="1829" />
  <row Id="6325" PostId="4250" Text="@JCL Renders fine for me in Chrome (v 5.0.375.125) on Ubuntu 10.04 running from a USB drive on my laptop." CreationDate="2010-11-06T09:06:51.530" UserId="449" />
  <row Id="6326" PostId="4247" Text="Srikant Vadali: I tried to remove the disambiguation between text and H0. mean_protein: protein ratio;  mean_population: the set of all protein ratios." CreationDate="2010-11-06T09:11:07.300" UserId="1829" />
  <row Id="6327" PostId="4268" Score="1" Text="I would say &quot;the&quot; textbook myself, with Stewart's *Matrix Algorithms* ( [both](http://books.google.com/books?id=XHOQ_HU-85IC) [parts](http://books.google.com/books?id=FtC9rHW7ORMC)) a close second. I would give a list of the pioneering papers myself, but the OP really should explain if he wants the numerics viewpoint or the stats viewpoint (I can help with the former, but not so much the latter)." CreationDate="2010-11-06T09:23:01.200" UserId="830" />
  <row Id="6328" PostId="4231" Text="@onestop..can you please elaborate on this ? whats wrong with selecting the variables which are most statistically significant." CreationDate="2010-11-06T09:34:04.070" UserId="1763" />
  <row Id="6329" PostId="3495" Text="-1 because this algorithm has been called out as erroneous but no modification to the response has been made to reflect that." CreationDate="2010-11-06T15:01:26.813" UserId="919" />
  <row Id="6330" PostId="3434" Score="1" Text="I'm missing something: if you have to &quot;maintain the entire vector in memory,&quot; how does this qualify as an &quot;online&quot; algorithm??" CreationDate="2010-11-06T15:03:17.920" UserId="919" />
  <row Id="6331" PostId="3389" Text="That's an interesting idea.  You could supplement it, perhaps, with online detection of outliers and use those to modify the estimate as you go along." CreationDate="2010-11-06T15:05:07.527" UserId="919" />
  <row Id="6332" PostId="4270" Text="Welcome to CrossValidated, Mark!" CreationDate="2010-11-06T15:11:08.567" UserId="919" />
  <row Id="6333" PostId="4271" Score="1" Text="Chen seems to focus on larger datasets, which makes sense because the fourth and sixth and higher moments involved in these tests are going to take some time to settle down to asymptotic levels.  But distributional tests are typically used for datasets smaller than 250 values (the minimum studied in this paper).  In fact, most of them become so powerful with larger amounts of data that they are little more than afterthoughts in such applications.  Or is there more going on here than I am seeing?" CreationDate="2010-11-06T15:18:43.720" UserId="919" />
  <row Id="6334" PostId="4269" Text="This solution ignores the structure of the experiment: it treats all subjects as identical.  If there is any possibility they are not, then at a minimum you should test for differences among their means." CreationDate="2010-11-06T15:22:09.613" UserId="919" />
  <row Id="6335" PostId="4255" Text="rattle requires XML which is not supported for Windows (and unavailable in a Windows binary) :-(.  http://cran.r-project.org/web/packages/XML/index.html" CreationDate="2010-11-06T15:38:37.923" UserId="919" />
  <row Id="6336" PostId="4044" Text="Dunno why, but a Mantel test flashed in front of my eyes when I read your post." CreationDate="2010-11-06T15:43:45.790" UserId="144" />
  <row Id="6337" PostId="4269" Text="By all means suggest an alternative but 1) given the hypothesis I think that it is not unreasonable to expect all subjects to score at chance (the null) and 2) I reread the paper I used as the inspiration and they used a one-sample t-test in this way. " CreationDate="2010-11-06T16:00:37.443" UserId="1614" />
  <row Id="6338" PostId="4047" Text="can one calculate the Kullback-Leibler divergence of points without making an assumption of the underlying probability density the points came from ?" CreationDate="2010-11-06T16:22:58.887" UserId="961" />
  <row Id="6339" PostId="4255" Text="@whuber: too bad! it's quite a neat package" CreationDate="2010-11-06T17:08:45.367" UserId="582" />
  <row Id="6340" PostId="4231" Text="Take a look at http://en.wikipedia.org/wiki/Stepwise_regression#Criticism" CreationDate="2010-11-06T17:24:16.900" UserId="449" />
  <row Id="6341" PostId="4252" Text="I'd say either http://en.wikipedia.org/wiki/Truncated_mean will do." CreationDate="2010-11-06T19:11:35.840" UserId="1833" />
  <row Id="6342" PostId="4268" Score="1" Text="+1 for Golub and Van Loan. And, yes, the definitive article is appropriate." CreationDate="2010-11-06T19:42:24.210" UserId="795" />
  <row Id="6343" PostId="4215" Text="as it turns out, for most weighting schemes, only the first few $w_i$ are non-zero, which should also simplify the computation." CreationDate="2010-11-06T19:54:17.677" UserId="795" />
  <row Id="6344" PostId="4268" Text="@onestop: I edited my question to clarify that I am focusing on the statistics part.&#xA;&#xA;I agree with everyone that Golub and Van Loan is the standard reference for matrix decompositions. But it's omitting the topic of very large scale decomposition through random projections. A survey paper I would put in my list is &quot;Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions&quot; by Halko et al." CreationDate="2010-11-06T20:16:37.943" UserId="30" />
  <row Id="6345" PostId="4276" Score="1" Text="How frequently is the sensor capturing its readings?  These graphs look like aliasing patterns where the gait frequency and the sensor frequency are only slightly different." CreationDate="2010-11-06T21:21:18.940" UserId="919" />
  <row Id="6346" PostId="4276" Text="Data is being sampled at 10 acceleration readings/second." CreationDate="2010-11-06T21:30:00.930" UserId="1224" />
  <row Id="6347" PostId="78" Text="@neil-mcguigan: Opps. Thanks." CreationDate="2010-11-06T21:56:59.753" UserId="8" />
  <row Id="6349" PostId="4222" Text="Hey, thanks! I think that point about not needing the norming constant for metropolis-hastings is exactly the information I needed to make sense of all of this.  I think, because the GS in WinBUGS stands for gibbs sampling, I was under the impression that gibbs superseded M-H and that the software was using gibbs exclusively. " CreationDate="2010-11-07T01:54:35.807" UserId="1795" />
  <row Id="6350" PostId="4218" Text="Can i suggest that we not answer core data analysis questions from new users with &quot;out of scope&quot; and/or &quot;try a web search?&quot; This is not just a core data analysis question (e.g., there are at least a dozen R packages directed either substantially or primarily to this area) but it is also among the most interesting and useful kind--i.e., how to quantify some interesting activity and encode it in a metric/kpi. " CreationDate="2010-11-07T06:01:19.373" UserId="438" />
  <row Id="6351" PostId="4218" Text="Sorry guys to be off topic, it wont happen again, I still did not understood how this site worked" CreationDate="2010-11-07T07:21:04.233" UserId="1808" />
  <row Id="6352" PostId="4274" Text="What if I have lots of observations with relatively few variables, but a very low signal-to-noise ratio? So low, in fact, that overfitting is a very real problem. Would regularization be a sensible thing to try and look at to improve predictive accuracy?" CreationDate="2010-11-07T10:50:45.730" UserId="439" />
  <row Id="6353" PostId="1726" Score="2" Text="This quote seems to be known only in Germany and there is doubt that it is authentic, see the link below where the State Office of Statistics in Baden-Württemberg show results of their research about this quote (sorry its only available in German). The Times, e.g., said that they never heard about it.&#xA;http://www.statistik.baden-wuerttemberg.de/Veroeffentl/Monatshefte/essay.asp?xYear=2004&amp;xMonth=11&amp;eNr=11&#xA;" CreationDate="2010-11-07T10:58:27.260" UserId="1573" />
  <row Id="6354" PostId="4274" Text="@aix It depends what you actually call few variables, and what kind of variables you are dealing with. But I think a ridge approach is to be preferred in your case. You can also look at [Boosting Ridge Regression](http://www.stat.uni-muenchen.de/sfb386/papers/dsp/paper418.pdf) (Tutz &amp; Binder, 2005). Penalized ML estimation was also proposed as a built-in method to prevent from overfitting; see e.g., Penalized Maximum Likelihood Estimation to predict binary outcomes: Moons KG, Donders AR, Steyerberg EW, Harrell FE. *J. Clin. Epidemiol.* 2004, 57(12): 1262–70." CreationDate="2010-11-07T12:03:58.450" UserId="930" />
  <row Id="6355" PostId="4257" Text="@gaetan how do you calculate the Chi square p value for the whole model ? Forgive my ignorance." CreationDate="2010-11-07T15:25:12.310" UserId="1763" />
  <row Id="6356" PostId="4209" Score="1" Text="@gabgoh I would like to hear more about which aspect(s) of the answer are weak. As far as scaling goes, you're stuck: you have already used up that possibility in standardizing the elements of the sequence. If (hypothetically) some form of scaling would keep the third moments from going to zero, then you would contradict the CLT because the limiting distribution would not be Normal. There's a related issue with asymptotics of estimators. Often you *can* adjust an estimator to kill higher moments asymptotically (e.g., with bootstrapping): but this still cannot be done by scaling alone." CreationDate="2010-11-07T16:54:07.813" UserId="919" />
  <row Id="6358" PostId="4235" Text="+1. It sounds like your &quot;negative probabilities&quot; are just signed measures, right?" CreationDate="2010-11-07T16:56:34.283" UserId="919" />
  <row Id="6359" PostId="4223" Text="I note that the example given on that wiki page uses probability densities in lieu of actual probabilities for the calculation of posteriors, presumably because the per unit aspect is not necessary for comparative purposes if the units being compared are the same. Extending this, if one doesn't want to assume normality but instead one has empirical data from which density can be estimated, e.g. a kernel density estimate, would it be valid to use a reading at a given value on the x-axis from this kde as input to calculating posteriors in a naive bayes classifier, assuming equal per units?" CreationDate="2010-11-07T17:08:00.227" UserId="226" />
  <row Id="6360" PostId="4223" Text="@babelproofreader I believe the posteriors are Bayesian updates, via the training data, of priors.  It's unclear how a kde could be construed similarly, but I'm no expert in this area.  Your question is interesting enough that you might consider posting it separately." CreationDate="2010-11-07T17:23:25.873" UserId="919" />
  <row Id="6361" PostId="4289" Text="Although not exactly the answer to my question, I am selecting since it gives pointers to techniques I didn't know and seem relevant. I know Jolliffe's book well, and it has nothing to that effect." CreationDate="2010-11-07T17:25:21.570" UserId="30" />
  <row Id="6362" PostId="4223" Score="1" Text="@babelproofreader BTW, you ought to consider voting up the questions and answers that interest you (*especially* any valid answers provided to your own questions!).  This not only generates goodwill but also is a good way to influence what appears on this site.  If you check you will notice that most people with high reputations vote *far* more often than they ask or answer questions." CreationDate="2010-11-07T17:31:01.977" UserId="919" />
  <row Id="6363" PostId="4292" Text="@mbq, thanks I wasn't exactly sure how to title it!" CreationDate="2010-11-07T17:48:53.547" UserId="776" />
  <row Id="6364" PostId="4292" Score="1" Text="Because this question appears to have nothing to do with R I have removed the &quot;r&quot; tag." CreationDate="2010-11-07T17:50:57.070" UserId="919" />
  <row Id="6365" PostId="4292" Score="1" Text="Please add to your question the N, and also what you predicted before you obtained these data.  If you predicted that later would always be greater than now that's very different from predicting that there would be 'some effect' of time." CreationDate="2010-11-07T17:59:29.630" UserId="601" />
  <row Id="6366" PostId="4292" Text="@whuber, I chose the R tag because a solution using R would be incredibly useful. " CreationDate="2010-11-07T17:59:55.827" UserId="776" />
  <row Id="6367" PostId="4293" Text="@whuber, yes, I did expect this pattern as it represents a well researched natural decline in a product life cycle. In this scenario the largest category represents the company that was first to market. As we would expect a larger and larger proportion of future expenditure is going to competitors with a smaller share going to the first to market company.  " CreationDate="2010-11-07T18:04:04.450" UserId="776" />
  <row Id="6368" PostId="4292" Text="@John, yes, it was expected please see the comment to @whuber in his answer below. The sample size has also been added to the description." CreationDate="2010-11-07T18:06:04.430" UserId="776" />
  <row Id="6369" PostId="2421" Text="@chl You are a great resource for new softwares. I had no idea about Orange! " CreationDate="2010-11-07T18:28:07.727" UserId="1307" />
  <row Id="6370" PostId="4293" Text="@whuber, that understood how would I implement this test using R?" CreationDate="2010-11-07T18:38:31.737" UserId="776" />
  <row Id="6371" PostId="4257" Score="1" Text="@ayush. I describe the calculation in the third paragraph.  In terms of actually calculating the resulting Chi Square p value, you can use an Excel formula.  If I remember correctly it is either CHIDIST or CHITEST.    " CreationDate="2010-11-07T18:59:44.297" UserId="1329" />
  <row Id="6372" PostId="4293" Score="2" Text="@Brandon I am glad to hear that this is not a post-hoc test.  You don't need R to implement it: the second paragraph of my response contains all the calculations needed!  In effect, this well-researched pattern would have a 1/512 chance of occurring by chance in the case that all nine possibilities (increase/decrease) were independent and equally likely.  Because 1/512 is so small (less than the standard threshold of 1/20), you have a valid claim that it is evidence that the pattern is not due to chance." CreationDate="2010-11-07T19:08:59.890" UserId="919" />
  <row Id="6373" PostId="4292" Score="2" Text="Sample size informs us about the *power* of any test but it is not needed to conduct a valid test." CreationDate="2010-11-07T19:09:46.473" UserId="919" />
  <row Id="6374" PostId="4235" Text="Thanks.  Yes, exactly right, mine are.  Those of Khrennikov mentioned in the Haug are entirely different, though, they are limits of relative frequencies in some sort of p-adic topology.  Wild, crazy stuff." CreationDate="2010-11-07T19:17:21.423" UserId="1108" />
  <row Id="6375" PostId="4223" Text="@whuber Thanks. I shall post a new question." CreationDate="2010-11-07T19:17:45.527" UserId="226" />
  <row Id="6376" PostId="4293" Text="@whuber, I should clarify that saying this movement or change in share of wallet is expected from secondary/tertiary/etc market entrants. As time passes the second/third/etc gain larger share or wallet and also take up some of the first entrants share of wallet. By well-researched, I meant that this phenomenon is well researched not that this specific instance or usage case is well researched beyond what I'm attempting to show right now." CreationDate="2010-11-07T19:47:46.440" UserId="776" />
  <row Id="6377" PostId="4295" Text="Thanks John, I guess what I'm trying to get at here is if the market leader has reached their potential or saturation point. The other market entrants are continuing to increase their share of wallet whereas the first to market's share of wallet is apparently decreasing or staying the same. To be able to say this with any confidence has substantial marketing and strategic consequences." CreationDate="2010-11-07T19:52:55.013" UserId="776" />
  <row Id="6378" PostId="4147" Text="I'm confused; doesn't this only give the power of the t.test?" CreationDate="2010-11-07T21:35:59.817" UserId="196" />
  <row Id="6379" PostId="4147" Text="@drknexus Of course it gives the power of the t-test only. whuber pointed out the impact of grouping data into sets and gave the output for certain specific situations. I simply coded up his simulation in R which demonstrates the point he made vis-a-vis the t-test." CreationDate="2010-11-07T21:38:33.343" />
  <row Id="6380" PostId="4299" Score="2" Text="Right.  Rewriting the x3=1 case as E[y] = B0' + B1'x1 + B2'x2 shows that we're really fitting two models (with a common error variance): one for non-smokers with parameters B0, B1, B2 and another for smokers with parameters B0', B1', B2'.  &quot;Parallelism&quot; means B1 = B1' and B2 = B2' (which says, geometrically, that plots of the fits are parallel planes).  In terms of the original variables these hypotheses are B4 = 0 and B5 = 0, exactly as Srikant has proposed." CreationDate="2010-11-07T21:42:33.850" UserId="919" />
  <row Id="6382" PostId="4299" Text="@Srikant, @whuber - Yeah this is what I was leaning towards (doing the `B1=B1'` and `B2=B2'` but wasn't sure where to go, but now I've got a good idea. Thank you both." CreationDate="2010-11-07T21:48:11.317" UserId="1894" />
  <row Id="6383" PostId="4287" Score="1" Text="@doug I'm having trouble understanding the equation; I can't find any way to justify it. Even a basic units analysis identifies problems. Suppose *y* is measured in parsecs and *x* in drams, for example, so that *f* and its estimator (is that what the little dot over the *f* means?) are also in parsecs. Then the lhs and \sigma^2 are squared parsecs; the middle term on the rhs is the square of a difference between a parsec (*f(x)*) and parsecs per dram (due to the division by *x*); and the last term on the rhs is squared parsecs per dram. It's not valid to add any of these terms to one another!" CreationDate="2010-11-07T21:58:10.927" UserId="919" />
  <row Id="6385" PostId="4276" Score="4" Text="Ah!  So the downward spikes are where a foot hits the ground and the rest is the &quot;jiggling&quot; in between.  If that's the case, even a crummy low-pass filter should do the job.  (http://en.wikipedia.org/wiki/Low-pass_filter )  *E.g.*, fdesign.lowpass (http://www.mathworks.com/help/toolbox/filterdesign/ref/fdesign.lowpass.html )." CreationDate="2010-11-07T22:08:15.143" UserId="919" />
  <row Id="6386" PostId="4287" Text="the equation's fine (the little greek letters in the numerator are *not* 'x' but 'kappa'). Try this: begin w/ a formula for SSE that you are comfortable with and just a few steps will get you to the one above." CreationDate="2010-11-08T00:49:50.667" UserId="438" />
  <row Id="6387" PostId="4147" Text="Ah; okay.  It doesn't compare the powers via simulation (as the question asked) then.  But the code you made was easy enough to modify; thanks." CreationDate="2010-11-08T01:20:52.783" UserId="196" />
  <row Id="6388" PostId="4295" Score="1" Text="The value didn't go up, which is what you predicted.  The amount it went down is insignificant.  No statistical test is going to allow you to say with more confidence that the value didn't go up than the value actually going down.  Some descriptive statistics, like confidence intervals of the effects would be good.  It would be best to graph all of your values as the difference scores and a confidence interval around those difference scores." CreationDate="2010-11-08T04:00:24.077" UserId="601" />
  <row Id="6389" PostId="4295" Text="(in addition to your current graph)" CreationDate="2010-11-08T04:06:02.440" UserId="601" />
  <row Id="6390" PostId="4295" Text="Thanks John, I'll plot that as well." CreationDate="2010-11-08T04:55:52.093" UserId="776" />
  <row Id="6391" PostId="808" Text="Nice quote, though I can't help imagine how it would sound being spoken by Chris Eubank..." CreationDate="2010-11-08T08:57:11.437" UserId="449" />
  <row Id="6392" PostId="4302" Text="Credit: Stephen Senn, *Statistical Issues in Drug Development*, page4.&#xA;http://media.wiley.com/product_data/excerpt/71/04700187/0470018771.pdf" CreationDate="2010-11-08T09:03:41.133" UserId="449" />
  <row Id="6393" PostId="4303" Text="From what I can understand, it's not clear what is the response variable and the explanatory variable. Can you please clarify this?" CreationDate="2010-11-08T09:25:32.883" UserId="1307" />
  <row Id="6394" PostId="4305" Text="Do you have a three measurements on one individual or three measurements on three individuals?" CreationDate="2010-11-08T12:08:50.470" UserId="8" />
  <row Id="6395" PostId="4309" Text="The size of population is important, for sure. Thus, for the  blood pressure example three individuals may be unrepresentative. My case is different, there is no a problem of sample size. Imagine you have a sample of fixed size (it is not possible to include more individuals) and you want to assess the effects of a drug on blood pressure. Would you realize the experiment only once or several times? I think that it should be replicated at least two (or three) times in order to consider the individual variability of each subject." CreationDate="2010-11-08T12:56:57.193" UserId="221" />
  <row Id="6396" PostId="4305" Text="I have three measurements in 20 individuals belonging to a small population of about 100 individuals. But my question seeks a more general rule. Csgillespie and you pointed out that the sample size is important when deciding whether or not the experiment should be replicated and how many times. However, it is not always possible to have a sufficiently large sample size, especially if you work with animals." CreationDate="2010-11-08T13:06:46.073" UserId="221" />
  <row Id="6397" PostId="4256" Text="@Ian Interesting, I hadn't heard of the singular Normal.  Still though, in this example I don't think this distribution is necessary, because as you said (and I implied), one of the events is degenerate.  Hence, you can lop off any row/col." CreationDate="2010-11-08T13:17:14.657" UserId="1835" />
  <row Id="6398" PostId="4309" Score="3" Text="@Manuel You are correct; you need to assess individual variability.  The number of repeated measurements needed depends on the size of that variability, how the variability translates to uncertainty in the inferences about the population, the cost of repeating the measurements, practical constraints such as the time needed for replication, and (perhaps) technical issues like the possibility of positive temporal correlation among the replicate measurements." CreationDate="2010-11-08T13:30:47.913" UserId="919" />
  <row Id="6399" PostId="4295" Score="2" Text="@Brandon A subtler point revealed by this conversation is that the effects are not independent: they are linked by the size of the market.  Thus, *given* that the market size is practically unchanged, a decrease in the leader's share *necessarily* leads to increases in the shares of at least one competitor.  This calls for a different model and a different test, as suggested by @John's comments: it would suffice to demonstrate that the leader's decrease is unlikely to be due to random market fluctuations or sampling error.  A better model would also account for changes in market size." CreationDate="2010-11-08T13:38:36.640" UserId="919" />
  <row Id="6400" PostId="4309" Text="So, the idea could be to replicate the experiment two or three times, evaluate the variability of the estimates and based on this variability decide whether more replicates will be necesary. " CreationDate="2010-11-08T13:39:29.790" UserId="221" />
  <row Id="6401" PostId="1996" Text="Google finds 12.3 million hits for this quotation.  After running down the first eight pages of them, I haven't found a single site that actually gives a source--they all seem to be quoting each other rather than Mr. Twain.  Does anyone know where he wrote this?  Or maybe it's apocryphal." CreationDate="2010-11-08T14:05:51.817" UserId="919" />
  <row Id="6402" PostId="854" Text="Now that I think of it: agreed!" CreationDate="2010-11-08T14:38:14.420" UserId="506" />
  <row Id="6403" PostId="4138" Text="@GJ Kerns: l is fixed but arbitrary, question updated" CreationDate="2010-11-08T14:53:39.793" UserId="767" />
  <row Id="6404" PostId="4138" Text="@girard: I am most interested in an analytical approach. $\Sigma$ can be arbitrary. Dimension can be small." CreationDate="2010-11-08T14:57:21.840" UserId="767" />
  <row Id="6405" PostId="4138" Text="Without any loss of generality you can take l = 1." CreationDate="2010-11-08T14:57:35.680" UserId="919" />
  <row Id="6406" PostId="4138" Text="@whuber: true, question updated" CreationDate="2010-11-08T14:58:20.563" UserId="767" />
  <row Id="6407" PostId="4247" Text="@Florian I am still confused about your hypothesis. I think it will help a lot if you could identify the variables in your set-up and explicitly establish the hypothesis in those terms. For example, my current understanding is as follows: We have a set of ratios $r_i$ where $i$ indexes the proteins. You want to identify the set of proteins ${i,j}$ such that ${r_i \ne r_j}$. Is the above an accurate translation of what you want to do?" CreationDate="2010-11-08T16:06:04.467" />
  <row Id="6408" PostId="4319" Text="Okay. That was my first ever attempt at an R function and I failed. I have added some updated information." CreationDate="2010-11-08T16:17:08.767" UserId="1614" />
  <row Id="6409" PostId="4319" Text="On page 128 of the DAAG book, the authors note that the one sample permutation test (onet.permutation) is an equivalent test to the one-sample t-test. I originally planned to use a one-sample t-test on my data, so I don't see how it is totally irrelevant. I understand that the procedure is a little unusual." CreationDate="2010-11-08T16:27:16.400" UserId="1614" />
  <row Id="6410" PostId="4321" Text="Okay. I will have to go away and have a think about all that. Thanks though. If you want more information about WHY then you can read the question that is linked in the first paragraph." CreationDate="2010-11-08T16:28:31.810" UserId="1614" />
  <row Id="6411" PostId="4319" Text="You are right in that this is what the book states. I find this misleading: of course, for each generated permutation of the signs of the given empirical difference-variable, a single sample results, and the one-sample t*-statistic can be calculated. But, in order to get the whole permutation distribution of t*, you have to repeatedly permute the signs of the difference variable in the first place. This assumes there are two dependent sample to begin with." CreationDate="2010-11-08T16:51:39.083" UserId="1909" />
  <row Id="6412" PostId="4320" Text="@Tim It sounds like the `Malmig` package is not available on official CRAN package releases, so you're better looking for a source version (that has to be compiled) or another mirror." CreationDate="2010-11-08T16:53:04.370" UserId="930" />
  <row Id="6414" PostId="4312" Text="P.S. The technique being described there is for a two-sample test; the approach you take with a one sample test has to be a bit different.  When the data is continuous you need to use a sign test as indicated by caracal.  I'm not sure a sign test works as expected when many of your values are 0 which is why I suggested the permutation approach I provide below." CreationDate="2010-11-08T17:12:01.160" UserId="196" />
  <row Id="6415" PostId="4324" Text="That all sounds very reasonable. Again, I will go away and think about it. Many thanks for the consideration and advice." CreationDate="2010-11-08T17:13:13.270" UserId="1614" />
  <row Id="6416" PostId="4321" Text="This research was an attempt at replicating a published result, claiming that their subjects scored above mu. It didn't claim that only some of them would do well, it claimed that there will be above chance scoring. As such this experiment tests that claim. Though I'm sure there are all manner of claims that could have been tested. Thanks for the very informative response. There is just one thing holding me back. Replication 3 had two-groups (effort vs no-effort). Is there a 2-sample binomial test? or is that another question? Sigh." CreationDate="2010-11-08T17:31:42.747" UserId="1614" />
  <row Id="6417" PostId="4319" Text="Could it not be compared to mu? If you subtract mu from the scores you are left with some positive and negative scores, and most should be at zero? I believe this is essentially what I was going to do originally with the one-sample t-test" CreationDate="2010-11-08T17:34:30.370" UserId="1614" />
  <row Id="6418" PostId="4258" Text="although not included in the original question, it would also be useful to automatically set the thinning interval as proposed in my answer. " CreationDate="2010-11-08T18:37:48.977" UserId="1381" />
  <row Id="6419" PostId="4295" Text="@Whuber, actually that's not true. There is significant revenue growth in this category. I also have access to the market size changes over the last few years (all positive, and expected to continue as such)" CreationDate="2010-11-08T18:56:26.027" UserId="776" />
  <row Id="6420" PostId="4295" Score="1" Text="@Brandon Good enough.  But now we are aware that the validity of the analysis may depend on this fact of significant growth, indicating it's an important thing to know and to document." CreationDate="2010-11-08T19:03:31.197" UserId="919" />
  <row Id="6421" PostId="4329" Text="@onestop (+1) I prefer your reference (I have it at home :)." CreationDate="2010-11-08T19:26:09.170" UserId="930" />
  <row Id="6422" PostId="4311" Text="I am not sure I follow the suggestion properly. How does the idea of a particle filter help here? " CreationDate="2010-11-08T19:50:15.913" />
  <row Id="6423" PostId="4304" Score="1" Text="Thanks all, the book  Dynamic Linear Models in R, by Petris et al has high S/N ratio." CreationDate="2010-11-08T20:15:20.547" UserId="1896" />
  <row Id="6424" PostId="4331" Text="Can you provide us with some context for your data?  At this point, I'm not really sure that the uniform distribution is the right one - but it's hard to tell what a more appropriate distribution would be." CreationDate="2010-11-08T20:30:37.500" UserId="71" />
  <row Id="6426" PostId="4331" Score="1" Text="More information is needed.  To check for a *uniform* distribution one would ordinarily check *all values,* not just those near a particular value.  The interval you request doesn't exist because it cannot answer questions about uniformity.  It would be better to state in plain non-statistical terms what data you have obtained and what you're trying to accomplish." CreationDate="2010-11-08T20:34:26.777" UserId="919" />
  <row Id="6428" PostId="4331" Text="Yes, there are four possible outcomes with equal probability (p=0.25). The data contains of authentication cokes which are examined for their randomness and/or patterns in it. I'm interested to see how often each possibility occurs and how far away this number is distanced from the calculated range (since the assumption already is that it is an uniform distribution with p=0.25)." CreationDate="2010-11-08T21:00:50.740" />
  <row Id="6429" PostId="4331" Text="To clarify; I want to be able to answer the following 2 questions: &quot;The character `c` is too rare at position `i`. It occurs only `no` times.&quot; and the other one is &quot;The character `c` is too common at position `i`. It occurs more than `no` times.&quot;" CreationDate="2010-11-08T21:08:36.583" />
  <row Id="6430" PostId="4095" Text="Thank you for all the references, especially the Judea Pearl responses (I will need to buy the book based on all the wonderful info in those short response essays.) " CreationDate="2010-11-08T21:09:29.780" UserId="1036" />
  <row Id="6431" PostId="4331" Text="You speak of 4 possibilities. What are those four possibilities? What is a sample for you? (A cookie string or a character in a cookie string?) I would suggest following whuber's suggestion and re-state the question without reference to statistical issues. Some sample data would also help immensely." CreationDate="2010-11-08T21:34:14.050" />
  <row Id="6432" PostId="4321" Score="1" Text="I may be missing something, but IMHO, permbinom() does not perform a permutation of its input (a re-arrangement of the input values), hence permtest() does not perform a permutation test. I think permbinom's random swapping of the input 1s and 0s with probability .5 gives you exactly what just calling rbinom(n, size=1, prob=.5) itself gives you: the probability for a 1 is .5 for each element, same for a 0 - regardless of the input vector. Your solution seems equivalent to the original one - it's a MC simulation with given priors. Sorry if I'm totally off base here, I'm by no means an expert." CreationDate="2010-11-08T21:39:10.053" UserId="1909" />
  <row Id="6433" PostId="4331" Text="ABCDBCDADBCDA,&#xA;BDCAADBCDADBA,&#xA;ADCDBDACDBDAD,&#xA;CDBDACDBDACDA,&#xA;&#xA;That's some sample data. A sample is one cookie string!&#xA;Now I want for each position in that cookie string determine if a character there is too rare or too common at that position. So I count, for all samples, the number of A's on positon 0, the number of B's, C's and D's. Suppose I get a count of 5 A's on position 0 and I would expect a count of roughly 50 A's then the character A is too rare at position 0. &#xA;&#xA;That's what I want to do for each character position." CreationDate="2010-11-08T21:47:16.940" />
  <row Id="6434" PostId="4320" Text="@Tim. Malmig is on the CRAN archive: http://cran.r-project.org/src/contrib/Archive/Malmig/. You would need to compile it yourself in order to install it." CreationDate="2010-11-08T22:30:41.973" UserId="159" />
  <row Id="6435" PostId="4276" Text="Thanks for the suggestion. I'll look into the use of a low-pass filter. " CreationDate="2010-11-09T02:44:55.553" UserId="1224" />
  <row Id="6437" PostId="4321" Text="I respectfully disagree with your assessment caracal.  I'll admit a permutation test in a one sample binomial case does look odd. So does a one sample continuous permutation test.  Rather than permute the values between groups the best you can do is change their outcomes to the inverse and see whether you observed something notable (which, as you note in your answer, is how a sign test works)." CreationDate="2010-11-09T04:13:29.830" UserId="196" />
  <row Id="6438" PostId="4321" Text="I admit, I may be wrong, but I think that permtest accomplishes this goal and in doing so it approximates an exact test.  That is, it finds what percentage of ways of picking between these options provides more successes than the observed successes.  Like a sign test, the probability of evidence being in the other direction is .5." CreationDate="2010-11-09T04:20:12.237" UserId="196" />
  <row Id="6439" PostId="4321" Text="@R Soul:  A permutation test is actually easier to conceive of in the two group case than in the one group case.  Exchangablity in this case is between groups whereas in the one sample case it had to be directly between outcomes.  You can randomly assign the group labels (e.g. with 4 in each group it would be newgroup &lt;- sample(c(rep(1,4),rep(2,4)),replace=FALSE).  Then get the difference between the averages of the two groups over many runs. This builds the empirical distribution you can test against." CreationDate="2010-11-09T04:26:05.203" UserId="196" />
  <row Id="6440" PostId="4319" Text="Caracal:  I misread your comment earlier; I thought you understood the role of a sign test in a single sample test.  Now, I'm not sure you do - so let me expand: a permutation test of dependent samples is the same as the permutation test of a single sample (just as a paired samples t-test is equivalent to a single sample test of the difference scores).  Thus, you can use a sign test on single sample data even though it doesn't look like the sort of permutation you are used to thinking about." CreationDate="2010-11-09T04:30:36.700" UserId="196" />
  <row Id="6441" PostId="4321" Text="My above comment assumes that Experiment 3 was done between subjects and that the difference between effort and no effort is the issue at hand." CreationDate="2010-11-09T04:31:56.983" UserId="196" />
  <row Id="6442" PostId="4321" Text="@caracal: Though, I'll yield your point - doing the sign test the way I do above is equivalent to using a .5 prior - it is only in concept that it varies from an MC simulation." CreationDate="2010-11-09T04:35:38.637" UserId="196" />
  <row Id="6443" PostId="4222" Score="1" Text="The term Gibbs sampling is often used to imply that you sample one parameter at a time, even if you don't use the original idea of sampling directly from the full conditionals.  All the software samples individual parameters or blocks of parameters in sequence, but the actual step type varies a lot depending on what works best." CreationDate="2010-11-09T04:49:05.357" UserId="493" />
  <row Id="6444" PostId="4317" Score="2" Text="Two principles apply: You can never know whether your chain has converged to its stationary distribution.  And any test for convergence you can do manually, you can automate.  So your approach seems sound enough." CreationDate="2010-11-09T05:07:47.363" UserId="493" />
  <row Id="6445" PostId="3434" Text="@whuber No, not missing something, I guess it isn't an online algorithm.  It requires `O(n)` memory, and in the worst case takes O(n) time for each item added.  In normally distributed data (and probably other distributions) it works quite efficiently though." CreationDate="2010-11-09T05:29:36.750" UserId="179" />
  <row Id="6446" PostId="3389" Text="You could probably use Welford's method for calculating the standard deviation online that I documented in my second answer." CreationDate="2010-11-09T05:38:19.327" UserId="179" />
  <row Id="6447" PostId="3495" Text="@whuber Yeah, point taken, I've been meaning to revise the algorithm itself but have been short on time.  Updated the quest to flag its problems." CreationDate="2010-11-09T05:41:24.227" UserId="179" />
  <row Id="6448" PostId="4303" Text="@suncoolsu I edited the question on Mengzhen's behalf to attempt to make it clearer" CreationDate="2010-11-09T07:00:03.917" UserId="183" />
  <row Id="6449" PostId="4334" Text="Dear Fred, I can merge your other account into this one if you wish. " CreationDate="2010-11-09T08:45:29.967" UserId="88" />
  <row Id="6450" PostId="4311" Text="@Srikant-vadali: Has my updated answer helped?" CreationDate="2010-11-09T09:34:44.230" UserId="8" />
  <row Id="6451" PostId="4334" Text="That would be great!" CreationDate="2010-11-09T09:40:37.843" UserId="1709" />
  <row Id="6452" PostId="4334" Text="Should be ok now." CreationDate="2010-11-09T10:30:17.860" UserId="88" />
  <row Id="6453" PostId="4334" Score="1" Text="Thanks! I feel... at home" CreationDate="2010-11-09T10:31:14.863" UserId="1709" />
  <row Id="6454" PostId="4337" Text="How many cinemas are there, roughly?" CreationDate="2010-11-09T11:03:10.540" UserId="449" />
  <row Id="6455" PostId="4340" Text="I'm not clear about using the daily takings, can you elaborate on that please?" CreationDate="2010-11-09T11:05:54.957" UserId="1922" />
  <row Id="6457" PostId="4337" Text="Oops sorry, five, one doesn't have gaming section" CreationDate="2010-11-09T11:08:14.067" UserId="1922" />
  <row Id="6458" PostId="4340" Text="@endy_c Does that help?" CreationDate="2010-11-09T11:52:14.637" UserId="8" />
  <row Id="6459" PostId="4335" Text="@Tal It seems to me that now mods have to make it CW themselves." CreationDate="2010-11-09T11:55:09.797" UserId="930" />
  <row Id="6460" PostId="4287" Text="What is 'kappa' in this context?" CreationDate="2010-11-09T12:12:08.173" />
  <row Id="6461" PostId="4311" Text="I am afraid my knowledge of particle filters is close to zero! How does this method of picking a particle with probability 1/M be then used with the new likelihood to walk over the posterior for the parameters?" CreationDate="2010-11-09T12:14:45.460" />
  <row Id="6462" PostId="4340" Text="OK so instead of trying to eliminate bias from the movie, I can include it in the experiment, got it, thanks!" CreationDate="2010-11-09T12:21:23.497" UserId="1922" />
  <row Id="6463" PostId="4345" Score="1" Text="@ucfagls (+1) Thanks for the hint about `nclass.*`. Why isn't your left-most interval left bounded (should set `include.lowest` to TRUE maybe)?" CreationDate="2010-11-09T13:06:09.803" UserId="930" />
  <row Id="6464" PostId="4346" Score="2" Text="@Jani The title seems clearly duplicated from here, http://stats.stackexchange.com/questions/4208/metric-based-random-forest." CreationDate="2010-11-09T13:12:09.997" UserId="930" />
  <row Id="6465" PostId="4346" Score="2" Text="@Jani I would also suggest to somehow rework your question so that it clearly reflects how it does relate to statistical computing and not only Java coding." CreationDate="2010-11-09T13:17:29.710" UserId="930" />
  <row Id="6466" PostId="4345" Text="@chl - that's what you get when doing something quickly. Will edit the code to add `include.lowest`" CreationDate="2010-11-09T13:56:29.973" UserId="1390" />
  <row Id="6468" PostId="4257" Text="@gaeton..Thanks." CreationDate="2010-11-09T14:42:56.483" UserId="1763" />
  <row Id="6469" PostId="4345" Score="1" Text="@ucfagls A good solution is to create an interactive application that dynamically displays a graphic of the binned counts (*e.g.*, a histogram) or appropriate statistics (*e.g.*, chi-squared) in response to the user's manipulation of the cutpoints, so that this issue can be quickly explored.  This is dead easy in Excel, the OP's platform of choice, and also fairly easy in some high-end software like *Mathematica*." CreationDate="2010-11-09T15:11:00.380" UserId="919" />
  <row Id="6470" PostId="4343" Text="Good start!  This solution *assigns* ranges; now you have to group them by following up your calculation with a summarize operation." CreationDate="2010-11-09T15:13:00.907" UserId="919" />
  <row Id="6471" PostId="4318" Text="+1 A breathtaking harbinger of the follies of the coming century; &quot;GIGO before its time.&quot;" CreationDate="2010-11-09T15:17:32.290" UserId="919" />
  <row Id="6472" PostId="4335" Score="1" Text="I don't think that this question really needs to be a CW. It is borderline, but it's not too bad." CreationDate="2010-11-09T15:22:59.883" UserId="8" />
  <row Id="6473" PostId="4311" Text="Suppose my prior on $\mu$ was N(0,1). A possible proposal would be to just select values from N(0,1) and accept/reject as necessary. In a particle representation, we could generate $M$ RNs from the N(0, 1) and select a particle at random. As M increases, the particle representation becomes as exact." CreationDate="2010-11-09T15:28:41.227" UserId="8" />
  <row Id="6474" PostId="2329" Text="Can you make vary A?" CreationDate="2010-11-09T15:32:27.320" UserId="1709" />
  <row Id="6475" PostId="3224" Text="didn't know as well... hehe thanks" CreationDate="2010-11-09T15:35:15.803" UserId="1709" />
  <row Id="6476" PostId="4311" Text="To be concrete, let: $f(\theta_c|-)$ be the current posterior for which I have MCMC samples. $L(y|\theta)$ be the likelihood corresponding to the new data point. Thus, the new posterior I wish to sample from is: $f(\theta_n |-) \propto f(\theta_c|-) L(y|\theta)$. Does your suggestion amount to implicitly setting the proposal density for the MH algorithm to be $f(\theta_c|-)$ and then use the particle filter approach as an approximation to $f(\theta_c|-)$?" CreationDate="2010-11-09T15:40:23.587" />
  <row Id="6477" PostId="4311" Text="Yes............" CreationDate="2010-11-09T15:45:30.077" UserId="8" />
  <row Id="6480" PostId="4345" Text="@whuber I voted up the response although it is certainly not the response expected by the OP, but I like the point about binning and density. Interactive visualization can be made really easy in R too, at least for this kind of viz. app. The Tk-based `rpanel` package provides basic functionalities (see e.g., `rp.slider()` for an illustration with a density plot); `playwith` is a Gtk-based interactive plotting device, and is very handy. Now, I have no doubt that this should be far more easier with Excel (which is the reason why I've also +1 your answer)." CreationDate="2010-11-09T16:33:01.603" UserId="930" />
  <row Id="6481" PostId="4335" Text="great information! (the time series package 'xts' has similar this metadata functionality.)" CreationDate="2010-11-09T17:38:11.687" UserId="438" />
  <row Id="6482" PostId="4343" Score="1" Text="@whuber Grouping is something I'm assuming lonut can do if he regularly works with Excel and SQL, and there are many ways to do it. In Excel, for example, he can use a PivotTable or the COUNTIF() function; in SQL he'll probably use COUNT(*) with a GROUP BY CATEGORY clause (where CATEGORY is the computed column)." CreationDate="2010-11-09T17:44:20.330" UserId="666" />
  <row Id="6483" PostId="4343" Text="@Carlos Thank you for making your answer complete!" CreationDate="2010-11-09T17:45:49.377" UserId="919" />
  <row Id="6484" PostId="4345" Text="@chl Thanks for the pointers to R interactivity tools." CreationDate="2010-11-09T17:53:46.993" UserId="919" />
  <row Id="6485" PostId="4319" Text="I think we just disagree about terminology. My narrow idea of a permutation test requires repeated re-assignment of observed units to an experimental condition consistent with the test's null hypothesis (H0) - as justified by assuming exchangeability under H0. Without different experimental conditions (pure 1 sample case), there is nothing to re-assign. You may still find the exact distribution of a certain test statistic in the 1 sample case by tabulating all possible outcomes, I just wouldn't call that &quot;permutations&quot; but &quot;subsets&quot;. This is certainly unimportant. Sorry for the confusion." CreationDate="2010-11-09T18:02:29.367" UserId="1909" />
  <row Id="6486" PostId="4354" Score="1" Text="I am not sure if I understand the question correctly. Are you asking if there are any distributions apart from the normal that are completely specified by the mean and the variance? In some sense, variance is a function of the mean as it is a measure of the dispersion around the mean but I guess this is not what you have in mind." CreationDate="2010-11-09T18:38:02.183" />
  <row Id="6487" PostId="4354" Text="you mean the sample mean $\bar{X}=\frac{1}{n}\sum_{i=1}^nX_i$ and sample variance $\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2$ are independant. Good question ! maybe projecting a gaussian random variable will keep independance ?   " CreationDate="2010-11-09T18:58:25.730" UserId="223" />
  <row Id="6488" PostId="4354" Score="4" Text="Srikant is right.  If the question is asking about &quot;sample mean and variance&quot; then the answer is &quot;no&quot;.  If the question is about population mean and variance, then the answer is yes;  David gives good examples below." CreationDate="2010-11-09T19:15:28.020" UserId="1108" />
  <row Id="6489" PostId="4356" Text="What do you precisely mean by multivariate split? Multivariate test which attribute should be used for split or split based on some function of few attributes?" CreationDate="2010-11-09T19:22:10.623" UserId="88" />
  <row Id="6490" PostId="4359" Score="1" Text="I'd upvote this if you'd given a full citation instead of just &quot;Feller (1966)&quot;. Why should you make many readers work out where this was published when you (presumably) already know?" CreationDate="2010-11-09T19:24:55.417" UserId="449" />
  <row Id="6491" PostId="4355" Score="1" Text="Could you provide a proper citation instead of just &quot;Carlin and Lewis (1996)&quot; ?" CreationDate="2010-11-09T19:26:01.360" UserId="449" />
  <row Id="6492" PostId="4359" Score="1" Text=" I did not know. I like that !  " CreationDate="2010-11-09T19:38:06.977" UserId="223" />
  <row Id="6493" PostId="4359" Score="1" Text="@onestop  I guess it is an unfortunate artifact of my age.  It is not an understatement to say that Feller's books revolutionized how probability was done -  worldwide. A large part of our modern notation is due to him. For decades, his books were **the** probability books to study. Maybe they still should be.  BTW: I've added the title for those that haven't heard of his books." CreationDate="2010-11-09T19:39:28.427" UserId="1108" />
  <row Id="6494" PostId="4355" Text="Done, sorry for the omission." CreationDate="2010-11-09T19:45:40.993" UserId="1381" />
  <row Id="6495" PostId="4359" Text="@robin yes, me too!  It's such a surprise, to me, at least. " CreationDate="2010-11-09T19:46:00.230" UserId="1108" />
  <row Id="6496" PostId="4359" Score="1" Text="I have aske the question about other funy characterisation ... http://stats.stackexchange.com/questions/4364/what-is-the-most-surprising-characterization-of-gaussian-distribution" CreationDate="2010-11-09T20:20:10.233" UserId="223" />
  <row Id="6497" PostId="4358" Text="Thank you very much" CreationDate="2010-11-09T20:23:02.073" />
  <row Id="6498" PostId="4362" Text="well explained. Thanks." CreationDate="2010-11-09T20:23:08.267" UserId="578" />
  <row Id="6499" PostId="4355" Score="4" Text="In *any* location-scale family the mean and variance will be functionally independent in this fashion!" CreationDate="2010-11-09T20:29:57.317" UserId="919" />
  <row Id="6502" PostId="4354" Score="1" Text="Just to clarify, what I meant is this. For the normal distribution, the mean $\mu$ and the variance $\sigma^2$ fully characterizes the distribution and $\sigma^2$ is not a function of $\mu$. For many other distributions, this is not so. For example, for the binomial distribution, we have the mean $\pi$ and the variance $n\pi(1-\pi)$, so the variance is a function of the mean. Other examples are the gamma distribution with parameters $\theta$ (scale) and $\kappa$ (shape), where the mean is $\mu = \kappa \theta$ and the variance is $\kappa theta^2$, so the variance is actually $\mu \theta$." CreationDate="2010-11-09T21:19:30.640" UserId="1934" />
  <row Id="6503" PostId="4309" Score="1" Text="@Manuel: This is a very dangerous strategy and you need to be careful. You can't just carry out a few experiments and stop when you like. Basically you can't change the sample size midway through your experiment unless you are very careful. " CreationDate="2010-11-09T21:25:21.437" UserId="8" />
  <row Id="6504" PostId="4355" Text="David, the double exponential is an excellent example. Thanks! I did not think of that one. The t-distribution is also a good example, but isn't E(X) = 0 and Var(X) = v/(v-2)? Or does Carlin et al. (1996) define a generalized version of the t-distribution that is shifted in its mean and scaled by sigma^2?" CreationDate="2010-11-09T21:30:17.757" UserId="1934" />
  <row Id="6505" PostId="4367" Score="1" Text="@user1566: I formatted your equations using LaTex. Would you double check that I didn't introduce any errors?" CreationDate="2010-11-09T21:31:21.050" UserId="8" />
  <row Id="6506" PostId="4359" Score="1" Text="Jay, thanks for the reference to the paper by Lukacs, who nicely shows that the sampling distributions of the sample mean and variance are only independent for the normal distribution. As for second central moment, there are some distributions where it is not a function of the first moment (David gave some nice examples)." CreationDate="2010-11-09T21:42:33.717" UserId="1934" />
  <row Id="6507" PostId="4355" Text="You are correct, the t-distribution appears to be frequently characterized with a mean = 0 and variance = 1, but the general pdf for t provided by Carlin and Louis explicitly includes both sigma and mu; the nu parameter accounts for the difference between the normal and the t." CreationDate="2010-11-09T21:48:37.643" UserId="1381" />
  <row Id="6508" PostId="4370" Text="ok... but I still don't understand why regressing sigma^2.... could you please elaborate on your last sentence &quot;An estimate of σ2 is given by the squared residuals as the mean of the residuals is zero&quot;." CreationDate="2010-11-09T21:52:12.803" UserId="333" />
  <row Id="6509" PostId="4355" Text="Thanks for the clarification." CreationDate="2010-11-09T21:52:12.663" UserId="1934" />
  <row Id="6510" PostId="4367" Text="Thanks, the problem is solved, so no longer a biggie, but, yes, everything looks OK." CreationDate="2010-11-09T22:11:59.353" UserId="1566" />
  <row Id="6511" PostId="4365" Score="4" Text="That they are sum stable and that they are the unique ones with finite variance are both forced on us by the CLT.  The interesting part of this assertion is that there exist *other* sum-stable distributions!" CreationDate="2010-11-09T23:12:03.263" UserId="919" />
  <row Id="6512" PostId="4370" Score="1" Text="Regression is used as it tests for one form of heteroscedasticity. In other words, it tests if error variance is a function of the regressors which if true would make the error heteroscedastic. Consider the OLS residuals. An estimate of error variance is: $\sum_i(e_i - \bar{e_i})^2 / n$. But, in OLS, $\bar{e_i} = 0$. Thus, an estimate of error variance from each observation is the square of the error residual." CreationDate="2010-11-10T00:33:46.250" />
  <row Id="6518" PostId="4377" Score="1" Text="Is d a parameter of the logistic regression or a predictor used in logistic regression? It sounds like a predictor but you call it a parameter. If it is a predictor, then its domain is irrelevant." CreationDate="2010-11-10T02:27:11.183" UserId="159" />
  <row Id="6520" PostId="4377" Text="d is the predictor.   Thanks for correcting me, I've updated the question." CreationDate="2010-11-10T02:29:26.743" UserId="1938" />
  <row Id="6522" PostId="4372" Text="Thank you for your answer. What I like about this model is that the state is directly understandable (mean and return to the mean coefficient of the mean reverting process)." CreationDate="2010-11-10T02:47:45.753" UserId="1709" />
  <row Id="6523" PostId="3542" Score="1" Text="Currently, most answers are books, so the bounty will go to someone who can provide the best Free resource! (@Jeromy, you could add this to the question if you want)" CreationDate="2010-11-10T11:12:31.647" UserId="190" />
  <row Id="6524" PostId="3542" Text="@Peter good idea. I've modified the question." CreationDate="2010-11-10T12:04:13.667" UserId="183" />
  <row Id="6525" PostId="4383" Text="@Chris Is this a validated scale, with known psychometrical properties?" CreationDate="2010-11-10T12:41:33.210" UserId="930" />
  <row Id="6527" PostId="4383" Text="Hi Chris, so you are not reducing the number of items in a likert scale, but rather using less questions/items (who are measured on a likert scale).  In general it sounds like it depends on your measures.  You could check the correlation of the items you intend to take down with the ones you are keeping.  It's actually an interesting how to measure how much to remove - it might be worth to reframe the question that way (if you won't, I might do it later).  Good question :)" CreationDate="2010-11-10T13:13:19.140" UserId="253" />
  <row Id="6528" PostId="4378" Text="Great answer, thanks!" CreationDate="2010-11-10T13:20:11.960" UserId="1938" />
  <row Id="6529" PostId="4378" Text="BTW, I just took the log of d.  d can include zeros, but the logistic function maps that to zero, which is what I wanted." CreationDate="2010-11-10T13:30:31.480" UserId="1938" />
  <row Id="6530" PostId="4383" Text="@Chris Three additional questions: (1) Is this an unidimensional scale or are there several subscales, (2) What is the No. individuals and the number of items, and (3) Do you work at the level of the items, or a total or mean score?" CreationDate="2010-11-10T13:37:51.280" UserId="930" />
  <row Id="6531" PostId="4384" Text="This kinda worked but Excel did sort the ranges as text generating a total mess. I wasn't able to convince Excel (2010) to show the ranges in the logical order." CreationDate="2010-11-10T14:29:25.027" UserId="1901" />
  <row Id="6532" PostId="4388" Text="I suppose the treatment cinema and the control cinema should be as similar as possible for this method? They are spread in 3 different cities and may play different movies." CreationDate="2010-11-10T15:23:18.887" UserId="1922" />
  <row Id="6533" PostId="4388" Text="@endy I am not sure those differences matter. The suggestion is to use *change in gaming sales* in treatment relative to control and not baseline sales as an indication of treatment effectiveness. Thus, while different cinemas, different movies etc may have different baseline gaming sales, the change in gaming sales would be a function of the presence or absence of the treatment." CreationDate="2010-11-10T15:29:12.897" />
  <row Id="6534" PostId="3542" Text="@Peter It will be terribly hard to get better references than those provided by @Rob and @ars. Good luck to all of you!" CreationDate="2010-11-10T15:36:27.323" UserId="930" />
  <row Id="6535" PostId="4388" Score="1" Text="(+1) I was going to suggest something similar. This is sometimes known as a 'difference in differences' estimator &#xA;http://en.wikipedia.org/wiki/Difference_in_differences" CreationDate="2010-11-10T16:56:36.447" UserId="449" />
  <row Id="6536" PostId="4365" Text="@whuber: indeed! this characterization is a bit contorted, and the other sum-stable distributions are perhaps more curious." CreationDate="2010-11-10T16:58:10.113" UserId="795" />
  <row Id="6537" PostId="4375" Text="Why the ratio? Why not just state that the probability of that exact draw is extremely low?" CreationDate="2010-11-10T17:06:13.190" UserId="1036" />
  <row Id="6543" PostId="4395" Text="I fixed the original question so that it make sense." CreationDate="2010-11-10T18:10:00.243" />
  <row Id="6544" PostId="4394" Text="biostat = application of statistical methodologies to biology ? " CreationDate="2010-11-10T18:16:20.980" UserId="223" />
  <row Id="6545" PostId="4394" Text="Right, but there are applications of statistical methodologies in every discipline.  Why does biostatistics exist (in the US, at least) as a semi-distinct discipline?" CreationDate="2010-11-10T18:24:06.167" UserId="71" />
  <row Id="6546" PostId="4396" Text="I'm pretty confused by what you are asking to do. Can you describe the format of your data and why using if statements won't accomplish what you are trying to do?" CreationDate="2010-11-10T18:40:39.657" UserId="1036" />
  <row Id="6547" PostId="4396" Text="Hi Andy. I've updated my question to include an example. Hope it's clearer now." CreationDate="2010-11-10T18:55:04.890" UserId="1950" />
  <row Id="6548" PostId="4363" Text="Actually I'm referring to splits using more than one variable.  If you read the Leo Breiman's CART book it refers to this as multi-variate splits as opposed to uni-variate where only one variable is consider.  Thanks for you answer." CreationDate="2010-11-10T19:10:45.297" UserId="1929" />
  <row Id="6549" PostId="4356" Text="This is a split that uses more than one variable.  Geometrically this means a splits along a hyperplane, rather than a plane perpendicular to one of the axis." CreationDate="2010-11-10T19:13:12.003" UserId="1929" />
  <row Id="6550" PostId="4403" Text="How do you decide whether the person's answer is 'right', once you have it?" CreationDate="2010-11-10T20:01:23.550" UserId="449" />
  <row Id="6551" PostId="4363" Score="1" Text="All variables are considered in `rpart`, the binary split is formed by searching over all variables and all possible split locations within each variable. `rpart` also stores information about surrogate splits which can be used when there is missing data in your variables. Does this help?" CreationDate="2010-11-10T20:04:31.173" UserId="1390" />
  <row Id="6552" PostId="4406" Text="I really don't think that biostatistics and biometrics are synonyms. Biometrics includes face recognition, finger print analysis, while biostatistics involved clinical trial design and so on... Similar names only." CreationDate="2010-11-10T20:14:08.933" UserId="1540" />
  <row Id="6553" PostId="4409" Text="Methinks you landed at the wrong site. This is about statistics." CreationDate="2010-11-10T20:20:25.663" UserId="334" />
  <row Id="6554" PostId="4363" Text="If you mean splits that are some combination of two or more variables, no, `rpart` doesn't handle that case." CreationDate="2010-11-10T20:25:28.713" UserId="1390" />
  <row Id="6555" PostId="3336" Text="+1 Agree with this interpretation about it looking like DIY Arellano-Bond. NB: I have found Arellano-Bond to be trustworthy only when the number of cross-sectional units is very large---as in, many hundreds.  Arellano hints as much in his articles and textbook by indicating that the consistency is in the number of cross section units, and the convergence rate is not all that fast." CreationDate="2010-11-10T20:30:43.890" UserId="96" />
  <row Id="6556" PostId="4406" Text="That usage of 'biometrics' is an unfortunate neologism. See http://www.tibs.org/interior.aspx?id=290" CreationDate="2010-11-10T20:32:25.197" UserId="449" />
  <row Id="6557" PostId="4407" Text="It would be easier if you where to provide the object, using ?dput .  My guess is that you changed the names, but that the level of the branches has changed, which you didn't recalculate now.  Offer some example data, and we'll find a solution :)" CreationDate="2010-11-10T20:39:47.217" UserId="253" />
  <row Id="6558" PostId="4406" Text="This isn't really addressing the question, however.  I know what the definition of biostatistics is, but I don't know how it differs from statistics in practice, in education, in philosophy, etc." CreationDate="2010-11-10T20:52:59.150" UserId="71" />
  <row Id="6559" PostId="4399" Text="hmmm.. an auto-regressive model isn't really a good fit for a binary time series is it? Auto-regressive on time between events? I hate the idea of variance of time between events." CreationDate="2010-11-10T21:09:31.483" UserId="1951" />
  <row Id="6560" PostId="4382" Score="4" Text="This is a property of the Normal distribution, to be sure, but it does not *characterize* it, because plenty of other distributions also have this property." CreationDate="2010-11-10T21:18:39.097" UserId="919" />
  <row Id="6561" PostId="4409" Text="One could argue (with a tiny bit of justification) that this question qualifies as statistical, but the answer is readily obtained by Googling &quot;blended interest rate&quot;." CreationDate="2010-11-10T21:20:36.697" UserId="919" />
  <row Id="6562" PostId="4411" Text="@aix (+1) A little bit hard to read, though (especially for someone seeking R applications)." CreationDate="2010-11-10T21:24:29.003" UserId="930" />
  <row Id="6563" PostId="4404" Text="Yes, this would definitely work but you have to admit that the syntax is somewhat unwieldy. dat[ , &quot;WEIGHT&quot;] is a lot less elegant compared to dat$WEIGHT.&#xA;I just wondered why there is no option to turn on the behavior above? This sounds like a very reasonable feature." CreationDate="2010-11-10T22:14:51.003" UserId="1330" />
  <row Id="6564" PostId="4404" Text="Also, on the heels of my last comment. I wonder if using attaching the data frame and then referencing the variables directly would mitigate this problem..." CreationDate="2010-11-10T22:17:00.863" UserId="1330" />
  <row Id="6565" PostId="4402" Text="Unfortunately, the snippet above would not help. Making sure that all columns (variables) are present in the data frame is one thing and it would never help against accidental misspelling of a variable name in some other corner of the code." CreationDate="2010-11-10T22:20:04.093" UserId="1330" />
  <row Id="6567" PostId="4404" Score="1" Text="I would definitely avoid using attach in all circumstances, but I think that's the right idea.  You can use `with()` to attach a data frame for a single command: `with(dat, weighted.mean(x, weight, na.rm = TRUE))`.  Also, many functions like weighted.mean include a `data=` option that essentially internalizes this." CreationDate="2010-11-10T22:39:35.110" UserId="71" />
  <row Id="6570" PostId="4375" Score="5" Text="The assertion that a particular probability is low out of context isn't convincing. The probability that I am exactly as tall as my height (whatever that may be) is zero. And, yes, it's problematic to even define height with infinite precision, yada, yada, yada...  My point is that the maelstrom of existence churns with events of infinitesimal probability happening all the time!  10,000 out of 20,000 - out of context - doesn't surprise me at all. Regardless of what its numerical probability may be." CreationDate="2010-11-11T00:12:19.400" UserId="1108" />
  <row Id="6571" PostId="4392" Text="You should ask another question on the new issue instead of editing the existing one." CreationDate="2010-11-11T01:24:38.437" />
  <row Id="6572" PostId="4392" Text="okay, reverted. will open a new question. thanks." CreationDate="2010-11-11T01:30:58.080" />
  <row Id="6573" PostId="4407" Text="Hi Tal, Is it possible to upload my dendrogram &quot;x&quot; as a &quot;.RData&quot; file/object? " CreationDate="2010-11-11T03:38:20.527" />
  <row Id="6574" PostId="4365" Text="@whuber actually, I do not see how the CLT implies this fact. It only seems to tell us that _asymptotically_, the sum of normals is normal, not that any finite sum is normally distributed. Or do you have to somehow use Slutsky's theorem as well?" CreationDate="2010-11-11T04:20:46.027" UserId="795" />
  <row Id="6576" PostId="4415" Text="Thanks Rob! I do want to note that I added my question comment before your answer so there is no way it was intended as an insult to your generous answer. I have 2 hesitations about variance of time between events (though I may go with it). There's the general &quot;does this assume too much normality.&quot; But my bigger concern is someone who does laundry twice a month, but misses a few (maybe due to travel) will have a series like &quot;7,7,14,7,7,7,21&quot; which would have a var similar to a more frequent but less regular individual. What do you think of multiplying the std dev by the frequency to norm'ze? " CreationDate="2010-11-11T05:24:34.560" UserId="1951" />
  <row Id="6577" PostId="4415" Text="oh... can we use Latex here?" CreationDate="2010-11-11T05:26:39.723" UserId="1951" />
  <row Id="6578" PostId="4422" Score="3" Text="@Brandon What is the aim of your analyses?" CreationDate="2010-11-11T05:38:33.487" UserId="183" />
  <row Id="6579" PostId="4415" Text="Variance does not assume normality. It is a general measure of spread. If you prefer a more robust measure that will allow for the occasional misses, uses the difference of two percentiles. I don't understand why you might want to multiply a measure of spread by the frequency. That would not normalize it. I've updated my answer." CreationDate="2010-11-11T05:45:06.423" UserId="159" />
  <row Id="6580" PostId="4415" Text="Of course the measurement itself doesn't assume normality, but if you are using it as a comparison of inner n-tile ranges you are at least assuming the distributions are of similar forms; if not normal. But yes, that's not a big deal. If you have 2 sets of diffs {6,6,6,12,18} and {2,2,2,4,6,2,2,2,4,6,2,2,2,4,6} you get standard deviations respectively of 1.66 and 5.37. Even though their regularities are similar. But multiply by washes/week you get 3.62 and 3.91. And these values will converge as you extend your sample period. Thank you for your further update." CreationDate="2010-11-11T07:16:59.540" UserId="1951" />
  <row Id="6581" PostId="4415" Text="Multiply by the frequency is just another way of saying divide by the mean of the differences, right?" CreationDate="2010-11-11T07:23:45.737" UserId="1951" />
  <row Id="6582" PostId="4415" Text="I think you were using &quot;frequency&quot; in the scientific sense (inverse of the period), whereas in time series &quot;frequency&quot; is usually a synonym of period. So my suggestion of dividing by $q_{0.5}$ will have the effect you describe." CreationDate="2010-11-11T08:13:39.153" UserId="159" />
  <row Id="6583" PostId="4422" Text="Yes, I did leave that out, didn't I. The goal is to find a predictive equation for share2 from the variables provided. " CreationDate="2010-11-11T08:16:58.340" UserId="776" />
  <row Id="6584" PostId="4425" Text="Thanks for the idea Steven, I'll check try that out" CreationDate="2010-11-11T08:17:20.740" UserId="776" />
  <row Id="6585" PostId="4426" Text="@Jeromy (+1) Good point." CreationDate="2010-11-11T08:23:12.790" UserId="930" />
  <row Id="6586" PostId="4425" Text="Hrmmm, prcomp returns ridiculous results. PC1 accounts for 100% of variance with a SD of 20800! No rotation mind you. Let me try that again" CreationDate="2010-11-11T08:23:21.857" UserId="776" />
  <row Id="6587" PostId="4430" Score="4" Text="@grautur (+1) He's also working on a set of tutorials on [Advanced R development](https://github.com/hadley/devtools/wiki), very handy!" CreationDate="2010-11-11T08:30:10.170" UserId="930" />
  <row Id="6588" PostId="4425" Text="Eh? PCA **is** a rotation." CreationDate="2010-11-11T08:43:32.317" UserId="1390" />
  <row Id="6589" PostId="4425" Score="1" Text="A problem with PCA is interpretation the final model. Say PC1 and PC2 are significant terms in the model. To explain the relationship between share2 and the original data to gain understanding on it, you need to pick apart the loadings of the variables on PC1 and PC2. This likely won't matter too much is prediction is all Brandon is interested in though." CreationDate="2010-11-11T08:49:38.923" UserId="1390" />
  <row Id="6590" PostId="4425" Text="Can't edit that comment now, but I meant scale, not rotation." CreationDate="2010-11-11T08:55:38.180" UserId="776" />
  <row Id="6591" PostId="4405" Text="I can only agree with chl's answer. Perhaps it can be helpful for you to have a look at the Psychometric Models Task View at CRAN: http://cran.r-project.org/web/views/Psychometrics.html" CreationDate="2010-11-11T09:10:29.647" UserId="442" />
  <row Id="6592" PostId="4430" Text="@chl: Thanks. I hadn't seen the tutorials." CreationDate="2010-11-11T09:13:51.633" UserId="183" />
  <row Id="6593" PostId="4425" Text="@Brandon - so that was going to be my other comment on the PCA you did - I think you should standardise (`scale = TRUE`) if the variables are in different units or take different ranges." CreationDate="2010-11-11T09:21:25.030" UserId="1390" />
  <row Id="6594" PostId="4430" Text="@Jeromy In fact, it seems this is merely a way to draft his future textbook (check HW's past tweets)." CreationDate="2010-11-11T09:26:07.113" UserId="930" />
  <row Id="6595" PostId="4437" Score="1" Text="@kwak What is the question exactly? You want (-0.921,0.389593) as a numeric vector?" CreationDate="2010-11-11T09:32:52.823" UserId="930" />
  <row Id="6596" PostId="4438" Text="@mbq Or use the `colClasses` argument in `read.table()`?" CreationDate="2010-11-11T09:36:50.970" UserId="930" />
  <row Id="6597" PostId="4438" Score="1" Text="@chl Possibly; depends on how this data look." CreationDate="2010-11-11T09:38:21.157" UserId="88" />
  <row Id="6598" PostId="4438" Text="&quot;This is because you have read the numbers as factors; if you use read.table, try header=T or restructure the data before read. Some sample of the file should be helpful to resolve it.&quot; nop. &quot;Workaround would be to first convert factors to strings using as.character and then back to numbers with as.numeric.&quot; nop. same result.&#xA;" CreationDate="2010-11-11T09:40:30.043" UserId="603" />
  <row Id="6599" PostId="4438" Score="2" Text="@kwak -- remove those &quot;V1 V2...&quot; from the file before doing anything. as.numeric won't work having unconvertable elements in vector." CreationDate="2010-11-11T10:02:49.937" UserId="88" />
  <row Id="6600" PostId="4438" Text="@mbq:&gt; that did the trick. I was expecting something else. " CreationDate="2010-11-11T10:19:27.483" UserId="603" />
  <row Id="6601" PostId="4438" Text="@mbq Very elegant! Cannot upvote more :( I initially thought lines with missing values were to be kept as is." CreationDate="2010-11-11T10:25:44.650" UserId="930" />
  <row Id="6602" PostId="4441" Text="@George Are you interested in studying possible attrition effect or do you only want to assess the overall trend?" CreationDate="2010-11-11T10:47:03.190" UserId="930" />
  <row Id="6603" PostId="4441" Text="I am not interested in the attrition effect. Only want to assess the trend" CreationDate="2010-11-11T10:49:35.807" UserId="1871" />
  <row Id="6604" PostId="4439" Score="2" Text="you can use data.matrix() for that http://stat.ethz.ch/R-manual/R-devel/library/base/html/data.matrix.html" CreationDate="2010-11-11T11:43:41.217" UserId="603" />
  <row Id="6605" PostId="4437" Text="yes (thanks again)" CreationDate="2010-11-11T11:44:37.753" UserId="603" />
  <row Id="6606" PostId="4439" Text="@kwak Thanks. Never used that function." CreationDate="2010-11-11T12:03:53.070" UserId="930" />
  <row Id="6607" PostId="4384" Text="Do you mean that Excel doesn't recognize that your data are numerical? If you look at the cell format, is it defined as numbers? Just to make sure I understand your comment." CreationDate="2010-11-11T13:16:02.163" UserId="1945" />
  <row Id="6608" PostId="4365" Score="1" Text="@shabbychef Adopting the usual standardization, a sum of two normals is the sum of one normal distribution X_0 plus the limiting distribution of a series X_1, X_2, ..., whence the sum is the limiting distribution of X_0, X_1, ..., which by the Lindeberg-Levy CLT is normal." CreationDate="2010-11-11T13:28:30.820" UserId="919" />
  <row Id="6609" PostId="4418" Text="Maybe this should be tagged as homework?" CreationDate="2010-11-11T13:30:22.300" UserId="919" />
  <row Id="6610" PostId="4443" Text="@Galit The preferred approach to answering questions is to provide one answer per question. Ideally, any new material should be integrated with an existing answer. You may want to look at this discussion on the meta site: http://meta.stats.stackexchange.com/q/635/28 FYI- the meta site is the place to go to if you need to know how the site works, have any questions about the site's functioning etc." CreationDate="2010-11-11T13:56:09.593" />
  <row Id="6613" PostId="4442" Text="@Galit How would you account for the within-person correlation between item responses within such models? (I'm assuming that items are ordered by &quot;difficulty&quot;)" CreationDate="2010-11-11T14:55:09.610" UserId="930" />
  <row Id="6616" PostId="4448" Text="Thanks.  Very pretty." CreationDate="2010-11-11T15:29:10.353" UserId="1959" />
  <row Id="6617" PostId="4456" Text="@dave To get code formatted the right way, insert 4 spaces or use the &quot;code formatting&quot; button in the edit bar." CreationDate="2010-11-11T20:55:56.743" UserId="930" />
  <row Id="6618" PostId="4454" Text="@Joe Are you interested in publishing (online or offline) or editing solutions, finally?" CreationDate="2010-11-11T21:05:35.700" UserId="930" />
  <row Id="6619" PostId="4455" Text="Django would be the Rails equivalent in Python; I haven't used but have heard good things about the django-survey module for it: http://code.google.com/p/django-survey/" CreationDate="2010-11-11T21:23:39.063" UserId="71" />
  <row Id="6620" PostId="4455" Text="@Matt Yes, I was thinking of this kind of stuff. Put it as an answer!" CreationDate="2010-11-11T21:29:23.010" UserId="930" />
  <row Id="6622" PostId="4455" Text="Sorry for any confusion. I'm more interested in knowing how a survey designer would communicate a survey to a programmer for them to create" CreationDate="2010-11-11T21:35:32.573" UserId="305" />
  <row Id="6623" PostId="4454" Text="I've changed the question wording to &quot;communicate&quot; to, hopefully, demonstrate what I mean a little clearer" CreationDate="2010-11-11T21:37:22.850" UserId="305" />
  <row Id="6624" PostId="4454" Text="@Joe I guess the &quot;software&quot; tag is no longer needed then?" CreationDate="2010-11-11T21:39:07.643" UserId="930" />
  <row Id="6625" PostId="4454" Score="3" Text="In retrospect (now that I've answered), StackOverflow is probably the ideal place for this question.  Who better to answer than a bunch of programmers?" CreationDate="2010-11-11T22:06:18.027" UserId="71" />
  <row Id="6626" PostId="4461" Text="How it looks visually isn't as important as the content and the rules that should be applied to a question. For instance, if question 1 should skip to question 5 if answers A or B are selected on question 1" CreationDate="2010-11-11T22:20:31.830" UserId="305" />
  <row Id="6628" PostId="4460" Text="R programming is at the edge of scope, but R administration is out. " CreationDate="2010-11-11T23:12:35.927" UserId="88" />
  <row Id="6629" PostId="4462" Score="4" Text="Can you please work on the style and clarity of your question?" CreationDate="2010-11-11T23:13:46.007" UserId="88" />
  <row Id="6630" PostId="4381" Text="This is a great set of lectures, thanks for the pointer." CreationDate="2010-11-11T23:17:09.577" UserId="493" />
  <row Id="6631" PostId="4465" Text="Roughly what proportion of the individuals died by the end of the experiment? If it's reasonably low there may be little advantage of survival analysis over ignoring the time to death and treating died/survived as a binary outcome, which could be considerably simpler." CreationDate="2010-11-11T23:46:22.160" UserId="449" />
  <row Id="6632" PostId="4452" Text="+1 Good point Harvey." CreationDate="2010-11-11T23:52:56.893" UserId="776" />
  <row Id="6633" PostId="4465" Text="A more complex option is frailty models - i'm too tired to attempt say any more now but i've added the 'frailty' tag above so you can click that and look at the answers to the other q with this tag." CreationDate="2010-11-11T23:53:33.010" UserId="449" />
  <row Id="6634" PostId="4459" Text="@Jake: You raise some interesting questions. I've posted a separate question quoting you: http://stats.stackexchange.com/questions/4466/how-to-increase-longer-term-reproducibility-of-research-particularly-using-r-and" CreationDate="2010-11-12T01:06:02.007" UserId="183" />
  <row Id="6635" PostId="4443" Text="Thanks for the reminder Galit, we will keep the big picture in mind." CreationDate="2010-11-12T01:14:14.843" UserId="1922" />
  <row Id="6636" PostId="4388" Text="Sorry I'm still not clear, the movies are not just different between cinemas, the movies change too. If one cinema starts playing a new interesting movie while the others haven't, this one cinema may have bigger change in gaming sales, no?" CreationDate="2010-11-12T01:50:08.037" UserId="1922" />
  <row Id="6637" PostId="4445" Text="Formatting issue now fixed. It was a bug." CreationDate="2010-11-12T03:27:59.160" UserId="159" />
  <row Id="6638" PostId="4470" Text="+1 Thanks for pointing that out. I somehow doubt that the implementation of weights within R is the same as that for SPSS" CreationDate="2010-11-12T04:06:40.907" UserId="776" />
  <row Id="6639" PostId="4470" Score="1" Text="I know SAS and SPSS are not the same, but that is all I know." CreationDate="2010-11-12T04:14:23.853" UserId="1036" />
  <row Id="6640" PostId="4471" Text="Is $X^2_\alpha$ meant to be a quantile of the chi-squared distribution?" CreationDate="2010-11-12T05:01:00.777" UserId="159" />
  <row Id="6641" PostId="4471" Text="@Rob Hyndman yes that is what it is mean to be" CreationDate="2010-11-12T05:06:35.247" />
  <row Id="6642" PostId="767" Text="great explanation Dave!" CreationDate="2010-11-12T05:28:33.160" UserId="1307" />
  <row Id="6644" PostId="4472" Text="+1 Thanks for the tip Jeromy, this would certainly help! The version I have access to is through my school PASW 18. Hopefully it also contains this functionality! Wouldn't that make life easy." CreationDate="2010-11-12T06:02:45.067" UserId="776" />
  <row Id="6645" PostId="4472" Text="And it works like a charm. You're a life saver." CreationDate="2010-11-12T06:20:00.333" UserId="776" />
  <row Id="6646" PostId="4456" Text="Thank you Dave. Will try this one. It seams to be an interesting software. What a pity they have not done a R package instead..." CreationDate="2010-11-12T07:07:03.770" UserId="1709" />
  <row Id="6647" PostId="4474" Text="Thanks Tal! Sometimes the answer stares you in the face and you don't see it. &quot;Let me put my glasses on, I can't hear you&quot;" CreationDate="2010-11-12T07:54:08.920" UserId="776" />
  <row Id="6648" PostId="4474" Text="You're welcome Brandon.  The 'newdata' thing is something that I failed to understand when I only started out using predict.  Cheers :)  Tal" CreationDate="2010-11-12T08:24:10.113" UserId="253" />
  <row Id="6649" PostId="4473" Text="I'm not a specialist of time series and forecasting, but does it make sense to use a model which assumes uncorrelated errors in this case?" CreationDate="2010-11-12T09:53:38.040" UserId="930" />
  <row Id="6650" PostId="4467" Text="@Jeromy (+1) Thanks for the references." CreationDate="2010-11-12T09:58:50.250" UserId="930" />
  <row Id="6651" PostId="4468" Text="@Dirk (+1) I can only agree with you. About R specifically, it seems very difficult to ensure that (a) some computations will remain reproducible after updating a package (which happens to me recently), and (b) no conflict with dependencies will emerge one day (it was the case e.g., for `lme4`)." CreationDate="2010-11-12T10:03:50.370" UserId="930" />
  <row Id="6652" PostId="4456" Text="I recommend the excellent package 'PBSadmb' available on CRAN." CreationDate="2010-11-12T10:12:30.367" UserId="1709" />
  <row Id="6653" PostId="4465" Text="@onestop- Thanks for contributing some perspective and ideas! I will look into the frailty model option. We had about 20 to 40% mortality in the treatment levels. After some discussion, we are now considering analyzing the replicates, not the individual deaths, since the replicates are independent from one another. Using this perspective, maybe we can arrive with a mean mortality value for each replicate and compare them (accounting for censoring also). Another option could be to compare the chance to survive in each replicate, then compare replicates to each other. " CreationDate="2010-11-12T11:02:12.567" UserId="1862" />
  <row Id="6654" PostId="4465" Text="...maybe Repeated Measures ANOVA could be an option, using replicate level information. We would need to use something that would allow for non-normal data distribution." CreationDate="2010-11-12T11:02:57.590" UserId="1862" />
  <row Id="6655" PostId="4465" Text="Any thoughts on these ideas? (thanks!)" CreationDate="2010-11-12T11:03:53.237" UserId="1862" />
  <row Id="6657" PostId="4309" Text="@csgillespie: I totally agree with you. We can not decide to stop an experiment because the results are what we wanted or otherwise continue. In my work usually replicate experiments 3 to 5 times depending on the availability of time. I think 3 replicas are sufficient. Also, if the estimates are accompanied by a measure of variability (standard errors and confidence intervals) the reader will have enough information to decide whether these estimates are more or less &quot;reliable&quot;.&#xA;" CreationDate="2010-11-12T12:32:30.100" UserId="221" />
  <row Id="6659" PostId="4476" Score="1" Text="But note that the first formula is less prone to round-off error." CreationDate="2010-11-12T13:25:01.933" UserId="449" />
  <row Id="6660" PostId="4470" Score="2" Text="Actually, they are the same for SPSS and R. Simple example:&#xA;yi &lt;- c(1,3,2,2,3,5,4,6); xi &lt;- c(1,2,2,3,3,4,5,5); wi &lt;- c(.2,.3,.2,.1,.4,.3,.2,.3); summary(lm(yi ~ xi, weights=wi))&#xA;&#xA;gives the exact same results as using the same data in SPSS with:&#xA;&#xA;REGRESSION&#xA;/MISSING LISTWISE&#xA;/REGWGT=wi&#xA;/STATISTICS COEFF OUTS R ANOVA&#xA;/CRITERIA=PIN(.05) POUT(.10)&#xA;/NOORIGIN&#xA;/DEPENDENT yi&#xA;/METHOD=ENTER xi  .&#xA;" CreationDate="2010-11-12T13:34:24.167" UserId="1934" />
  <row Id="6661" PostId="4470" Text="@Wolfgang, thank you you pointing that out." CreationDate="2010-11-12T14:24:11.910" UserId="1036" />
  <row Id="6662" PostId="4445" Text="@Rob Thank you, and thanks to all the people who looked at this: I saw your comments before they were deleted and am grateful for your efforts." CreationDate="2010-11-12T16:25:09.873" UserId="919" />
  <row Id="6663" PostId="4478" Text="Agreed, solid data and metadata first." CreationDate="2010-11-12T17:04:25.430" UserId="1965" />
  <row Id="6664" PostId="4354" Score="2" Text="@Wolfgang Please consider modifying your question, then, because the response you checked as your preferred answer does *not* answer the question as it stands (and the other one does).  Currently you are using the word &quot;independent&quot; in an idiosyncratic way.  Your example with Gamma shows this: one could simply reparameterize Gamma in terms of the mean (mu) and variance (sigma), because we can recover theta = sigma/mu and kappa = mu^2/sigma.  In other words, *functional* &quot;independence&quot; of the parameters is usually meaningless (except for single-parameter families)." CreationDate="2010-11-12T18:15:05.757" UserId="919" />
  <row Id="6665" PostId="4488" Text="Hi @Aniko, Thanks for the feedback on this and for the reference. If my goal is to estimate the variance that would be measured on a single day in a single treatment, which term would be approapriate?" CreationDate="2010-11-12T21:44:00.697" UserId="1381" />
  <row Id="6666" PostId="3542" Text="Portions of some table design books are free :-); e.g. http://books.google.com/books?id=0sCkxy_1ChMC&amp;printsec=frontcover&amp;dq=related:ISBN1561583464" CreationDate="2010-11-12T23:24:40.063" UserId="919" />
  <row Id="6667" PostId="4436" Text="With 21 subjects and 8 variables, and without any stated question, hypothesis or model .... what question would ridge regression answer?" CreationDate="2010-11-13T00:46:56.177" UserId="25" />
  <row Id="6668" PostId="4490" Score="1" Text="Great! Can you make it work with timeseries as a bonus? When using a zoo the border insists on appearing." CreationDate="2010-11-13T02:12:46.567" UserId="990" />
  <row Id="6669" PostId="4490" Score="1" Text="See updated answer" CreationDate="2010-11-13T03:17:43.040" UserId="159" />
  <row Id="6670" PostId="4494" Text="Thnks! Any way to add a small, filled triangle?" CreationDate="2010-11-13T03:52:53.607" UserId="990" />
  <row Id="6671" PostId="4501" Score="1" Text="This question seems to me to be straying beyond the bounds of this site. Not only is it subjective, it seems to me a question of page layout graphic design rather than statistics or data visualisation. While the design of figures displaying quantitative information is certainly within our bounds, I'd say the placement and sizing of such figures is straying outside." CreationDate="2010-11-13T07:26:06.577" UserId="449" />
  <row Id="6672" PostId="4501" Score="3" Text="Try on tex.stackexchange.com." CreationDate="2010-11-13T09:04:28.320" UserId="88" />
  <row Id="6673" PostId="4494" Text="@Hugo See my response with `pch`." CreationDate="2010-11-13T09:19:36.997" UserId="930" />
  <row Id="6674" PostId="4436" Score="1" Text="@Harvey - Brandon stated in a comment to his Q that he wanted a model for prediction purposes." CreationDate="2010-11-13T09:28:49.263" UserId="1390" />
  <row Id="6675" PostId="4502" Score="1" Text="+1 and thanks for a very nice link." CreationDate="2010-11-13T10:05:24.030" UserId="88" />
  <row Id="6676" PostId="4511" Score="3" Text="(+1) [Linear Models with R](http://www.maths.bath.ac.uk/~jjf23/LMR/), by Julian J. Faraway is also a good starting point." CreationDate="2010-11-13T18:07:39.247" UserId="930" />
  <row Id="6678" PostId="4509" Text="@ucfagls (+1) Good to know, thx." CreationDate="2010-11-13T18:12:20.397" UserId="930" />
  <row Id="6679" PostId="4462" Score="1" Text="if I understand this correctly, dartdog wants to know whether the median of medians is the can be used to calculate the median of the whole sample, i.e. if $M=M_1 \cup M_2 \cup \ldots \cup M_k$ then median($M$)=median(median$(M_1)$,median$(M_2)$,$\ldots$,median$(M_k)$)..." CreationDate="2010-11-13T19:23:20.857" UserId="1573" />
  <row Id="6680" PostId="4511" Text="Yeah, I guess there's really no escaping me actually digging down and figuring out what exactly I want to compute. I was just hoping there was some sort of standard first approach to use." CreationDate="2010-11-13T21:14:05.973" UserId="1973" />
  <row Id="6681" PostId="4511" Score="1" Text="Yup. _With great freedom comes great responsibility._ It is easy to just mechanically compute something. It may be much harder to come up something meaningful. Damn _No Free Lunch_ theorem again..." CreationDate="2010-11-13T22:15:30.957" UserId="334" />
  <row Id="6682" PostId="4496" Text="Don't confuse the pretty label locations for the data. The figure region has been set to the range of the data. If you want axes with labels at specific, predefined points, turn off axis drawing in the `plot()` call with `axes = FALSE` and then draw the axes using `axis(side = 2)` for the y axis with suitable `at`, and the x-axis can be drawn using `axis.Date()` as per my answer below." CreationDate="2010-11-13T22:48:31.523" UserId="1390" />
  <row Id="6683" PostId="4258" Text="I would just like to mention that as someone interested in making generic MCMC algorithms, easily applicable to many problems, I am very interested in this topic. " CreationDate="2010-11-13T23:01:30.183" UserId="1146" />
  <row Id="6684" PostId="4519" Text="@Henry. If I understand you correctly. In your case, for the tuple (a,b), a and b are both observation (may be missing)?" CreationDate="2010-11-14T05:54:00.287" UserId="1307" />
  <row Id="6685" PostId="4515" Text="@galois. Is your question related to topic models?" CreationDate="2010-11-14T05:59:26.850" UserId="1307" />
  <row Id="6686" PostId="4502" Text="@carlo. nice link. thx " CreationDate="2010-11-14T06:02:23.663" UserId="1307" />
  <row Id="6687" PostId="4519" Score="3" Text="*Why* did you want to do a regression of one coordinate on the other? The better way to do this depends on what is is you're trying to do. What's the aim? What question are you trying to answer?" CreationDate="2010-11-14T07:39:10.820" UserId="449" />
  <row Id="6688" PostId="4513" Text="Is that the statistical query model of Kearns (1998) http://dx.doi.org/10.1145/293347.293351 ? If so, sorry, i'd never even heard of it before." CreationDate="2010-11-14T07:54:21.400" UserId="449" />
  <row Id="6689" PostId="4501" Text="Okay. I've posted the question on Tex.SE: http://tex.stackexchange.com/questions/5351/aesthetically-pleasing-width-for-figures-in-latex-documents" CreationDate="2010-11-14T08:10:38.220" UserId="183" />
  <row Id="6690" PostId="4503" Text="Thanks. That's really useful." CreationDate="2010-11-14T08:12:41.993" UserId="183" />
  <row Id="6691" PostId="4506" Text="@Bernd. Thanks for that. With regards to the second question, I was thinking of several scenarios. For example, you conduct two studies and they get written up in a journal article and conference proceedings. Then you do a third study and present a second conference paper using all three studies. Perhaps you can see where I'm going with this. There are connections between the set of studies and the set of publications and presentations. Should they all be in one repository, or would you split them up in some way (e.g., one repository per publication)?" CreationDate="2010-11-14T08:18:24.343" UserId="183" />
  <row Id="6692" PostId="4506" Text="@Bernd; thanks also for the link to the SO question. It suggests having one project per repository. However, for me it still doesn't quite answer the question: &quot;what is a project?&quot; In particular, it seems to me that there are a few many-to-many relationships going on between studies and publications that can sometimes complicate things." CreationDate="2010-11-14T08:21:49.037" UserId="183" />
  <row Id="6693" PostId="4514" Text="Wow! Thanks a lot!" CreationDate="2010-11-14T08:33:52.833" UserId="1351" />
  <row Id="6694" PostId="4513" Text="@onestop yes it is." CreationDate="2010-11-14T08:35:31.400" UserId="1961" />
  <row Id="6695" PostId="4514" Text="This is the perfect answer, I realize now that I was getting lost with my own proof. :)" CreationDate="2010-11-14T08:42:45.803" UserId="1351" />
  <row Id="6696" PostId="4520" Text="@mariana soffer this is a very good idea but for my reasons I would not like to go through articles distance. I want the probability/weight not for the actual tag suggestion which it can be done as you suggested, but as a solid tag score that I will use in my system for many other purposes." CreationDate="2010-11-14T10:00:04.683" UserId="1975" />
  <row Id="6697" PostId="4506" Text="(Sorry, just realized that comments are limited to 600 characters)&#xA;&#xA;@Jeromy (part 1/3): You wrote: &quot;There are connections between the set of studies and the set of publications and presentations. Should they all be in one repository, or would you split them up in some way (e.g., one repository per publication)?&quot; When you write &quot;split them up in some way&quot;, do you mean sth like forking? And &quot;studies&quot; are datasets? As I wrote before, each publication, each presentation etc. has its own repository." CreationDate="2010-11-14T11:04:28.490" UserId="307" />
  <row Id="6698" PostId="4506" Text="@Jeromy (part 2/3): &#xA;However, there is one real downside: using the same dataset in different publications means to maintain different versions of &quot;initialization code&quot; (define missing values, generate new variables etc.). To overcome this problem, I decided to maintain /one/ study/dataset-related repository which contains the original init-file. For each publication, presentation etc. I am using a copy of the original data-file as well as of the init-file (in R via file.copy())." CreationDate="2010-11-14T11:05:49.893" UserId="307" />
  <row Id="6699" PostId="4506" Text="@Jeromy (part 3/3): Of course, whenever I create a new variable I need to modify my original init-file and do a file.copy() (which is the most annoying part of my approach).&#xA;&#xA;Regarding your question &quot;what is a project?&quot;, I would say that project = (primary) study/dataset.  " CreationDate="2010-11-14T11:06:34.777" UserId="307" />
  <row Id="6700" PostId="4501" Text="@onestop I changed the title to try to make it less subjective" CreationDate="2010-11-14T11:33:03.083" UserId="183" />
  <row Id="6701" PostId="4501" Text="I was more worried about the subject matter than the subjectivity to be honest. How about proposing it as a suitable question for http://area51.stackexchange.com/proposals/1924/graphic-design ?" CreationDate="2010-11-14T11:39:41.877" UserId="449" />
  <row Id="6702" PostId="4501" Text="@onestop I can see where you are coming from. I can see how general graphic design, and tex sites are relevant. However, I have a specific interest in what other users of Sweave and R do, and how they justify their decisions. Thus, I thought I'd get more relevant answers here. Perhaps this is an issue other users of Sweave and R encounter." CreationDate="2010-11-14T11:46:58.687" UserId="183" />
  <row Id="6703" PostId="4518" Text="One might add that fitting the regressions separateley is indeed equivalent to the multivariate formulation with a matrix of dependent variables. In R with package mvtnorm installed (1st: multivariate model, 2nd: separate univariate models):&#xA;library(mvtnorm);&#xA;X &lt;- rmvnorm(100, c(1, 2), matrix(c(4, 2, 2, 3), ncol=2));&#xA;Y &lt;- X %*% matrix(1:4, ncol=2) + rmvnorm(100, c(0, 0), diag(c(20, 30)));&#xA;lm(Y ~ X[ , 1] + X[ , 2]);&#xA;lm(Y[ , 1] ~ X[ , 1] + X[ , 2]);&#xA;lm(Y[ , 2] ~ X[ , 1] + X[ , 2])" CreationDate="2010-11-14T12:26:00.207" UserId="1909" />
  <row Id="6704" PostId="4288" Score="1" Text="I don't understand what you're trying to do here. To estimate propensity scores you need to know whether each individual received the treatement or not, in which case you can simply count up how many received the treatment. Why do you want to aggregate propensity scores?" CreationDate="2010-11-14T14:04:56.463" UserId="449" />
  <row Id="6705" PostId="4456" Text="If you post some data, I'll run the model for you and see what happens." CreationDate="2010-11-14T16:45:09.640" UserId="1585" />
  <row Id="6706" PostId="4451" Text="what are datasets ?" CreationDate="2010-11-14T17:23:04.820" UserId="1983" />
  <row Id="6707" PostId="4516" Text="Hi Jon, could you explain how to generate ranks within groups?" CreationDate="2010-11-14T17:32:48.523" UserId="1950" />
  <row Id="6709" PostId="4479" Text="Thanks onestop! We are definitely going to use each aquarium as the unit of analysis so we will not have dependent data. We are still formulating the best approach. I will post the final decision once it is made." CreationDate="2010-11-14T19:45:01.513" UserId="1862" />
  <row Id="6710" PostId="4506" Text="@Bernd. Can you please point me to a tutorial (for dummies) which explains the basic use of git+github+ProjectTemplate-like package? Thx for the help." CreationDate="2010-11-14T19:49:31.350" UserId="1307" />
  <row Id="6711" PostId="4519" Text="Post-edit: Your statements together seems to imply that the number of times *i* attacks *j* is always either 0 or 1. And that if *i* attacks *j* then *j* attacks *i*. So relating the number of attackes that *i* makes on *j* to the number *j* makes on *i* seems trivial. Sorry, I'm obviously not understanding something here." CreationDate="2010-11-14T19:51:21.973" UserId="449" />
  <row Id="6713" PostId="4527" Text="@annemphillip If you mean the column headed &quot;Value&quot; as returned by a call to `summary`, then it is *a priori* a regression coefficient (for your survival model). Its interpretation depends on the fitted distribution." CreationDate="2010-11-14T20:21:24.817" UserId="930" />
  <row Id="6714" PostId="4519" Text="@onestop: Thanks.  Good point.  I was sloppy in my description there." CreationDate="2010-11-14T20:51:19.613" UserId="847" />
  <row Id="6715" PostId="4518" Text="Thanks, this is what I was looking for! There doesn't seem to be an option for this in SPSS, though I feel like I should be able to determine the coefficients of my equation from a multivariate GLM, no?" CreationDate="2010-11-14T20:53:22.263" UserId="1977" />
  <row Id="6716" PostId="4521" Score="1" Text="This seems conceptually different than the answer above.  I'm still not clear as to how transforming my variables to PCA coefficients allows me to regress on 2+ dependent variables?" CreationDate="2010-11-14T20:55:10.980" UserId="1977" />
  <row Id="6717" PostId="4522" Score="2" Text="I could be wrong, but I don't think this is the same thing.  From SEM graphs that I've seen, it looks like SEM uses multiple regression equations to determine the values of latent factors, and then another regression is run on the value of those latent factors to determine a higher-order factor.  Maybe this is wrong, but I've never seen an SEM graph that links several IVs to multiple DVs-- everything is hierarchical." CreationDate="2010-11-14T20:59:25.597" UserId="1977" />
  <row Id="6718" PostId="4526" Text="Excellent answer, thanks! Since there where no answer for some time, I also asked the question over at [math](http://math.stackexchange.com/questions/10124/), and they have some different approaches there, that might interest you." CreationDate="2010-11-14T21:36:07.493" UserId="977" />
  <row Id="6719" PostId="4526" Text="Thank you!  @Mike Spivey gives a nice answer there.  However, it's for *choosing without replacement,* which answers a different (and seemingly easier) question." CreationDate="2010-11-14T21:39:15.187" UserId="919" />
  <row Id="6722" PostId="4506" Text="@ suncoolsu: git: Pro Git (http://progit.org/book), Git References (http://gitref.org/), I also bought O'Reilly's &quot;Version Control with Git&quot;. Github also offers some learning resources (http://help.github.com/). What do you mean by &quot;ProjectTemplate-like package&quot;? I know only the original package by JMW." CreationDate="2010-11-15T10:51:16.090" UserId="307" />
  <row Id="6723" PostId="4541" Text="http://www.healthknowledge.org.uk/public-health-textbook looks like a useful resource -- surprised i'd never come across it before. Thanks chl." CreationDate="2010-11-15T11:55:32.213" UserId="449" />
  <row Id="6724" PostId="4497" Score="1" Text="Great, they seem to be both available for free on the web!" CreationDate="2010-11-15T14:50:54.277" UserId="190" />
  <row Id="6725" PostId="3671" Text="The linked post does provide some good insight into the challenges that readers face when confronted with a table; this can be good information when designing tables or choosing alternatives. " CreationDate="2010-11-15T15:22:26.290" UserId="1381" />
  <row Id="6726" PostId="4551" Text="I'm aware that &quot;sin&quot; is possibly inflammatory and that that some aspects of statistical analysis are not black-and-white. My intention is to solicit cases where a given commonly-taught practice is pretty clearly inappropriate." CreationDate="2010-11-15T18:53:14.270" UserId="364" />
  <row Id="6727" PostId="4544" Score="2" Text="@drknexus .. I understand that you need Type III sum, but this (www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf) article is a good read. It illustrates some interesting points." CreationDate="2010-11-15T19:03:03.193" UserId="1307" />
  <row Id="6728" PostId="4551" Text="You can also add biology/life sciences students to the mix if you like ;)" CreationDate="2010-11-15T19:03:17.187" UserId="582" />
  <row Id="6730" PostId="4552" Text="but sometimes... sometimes the potential directions of causation have highly disparate probabilities.  I'm certainly not going to think that a correlation between age and height could be caused by the height... or some intervening variable either.  Also, I think that this is one that behavioural science training is generally quite sensitive to." CreationDate="2010-11-15T19:21:49.960" UserId="601" />
  <row Id="6731" PostId="4551" Text="maybe retitle it life science statistical sins?... or something else more specific..." CreationDate="2010-11-15T19:27:28.377" UserId="601" />
  <row Id="6733" PostId="4529" Text="@onestop- Thanks again for your input and your help!" CreationDate="2010-11-15T20:14:59.897" UserId="1862" />
  <row Id="6734" PostId="4534" Text="@chi- I appreciate the thorough and helpful answer. Thanks!" CreationDate="2010-11-15T20:15:30.320" UserId="1862" />
  <row Id="6736" PostId="3214" Text="Sorry about the late comment. I just received the book that Thylacoleo recommended. To answer your questions: Chl - the outcome is survival (i.e. no death) over a period of time. The sample size in each group varies from, say, 5-8. Andy - survival experiments suffer from inherent variabilities. I do indeed examine the Kaplan Meier plots of the groups by a log rank test. But I am more interested in finding out if there is a test that can evaluate inter-experimental variabilities of survival experiments." CreationDate="2010-11-15T20:47:54.700" UserId="1464" />
  <row Id="6738" PostId="4550" Text="I know Moore is in the car library, but When sample data is provided it is easier for the question asker to understand your response if you use the sample data." CreationDate="2010-11-15T21:23:59.367" UserId="196" />
  <row Id="6740" PostId="4549" Text="I don't think this answer uses the -3,-1,1,3 contrast I specified nor does it seem to provide a 1 df test of the contrast." CreationDate="2010-11-15T21:33:58.300" UserId="196" />
  <row Id="6741" PostId="4562" Text="Thanks, sounds reasonable, but is there any other possibility to further constrain the definition of different... For instance, a difference of 8 byte is statistically significant, but it is possibly not in practice. Such a constraint would be useful for hypothesis testing." CreationDate="2010-11-15T21:34:15.837" UserId="1992" />
  <row Id="6743" PostId="4549" Text="@drknexus Yes, you're right. Wrote too quickly. Something like `Anova(lm(DV ~ C(IV, c(-3,-1,1,3),1), data=sample.data), type=&quot;III&quot;)` should be better. Please let me know if this ok with you." CreationDate="2010-11-15T22:23:19.763" UserId="930" />
  <row Id="6744" PostId="4549" Text="Thanks!  That looks okay I'll validate it against SPSS tomorrow and get back to you.  " CreationDate="2010-11-15T22:35:00.313" UserId="196" />
  <row Id="6745" PostId="4553" Score="4" Text="I don't think this is really a &quot;sin&quot; as the results obtained are not wrong. However, it does throw away a lot of useful information so is not good practice." CreationDate="2010-11-15T22:51:35.173" UserId="159" />
  <row Id="6746" PostId="4562" Text="@stefan-ock Hypothesis testing is only about approximating chance that your result is just a lucky coincidence; here the result is absolutely clear. What you can do is to write, for instance, that the difference is so small that it is not an important factor for comparison; yet judging how many bytes is equally important as 1 ms of computation is purely use-dependent and a bit subjective in its nature." CreationDate="2010-11-15T23:39:43.497" UserId="88" />
  <row Id="6747" PostId="4553" Score="1" Text="Along these lines, using extreme groups designs over-estimates effect sizes whereas the use of a mean or median split under-estimates effect sizes." CreationDate="2010-11-16T02:47:46.777" UserId="196" />
  <row Id="6748" PostId="4557" Score="3" Text="Would you please explain the consequences of this sin?" CreationDate="2010-11-16T02:50:28.617" UserId="196" />
  <row Id="6749" PostId="4560" Score="3" Text="&quot;Good idea&quot; depends on the situation.  When you want to maximize prediction it isn't a horrible idea - though it may lead to over fitting.  There are some rare cases where it is inevitable - where there is no theory to guide the model selection.  I wouldn't count stepwise regression as a &quot;sin&quot; but using it when theory is sufficient to drive model selection is." CreationDate="2010-11-16T02:52:32.267" UserId="196" />
  <row Id="6750" PostId="4563" Score="3" Text="What's inappropriate about least squares estimation when the data are non-normal or heteroskedastic? It is not fully efficient, but it is still unbiased and consistent." CreationDate="2010-11-16T03:18:20.200" UserId="159" />
  <row Id="6751" PostId="4560" Score="7" Text="Perhaps the sin is doing statistical tests on a model obtained via stepwise regression." CreationDate="2010-11-16T03:19:25.860" UserId="159" />
  <row Id="6752" PostId="4554" Text="Thanks for pointing out the `penalized` package." CreationDate="2010-11-16T07:02:09.293" UserId="439" />
  <row Id="6753" PostId="4573" Text="link: http://socialnetworks.mpi-sws.org/" CreationDate="2010-11-16T07:19:07.930" />
  <row Id="6754" PostId="4555" Score="2" Text="How much does this decrease the power of the analysis? In what conditions is it most problematic? &#xA;In many cases deviations from the assumptions of ANOVA do not substantially affect the outcomes to an important extent." CreationDate="2010-11-16T07:48:44.663" UserId="1679" />
  <row Id="6755" PostId="4559" Text="@Gottfried Canonical correlation is the correlation between factor scores computed from two-block structures, as with CCA or PLS. This is exactly what I described in my response (PLS regression), although PLS is more appropriate than CCA when the variables play an asymmetrical role, which is likely to be the case here. This is because there's an asymmetric deflation process and we work with the covariance instead (with CCA, we deflate both blocks at the same time, and we seek to maximize the correlation, instead of the covariance)." CreationDate="2010-11-16T07:50:59.637" UserId="930" />
  <row Id="6756" PostId="4576" Text="@suncoolsu +1 for pointing out Hadley's wiki." CreationDate="2010-11-16T08:28:01.213" UserId="930" />
  <row Id="6757" PostId="4579" Text="The tag &quot;machine learning&quot; doesn't seem very appropriate there. Maybe &quot;optimization&quot; or something like that?" CreationDate="2010-11-16T08:31:25.047" UserId="930" />
  <row Id="6758" PostId="4580" Text="A possible related question, http://stats.stackexchange.com/questions/825/any-suggestions-for-making-r-code-use-multiple-processors." CreationDate="2010-11-16T08:33:28.243" UserId="930" />
  <row Id="6760" PostId="4579" Text="I agree, but as I am new here I couldn't create that tag." CreationDate="2010-11-16T08:36:25.650" UserId="2005" />
  <row Id="6761" PostId="4579" Text="they are not the same but if you minimize them with respect to lambda in the whole real line changing lambda in -lambda does not change the solution. Not that there is a mathematic stackoverflow somewhere (This site is more for statistics)" CreationDate="2010-11-16T08:54:40.620" UserId="223" />
  <row Id="6762" PostId="4571" Text="Why check against a log odds ratio *z*-test? That's a Wald test, and Wald tests typically perform worse than score tests such as Pearson's chi-squared test. Is this known to be an exception?" CreationDate="2010-11-16T09:11:04.647" UserId="449" />
  <row Id="6763" PostId="4557" Text="If the data are generated by a process with a heteroscedastic noise process, the regression model is likely to give very inacurate out-of-sample predictions." CreationDate="2010-11-16T09:26:46.770" UserId="887" />
  <row Id="6764" PostId="4582" Text="But indeed I am not sure http://stats.stackexchange.com is the best place to ask your question" CreationDate="2010-11-16T09:27:34.477" UserId="1709" />
  <row Id="6765" PostId="4563" Score="1" Text="If the data are heteroscedastic you can end up with very innacurate out of sample predictions because the regression model will try too hard to minimise the error on samples in areas with high variance and not hard enough on samples from areas of low variance.  This means you can end up with a very badly biased model.  It also means that the error bars on the predictions will be wrong." CreationDate="2010-11-16T09:31:44.380" UserId="887" />
  <row Id="6767" PostId="4582" Text="@fRed ... and it is not the best place to comment the question ;) " CreationDate="2010-11-16T09:54:59.537" UserId="223" />
  <row Id="6768" PostId="4584" Text="Thanks for your answer! Do you do the computation for your work/academic research or for own projects on a own PC?" CreationDate="2010-11-16T09:59:35.063" UserId="2006" />
  <row Id="6769" PostId="4586" Text="Sorry if I'm being a bit slow.  How do I do the minimization? Is there a formula or an algorithm that I need to use?" CreationDate="2010-11-16T10:15:20.187" UserId="847" />
  <row Id="6770" PostId="4570" Score="3" Text="Perhaps I don't see why this is such a problem. Hypothesis testing a small sample size using a normal distribution, sure, but using a more conservative/nonparametric test, is this so bad?" CreationDate="2010-11-16T10:18:30.767" UserId="1118" />
  <row Id="6771" PostId="4586" Text="You need to use a nonlinear optimization algorithm." CreationDate="2010-11-16T10:19:26.710" UserId="159" />
  <row Id="6772" PostId="4567" Text="@Michael (+1) I added links to abstracts and ungated PDFs. Hope you don't mind." CreationDate="2010-11-16T10:24:17.297" UserId="930" />
  <row Id="6773" PostId="4587" Text="@mbq, thanks for your answer!" CreationDate="2010-11-16T10:53:00.153" UserId="2006" />
  <row Id="6774" PostId="4584" Text="This is done in a commercial setting. For this task, I am using a single Intel box with 32GB of RAM and RAIDed disks (the main difficulty is lots of data, while the processing itself is not very computationally demanding.)" CreationDate="2010-11-16T11:01:33.627" UserId="439" />
  <row Id="6775" PostId="4582" Text="Thus it does not even play a role whether I want to maximize or minimize the function A?" CreationDate="2010-11-16T11:07:32.807" UserId="2005" />
  <row Id="6776" PostId="4588" Score="1" Text="@user1806 This is basically what I answered in my comment. Now, the point is that the OP want to know how it reads in plain english, which depends *a priori* of the kind of parametric assumptions that are made in the model." CreationDate="2010-11-16T12:21:49.337" UserId="930" />
  <row Id="6777" PostId="4584" Text="Alright @aix, how often do you perform these calculations. Is you box running all the day or more idle?" CreationDate="2010-11-16T12:38:19.517" UserId="2006" />
  <row Id="6778" PostId="4563" Score="2" Text="No, it is unbiased, but the variance is larger than if you used a more efficient method for the reasons you explain. Yes, the prediction intervals are wrong." CreationDate="2010-11-16T12:39:49.377" UserId="159" />
  <row Id="6779" PostId="4549" Text="BTW, have a look at the ez package (http://cran.r-project.org/web/packages/ez/index.html) for wrapping the Anova code..." CreationDate="2010-11-16T12:45:08.080" UserId="253" />
  <row Id="6780" PostId="4555" Text="What is the alternative do the ANOVA procedure?" CreationDate="2010-11-16T12:55:24.730" UserId="442" />
  <row Id="6781" PostId="4587" Text="We here in Hamburg always have a problem that the waiting time for the academic data centers are really long. is it the same for you?" CreationDate="2010-11-16T13:16:14.660" UserId="2006" />
  <row Id="6782" PostId="4587" Text="@Heinrich I work for a kind of academic data center, so I don't have such problems (-; Seriously, in Warsaw the scientific CPU time supply is larger than demand, so I believe it is quite easy to get a grant. And I think you should try D-Grid or EGEE, my experience is that grids in general are very underused." CreationDate="2010-11-16T14:16:36.007" UserId="88" />
  <row Id="6783" PostId="4575" Score="2" Text="That is either a fairly arrogant comment, or just a fairly naive comment given how much or little you know about details of the R / S languages." CreationDate="2010-11-16T15:03:33.997" UserId="334" />
  <row Id="6784" PostId="4555" Text="@Michael Lew &amp; Henrik: I just updated this entry to include a link to Dixon (2008)" CreationDate="2010-11-16T15:48:33.303" UserId="364" />
  <row Id="6785" PostId="4557" Text="This entry was originally inspired by the observation that some folks estimate non-linear psychometric functions by minimizing least squares. For example, Murd et al (2009, http://www.perceptionweb.com/abstract.cgi?id=p6145 , free pdf available by googling the title) fit a probit function through data by minimizing least-squares." CreationDate="2010-11-16T16:04:32.697" UserId="364" />
  <row Id="6786" PostId="4587" Text="Oh. That is interesting. Dow you know in what kind of businesses R is used in these extends?" CreationDate="2010-11-16T16:22:52.823" UserId="2006" />
  <row Id="6787" PostId="4288" Text="I suppose I worded it poorly; the data I have is all either binary or multinomial and each observation is accompanied by the probability it was observed correctly. So suppose I have 5 persons; 3 of which had a value of 1 for an attribute, 2 of which had a value of 0 for that attribute, each of which had a probability .8,.81,.82.,.83,.84 respectively of being observed correctly. What is the p(having that attribute)?" CreationDate="2010-11-16T17:27:10.687" UserId="1893" />
  <row Id="6788" PostId="4546" Text="I have realizations of Turkey and probabilities I observed that correctly; I edited the question to better reflect this. Thanks!" CreationDate="2010-11-16T17:28:23.947" UserId="1893" />
  <row Id="6789" PostId="4596" Text="@Maiasaura. Can you pls tell what is the size of your data set." CreationDate="2010-11-16T19:28:27.417" UserId="1307" />
  <row Id="6790" PostId="4567" Text="@Chi Thanks! I should have done that myself. Next time..." CreationDate="2010-11-16T19:44:01.083" UserId="1679" />
  <row Id="6791" PostId="4598" Score="2" Text="Perhaps moving standard deviations would be more informative." CreationDate="2010-11-16T22:03:11.017" UserId="1679" />
  <row Id="6792" PostId="4563" Score="1" Text="Yes (I was using biased in a colloquial rather than a statistical sense to mean the model was systematically biased towards observations in high-variance regions of the feature space - mea culpa!) - it would be more accurate to say that the higher variance means there is an increased chance of getting a poor model using a finite dataset.  That seems a reasonable answer to your question. I don't really view unbiasedness as being that much of a comfort - what is important is that the model should give good predictions on the data I actually have and often the variance is more important." CreationDate="2010-11-16T22:18:23.977" UserId="887" />
  <row Id="6794" PostId="4602" Text="I know many problems not related to econometry which are used as a benchmark, but I want to focus on applications of GA in economics, and I thought that estimation of econometry model is good start." CreationDate="2010-11-16T22:37:18.667" UserId="1643" />
  <row Id="6795" PostId="4590" Score="1" Text="The same logic (taking &quot;absence of evidence in favor H1&quot; as &quot;evidence of absence of H1&quot;) essentially underlies all goodness-of-fit tests. The reasoning also often crops up when people state &quot;the test was non significant, we can therefore conclude there is no effect of factor X / no influence of variable Y&quot;. I guess the sin is less severe if accompanied by reasoning about the test's power (e.g., a-priori estimation of sample size to reach a certain power given a certain relevant effect size)." CreationDate="2010-11-16T23:07:06.833" UserId="1909" />
  <row Id="6796" PostId="4597" Text="What do you mean by needs to implement? Do you mean you simply need to report the &quot;wiggliness&quot;. Also I would take wiggliness to mean the variance of your series (or intervals within the series.)" CreationDate="2010-11-16T23:33:52.943" UserId="1036" />
  <row Id="6797" PostId="4598" Text="Perhaps I misunderstood the question, I thought he wanted to smooth the data, and was underwelmed by his current results. If he wants to extract the conditional volatility I would use GARCH, or squared logged differences as a first step. " CreationDate="2010-11-17T00:05:17.777" UserId="1766" />
  <row Id="6798" PostId="4597" Text="I need to find the 'wiggly' intervals, as those are text blocks. I'm using this to find blocks of text in a scanned document, and I found that this is the easiest way for the computer to 'see' deformed blocks of text on a sheet of badly-scanned paper." CreationDate="2010-11-17T00:06:58.243" UserId="2002" />
  <row Id="6799" PostId="4598" Text="How would I go about implementing a moving standard deviation? I'm not familiar with statistics (I'm studying theoretical mathematics at the moment), but I think that's exactly what I need. Thanks a lot!" CreationDate="2010-11-17T00:13:45.410" UserId="2002" />
  <row Id="6800" PostId="4598" Text="@Blender I've never actually calculated a moving stdev, but I expect that you could simply choose a window size (the number of data points to include in the stdev) and calculate a stdev at each point along the dataset. You would have to start half of the window length in from one end of the data, and stop half from the other end. Probably an easy calculation in a spreadsheet." CreationDate="2010-11-17T00:45:13.093" UserId="1679" />
  <row Id="6801" PostId="4570" Text="I agree that using a more conservative model to fit the data is the best we can do. But in any case you will have to trust this model. It will be a fitting, not a model. A model requires a representative set of data otherwise it may not work in the future." CreationDate="2010-11-17T02:17:33.240" UserId="1709" />
  <row Id="6802" PostId="4571" Text="@Mike - thanks for the information! Fisher's test does seem to be a more robust method for questions like these. I don't think the course I am currently taking will address Fisher's test, but I'll definitely keep it in mind for real world applications." CreationDate="2010-11-17T03:52:29.303" UserId="696" />
  <row Id="6803" PostId="4577" Text="@chl - Thanks for the references. I was able to find a &quot;pre-print&quot; version of the Campbell paper [here](http://www.iancampbell.co.uk/twobytwo/files/campbell_twobytwo_preprint.pdf) for those who can't access Pub Med." CreationDate="2010-11-17T03:56:05.740" UserId="696" />
  <row Id="6804" PostId="2172" Text="@shane thanks for the links " CreationDate="2010-11-17T05:57:00.883" UserId="1307" />
  <row Id="6805" PostId="4608" Text="The gradient is a vector since your loss function has real values." CreationDate="2010-11-17T07:04:51.847" UserId="1351" />
  <row Id="6806" PostId="4605" Text="Thanks for your info. What do you mean with course parallelization?" CreationDate="2010-11-17T07:13:13.713" UserId="2006" />
  <row Id="6807" PostId="4586" Text="Okay. I've got as far as using ARIMA(0,1,1).  I am assuming that the code, arima(x, order = c(0, 1, 1)) is how one does that.  (I spent a few hours learning R first so I am completely new to this.) How do I apply the output to my smoothing problem?" CreationDate="2010-11-17T07:22:48.937" UserId="847" />
  <row Id="6808" PostId="4590" Text="If you do not make any concideration about the power, I would say claming $H_0$ is true when it is not rejected is very very bad while claming $H_1$ is true while $H_0$ is rejected is just a little wrong :). " CreationDate="2010-11-17T07:22:53.877" UserId="223" />
  <row Id="6809" PostId="4586" Score="1" Text="Using R's parameterization, $\alpha = 1+\theta$ where $\theta$ is the moving average parameter." CreationDate="2010-11-17T07:32:14.803" UserId="159" />
  <row Id="6810" PostId="4600" Text="@Lynn Do you mean that you want to know the % of variance ($R^2$) in your DV explained by all predictors in your final model, or during model selection?" CreationDate="2010-11-17T07:45:39.563" UserId="930" />
  <row Id="6813" PostId="4586" Text="Thanks! Great! I get it now.  My last hurdle is the missing values.  Now I have $\alpha$.  How do handle the missing values? Using the notation on Wikipedia, does this mean I have several $x_0$'s and I just restart the exponential smoothing algorithm after each gap?" CreationDate="2010-11-17T08:03:51.523" UserId="847" />
  <row Id="6814" PostId="4602" Text="Surely the travelling salesman problem has huge application within economics. Perhaps you could clarify what you mean by an 'econometry model'." CreationDate="2010-11-17T08:09:00.483" UserId="449" />
  <row Id="6816" PostId="1882" Text="Domo arigato, Mr. Roboto :)" CreationDate="2010-11-17T09:13:38.573" UserId="339" />
  <row Id="6818" PostId="4602" Text="Sorry it was late, of course I meant econometric model. Example: Y=b1*x1+b2*x2^b3+(b4/x3+b5*x4)^b6 , but I'm not sure if this is hard to estimate using non-linear regression and secondly whether it has any real-world application. " CreationDate="2010-11-17T09:36:23.263" UserId="1643" />
  <row Id="6819" PostId="4524" Text="Your first suggestions worked great. Your second, I wasn't able to achieve what I wanted. Could you give me an example? Thx." CreationDate="2010-11-17T10:17:22.010" UserId="990" />
  <row Id="6820" PostId="4610" Text="can you use R ? (if yes i'll post a code that'll solve this problem)." CreationDate="2010-11-17T11:05:19.737" UserId="603" />
  <row Id="6821" PostId="4626" Text="Thanks, but I am afraid I don't follow. LARS produces a piecewise-linear path (with exactly $p+1$ points for the least angles and possibly more points for the lasso.) Each point has its own set of $\beta$. When we add more observations, all the betas can move (except $\beta^0$, which is always $0_p$.) Please could you expand on your answer? Thanks." CreationDate="2010-11-17T11:34:29.510" UserId="439" />
  <row Id="6822" PostId="4608" Score="1" Text="your function is not differentiable everywhere." CreationDate="2010-11-17T11:40:22.203" UserId="223" />
  <row Id="6824" PostId="4621" Text="Thanks! That clears things up for me. Now I just have to get it right in a practical setting. You wouldn't happen to have any idea why the above code doesn't work? It seems to converge in 4 iterations with the loss starting at 1 and going down 0.25 each time and converging at 0. However, the weights it produces seem quite wrong." CreationDate="2010-11-17T11:56:38.627" UserId="2023" />
  <row Id="6826" PostId="4586" Score="1" Text="arima() will handle the missing values appropriately. There's no need for you to do anything. Just use the forecasts it provides." CreationDate="2010-11-17T12:37:05.900" UserId="159" />
  <row Id="6827" PostId="4630" Score="2" Text="+1 Note that the MSE comes with a built-in goodness of fit estimate, so it solves problems #1 and #2 simultaneously." CreationDate="2010-11-17T13:28:21.080" UserId="919" />
  <row Id="6829" PostId="4567" Score="2" Text="+1, but I would like to make some critical comments.  Regarding the opening line, one could just as well say that &quot;almost all&quot; (in the measure theoretic sense) interpretations of *any* well-defined concept are incorrect, because only one is correct.  Second, to what do you refer when you say &quot;the conventional usage&quot; and &quot;standard approaches&quot;? These vague references sound like a straw man.  They do not accord with what one can find in the literature on statistics education, for example." CreationDate="2010-11-17T13:43:58.040" UserId="919" />
  <row Id="6830" PostId="4630" Text="@whuber (+1) Thanks, I hadn't noticed that. I admit I've never actually *used* maximum spacing estimation; I'd *heard* of it in connection with estimation for a uniform distribution and just checked Wikipedia." CreationDate="2010-11-17T14:26:41.310" UserId="449" />
  <row Id="6831" PostId="4613" Text="Thanks for the response. Could you elaborate any more on what these equations actually say? They may well be useful, but I've sort of got to convince myself. :)" CreationDate="2010-11-17T14:45:19.760" UserId="1810" />
  <row Id="6832" PostId="4636" Text="Ouch.  In the real world the incidence of people who can't add correctly to 100 is going to be high.  Even so, in their minds most people can't tell the difference between 90% and 95% so providing them a scale where they can do so is pointless.  I'd suggest you'd be better off giving a Likert scale for each of your major competitors and one marked other.  You can then look at the proportion of your likert scale over the sum of all likert scores (lower bound should have a value of zero) as an indication of market share." CreationDate="2010-11-17T15:07:42.643" UserId="196" />
  <row Id="6833" PostId="4557" Text="So, should the &quot;answer&quot; be amended to &quot;application of least-squares minimization on heteroscedatic data&quot;?" CreationDate="2010-11-17T15:13:00.360" UserId="196" />
  <row Id="6834" PostId="4613" Score="1" Text="@Noldorin answer updated" CreationDate="2010-11-17T15:13:50.523" UserId="223" />
  <row Id="6835" PostId="4555" Text="But in short, it is most problematic when probabilities observed are low or high as the range of values are constricted and unable to meet Gaussian assumptions. " CreationDate="2010-11-17T15:14:31.483" UserId="196" />
  <row Id="6836" PostId="4632" Text="As it happens, I am using this text in a course that I am TAing. The students hate this book. Stock and Watson is a common intro book, with Wooldridge's Introductory Econometrics a bit more sophisticated, but very clear." CreationDate="2010-11-17T15:16:27.550" UserId="401" />
  <row Id="6837" PostId="4636" Text="@drknexus Adding up to 100 is an issue which is mitigated to some extent if you do online surveys where the software can check if the total adds up to 100. Not being able to tell the difference between 90% and 95% is measurement error and that can be dealt with if necessary. The question form I suggested is one of the standard question types used in market research but has not been used much in the past because of the issue about totals that you mentioned. I feel that the issue of totals is much less pertinent in online surveys." CreationDate="2010-11-17T15:24:35.387" />
  <row Id="6838" PostId="4613" Text="@robin: Thanks for that. Probabilities of probabilities - that may take a little while for me to get my head around! Final question: is the second equation effectively a simplified version of the first (which is always better)?" CreationDate="2010-11-17T15:38:30.503" UserId="1810" />
  <row Id="6839" PostId="4610" Text="Heard of it, but in this case, not relevant - application code is C++, the OpenFOAM CFD package, and runs on a super computer." CreationDate="2010-11-17T15:44:24.450" />
  <row Id="6840" PostId="4636" Text="I seldom think being a &quot;standard&quot; makes something a defensible practice.  That being said, I suppose even in offline surveys I suppose participants responses can be rescaled to total to 100 if they make some sort of math error." CreationDate="2010-11-17T15:53:21.347" UserId="196" />
  <row Id="6841" PostId="4631" Score="1" Text="But what if an individual goes to more than one coffee house?  Or what if Person A goes once per month to five different coffeehouses, and Person B goes five times per month to just one coffeehouse?" CreationDate="2010-11-17T16:03:20.050" UserId="71" />
  <row Id="6842" PostId="4639" Score="6" Text="the help in R is meant to explain you how to use the function. It's not meant to be a course on statistics. And regarding that, in general I believe the R help pages are among the most complete and handy from all open source packages I know of. And paying packages for that matter. SPSS and SAS give you a lot of mumbo-jumbo with half-truths and complete nonsens as a &quot;guide for interpretation&quot;. " CreationDate="2010-11-17T16:03:26.087" UserId="1124" />
  <row Id="6844" PostId="4641" Text="I like this sentiment, &quot;if I don't understand what I'm doing already, I shouldn't try to learn it...&quot; This is also the approach taken in the R help - it's not helpful unless you already know what's going on. I was hoping this could be the start of something different. " CreationDate="2010-11-17T16:37:55.467" UserId="1994" />
  <row Id="6845" PostId="4641" Text="But I can use this part of your answer: &quot;Interprete it in exactly that way: it expresses if the model without that term is significantly different from the model with that term.&quot;  To me this means that the Pr(F) values are the significance of each of these terms, and a small value means that this variable is important. So, a good model should include the &quot;***&quot; variables and not the ones that have no stars. " CreationDate="2010-11-17T16:41:04.027" UserId="1994" />
  <row Id="6847" PostId="4626" Text="@aix: do you want to update the whole lasso path or just the solution ? (i.e. is the sparsity penalty fixed?).  " CreationDate="2010-11-17T16:57:53.280" UserId="603" />
  <row Id="6848" PostId="4634" Text="That one is a bit too superficial and 'hand-on' imho." CreationDate="2010-11-17T17:00:36.380" UserId="603" />
  <row Id="6849" PostId="4641" Score="2" Text="@gakera: You got me wrong. If you don't understand what you're doing, you should definitely try to learn it **before using it**. That means, reading up on statistics and following a course. So, a good model should include the variables that are formulated in the hypothesis. If you base yourself on the &quot;***&quot; variables, you need a thorough course on modeling first. You obviously didn't understand my last comment. Sorry for the direct communication, comes with the guy. Nothing personal." CreationDate="2010-11-17T17:01:14.027" UserId="1124" />
  <row Id="6850" PostId="4580" Text="Exact duplicate of https://stat.ethz.ch/pipermail/r-help/2010-November/259921.html" CreationDate="2010-11-17T17:03:45.620" UserId="1657" />
  <row Id="6851" PostId="4641" Text="@gakera: I updated my answer to clarify some points that are important. Mainly because you misinterpreted the part you thought you could use." CreationDate="2010-11-17T17:09:10.180" UserId="1124" />
  <row Id="6852" PostId="4634" Score="1" Text="@user603 The question doesn't specify anything like that; I also don't see how it's &quot;superficial&quot;.  It is very high level and shows how to do analysis in R.  I wouldn't recommend it as a textbook for an econometrics course, but it certainly serves a purpose." CreationDate="2010-11-17T17:18:05.800" UserId="5" />
  <row Id="6853" PostId="4605" Text="Course parallelization would be something like independent runs of a MCMC change., i.e. very large chucks that can be ran in parallel without syncing threads.  An example of fine grain is computing the likelihood where calculations can be performed on the data points independently." CreationDate="2010-11-17T17:23:46.640" UserId="1364" />
  <row Id="6855" PostId="4626" Text="I was looking to update the entire path. However, if there's a good way to do it for a fixed penalty ($\lambda$ in the formula below), this may be a good start. Is this what you are proposing?&#xA;&#xA;$$ \hat{\beta}^{lasso} = \underset{\beta}{\operatorname{argmin}} \left \{ {1 \over 2} \sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p x_{ij} \beta_j)^2 + \lambda \sum_{j=1}^p |\beta_j| \right \} $$" CreationDate="2010-11-17T17:48:12.177" UserId="439" />
  <row Id="6856" PostId="4631" Text="I agree with Matt, and this is probably not the best measure per say to assess market share (particularly as people have less loyalty). But this approach is enlightening as to consumers preferences. It will diverge from market share depending on how much people choose multiple vendors and if frequency of coffee buying varies between preferred vendors (i.e. do people who prefer A overall drink more coffee than people who prefer B.)" CreationDate="2010-11-17T18:05:13.360" UserId="1036" />
  <row Id="6857" PostId="4613" Text="@Noldorin the first is not always better (take $p=0$). " CreationDate="2010-11-17T18:15:48.640" UserId="223" />
  <row Id="6858" PostId="4641" Text="I'm learning by doing, this is homework after all, nobody is going to die if I don't get this right - the fish are already dead :P    Thanks for the help so far, and don't worry, this is not my first time on the internet :)" CreationDate="2010-11-17T18:16:07.590" UserId="1994" />
  <row Id="6859" PostId="4644" Text="@gakera [Practical Regression and Anova using R](http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf) is a good starting point for understanding linear models, and methods related to variables/model selection. As pointed by @Joris, stepwise regression is rarely the panacea." CreationDate="2010-11-17T18:33:24.510" UserId="930" />
  <row Id="6860" PostId="4644" Text="hah, thanks for adding the links @chl while maintaining my disclaimer as to why I can't post them. You must agree that I suck :D" CreationDate="2010-11-17T18:36:29.607" UserId="1994" />
  <row Id="6861" PostId="4381" Text="Nice set of lectures indeed, thanks!" CreationDate="2010-11-17T18:47:28.350" UserId="439" />
  <row Id="6862" PostId="4596" Text="Sure. The datasets I am currently working with are ~14 gigs " CreationDate="2010-11-17T18:55:40.557" UserId="1451" />
  <row Id="6863" PostId="4644" Score="1" Text="@gakera I think you need to have more rep to add more than one link per edit -- I can understand this is not very pleasant when starting on an Q&amp;A website. I was assuming that you would remove your last sentence yourself. On the other hand, I feel you shouldn't expect too much upvotes for providing an answer to your own question, since it is a sort of recap' (useful, though)." CreationDate="2010-11-17T19:02:39.000" UserId="930" />
  <row Id="6864" PostId="4645" Text="I didn't fully digest the &quot;multiple choice&quot; aspect of the question previously. I would look further into the Van Westendorp methods if that is what you are limited to..." CreationDate="2010-11-17T19:27:30.327" UserId="696" />
  <row Id="6865" PostId="4644" Text="I'm not doing this for upvotes (that's so Reddit :P  ) - useful recap is exactly what I'm going for - mainly for myself but probably useful for others as well. " CreationDate="2010-11-17T20:03:14.960" UserId="1994" />
  <row Id="6866" PostId="4621" Text="You could check what predictions it gives to your training data. If loss goes down to zero, all the instances should be classified perfectly" CreationDate="2010-11-17T20:33:26.607" UserId="511" />
  <row Id="6867" PostId="4642" Text="I wouldn't call this data &quot;malformed&quot; - it just has a zero-frequency cell, which in large part is due to the effect being large. From the point of view of the application this is a &quot;good thing&quot;." CreationDate="2010-11-17T21:14:34.500" UserId="279" />
  <row Id="6868" PostId="4622" Text="@user603 (welcome back :) I've +1, but I still like Greene's book (maybe just because I'm not an econometrician and it highlights another approach compared to that found in psychophysics or psychometrics)" CreationDate="2010-11-17T21:19:46.753" UserId="930" />
  <row Id="6869" PostId="4650" Text="Thanks Raskolnikov dont worry about time or the &quot;market&quot; it could be a five sided dice if you prefer. I'm just trying to understand the math of combining 2 (or 20) x 5 sided dice showing the same side." CreationDate="2010-11-17T21:45:48.937" />
  <row Id="6870" PostId="4650" Text="So, you are really asking about what happens when you have a lot of variables that behave in the same way? You want to know how the sum behaves in particular, right? If it is for dice, you might want to look up the multinomial distribution then." CreationDate="2010-11-17T21:51:06.840" UserId="2036" />
  <row Id="6871" PostId="4650" Text="Raskolnikov please ignore my erroneous use of %.  If i have two five sided dice what is the probability of both dice having the same outcome? And what would the math be if I had 3 dice etc" CreationDate="2010-11-17T21:57:20.043" />
  <row Id="6873" PostId="4650" Text="Sorry our posts are crossing. &quot;multinomial distribution&quot; errr ok" CreationDate="2010-11-17T21:58:28.047" />
  <row Id="6875" PostId="4650" Text="Ok if I play a game whith a dice such that when the side with 5 on lands upper most I loose. I can play the game with more dice but if they all show the same side &quot;5&quot; I loose. Any other combination I win or at least dont loose.How can I express or calculate the difference?" CreationDate="2010-11-17T22:14:31.630" />
  <row Id="6876" PostId="4650" Text="Brilliant so with more dice you simply multiply the possible outcomes.  If you want to express the probability of faces 1,2 and 3 landing face up how would you do so?  This is excellent Raskolnikov I'm learning thank you." CreationDate="2010-11-17T22:28:05.543" />
  <row Id="6877" PostId="4650" Text="Just the product of the probabilities for one die: $1/125$. What makes it different when two or more die come out the same, is that you have to account for the different possible arrangements of these outcomes. Since you're only interested in what comes out and not in what die gives what number, the probabilities are higher, like in the example I gave with two ones and one two." CreationDate="2010-11-17T22:29:29.380" UserId="2036" />
  <row Id="6878" PostId="4586" Text="I see. Yes, this works. Thanks." CreationDate="2010-11-17T22:33:14.103" UserId="847" />
  <row Id="6879" PostId="4650" Text="You are very kind thank you very much. Rigorous understanding. Excellent. Kind regards Medius." CreationDate="2010-11-17T22:41:44.887" />
  <row Id="6880" PostId="4642" Text="@Aniko : I agree, malformed is a wrong word, but I didn't really know how to say it different." CreationDate="2010-11-17T23:12:13.627" UserId="1124" />
  <row Id="6881" PostId="4646" Text="Thanks for your reply.  I've done (1) and it seems pretty reasonable.  As for dealing with the presence of C - I neglected to mention that the number of flips, or number of heads, of C should be fairly constant, which is why I thought including the intercept term might work." CreationDate="2010-11-17T23:17:47.937" UserId="1777" />
  <row Id="6882" PostId="4598" Text="In the trading world a moving standard deviation is known as Bollinger Bands. A web query on this will generate thousands of hits." CreationDate="2010-11-17T23:53:00.533" UserId="226" />
  <row Id="6883" PostId="4635" Text="(+1) Worth reading, yes, entertaining, yes. But a textbook? Not exactly... I take the subtitle to mean it's intended as a *companion* to a textbook. Andrew Gelman's review of it is worth reading too: http://www.stat.columbia.edu/~gelman/research/published/angristpischke2.pdf" CreationDate="2010-11-18T00:56:19.357" UserId="449" />
  <row Id="6886" PostId="4649" Text="Not clear what you are asking...what do you want to add another market to?" CreationDate="2010-11-18T01:51:40.613" UserId="511" />
  <row Id="6888" PostId="4635" Text="@onestop: That's a fair point; I wouldn't regard this to be a textbook, but worth considering for someone wanting to learn Econometrics, regardless." CreationDate="2010-11-18T01:58:13.210" UserId="5" />
  <row Id="6889" PostId="4567" Text="@Whuber Have a look at the Goodman paper. It accords pretty well with my experience in the field of pharmacology.&#xA;Methods say &quot;Results where P&lt;0.05 were taken as statistical significant&quot; and then results are presented with + for p&lt;0.05, ++ for p&lt;0.01 and +++ for p&lt;0.0001. The statement implies the control of error rates a la Neyman and Pearson, but the use of different levels of p suggest Fisher's approach where the p value is an index of the strength of evidence against the null hypothesis. As Goodman points out, you cannot simultaneously control error rates and assess strength of evidence." CreationDate="2010-11-18T02:41:07.073" UserId="1679" />
  <row Id="6890" PostId="4655" Score="2" Text="Thanks for the clarification and for encouraging me to read up, think a bit, and revise my question. It led me to the Merkle reference. Additional feedback appreciated." CreationDate="2010-11-18T03:03:03.407" UserId="1381" />
  <row Id="6891" PostId="4549" Text="@Tal:  I really like ez for most of my ANOVA analyses, but I couldn't see a way to use it for contrast codes; I think I've seen that the author uses this site, perhaps he'll consider adding it to his wrapper." CreationDate="2010-11-18T03:14:19.860" UserId="196" />
  <row Id="6892" PostId="4655" Score="1" Text="(+1) wow, that's great improvement.  I'll delete my earlier comment." CreationDate="2010-11-18T04:07:55.930" UserId="1108" />
  <row Id="6894" PostId="4605" Text="Alright, thanks!" CreationDate="2010-11-18T08:23:40.263" UserId="2006" />
  <row Id="6895" PostId="4625" Score="1" Text="Thanks Graham, I was looking everywhere for an option to select, but I see you need to double click on the output and a model viewer comes up with the information. Thanks." CreationDate="2010-11-18T08:41:52.240" UserId="2025" />
  <row Id="6897" PostId="4664" Text="Thanks, indeed it is almost the same question, but on the kurtosis instead of the skewness." CreationDate="2010-11-18T08:54:27.743" UserId="1709" />
  <row Id="6898" PostId="4653" Text="@user2040 It is not the place for asking about complementary information. Just put it as a comment under @whuber's response (see &quot;add comment&quot;)." CreationDate="2010-11-18T09:25:53.137" UserId="930" />
  <row Id="6899" PostId="4644" Text="@gakera I'm sure this was not for getting upvotes. Most of the times, we set our own response as Community Wiki (CW), when they don't add further or contradictory information. This is a neutral way to sum up or aggregate others' responses." CreationDate="2010-11-18T09:30:52.637" UserId="930" />
  <row Id="6901" PostId="4666" Score="2" Text="With respect, this looks like a direct copy-and-paste from Wikipedia, and doesn't really answer the question." CreationDate="2010-11-18T09:51:13.590" UserId="439" />
  <row Id="6902" PostId="4639" Score="1" Text="This question has been downvoted. I didn't intend to give my +1, but it seems to me now that voting it down is not very constructive: (1) the OP makes clear this is homework and uses an R built-in data set for illustration, not his data, (2) a related question with `step()` has been rated +2 at the time of this writing (so why?!), (3) the OP acknowledged the usefulness of @Joris's response." CreationDate="2010-11-18T09:52:35.903" UserId="930" />
  <row Id="6903" PostId="4644" Text="ah, I didn't notice that option, sorry. I changed this to CW. I assumed everybody could edit, since you could. " CreationDate="2010-11-18T09:53:38.737" UserId="1994" />
  <row Id="6904" PostId="4651" Text="@david Answer to a similar question has been provided here, http://stats.stackexchange.com/questions/4009/intra-class-correlation. BTW, I think you meant &quot;inter-observer variability&quot;, not &quot;intra&quot;." CreationDate="2010-11-18T09:58:54.467" UserId="930" />
  <row Id="6905" PostId="4667" Text="@Tal Which other languages are you interested in?" CreationDate="2010-11-18T10:00:34.527" UserId="930" />
  <row Id="6906" PostId="4667" Text="Dear chl - any of them.  My real motivation is to find bloggers in other languages, but I would feel fine with other websites from which to find bloggers (see more about my motivation here:  http://www.r-bloggers.com/r-bloggers-now-in-your-language/)" CreationDate="2010-11-18T10:01:50.150" UserId="253" />
  <row Id="6907" PostId="4666" Text="@mariana (-1) At the very least, you should acknowledge the quoting from Wikipedia, § on LASSO method at http://en.wikipedia.org/wiki/Least_squares!!! BTW you forgot to paste the 11th reference." CreationDate="2010-11-18T10:15:46.470" UserId="930" />
  <row Id="6908" PostId="4644" Text="@gakera No, one needs enough rep to edit others' post, see the [FAQ](http://stats.stackexchange.com/faq). Good to hear you edited your post yourself. Welcome to CrossValidated!" CreationDate="2010-11-18T10:20:37.293" UserId="930" />
  <row Id="6909" PostId="4666" Text="I forgot to put the link, it is true, but anyway I think that is a good reply for this questions. Sorry if I made you think I wrote that" CreationDate="2010-11-18T10:25:34.123" UserId="1808" />
  <row Id="6910" PostId="4668" Text="Can you be more explicit? Are you looking for non linear relationship between two time series?" CreationDate="2010-11-18T10:30:18.847" UserId="1709" />
  <row Id="6911" PostId="4668" Text="Can you make this question clearer by providing more context -- this is about non-linear ts, mutual information or differentiation of nonlinear and linear relations? What plot? " CreationDate="2010-11-18T10:30:22.097" UserId="88" />
  <row Id="6912" PostId="4639" Text="@chl : seems like I'm not the only one with sensitive toes when it comes to the R help pages :-). But I agree wholeheartedly with you. The question is valid, asked in a clear way and hence there is absolutely no reason whatsoever to downvote it." CreationDate="2010-11-18T10:31:00.360" UserId="1124" />
  <row Id="6913" PostId="4667" Text="@Tal The only French blogger I know is writing in... English (and isn't very productive anymore :) http://zoonek.free.fr/" CreationDate="2010-11-18T10:43:00.607" UserId="930" />
  <row Id="6914" PostId="4630" Score="1" Text="Whuber and onestop thanks for the answers. Now I understand how to check that my marginal distributions X, Y and Z are uniform. However to prove that the joint distribution of (X,Y,Z) is uniform in R^3 we have to prove, that (X,Y,Z) are independent. So the question is: by given sample (X_k, Y_k, Z_k) how can I check whether random variables (X,Y,Z) are independent? Thanks!" CreationDate="2010-11-18T10:43:28.767" UserId="2029" />
  <row Id="6915" PostId="4667" Text="Dear chl, I actually got his permission some time ago to add him to R-bloggers.  I do hope he'll go back to writing one day.  Thanks for the pointers so far." CreationDate="2010-11-18T10:45:25.073" UserId="253" />
  <row Id="6916" PostId="4670" Text="Wonderful link to that blogger - thank you aix !" CreationDate="2010-11-18T10:47:09.690" UserId="253" />
  <row Id="6917" PostId="4639" Text="Heh, I'm sorry if I stepped on your toes with my snide at the help, I'm just not very patient when it comes to anything with a command line really. I'm weird that way, I know. You wouldn't be the first ones to call me out on it :) I like this place, people are honest. " CreationDate="2010-11-18T10:49:15.977" UserId="1994" />
  <row Id="6918" PostId="4666" Text="@mariana It would be more helpful to refer to [The Lasso Page](http://www-stat.stanford.edu/~tibs/lasso.html) in that case. Now, the question is about pros and cons of LAR and Lasso, not about what Lasso actually does. The LARS algorithm might easily be modified to produce solutions for other estimators, like the Lasso; it works well in the $n\ll p$ case, but it is sensitive to the effects of noise (because it is based upon an iterative refitting of the residuals), as quoted from http://scikit-learn.sourceforge.net/modules/glm.html." CreationDate="2010-11-18T10:50:03.287" UserId="930" />
  <row Id="6919" PostId="4639" Text="There we go, I edited the question so that it's not as off-putting to advocates of R and the R help :) And reworded the question on AIC to avoid misleading OP only readers. " CreationDate="2010-11-18T10:55:24.253" UserId="1994" />
  <row Id="6920" PostId="4668" Text="NO SINGLE TIME SERIES ONLY " CreationDate="2010-11-18T11:29:02.150" />
  <row Id="6921" PostId="4668" Text="I WANT TO IDENTIFY A TIME SERIES, WHETHER IT IS A NONLINEAR ONE OR LINEAR ONE, FOR THAT I HAVE TO USE MUTUAL INFORMATION WHICH IS JUST LIKE AUTO CORRELATION FUNCTION,  BUT IT WILL GIVE BOTH THE NONLINEAR AND LINEAR RELATIONSHIPS. SO I WANT TO INTERPRET THAT. PLEASE HELP ME   " CreationDate="2010-11-18T11:32:09.027" />
  <row Id="6922" PostId="4668" Text="I am concerned about mutual plot" CreationDate="2010-11-18T11:37:01.973" />
  <row Id="6923" PostId="4668" Text="and iam asking about the nonlinear relationship" CreationDate="2010-11-18T11:37:26.433" />
  <row Id="6924" PostId="4664" Text="I meant L-kurtosis, but somehow wrote L-skewness..." CreationDate="2010-11-18T12:46:07.420" UserId="449" />
  <row Id="6925" PostId="4630" Text="Good question (i.e. i can't immediately think of an answer)." CreationDate="2010-11-18T12:48:13.253" UserId="449" />
  <row Id="6926" PostId="4673" Text="No, i wish to construct a function of one variable. Something like &quot;ff&lt;-function(x) 3.23+2.78x-3.45x^4&quot;." CreationDate="2010-11-18T12:49:55.920" UserId="2043" />
  <row Id="6927" PostId="4676" Text="@Chase (+1) Good to know, thanks." CreationDate="2010-11-18T13:42:05.983" UserId="930" />
  <row Id="6928" PostId="4673" Score="1" Text="@Roah Did you try the answer? Because that's exactly what it will give you (you won't see the coefficients, but why do you care about the internals of the function?" CreationDate="2010-11-18T13:54:09.783" UserId="279" />
  <row Id="6929" PostId="4673" Text="@Roah - Could you explain in what way the function I suggested does not do what you expect? It certainly accepts scalars so you can call it with a single value. Is the return value not what you are expecting?" CreationDate="2010-11-18T13:55:40.213" UserId="439" />
  <row Id="6930" PostId="4567" Text="@Michael There are alternative, more generous interpretations of that kind of reporting.  For example, the author might be aware that readers might want to apply their own thresholds of significance and therefore do the flagging of p-values to help them out.  Alternatively, the author might be aware of possible multiple-comparisons problems and use the differing levels in a Bonferroni-like adjustment.  Perhaps some portion of the blame for misuse of p-values should be laid at the feet of the reader, not the author." CreationDate="2010-11-18T14:04:04.577" UserId="919" />
  <row Id="6931" PostId="4454" Text="@Matt this question is not intended to have anything to do with programming. It's about communicating a questionnaire, strictly from a statisticians point of view, to a programmer" CreationDate="2010-11-18T14:31:27.347" UserId="305" />
  <row Id="6933" PostId="4673" Text="Sorry, aix! I was too bothered by the internal structure of the algorithm... And it is very simple! Thanks a lot, because i was going to parse the output..." CreationDate="2010-11-18T15:04:00.280" UserId="2043" />
  <row Id="6934" PostId="4575" Score="1" Text="Please accept my apologies for this rather infantile comment. I am not even capable of telling what I thought in this moment. " CreationDate="2010-11-18T15:08:22.290" UserId="264" />
  <row Id="6936" PostId="4614" Text="Nice, thanks for the pointer." CreationDate="2010-11-18T16:36:32.820" UserId="439" />
  <row Id="6937" PostId="4684" Text="(+1) very nice!" CreationDate="2010-11-18T16:51:58.020" UserId="1108" />
  <row Id="6938" PostId="4684" Text="I appreciate the effort. I'm going to have to really study this because it's not my &quot;native tongue&quot;. Also, I'm seeing a lot of dollar signs and formatting stuff. Is there something I don't know about that makes it look like real math?" CreationDate="2010-11-18T17:17:27.467" UserId="1270" />
  <row Id="6939" PostId="4684" Text="What happened? All of a sudden the math showed up, and typing in here got real slow." CreationDate="2010-11-18T17:25:57.910" UserId="1270" />
  <row Id="6940" PostId="1146" Text="+1 &quot;educate the public&quot; == *communicate* models. (To whom, how ? papers, pictures / infographics, interactive models ...)" CreationDate="2010-11-18T17:31:52.173" UserId="557" />
  <row Id="6941" PostId="4657" Text="Thanks for your answer, but what I am really interested in knowing is under what cases using medians rather than means provides better estimates the central tendency (median or mean) of the function." CreationDate="2010-11-18T17:55:23.833" UserId="1381" />
  <row Id="6942" PostId="4684" Text="@Mike See http://meta.stats.stackexchange.com/q/218/919 ." CreationDate="2010-11-18T18:12:21.460" UserId="919" />
  <row Id="6943" PostId="4688" Text="@aix (+1) Yes, I'm aware of that project too. But I know that those that are currently developing the `scikit.learn` are serious guys working in neuroimaging, and that the project will evolve and include ever growing features." CreationDate="2010-11-18T18:45:59.423" UserId="930" />
  <row Id="6944" PostId="4660" Score="1" Text="It seems to me that the Chi Square test will be particularly *poor* at identifying differences in tails.  If the tails are covered by many bins, then--because they are tails!--there may be few data in any of the bins, invalidating the chi-squared approximation.  If the tails are covered by few bins, then you lose almost all power to discriminate their shapes, and what you do manage to discriminate might not be terribly relevant or useful.  (One problem we're up against here is that &quot;fatness of tail&quot; has not been defined, so the question is really too vague to answer well.)" CreationDate="2010-11-18T19:22:27.073" UserId="919" />
  <row Id="6945" PostId="4661" Text="@Mike Why would this family of distributions be particularly good for this problem and not some other family like the Pearson distributions?" CreationDate="2010-11-18T19:24:55.417" UserId="919" />
  <row Id="6946" PostId="4686" Text="Is anything known / can be assumed about the $b_i$'s?" CreationDate="2010-11-18T19:48:40.640" UserId="279" />
  <row Id="6947" PostId="4686" Text="Each $b_i$ is a binary bit -- that is, $b_i \in \{0,1\}$. But we don't assume anything about how the $b_i$s correlate to the costs $c_i$. " CreationDate="2010-11-18T19:50:36.507" />
  <row Id="6948" PostId="4688" Text="@chi I've decided to go with `scikit.learn` as my reference implementation (and have accepted your answer.) It looks like high-quality code." CreationDate="2010-11-18T20:27:15.820" UserId="439" />
  <row Id="6949" PostId="4691" Score="1" Text="Not sure if this helps, but this article claims using Cronbach on ordinal data will underestimate the true value: http://www.amstat.org/meetings/jsm/2007/OnlineProgram/index.cfm?fuseaction=abstract_details&amp;abstractid=308244" CreationDate="2010-11-18T20:57:36.007" UserId="1118" />
  <row Id="6950" PostId="4660" Text="@whuber, I can't say whether I concur with your comment because I don't fully understand one of your points.  What do you mean exactly by &quot;invalidating the chi-squared approximation&quot;? " CreationDate="2010-11-18T21:04:02.210" UserId="1329" />
  <row Id="6951" PostId="4679" Text="I know those look almost the same, but if I substitute y for n-x, and if I take the Beta pdf and substitute x for a-1 and y for b-1 I get an extra factor of (x+y+1), or n+1. i.e. (x+y+1)!/x!/y!*p^x*q^y. That seems to be enough to throw me off." CreationDate="2010-11-18T21:10:19.257" UserId="1270" />
  <row Id="6952" PostId="4686" Text="are the costs $c_i$ known _a priori_?" CreationDate="2010-11-18T21:12:53.987" UserId="795" />
  <row Id="6953" PostId="4686" Text="Yes, we know the $c_i$." CreationDate="2010-11-18T21:20:21.137" />
  <row Id="6954" PostId="4693" Score="1" Text="No, that doesn't seem right. Suppose we have many $c_i = \epsilon$, and then some more expensive ones: a greedy sampling procedure would buy -all- of the cheap bits before the expensive ones, but there is no reason to buy all of the cheap bits -- we quickly have high accuracy there." CreationDate="2010-11-18T21:21:45.317" />
  <row Id="6955" PostId="4679" Text="Maybe somebody will chime in with a full response, but in an &quot;intuitive&quot; explanation we can always hand-wave away constants (like $n+1$) that do not depend on the variables of interest ($x$ and $p$), but are required to make the pdf add/integrate to 1. Feel free to replace the &quot;equality&quot; signs with &quot;proportional to&quot; signs." CreationDate="2010-11-18T21:37:02.647" UserId="279" />
  <row Id="6956" PostId="4679" Text="Good point. I think I'm getting closer to an understanding. I'm still trying to be able to say what x tells you about the p distribution, and why those two cdfs sum to 1." CreationDate="2010-11-18T21:43:00.413" UserId="1270" />
  <row Id="6957" PostId="4693" Score="1" Text="You have high accuracy only if $b_i$ are iid. If anything goes, then the only way to make inference is to almost exhaust the population." CreationDate="2010-11-18T21:45:26.840" UserId="279" />
  <row Id="6958" PostId="4693" Text="The $b_i$ can be set by any arbitrary (adversarial) process, but must be fixed before we begin sampling. At that point, they just form a fixed population, and we can, for example, sample uniformly at random among the individuals with low $c_i$ to obtain high accuracy among one subset of the population. " CreationDate="2010-11-18T21:52:09.380" />
  <row Id="6959" PostId="4691" Text="@Christopher (+1) Thanks for this link." CreationDate="2010-11-18T22:18:27.367" UserId="930" />
  <row Id="6960" PostId="4658" Text="@fRed Is the &quot;fat-tail&quot; tag really meaningful (for future questions)?" CreationDate="2010-11-18T22:22:31.310" UserId="930" />
  <row Id="6961" PostId="4693" Text="I'm starting to think I do not understand the question. Is it the case that we have observations of $m$ individuals, each with $n$ bits of data associated? And then we can sample one bit of each individual with a known cost? Otherwise, I don't understand the 'subset of the population' part." CreationDate="2010-11-18T22:23:15.677" UserId="795" />
  <row Id="6962" PostId="4653" Text="That link is not available as it was on this one..... I was confused where to add comments. Not sure why I dont see that link." CreationDate="2010-11-18T22:32:27.547" UserId="2040" />
  <row Id="6963" PostId="4693" Text="@Frank That's right, but if there's any association between the values of the bits and the costs, then your estimate based on sampling only the low-cost bits will be biased and thereby might *never* attain the targeted low error rate (except, as @Aniko points out, when this almost exhausts the population).  The way out is to sample the bits with probabilities that depend on the costs (and, as it turns out, on the value of k) and to use an estimator that accounts for the differing probabilities." CreationDate="2010-11-18T22:42:20.207" UserId="919" />
  <row Id="6964" PostId="4660" Text="The chi-squared test is based on a Normal-theory approximation to the true distribution of the chi-squared statistic.  Typically this approximation gets poor when bin populations drop below 5." CreationDate="2010-11-18T22:47:14.900" UserId="919" />
  <row Id="6965" PostId="4679" Text="@Aniko I take a different view of &quot;intuitive&quot; explanations.  In some cases we don't care too much about constants, but in this case the *crux* of the matter is to see why an n+1 appears and not an n.  If you don't understand that then your &quot;intuition&quot; is incorrect." CreationDate="2010-11-18T22:49:23.960" UserId="919" />
  <row Id="6966" PostId="4693" Text="@Shabbychef Think of a box containing n envelopes.  On the outside of each envelope is written a cost c_i and inside the envelope is a slip of paper on which is written either a zero or a one: the b_i.  You are allowed to peek into the box where you can see all the c_i, but in order to see any of the b_i you need to pay the value of c_i written on its envelope.  The objective is to obtain an acceptably accurate estimate of the total (B) of all the zeros and ones in the box and to do so by paying as little as possible." CreationDate="2010-11-18T23:10:05.737" UserId="919" />
  <row Id="6967" PostId="4684" Text="Boy, you're giving me a lot to chew on" CreationDate="2010-11-18T23:34:14.843" UserId="1270" />
  <row Id="6968" PostId="4660" Text="@whuber, thanks for the explanation.  In view of it, I feel like your initial comment's first phrase may not be as nuanced as you may have cared to (&quot;the Chi Square test will be particularly poor at identifying difference in tails&quot;).  Maybe the more appropriate statement would have been &quot;it depends...&quot;  This test has several merits, including forcing you to define the relevant bins.  And, just as importantly facilitate the construction of a histogram.  Granted if you have fewer than 5 observation in a bin, you will lose accuracy as you well explained.     " CreationDate="2010-11-19T00:16:50.433" UserId="1329" />
  <row Id="6969" PostId="4699" Text="I should note that my problem is not 'online': I have all the data at my disposal, and do not need to e.g. identify the first time my process goes haywire." CreationDate="2010-11-19T01:36:32.127" UserId="795" />
  <row Id="6970" PostId="4693" Text="@whuber: this is what I thought. However, I am still confused by 'subset of the population'." CreationDate="2010-11-19T01:38:26.837" UserId="795" />
  <row Id="6971" PostId="1036" Text="This book it excellent. It's short and practical." CreationDate="2010-11-19T01:48:48.337" UserId="1146" />
  <row Id="6972" PostId="4684" Text="good description!" CreationDate="2010-11-19T02:22:47.770" UserId="1307" />
  <row Id="6974" PostId="4635" Text="I would say this coupled with Wooldridge's Econometric Analysis of Cross Section and Panel Data would provide the most updated perspective." CreationDate="2010-11-19T03:02:32.527" UserId="96" />
  <row Id="6975" PostId="4615" Text="Greene's book is encyclopedic and a must-have reference.  But I have two quibbles: (i) I feel it is a bit out of date relative to current econometric practice; (ii) the presentation is very non-linear, but at the same time, in his exposition Greene will use symbols that were defined maybe 100 pages prior in a completely different context, making things horribly confusing at times.  As mentioned below, I would vote for Wooldridge coupled with Angrist and Pischke for an up to date and thorough perspective." CreationDate="2010-11-19T03:05:59.497" UserId="96" />
  <row Id="6976" PostId="4617" Text="My sense is that this, coupled with Angrist and Pischke, would provide the most up to date perspective, although both are geared more toward micro analysis, rather than the types of time series or time-series cross-section analysis that macro analyses entail." CreationDate="2010-11-19T03:09:27.527" UserId="96" />
  <row Id="6977" PostId="4567" Score="1" Text="@Whuber I agree entirely, but only that what you suggest is true in some small fraction of cases (a restricted version of 'entirely'). There are some journals that specify that p values should be reported at one, two or three star levels rather than exact values, so those journals share some responsibility for the outcome. However, both that ill-considered requirement and the apparently naive use of p values might be a result of the lack of a clear explanation of the differences between error rates and evidence in the several introductory statistics texts that are on my shelves." CreationDate="2010-11-19T03:40:25.297" UserId="1679" />
  <row Id="6978" PostId="4702" Text="+1: very nice link! I guess the definition also varies depending on the field (e.g. #4 is very mathematical/statistical, but #1 and #2 are more &quot;understandable&quot; from a life science point of view)" CreationDate="2010-11-19T06:39:42.860" UserId="582" />
  <row Id="6980" PostId="4588" Text="@chi Thanks very much for the interpretation. That makes sense." CreationDate="2010-11-19T07:21:09.993" UserId="1862" />
  <row Id="6981" PostId="4665" Score="1" Text="@mariana At risk of being repetitive, could you pls give a proper citation to Wikipedia when you copy/paste entire paragraphs on Lasso (http://en.wikipedia.org/wiki/Least_squares)?" CreationDate="2010-11-19T07:54:19.280" UserId="930" />
  <row Id="6982" PostId="4658" Text="@chl You tell me, I am certainly not as experienced as you in statistics. But IMO it is a classic bias to underestimate the importance of tails. Have you read the work of Mandelbrot? Fat tails are very important in applied statistics for finance and the credit crisis of 2008 came for some part from some pricing models that were assuming normality and underestimating the fat tails of some correlation distribution. We can discuss that in another thread :)" CreationDate="2010-11-19T08:23:32.003" UserId="1709" />
  <row Id="6983" PostId="4570" Text="If you use Bayesian regression then the error bars also indicate the uncertainty due to the finite nature of the dataset (given the prior), you only trust the model as far as the error bars suggest you ought to trust it. If you don't have enough data to make a useful inference it will generally be evident in the posterior distributions for the parameters and/or predictions.  The usual frequentist error bars will probably say pretty much the same thing.  At the end of the day, sometimes only a small dataset is available, it just limits the confidence in your conlcusions." CreationDate="2010-11-19T09:14:11.553" UserId="887" />
  <row Id="6984" PostId="4570" Text="Agreed for the Bayesian regression. Thanks for pointing that out. But if you have two points that form a straight line, how do you calculate the frequentist error bars? And let's say that you have enough points to calculate the frequentist error bars, from how many points can you trust them (should we use the error bars of the error bars?)" CreationDate="2010-11-19T09:26:57.503" UserId="1709" />
  <row Id="6985" PostId="4658" Text="@fRed It was just a question :) If you feel it's an interesting and relevant tag, let's go with it (BTW, I don't know anything about quantitative finance)." CreationDate="2010-11-19T09:30:54.700" UserId="930" />
  <row Id="6986" PostId="4570" Text="Sorry, I am not that familiar with the limitations of frequentist error bars (I only mentioned them so as not to appear bigoted ;o).  IIRC frequentist error bars for ridge regression are very similar to the Bayesian ones for a fixed value of the ridge parameter (a full Bayesian analysis would marginalise over the ridge parameter as well).  " CreationDate="2010-11-19T09:50:49.527" UserId="887" />
  <row Id="6989" PostId="4689" Text="Sorry, I don't understand what you mean by the second sentence. Would you try and reword it?" CreationDate="2010-11-19T10:06:13.320" UserId="8" />
  <row Id="6990" PostId="4668" Text="There is no reason to shout or click everything around, I'm here to help. Click edit below your question and fix it, then your question will jump to the top of question list and because it will be hopefully clear then your chances to get an answer will significantly increase." CreationDate="2010-11-19T10:26:21.877" UserId="88" />
  <row Id="6991" PostId="4606" Text="Here is the link to implementation in R (http://www-etud.iro.umontreal.ca/~botev/kde.R). I understand your question now. Mine is why you want to do that ? I would perform a MC as described to get an estimate of the error of the kde, but not to calculate a kde for NB, which uses maximum likelihood estimates of all variables per definition ! I think that performing the algorithm as described by you and calculating the kde directly from the training data (without any subsampling) will not affect the result of NB." CreationDate="2010-11-19T10:43:11.947" UserId="264" />
  <row Id="6992" PostId="4685" Text="You may need to explain what you mean by soft and hard EM, or provide a link." CreationDate="2010-11-19T11:02:03.530" UserId="495" />
  <row Id="6993" PostId="4689" Text="ohu, i'm just joined the world of statistic and machine learning, sorry i didin't find out how to link unsupervised learning with decision theory. but i'm still studing!" CreationDate="2010-11-19T11:23:27.670" UserId="2046" />
  <row Id="6994" PostId="2365" Text="I was recently given a copy of Felsenstein's book.  In chapter 18 he does not say why you can't use an improper flat prior on 0-infinity.  Neither does he mention MaxEnt or transformation groups in his criticism of uniformative priors.  While the rest of the book may be very good; this suggests inadequate scholarship on that particular issue.  Caveat lector - just because something appears in a text book or journal paper, doesn't mean that it is correct." CreationDate="2010-11-19T13:06:00.873" UserId="887" />
  <row Id="6995" PostId="4557" Text="if the noise process is skewed but homoscedastic then it is still inappropriate if you need the error bars.  " CreationDate="2010-11-19T13:10:33.167" UserId="887" />
  <row Id="6996" PostId="4570" Text="Isn't performing a test implicitly looking at the confidence intervals?  Perhaps it would be that the sin test-wise is ignoring the power of the test?" CreationDate="2010-11-19T13:11:43.460" UserId="887" />
  <row Id="6997" PostId="4711" Text="Than you for comment. " CreationDate="2010-11-19T13:17:10.103" />
  <row Id="6998" PostId="4711" Text="Then, do two datasets having the same eigenvectors means they are same or similar data sets?" CreationDate="2010-11-19T13:23:03.217" />
  <row Id="6999" PostId="2365" Text="@Dikran: entropy maximization without testable information gets only one constraint: probabilities sum up to one. Most often the uniform distribution is taken there. I don't take it as granted, but I do agree with Felsensteins calculations and reasoning. So we disagree, like more people in that field. Felsenstein if far from accepted by everybody, and I'm not accepting everything he says. But on this point, I follow him. Sometimes a Bayesian approach is not superior to another one. And the case he describes is one such a case according to me. YMMV." CreationDate="2010-11-19T13:32:08.170" UserId="1124" />
  <row Id="7000" PostId="4706" Text="I think I'm going to need to see that presentation in person to get the full impact." CreationDate="2010-11-19T13:36:07.997" UserId="1036" />
  <row Id="7001" PostId="4689" Text="I'm just confused how it fits in with the question. For example, the words &quot;prediction&quot;, &quot;decision theory&quot; or &quot;unsupervised&quot; don't appear in the accepted answer" CreationDate="2010-11-19T13:43:02.123" UserId="8" />
  <row Id="7002" PostId="4703" Text="Your not wrong, but your definition of what a fixed effect is is not what I would think of when someone says fixed effect. Here is what I think of when someone says fixed effect http://en.wikipedia.org/wiki/Difference_in_differences , or this http://www.stata.com/support/faqs/stat/xtreg2.html (particularly equation 3 on the Stata page)" CreationDate="2010-11-19T13:44:08.363" UserId="1036" />
  <row Id="7003" PostId="4567" Text="@Michael All good points; thank you.  And thank you for sharing the references: they make good reading (even though the papers addressed to medical practitioners belabor their arguments)." CreationDate="2010-11-19T13:49:32.273" UserId="919" />
  <row Id="7004" PostId="4725" Score="1" Text="(+1) Nice that you cite Laura's textbook and add this complement of information about Wilson's CIs." CreationDate="2010-11-19T13:52:02.693" UserId="930" />
  <row Id="7005" PostId="4660" Text="@Gaetan I do appreciate attention to nuance, but in this case the judgment seems justified.  Compared to the many other methods available for comparing distributions, the Chi Squared test doesn't hold up well.  If you &quot;define relevant bins&quot; based on the data themselves, the test is invalid.  Also, a histogram is not usually a useful way to look at tails of distributions.  However, I'm reluctant to propose alternatives because the problem is ill-defined: just what might it mean for two distributions to have the same &quot;fatness of tails&quot;?  Kurtosis is one possibility, but it's a crude measure." CreationDate="2010-11-19T13:55:20.570" UserId="919" />
  <row Id="7006" PostId="4658" Score="1" Text="This question is potentially interesting but some clarification would be welcome.  Are you concerned about one tail or both?  How do you measure &quot;fatness&quot;?  (Are you willing to shift and rescale the two distributions to make the comparison, for instance?)  How do you measure deviations in &quot;fatness&quot;?  If you contemplate a hypothesis test, then what will the alternative hypothesis be, precisely?" CreationDate="2010-11-19T13:57:50.223" UserId="919" />
  <row Id="7007" PostId="4525" Text="Thanks. That's helpful." CreationDate="2010-11-19T14:07:55.643" UserId="183" />
  <row Id="7008" PostId="4690" Text="Nice answer. With respect to comparing canonical examples of discriminative vs generative classifiers (logistic regression and Gaussian naive Bayes respectively), I found this book chapter to be very accessible than the Ng: http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf" CreationDate="2010-11-19T14:25:56.150" UserId="1080" />
  <row Id="7009" PostId="4725" Score="2" Text="Thanks.  I would like to point out that the Wilson interval is discussed in the article that @Joris referenced." CreationDate="2010-11-19T14:35:02.550" UserId="1108" />
  <row Id="7010" PostId="4701" Text="I appreciate your take on it. All the answers are helping me to think about the question and possibly understand better what I'm asking." CreationDate="2010-11-19T14:56:34.260" UserId="1270" />
  <row Id="7011" PostId="4697" Text="Thanks -- your example is elucidating. You say that methods for solving these kinds of problems are well known -- do you have a reference that discusses a general solution?" CreationDate="2010-11-19T15:09:51.840" />
  <row Id="7012" PostId="4693" Text="shabbychef -- in the simplest case, imagine that 90% of the population has cost &quot;Low&quot; to sample from, and 10% of the population has cost &quot;High&quot; to sample from. We could sample uniformly from the Low individuals to get a high confidence within this segment of the population, and do the same thing among the High individuals, and then combine estimates. But because the &quot;High&quot; individuals represent only 10% of the population, we can tolerate higher error in this segment of the population, and so can sample them more sparingly. Our total error will be .9*error(Low) + .1*error(High). " CreationDate="2010-11-19T15:12:39.017" />
  <row Id="7014" PostId="4705" Text="@chl. Thanks a lot. I was dumb not to look at the source code. It mentions this clearly... " CreationDate="2010-11-19T15:52:11.337" UserId="1307" />
  <row Id="7015" PostId="4697" Text="@Frank This is a straightforward, if messy, application of methods of decision theory described in textbooks on the subject by E.S. Lehman (*The Theory of Point Estimation*) and Jack Kiefer (*Introduction to Statistical Inference*), among others." CreationDate="2010-11-19T16:02:02.910" UserId="919" />
  <row Id="7016" PostId="4693" Text="@Frank My second reply gives some quantitative support to your analysis.  However, the optimal solutions conform to @shabbychef's initial description: they tend to sample all the cheapest bits and then obtain a subset of the more expensive ones, which is exactly what a greedy algorithm would do." CreationDate="2010-11-19T16:14:39.450" UserId="919" />
  <row Id="7017" PostId="4711" Score="3" Text="@Peter if the eigenvectors are identical and so are their corresponding eigenvalues, then you can recreate the covariance (or correlation) matrix you used for PCA.  However, such a matrix obviously does not determine the data.  For instance, subtracting a constant from all values won't change the covariances." CreationDate="2010-11-19T16:18:17.093" UserId="919" />
  <row Id="7018" PostId="4711" Score="1" Text="@Dikran re: your last comment.  Take any k datasets you like, each with n values, and combine them into one k-variate dataset of n records after first randomly permuting the order of each dataset.  The random permutations assure the correlation matrix will be close to the identity.  Do the same for another collection of k completely different datasets.  The two PCAs will be essentially the same, but by construction there is *no* relationship between the two collections of data.  What is common is the mutual independence of the coordinates in each k-variate combination, nothing more." CreationDate="2010-11-19T16:23:04.187" UserId="919" />
  <row Id="7019" PostId="4567" Text="@Michael: (+1). The Goodman article was especially insightful in supporting your argument. " CreationDate="2010-11-19T16:36:04.347" UserId="1118" />
  <row Id="7020" PostId="4143" Text="@suncoolsu: Hi, can you suggest some reference for Bayes Factor that you mentioned? Thanks." CreationDate="2010-11-19T16:52:57.187" UserId="528" />
  <row Id="7021" PostId="4729" Text="Being not in your area I'm not sure what you mean by &quot;each treatment should be replicated three times&quot; or the precise nature of treatment; are they between or within samples?  To me it seems practical to reduce to a treatment and control (after all more samples in a design cell the better), but that is a trade off you should determine based on your needs.  Which is more important, experimental power or evaluating both treatments?" CreationDate="2010-11-19T16:56:43.303" UserId="196" />
  <row Id="7022" PostId="4660" Text="@whuber, your comments are well put.  We just disagree on the value of histograms and actually graphing and visualizing what the tails actually look like.  It counts for a lot, and it is often much more informative than a test that gives you a statistical significance that may often be more a function of n than any real difference in the data set.  Given that, I appreciate your time on this.  And, I will not debate my points any further as I would only repeat myself.   " CreationDate="2010-11-19T17:02:33.777" UserId="1329" />
  <row Id="7023" PostId="2365" Text="I am not suggesting a Bayesian approach is any better than a frequentist one - horses for courses.  In this case it is probably transformation groups that hold the key.  It is quite possible that a prior on branch length that is invariant to the units used is equivalent to a flat prior on the probability of a change - in which case Felsensteins criticism is badly misguided.  Uninformative priors are not necessarily flat and it is inappropriate to criticize uninformative priors without mentioning the standard procedures for finding them!  Not that this means Bayesian is better, of course." CreationDate="2010-11-19T17:32:52.707" UserId="887" />
  <row Id="7024" PostId="4143" Text="@Samik. Check this out http://bit.ly/9YXqCy" CreationDate="2010-11-19T17:38:39.710" UserId="1307" />
  <row Id="7025" PostId="4660" Text="@Gaetan Just to clarify a possible misapprehension: I am not advocating not graphing or exploring the data, nor have I ever said anywhere that there is no value in graphing or visualizing data--just the opposite, in fact.  However, histograms often don't do a good job of displaying tails in data.  Probability plots, survival curves, and related approaches do much better.  A histogram is naturally associated with a chi-squared test while a probability plot is naturally associated with KS-like tests and their ilk." CreationDate="2010-11-19T17:44:22.913" UserId="919" />
  <row Id="7026" PostId="4734" Text="@Neil I don't think the preceding question has anything to do with CFA. It was clearly about EFA vs. PCA, with the possible confusion that EFA can actually be based on &quot;principal components&quot;. But maybe you could reword your question if you have a specific application in mind?" CreationDate="2010-11-19T19:18:52.010" UserId="930" />
  <row Id="7027" PostId="4733" Text="It is hard to imagine data with exponentially distributed &quot;noise&quot;, as noise is supposed to average to 0. Perhaps you could give more details, and get a more meaningful answer." CreationDate="2010-11-19T19:28:10.890" UserId="279" />
  <row Id="7029" PostId="4702" Text="Thanks - this is really helpful. " CreationDate="2010-11-19T19:52:30.100" UserId="1991" />
  <row Id="7030" PostId="4630" Text="And is it possible to check whether say (X_k) and (Y_k) are independent?" CreationDate="2010-11-19T20:53:05.280" UserId="2029" />
  <row Id="7031" PostId="4733" Text="@Denis Do you perhaps mean &quot;double exponential&quot; instead of exponential?  http://en.wikipedia.org/wiki/Laplace_distribution .  When errors have a double exponential distribution it is optimal to minimize the sum of absolute values of residuals, leading to median regression.  (Normally one would not model &quot;errors&quot; with an exponential distribution because an &quot;error&quot; should have zero expectation.)" CreationDate="2010-11-19T21:33:45.040" UserId="919" />
  <row Id="7032" PostId="4740" Text="+1.  Actually, kurtosis and skewness aren't any more of a problem to compute this way: typically, wherever division by N is called for, one divides by the sum of weights.  The median can be obtained by sorting the data, computing the cumulative sum of weights, and picking the value where half the total sum is crossed.  Weighted percentiles are obtained similarly." CreationDate="2010-11-19T21:52:07.803" UserId="919" />
  <row Id="7033" PostId="4734" Score="3" Text="http://www2.sas.com/proceedings/sugi30/203-30.pdf" CreationDate="2010-11-19T21:54:20.217" UserId="919" />
  <row Id="7034" PostId="4740" Score="2" Text="I agree that it depends entirely on what sort of weights you have. Decent statistical packages distinguish between several (sampling weights, frequency weights, inverse-variance weights...) and treat them differently." CreationDate="2010-11-19T22:04:42.683" UserId="449" />
  <row Id="7035" PostId="4720" Text="Did you really mean to say that the logit &quot;transforms the binomial distribution in a normal distribution&quot;??" CreationDate="2010-11-19T22:06:46.883" UserId="919" />
  <row Id="7036" PostId="4734" Score="1" Text="You may want to rethink the question.  Whether you put &quot;Exploratory&quot; or &quot;Confirmatory&quot; in front of it, C(ommon)FA is based on some common portion of the variance presumed to represent latent constructs.  Principal components is based on the total variance and component scores are a unique linear combination that maximizes that total variance explained by the factors.  These are different models." CreationDate="2010-11-19T22:17:32.110" UserId="485" />
  <row Id="7040" PostId="4720" Text="@whuber: nice catch of the formula, and nice catch of the formulation. Pretty much not. It makes sure the errors in a logistic regression follow the normal distribution. Thx for the correction." CreationDate="2010-11-20T00:18:56.950" UserId="1124" />
  <row Id="7041" PostId="4743" Text="That my null hypothesis is flat in $x_1$ and $x_2$ doesn't mean that it is flat in some transformed quantities which lead to my measurements being distributed flat in the available parameter space, right? Why do I not need to worry about this? And I am dealing with hundred millions of observations, so histograms are just easier to use for me." CreationDate="2010-11-20T03:07:22.160" UserId="56" />
  <row Id="7042" PostId="4693" Text="@shabbychef I believe @Frank is thinking in terms of stratifying the population according to costs, in an analogy to importance sampling.  Thus, if we preferentially sample the lowest-cost bits we might consider that we have a good idea of the contribution to B of the &quot;low cost&quot; subset of the population but only a poor idea of the contribution from the &quot;high cost&quot; subset.  This reflects a suspicion that the frequency of 1's might vary by cost." CreationDate="2010-11-20T03:46:20.780" UserId="919" />
  <row Id="7043" PostId="4693" Text="@Frank, @whuber I see it now. I had problems imagining why one would be interested in the contribution to $B$ of a sub-population. Apparently this is for good reason because the OP is interested in $B$ for the entire population." CreationDate="2010-11-20T05:11:53.033" UserId="795" />
  <row Id="7045" PostId="4597" Text="I am not an expert in computer vision, but I would bet there are better and faster algorithms to do that." CreationDate="2010-11-20T10:23:05.457" UserId="582" />
  <row Id="7046" PostId="4598" Score="1" Text="You can look at the rollapply function in the zoo package.  And use sd as the function." CreationDate="2010-11-20T11:09:54.873" UserId="253" />
  <row Id="7048" PostId="4757" Text="+1, but please http://en.wikipedia.org/wiki/Wikipedia:Don%27t_call_it_Wiki !" CreationDate="2010-11-20T13:17:24.483" UserId="449" />
  <row Id="7053" PostId="4757" Text="Point taken but, don't google it as well? :-). " CreationDate="2010-11-20T15:01:32.960" />
  <row Id="7054" PostId="4684" Text="I revised the question, if you care to take a look. Thanks." CreationDate="2010-11-20T15:02:56.967" UserId="1270" />
  <row Id="7055" PostId="4679" Text="I revised the question, if you care to take a look. Thanks." CreationDate="2010-11-20T15:03:58.323" UserId="1270" />
  <row Id="7056" PostId="4701" Text="I revised the question, if you care to take a look. Thanks." CreationDate="2010-11-20T15:05:48.727" UserId="1270" />
  <row Id="7057" PostId="4759" Score="5" Text="pretty much solved here: http://stackoverflow.com/questions/2200460/what-programming-languages-are-good-for-statistics" CreationDate="2010-11-20T16:06:51.820" UserId="22" />
  <row Id="7058" PostId="4626" Text="@aix. Yes, it all depends on the implementation you use and the facilities you have access to. For example: if you have access to a good lp solver, you can feed it with the past optimal values of $\beta$ and it'll carry the 1-2 step to the new solution very efficiently. You should add these details to your question." CreationDate="2010-11-20T16:27:51.053" UserId="603" />
  <row Id="7059" PostId="4684" Text="I'm a little slow, but you're getting through to me.&#xA;The denominator of the Beta integral is exactly what I came up with." CreationDate="2010-11-20T17:48:36.443" UserId="1270" />
  <row Id="7060" PostId="4759" Score="1" Text="Made wiki since this is entirely subjective." CreationDate="2010-11-20T19:17:39.027" UserId="5" />
  <row Id="7062" PostId="4741" Score="1" Text="What do you mean by assigning an uncertainty? Do you want something like a standard error or confidence interval for each bin? " CreationDate="2010-11-20T19:43:54.957" UserId="279" />
  <row Id="7063" PostId="4754" Text="Wow, thanks! This is a lot more efficient than recursively counting minima/maxima within a set interval... I'll have to go and check it out." CreationDate="2010-11-20T21:58:13.210" UserId="2002" />
  <row Id="7065" PostId="126" Text="This is an introductory book for people who have a decent amount of statistical background already. " CreationDate="2010-11-20T23:41:08.573" UserId="1146" />
  <row Id="7066" PostId="4762" Score="1" Text="Of course now the SPSS way can also be the R way: &quot;Starting with Version 16, SPSS offers a free plug-in that lets users run R code within SPSS having full access to the active SPSS Statistics data, and writing its output to the SPSS Statistics Viewer&quot; (http://insideout.spss.com/2009/01/13/spss-statistics-and-r/)" CreationDate="2010-11-21T01:22:33.923" UserId="196" />
  <row Id="7068" PostId="4766" Text="As a first stab, I would guess you should normalize the vectors you are projecting onto, otherwise their magnitude could screw things up." CreationDate="2010-11-21T04:50:43.183" UserId="795" />
  <row Id="7069" PostId="4768" Text="are we to suppose it does each of these with probability $1/4$?" CreationDate="2010-11-21T05:40:42.343" UserId="795" />
  <row Id="7070" PostId="4762" Text="Your link seems like it explains it fairly well, gd. It's a little verbose, but it does the job. If you really feel most comfortable in R and need the output in SPSS, DrNexus' suggestion about the SPSS-R connection is sufficient." CreationDate="2010-11-21T06:06:00.740" UserId="1118" />
  <row Id="7071" PostId="4768" Text="yes, each outcome has a 1/4 probability. " CreationDate="2010-11-21T06:10:21.813" UserId="2079" />
  <row Id="7072" PostId="4766" Text="Thanks for that.&#xA;&#xA;See &quot;A randomized algorithm for the decomposition of matrices by Per-Gunnar Martinssona, Vladimir Rokhlinb and Mark Tygert&quot; for more details" CreationDate="2010-11-21T09:51:39.353" UserId="2078" />
  <row Id="7073" PostId="4701" Text="Regarding your revision: Yes, $F\sim Beta(I+1,N-I+1)$, as long as your sampling intervals are long enough that each observation is independent and identically distributed.&#xA;&#xA;Note that if you want to be Bayesian about it and specify a nonuniform prior distribution for what you expect the actual proportion to be, you can add something else to both parameters." CreationDate="2010-11-21T11:51:19.177" UserId="2456" />
  <row Id="7074" PostId="4768" Score="5" Text="@AME : from a biological point of view, that chance is 1. The environment is bound to change to a point that no population can survive, given that in x billion years the sun is to explode. But I guess that's not really the answer he was looking for. ;-) The question doesn't make sense either. An amoebe can only divide into 2 or 0. Moral: traders shouldn't ask questions about biology." CreationDate="2010-11-21T12:34:13.677" UserId="1124" />
  <row Id="7075" PostId="4768" Score="1" Text="Such a question on interview for a such position? Maybe it is something like http://dilbert.com/strips/comic/2003-11-27/ ?" CreationDate="2010-11-21T14:02:48.010" UserId="88" />
  <row Id="7076" PostId="4773" Score="1" Text="$\sqrt{2}-1$ is 0.4142, so it is in agreement with Mike's analytical result. And +1, because I like simulations ;-)" CreationDate="2010-11-21T14:04:36.493" UserId="88" />
  <row Id="7077" PostId="4772" Score="1" Text="The reason 1 is not a root is easily seen by considering the expected number of Amoeba after $k$ steps, call it $E_k$. One can easily show that $E_k = E_1^k$. Because the probability of each outcome is $1/4,$ we have $E_1 = 3/2$, and so $E_k$ grows without bound in $k$. This clearly does not gibe with $P = 1$." CreationDate="2010-11-21T18:03:55.047" UserId="795" />
  <row Id="7078" PostId="4778" Score="6" Text="Well the original question was about R though." CreationDate="2010-11-21T18:42:31.037" UserId="334" />
  <row Id="7079" PostId="4780" Text="Actually it does a surprisingly good job. For the matrix B in my example, you can very accurately approximate the 50 singular values using the algorithm with k=20 p=35" CreationDate="2010-11-21T18:53:17.573" UserId="2078" />
  <row Id="7081" PostId="4772" Score="2" Text="@shabbychef It's not so obvious to me.  You can have the expectation grow exponentially (or even faster) while the probability of dying out still approaches unity.  (For example, consider a stochastic process in which the population either quadruples in each generation or dies out entirely, each with equal chances.  The expectation at generation n is 2^n but the probability of extinction is 1.)  Thus there is no inherent contradiction; your argument needs something additional." CreationDate="2010-11-21T21:02:38.383" UserId="919" />
  <row Id="7082" PostId="4659" Text="If &quot;what you actually wanted to know&quot; is &quot;the actual fraction of time that Foo is in execution,&quot; then you are asking about a Binomial confidence interval or a (Bayesian) Binomial credible interval." CreationDate="2010-11-21T21:12:33.740" UserId="919" />
  <row Id="7083" PostId="4659" Text="@whuber: Well I've used the random-pause method of performance tuning for over 3 decades, and some other people have discovered it too. I've told people that if some condition is true on 2 or more random-time samples, then removing it would save a good fraction of time. HOW good a fraction is what I've tried to be explicit about, assuming we don't know a Bayesian prior. Here's the general flame: http://stackoverflow.com/questions/375913/what-can-i-use-to-profile-c-code-in-linux/378024#378024 and http://stackoverflow.com/questions/1777556/alternatives-to-gprof/1779343#1779343" CreationDate="2010-11-21T21:54:34.670" UserId="1270" />
  <row Id="7084" PostId="4659" Score="1" Text="@Mike Nice idea.  The statistical assumption is that the interruption is independent of the execution state, which is a reasonable hypothesis.  A **binomial confidence interval** is a good tool to use to represent the uncertainty.  (It can be an eye-opener, too: in your 3/10 situation, a symmetric two-sided 95% CI for the true probability is [6.7%, 65.2%].  In a 2/10 situation the interval is [2.5%, 55.6%].  These are wide ranges! Even with 2/3, the lower limit is still less than 10%.  The lesson here is that something fairly rare can happen twice.)" CreationDate="2010-11-21T22:16:52.733" UserId="919" />
  <row Id="7085" PostId="4766" Text="you might add a link to the related question on stackoverflow : http://stackoverflow.com/questions/4224031/randomized-svd-singular-values/4226055#4226055 . You might also want to refer to the relevant article, as given there in my answer." CreationDate="2010-11-21T22:26:45.843" UserId="1124" />
  <row Id="7086" PostId="4659" Text="@whuber: Thanks. You're right. Something more useful is the expected value. As far as priors go, I point out that if you only see something once, it doesn't tell you much *unless* you happen to know the program is in an infinite (or exceedingly long) loop." CreationDate="2010-11-21T22:44:02.680" UserId="1270" />
  <row Id="7087" PostId="4772" Text="@shabbychef -- thanks for the edit.  I didn't realize we could use embedded TeX for math!  @whuber -- shabbychef's statement $E_k = E_1^k$ is just a variation on my statement about the extinction probability, just add expectations instead of multiplying probabilities.  Nice work, shab." CreationDate="2010-11-21T23:04:10.770" UserId="1920" />
  <row Id="7088" PostId="4782" Score="1" Text="I must say I have to disagree about R being simpler to learn than SAS. It may be because I learned SAS and SPSS first, but I think SAS, SPSS(PASW now), and Stata are all easier languages to pick up than R. It is a subjective argument though." CreationDate="2010-11-22T00:01:41.193" UserId="1036" />
  <row Id="7089" PostId="4782" Text="I feel like R gives the user so much more in terms of functionality. It goes above and beyond what SAS/SPSS can do." CreationDate="2010-11-22T00:23:18.063" UserId="2078" />
  <row Id="7090" PostId="4766" Text="added it. thanks for the reminder." CreationDate="2010-11-22T00:23:48.470" UserId="2078" />
  <row Id="7091" PostId="4782" Text="I don't disagree with that, but that doesn't make it easier to understand. I think it is pretty transparent what objects I am working with in SAS, SPSS, or Stata and the format/nature of those objects, but it isn't as transparent in R. Although R may be more cutting edge, I rarely have a need for cutting edge statistical techniques in my day to day work." CreationDate="2010-11-22T01:21:39.127" UserId="1036" />
  <row Id="7092" PostId="4787" Text="I know the various methods (gradient based) that can be applied. I am working on comparing various optimization methods for this model for a comprehensive theoretical evaluation. Therefore, I want to try LP-based methods as well." CreationDate="2010-11-22T03:43:04.873" UserId="2071" />
  <row Id="7093" PostId="4787" Text="Sounds interesting. In regards to Fisher scoring, you may want to check out &quot;Meenakshi Devidas and E. Olusegun George, Monotonic Algorithms for Maximum Likelihood Estimation in Generalized Linear Models. Sankhy\-{a}: The Indian Journal of Statistics (Series B), Vol. 61, 382--396, 1999.&quot; &#xA;" CreationDate="2010-11-22T05:21:15.817" UserId="530" />
  <row Id="7094" PostId="3763" Score="1" Text="-1; No need to be acerbic." CreationDate="2010-11-22T07:04:39.430" UserId="196" />
  <row Id="7095" PostId="4788" Text="Interior point can be applied to nonlinear problems as well. (wikipedia). This is not an exercise to find the best solution. It is more of an exercise in exploration. May be if I knew as much as you do, I won't even try it. But I don't. And this is how I will. :)" CreationDate="2010-11-22T07:50:50.423" UserId="2071" />
  <row Id="7096" PostId="4788" Text="@euphoria83, sorry I didn't mean to be obnoxious or anything. I've edited my answer." CreationDate="2010-11-22T08:54:51.373" UserId="1540" />
  <row Id="7097" PostId="4789" Score="1" Text="Well, R support are places like here, which are often more effective that a paid support. For Googling, there is rseek.org, works very nice. 99% of R-is-slow cases can be solved with some thinking; there are also packages to deal with huge data (it is not straightforward in SAS neither). R is a programming language, SAS is an extended SQL." CreationDate="2010-11-22T09:30:48.677" UserId="88" />
  <row Id="7098" PostId="4787" Text="@euphoria83: by 'LP based methods' do you just mean interior points or also the 'regular' simplex?" CreationDate="2010-11-22T10:09:30.497" UserId="603" />
  <row Id="7099" PostId="4793" Score="1" Text="+1 for good advice.  But it's unclear whether the implicit assumptions in your reply match the question: it sounds like the OP is not actually adding any constraints, but only is asking whether *methods* of constrained optimization could be used to solve the *unconstrained* problem.  (Of course my interpretation could be wrong, too...)" CreationDate="2010-11-22T15:02:29.993" UserId="919" />
  <row Id="7100" PostId="4772" Text="@Mike That's clear, Mike, but what's your point?  Aren't we talking about how to rule out 1 as a solution?  By the way, it's obvious (by inspection and/or by understanding the problem) that 1 will be a solution.  This reduces it to a quadratic equation which one can easily solve on the spot.  That's not usually the point of an interview question, though.  The asker is probably probing to see what the applicant actively knows about stochastic processes, Brownian motion, the Ito calculus, etc., and how they go about solving problems, not whether they can solve this particular question." CreationDate="2010-11-22T15:18:59.977" UserId="919" />
  <row Id="7101" PostId="4789" Score="1" Text="+1 because this answer *is* useful, but I think your points about R's support, speed, and ability to handle large data are out of date or becoming so fairly quickly." CreationDate="2010-11-22T16:00:46.713" UserId="71" />
  <row Id="7102" PostId="4797" Text="Thanks! What kinds of problem can be solved by Gibbs but not or not easily by slice sampling? What by slice but not (easily) by Gibbs sampling?" CreationDate="2010-11-22T16:05:54.157" UserId="1005" />
  <row Id="7103" PostId="4797" Text="If you can use gibbs you can use the slice as well. See the wikipedia link for an example of a slice sampler for the normal(0,1) density. An example where the gibbs is not possible is when you cannot identify the distribution of interest but the density function can be inverted." CreationDate="2010-11-22T16:24:44.930" />
  <row Id="7104" PostId="4793" Score="1" Text="@whuber: It's a bit difficult. He refers to the Interior point method, which is a constrained optimization in the sense that it constrains the search space. But penalty methods are also considered constrained methods, and actually one can say that a binary logistic regression is constrained as well, in the sense that it is built upon a set of assumptions. So I find the question of OP a bit confusing. " CreationDate="2010-11-22T16:51:40.677" UserId="1124" />
  <row Id="7105" PostId="4793" Text="@Joris Thank you: that perspective helps me appreciate your answer even more." CreationDate="2010-11-22T17:00:23.963" UserId="919" />
  <row Id="7106" PostId="4770" Score="1" Text="Thanks. I know both papers. I am not a great fan of Witten [not Whitten] et al., as i think there are more important papers on sparse decompositions. On randomized SVD, I especially like the review paper &quot;Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions&quot; (http://arxiv.org/abs/0909.4061) also coauthored by Martinsson." CreationDate="2010-11-22T17:17:49.607" UserId="30" />
  <row Id="7107" PostId="4787" Text="@user one of those. mostly interior point." CreationDate="2010-11-22T17:48:49.223" UserId="2071" />
  <row Id="7108" PostId="4793" Text="@Joris. I am not an expert in constrained methods, but from what I have read till now, the constraints in constrained optimization problems are hard inequalities defined by Ax &lt; b, where A is a matrix and b is a vector. Penalized log likelihood (I understand you mean, regularization) does not cut the space, it just modifies the contours of the objective function. The objective function is still defined for all values, even Infinity. But the hard constraints that generally are part of constrained problems will not allow all points in the space to be a valid solution." CreationDate="2010-11-22T17:56:03.370" UserId="2071" />
  <row Id="7109" PostId="4793" Text="@euphoria83 That's how I understood it as well, which is why I con't consider penalized methods constrained methods. But they tend to end up under the constrained methods in some textbooks, together with Lagrange multipliers and the likes." CreationDate="2010-11-22T18:25:41.220" UserId="1124" />
  <row Id="7110" PostId="4770" Text="i agree. i was just putting out there 2 papers no one had mentioned." CreationDate="2010-11-22T19:19:42.567" UserId="2078" />
  <row Id="7111" PostId="4801" Score="2" Text="I'm confused about what x and y are.  The first sentence characterizes them as &quot;estimates,&quot; implying they are statistics or functions thereof.  The third sentence refers to them as objects to be estimated, implying they are parameters.  Could you clarify?  Perhaps provide an example?" CreationDate="2010-11-22T19:45:08.173" UserId="919" />
  <row Id="7112" PostId="4803" Score="2" Text="Psst... you might want to ask a question in your question." CreationDate="2010-11-22T21:45:56.557" UserId="71" />
  <row Id="7113" PostId="4772" Score="1" Text="@Mike, @shabbychef: One way to rule out P=1 is to study the evolution of the probability generating function.  The pgf is obtained by starting with t (representing an initial population of 1) and iteratively replacing t by (1+t+t^2+t^3)/4.  For any starting value of t less than 1, a graphic easily shows the iterates converge to Sqrt(2)-1.  In particular, the pgf is staying away from 1, showing it cannot converge to 1 everywhere, which would represent complete extinction.  This is why &quot;the 1 isn't plausible.&quot;" CreationDate="2010-11-22T21:53:26.423" UserId="919" />
  <row Id="7114" PostId="4805" Score="2" Text="&quot;The&quot; &quot;best-fit&quot; transformation is the inverse cumulative normal integral of the percentiles (readily computed with almost anything more sophisticated than an adding machine), but I doubt this is what you are referring to.  For this problem to be well-posed, you need to stipulate (a) what family of transformations you have in mind and (b) how to measure the goodness of fit.  You are correct that this problem has been addressed, but as you might guess, it has many different solutions depending on what is assumed in (a) and (b)." CreationDate="2010-11-22T21:57:12.367" UserId="919" />
  <row Id="7115" PostId="4803" Text="Heh, sorry about that. " CreationDate="2010-11-22T22:03:39.053" UserId="996" />
  <row Id="7116" PostId="4789" Score="1" Text="I'll second @Matt and @mbq's comment about R performance, but I'd like to add that R is pretty good actually for (N)LMEs. I can remember a talk from Doug Bates at the [DSC 2009 conference](http://j.mp/ht1oTG) where he showed how the `lme4` package easily handles a lot of random effects (as encountered e.g., in educational assessment). My own (but limited) experience (SAS `NLMIXED` *vs.* R `lme4`) confirms that point: R is by no way slower than SAS when it comes to apply complex [IRT](http://j.mp/hVEMB5) models, and it handles large data genetic sets as well (thanks to clever C implementation)." CreationDate="2010-11-22T22:25:14.123" UserId="930" />
  <row Id="7117" PostId="4790" Text="@Dikran Can you expand on possible 'ensemble method' in this particular case?" CreationDate="2010-11-22T22:35:25.483" UserId="930" />
  <row Id="7118" PostId="4805" Score="2" Text="I'm assuming you're doing this so that your data might meet the assumptions of standard least-squares statistics - regression, ANOVA, etc.  Be careful here, though.  Make sure you know what the transformed values actually mean lest your inferences are wrong.  Similarly, there may be meaning in the non-normality of your data.  Perhaps a generalized linear model might be more appropriate for you." CreationDate="2010-11-22T23:49:25.247" UserId="101" />
  <row Id="7119" PostId="4805" Text="By 'SAS EM' do you mean SAS Enterprise Miner or the SAS implementation of the Expectation-Maximization algorithm?" CreationDate="2010-11-22T23:53:02.000" UserId="449" />
  <row Id="7120" PostId="4120" Text="Yes.  It does give the means." CreationDate="2010-11-23T00:15:32.120" UserId="101" />
  <row Id="7121" PostId="4805" Text="@onestop By SAS EM I mean Enterprise Miner. @jebyrnes In this case I'm looking at variables to be used in a clustering analysis. But in general I don't like to use techniques that I don't have a basic understanding of." CreationDate="2010-11-23T05:20:05.097" UserId="1951" />
  <row Id="7122" PostId="4805" Text="@whuber for (b) KS seems natural enough. There's also quantile matching. (a) I would think some linear combination of exponentiation, log, and taking to various powers, with a penalty for complexity. But I do think the details are putting the cart in front of the horse. What I want is a general discussion of the topic that may have its own recommendations for answers to your questions." CreationDate="2010-11-23T05:25:40.517" UserId="1951" />
  <row Id="7123" PostId="4813" Text="Thanks Chris.  This should be a wiki - but I don't have permissions to make it so :)" CreationDate="2010-11-23T07:53:26.577" UserId="253" />
  <row Id="7124" PostId="4812" Text="Community Wiki ? " CreationDate="2010-11-23T08:50:51.973" UserId="223" />
  <row Id="7125" PostId="4812" Text="As said - I'd love it to be." CreationDate="2010-11-23T09:01:39.483" UserId="253" />
  <row Id="7126" PostId="4812" Text="Changed the question to a CW" CreationDate="2010-11-23T10:23:23.900" UserId="8" />
  <row Id="7127" PostId="4815" Text="@fabians: thx for pointing out the obvious. I had my aha-moment when I calculated the df. I should have more coffee before I post a question. :-)  The only thing I fail to see is how the AIC ends up equal, as k is the number of parameters and not the df. Regarding the model specification: collectorID is actually nested in Crownlevel, and as far as I'm concerned one shouldn't omit a main effect when the term is included in an interaction, unless we're talking about purely nested factors. But that's not the case for Season and Crownlevel." CreationDate="2010-11-23T10:51:55.973" UserId="1124" />
  <row Id="7128" PostId="4815" Score="1" Text="i'd think you have 16 fixed effects, one for each Season:Crownlevel combination (i.e. df=16) for both models plus two variance parameters (error + ID), so k has to be the same for both (they ARE exactly the same models, just with different parameterizations...). " CreationDate="2010-11-23T11:04:58.443" UserId="1979" />
  <row Id="7129" PostId="4819" Text="Thanks mbq, but that will give us a test on the rank, not on the median - wouldn't it?" CreationDate="2010-11-23T11:07:10.563" UserId="253" />
  <row Id="7130" PostId="4820" Text="Hi psj, your answer gives a good explanation as to why I asked my question :) - exactly because I am not aware of how to generalize the sign test to unpaired data.  However, there might be a solution out there, thus my question :)" CreationDate="2010-11-23T12:05:18.437" UserId="253" />
  <row Id="7131" PostId="4819" Text="@Tal Yes, but as I wrote -- this is all one can get. Yet they both answer the same holistic question &quot;Is there a significant difference?&quot;, so both are considered nonparametric t-test variants." CreationDate="2010-11-23T12:40:28.313" UserId="88" />
  <row Id="7132" PostId="4813" Text="I'd vouch for those materials as well. They also have intro tutorials and not just multi level modelling resources." CreationDate="2010-11-23T13:16:08.913" UserId="1036" />
  <row Id="7133" PostId="4827" Text="+1 I supposed something like this may exist." CreationDate="2010-11-23T15:15:33.937" UserId="88" />
  <row Id="7134" PostId="4827" Text="@mbq: thanks for the formatting! So that's how you can LaTeXify your answers..." CreationDate="2010-11-23T15:36:54.713" UserId="1909" />
  <row Id="7135" PostId="4807" Score="1" Text="my son was born by Caesarian. I assure you it was traumatic in the casual sense of the word!" CreationDate="2010-11-23T15:56:49.947" UserId="795" />
  <row Id="7136" PostId="4819" Score="2" Text="@mbq, @Tal The Wilxocon-Mann-Whitney test is an unpaired test based on ranks.  It has two interpretations (the Wikipedia article has a good discussion).  One of them is that it's a test of equality of *medians* relative to an alternative hypothesis that's essentially a shift of median.  BTW, the Wilcoxon test (not to be confused with the Mann-Whitney test!) is a sign test for paired data.  Is this the &quot;Sign Test&quot; referred to in the question?" CreationDate="2010-11-23T16:01:03.087" UserId="919" />
  <row Id="7137" PostId="4819" Text="Hi Whuber, this is the sign test:  http://en.wikipedia.org/wiki/Sign_test" CreationDate="2010-11-23T16:03:53.317" UserId="253" />
  <row Id="7138" PostId="4816" Text="If you define the deltas to be complementary incomplete gamma functions, you obtain exact equalities.  Obviously these are the sharpest possible bounds!  I guess the point of this question is that your calculator doesn't compute incomplete gammas and you're looking for an approximation, but that still omits essential information: how can we answer this question until we know just what your calculator *can* compute?" CreationDate="2010-11-23T16:04:30.073" UserId="919" />
  <row Id="7139" PostId="4827" Text="Thank you Caracal, that was a great explanation/reference of the sign test.  I guess the answer to my question is &quot;no&quot;..." CreationDate="2010-11-23T16:06:59.897" UserId="253" />
  <row Id="7140" PostId="4819" Score="2" Text="Perhaps it's useful to distinguish the three Wilcoxon tests: 1. one-sample test: &quot;Wilcoxon signed rank test&quot; 2. two dependent samples -&gt; can be reduced to 1. by taking the observation-wise difference and testing for location parameter of 0. 3. two independent samples: &quot;Wilcoxon rank sum test&quot;. This is equivalent to the Mann-Whitney U-test: both test statistics are conceptually different but numerically identical." CreationDate="2010-11-23T16:10:49.847" UserId="1909" />
  <row Id="7141" PostId="4826" Text="Thanks. I'll look at the package. This is stackexchange isn't it?" CreationDate="2010-11-23T16:23:50.277" UserId="261" />
  <row Id="7142" PostId="4820" Score="1" Text="@psj You have just described the Mann-Whitney U test!  Also, you use &quot;every value in the other group [as] a potential counterpart for comparison&quot; and compute the median of all such paired differences you obtain the Hodges-Lehman estimate of the difference in medians of the two datasets: http://en.wikipedia.org/wiki/Hodges%E2%80%93Lehmann_estimate" CreationDate="2010-11-23T16:31:14.213" UserId="919" />
  <row Id="7143" PostId="4819" Score="2" Text="@Tal Yep, your &quot;sign test&quot; is also known as the Wilcoxon test.  The Mann-Whitney U test is its unpaired counterpart.  The *Mann-Kendall* test is its extension to more than two sets of data.  Although it doesn't seem to be commonly known, the U statistic can be obtained as the t-statistic for the ranks of the *combined* dataset (using midranks for tied groups) and the Mann-Kendall test is really just ANOVA on the ranks of the combined data.  This insight allows you to avoid the complex formulas often cited and use simple software to do the calculations." CreationDate="2010-11-23T16:35:12.823" UserId="919" />
  <row Id="7144" PostId="4826" Text="I think fabians meant Stackoverflow: http://stackoverflow.com which is for R programming, how do I do something in R questions. This place is more generally about stats. The two can blend a lot, however." CreationDate="2010-11-23T16:45:58.553" UserId="1390" />
  <row Id="7145" PostId="4815" Text="@fabians : but isn't k the number of **parameters**, and not the number of df? The different parametrization should give a different number of parameters, no?" CreationDate="2010-11-23T16:48:14.417" UserId="1124" />
  <row Id="7146" PostId="4822" Score="2" Text="That's right.  The connection can perhaps be seen more clearly by considering Sqrt(M) = (b-c) / Sqrt(b+c).  Approximating the variance of b as b and the variance of c as c (as is usual with counted data), we see that Sqrt(M) looks like an approximately normal variate (b-c) divided by its standard deviation: in other words, it looks like a *standard* normal variate.  In fact, we could conduct an equivalent test by referring Sqrt(M) to a table of the standard normal distribution.  Squaring it effectively makes the test symmetric two-tailed.  Obviously this breaks down if either b or c is small." CreationDate="2010-11-23T17:11:49.430" UserId="919" />
  <row Id="7147" PostId="4815" Score="2" Text="@Joris The df is the correct value to use, not the nominal number of parameters.  The former has mathematical and statistical meaning whereas the latter does not.  If you use more parameters than there are df, then one or more of those parameters will not be identifiable." CreationDate="2010-11-23T17:19:43.593" UserId="919" />
  <row Id="7148" PostId="4816" Text="I am not interested in computing an upper bound, but obtaining something that I can control analytically. The answer that robin has provided is exactly what I was looking for. The question is, are there more precise bounds than those provided by Massart and Laurent? " CreationDate="2010-11-23T17:30:10.510" UserId="168" />
  <row Id="7149" PostId="4821" Text="@robin the second inequality seems to be incorrect or am I missing something?" CreationDate="2010-11-23T17:32:43.910" UserId="168" />
  <row Id="7152" PostId="4830" Text="Related: http://stats.stackexchange.com/questions/4831/logistic-regression-transforming-variables and; http://stats.stackexchange.com/questions/4832/logistic-regression-classification-tables-a-la-spss-in-r" CreationDate="2010-11-23T17:46:50.293" UserId="776" />
  <row Id="7153" PostId="4821" Text="@mkolar sorry about that, now corrected" CreationDate="2010-11-23T17:47:47.813" UserId="223" />
  <row Id="7155" PostId="4819" Text="Whuber, if I understand correctly, it can be called wilcoxon-sign-test, but not wilcoxon-sign-rank-test.  Other then that - thanks for your references - all are interesting to know." CreationDate="2010-11-23T17:48:11.477" UserId="253" />
  <row Id="7156" PostId="4832" Text="You should post an example of the SPSS output you want to replicate in R, or explicitly detail what the SPSS output is reporting" CreationDate="2010-11-23T17:48:33.417" UserId="1036" />
  <row Id="7157" PostId="4832" Text="Good idea, output added for clarity. Thanks for the suggestion Andy." CreationDate="2010-11-23T17:55:29.180" UserId="776" />
  <row Id="7158" PostId="4362" Text="Very well explained!  What a wonderful example to the Bayes theorem approach." CreationDate="2010-11-23T18:04:36.970" UserId="253" />
  <row Id="7159" PostId="4835" Text="Thanks Caracal, let me try this out on my data" CreationDate="2010-11-23T18:05:29.967" UserId="776" />
  <row Id="7160" PostId="4822" Text="Thank you for the intuitive answer Joris.  Still, why is it more common to use this approximation rather then using the normal approximation to the exact binomial test of McNemar? " CreationDate="2010-11-23T18:08:09.840" UserId="253" />
  <row Id="7161" PostId="4834" Text="Likert scale is actually a good example of an underlying continuous variable that is collected in discrete fashion." CreationDate="2010-11-23T19:38:16.297" UserId="279" />
  <row Id="7162" PostId="4839" Text="Are these measurements binary i.e., 1 (for yes), 0 (for no)? " CreationDate="2010-11-23T19:51:57.487" />
  <row Id="7163" PostId="3129" Text="I agree about the bootstrap approach.  But power is not a function of degrees of freedom.  In many cases, including this one, the Mann-Whitney test often has *greater* power than the t-test.  See http://tbf.coe.wayne.edu/jmasm/sawilowsky_misconceptions.pdf .  In general, the power of a parametric test is good when the parametric assumptions are true but can be lower--sometimes drastically so--when those assumptions are violated, whereas good nonparametric tests maintain their power." CreationDate="2010-11-23T19:53:08.280" UserId="919" />
  <row Id="7164" PostId="4836" Text="(+1) Agree, i think the assumption of a continuous distribution may affect the standard way of calculating p-values, but simulation as suggested by whuber gets around that (i'd call this a permutation test)." CreationDate="2010-11-23T20:19:26.417" UserId="449" />
  <row Id="7165" PostId="4809" Text="In fitting the nonlinear curves have you modelled the unstransformed response or the log of the response, or something else?" CreationDate="2010-11-23T20:27:57.940" UserId="449" />
  <row Id="7166" PostId="4840" Score="2" Text="Your question lacks some critical details -- first of all, what should be the protocol of this forecast, i.e. you want to predict the ozone level for next day every day? Predict the mean level in 2020 from 2004-2006? Next, is the ozone levels the only data you have? How it is discretized, one measurement per week/day/hour? Maybe you can pull some other relevant data from other sources? What can be relevant from chemical point of view?" CreationDate="2010-11-23T20:31:48.333" UserId="88" />
  <row Id="7167" PostId="4774" Text="Thanks for the intricate response. The new tool provides more data and costs less, yes. I ended up doing a lot of what you describe including a correlogramme of the values after matching the data points. " CreationDate="2010-11-23T20:34:30.437" UserId="1784" />
  <row Id="7168" PostId="4836" Score="3" Text="@onestop That is correct.  Indeed, the Wilcoxon test *is* a permutation test.  All we're discussing is how to compute (or estimate) the distribution of the test statistic.  The distribution can be computed exactly using combinatorial techniques; it can be estimated through simulation; and it can be approximated, usually with a Normal approximation when the amount of data is large enough (m+n of 10 or greater is usually good enough) and the data can be thought of as realizations of a continuous RV." CreationDate="2010-11-23T20:38:58.477" UserId="919" />
  <row Id="7169" PostId="4842" Text="@onestop : see my answer for further elaboration" CreationDate="2010-11-23T20:40:10.737" UserId="1124" />
  <row Id="7170" PostId="4839" Text="It seems feasible to make some clustering; can you state what is the aim of this analysis?" CreationDate="2010-11-23T20:42:56.673" UserId="88" />
  <row Id="7171" PostId="4816" Text="@mkolar Gamma integrals can be &quot;controlled analytically,&quot; so what distinction are you making?" CreationDate="2010-11-23T20:44:09.220" UserId="919" />
  <row Id="7172" PostId="4842" Text="Hi onestop - Since both are asymptotic, then for smaller N's they might yield somewhat different results.  In such a case, I wonder if the choice of going with chi-square is because it is better then the normal approximation, or because of historical reasons (or maybe, as you suggested - they always yield identical results)" CreationDate="2010-11-23T20:51:43.987" UserId="253" />
  <row Id="7173" PostId="4845" Text="Thanks mbq, I'll try that out!" CreationDate="2010-11-23T20:55:55.187" UserId="776" />
  <row Id="7174" PostId="4833" Text="Very helpful description, thanks for the direction and the detail on my subquestion as well." CreationDate="2010-11-23T20:57:12.563" UserId="776" />
  <row Id="7175" PostId="4830" Text="@Brandon The raw proportion of defaulters is apparently 1 case out of 4, but it seems you also have a lot of variables. Did you try them all, have you any set of primary variables of interest? " CreationDate="2010-11-23T21:00:47.440" UserId="930" />
  <row Id="7176" PostId="4839" Text="To echo @mbq's comment, are you interested in finding clusters of variables, of individuals, or both? Do you want to build a scoring rule, or to find those computer games that favor &quot;hard&quot;/&quot;soft&quot; self-perception wrt. game addiction?" CreationDate="2010-11-23T21:05:16.873" UserId="930" />
  <row Id="7177" PostId="4822" Text="@Tal: It's the same. See nonstops answer and my edit." CreationDate="2010-11-23T21:10:44.987" UserId="1124" />
  <row Id="7178" PostId="4842" Text="@Tal: for smaller N, neither of both hold. And as shown in my edit, they are exactly the same." CreationDate="2010-11-23T21:12:44.937" UserId="1124" />
  <row Id="7179" PostId="4834" Text="@Aniko Right, but the problem is that we're not sure how well the original interval scale fits into the discretized one, unless making strong assumptions. Optimal scaling might be an option there." CreationDate="2010-11-23T21:13:35.380" UserId="930" />
  <row Id="7180" PostId="4654" Text="@onestop: risk differences is indeed the way to go, as that one can be understood by non-statisticians as well. Accepted." CreationDate="2010-11-23T21:15:20.537" UserId="1124" />
  <row Id="7181" PostId="4830" Text="The original variables are at the front of the file. The rest are transformations of the same identified by x_ (where x = log, ln, inv, sqrt). I've tried a mixture of these. But I'm a little bit miffed as to how to interpret or create residual plots where the predictor is 0,1" CreationDate="2010-11-23T21:16:32.127" UserId="776" />
  <row Id="7182" PostId="4647" Text="@Aniko: +1 for using a binomial mixed effect models. Alas it was rejected as &quot;not a standard method&quot;. If you can give me some links to papers where this approach is used in a meta-analysis setup, you would help me greatly. Thank you in advance." CreationDate="2010-11-23T21:16:33.763" UserId="1124" />
  <row Id="7183" PostId="4830" Text="In terms of variables of interest, I've tried all of them primaries and a number of different combinations of the transformed variables as well as mixed models that include interaction effects. Still, nothing higher than 81.7% overall efficiency." CreationDate="2010-11-23T21:18:03.560" UserId="776" />
  <row Id="7184" PostId="3562" Text="@whuber, thanks for this, helful for a project I'm working on currently as well - and totally makes sense." CreationDate="2010-11-23T21:37:44.150" UserId="776" />
  <row Id="7185" PostId="4815" Text="@whuber: see your point. Thx for the clarification" CreationDate="2010-11-23T21:38:29.627" UserId="1124" />
  <row Id="7186" PostId="4844" Text="I guess, even if you drive randomly, temperature will be smooth along time and you won't get independence ... " CreationDate="2010-11-23T21:41:06.253" UserId="223" />
  <row Id="7187" PostId="4844" Text="ok, that's what I thought... so there is no way to derive confidence intervals for the proportions ? In reality, I'm benchmarking two appartuses : is it meaningful to compare the proportions using R's prop.test then ?" CreationDate="2010-11-23T21:43:38.323" UserId="1784" />
  <row Id="7188" PostId="4647" Text="I have edited my answer with some references." CreationDate="2010-11-23T21:43:55.520" UserId="279" />
  <row Id="7189" PostId="2697" Score="1" Text="@Freya Eigenvectors are just the linear combinations of the original variables (in the simple or rotated factor space); they described how variables &quot;contribute&quot; to each factor axis. Basically, think of PCA as as way to construct new axes that point to the directions of maximal variance (in the original variable space), as expressed by the eigenvalue, and how variables contributions are weighted or linearly transformed in this new space." CreationDate="2010-11-23T21:46:15.793" UserId="930" />
  <row Id="7190" PostId="4811" Text="@James It looks like [@G. Jay Kerns's response](http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues/2704#2704), isn't it?" CreationDate="2010-11-23T21:49:25.180" UserId="930" />
  <row Id="7191" PostId="4834" Text="@chl That's why you are using Wilcoxon's test instead of a t-test. Wilcoxon's test does not assume equally spaced intervals, etc, so scaling is not an issue." CreationDate="2010-11-23T22:11:50.827" UserId="279" />
  <row Id="7192" PostId="4845" Text="I tried it out, presented no substantial increases or decreases in overall efficiency (efficiency being how well it predicted defaulters/non-defaulters in the absence of false-positives,false-negatives)" CreationDate="2010-11-23T22:16:46.020" UserId="776" />
  <row Id="7193" PostId="4848" Text="I certainly appreciate the responses - thanks very much. As one follow-up, how this question came about was the accuracy of the random table above is 0.41 (82/200). But, if we wanted to compare our model against random guess, using only the priors, we'd have an accuracy of 0.38 ((100/200)^2+(60/200)^2+(40/200)^2. Doesn't it seem that this should match the random matrix?" CreationDate="2010-11-23T22:20:48.077" UserId="2040" />
  <row Id="7194" PostId="4835" Text="How would I generate Yfac if my variable is already a 1/2 dichotomous variable saved as a factor in R?" CreationDate="2010-11-23T22:27:53.000" UserId="776" />
  <row Id="7195" PostId="4835" Text="Nevermind, worked like a charm, just set Yfac &lt;- df$dichotomous.var" CreationDate="2010-11-23T22:32:03.527" UserId="776" />
  <row Id="7196" PostId="4845" Score="1" Text="@Brandon I've tried few other ideas and it doesn't seem to help. This suggests that this set is just hard enough to make this happen (possibly default is just driven by some unpredictable random factors)." CreationDate="2010-11-23T22:32:52.487" UserId="88" />
  <row Id="7197" PostId="4845" Text="@mbq, thanks for taking the time! Much appreciated." CreationDate="2010-11-23T22:34:18.097" UserId="776" />
  <row Id="7198" PostId="4841" Text="Thank you! Is that an asymptotic result? How high does the trigger have to be? " CreationDate="2010-11-23T22:45:06.137" UserId="996" />
  <row Id="7199" PostId="4819" Text="@Tal There is a total chaos with these nomenclature; I used Wilcoxon as a keyword, since it is most common. And the whole bundle of such tests is located under `wilcox.test` function in R." CreationDate="2010-11-23T22:53:03.617" UserId="88" />
  <row Id="7200" PostId="4836" Text="You're right of course @whuber, i meant &quot;i'd call that a *Monte-Carlo* permutation test*&quot;. (Sorry was in too much of a hurry for no good reason really)" CreationDate="2010-11-23T22:58:41.913" UserId="449" />
  <row Id="7201" PostId="4836" Text="@onestop I didn't mean to sound critical.  I was agreeing with you and felt your meaning was clear." CreationDate="2010-11-23T23:07:15.167" UserId="919" />
  <row Id="7202" PostId="4851" Text="Further info: the functions represent different measures of proximity to various types of genes. I want to try out different metrics, but they all can be boiled down to &quot;(some kind of) distance to the nearest gene of type X&quot;. So the values would probably look something like this [graph](http://i.imgur.com/hVyIO.png) repeated over and over again (with different shapes), and a histogram could have pretty much any shape, depending on how far apart the relevant genes are. Using the CDF, I could compute a score for a position and the probability of getting this high a score by chance." CreationDate="2010-11-23T23:14:22.700" UserId="2111" />
  <row Id="7203" PostId="4851" Score="1" Text="@bnaul It sounds like you really need only to store one tail accurately.  That could simplify your precalculations somewhat.  After all, there's little practical difference between a p-value of 50% and 25%, but a huge difference between 5% and 0.001%." CreationDate="2010-11-23T23:33:31.913" UserId="919" />
  <row Id="7204" PostId="4845" Score="1" Text="Personally, I think this is live data that my prof is getting paid to model in one of his consulting jobs... but that's another issue entirely" CreationDate="2010-11-23T23:38:57.260" UserId="776" />
  <row Id="7205" PostId="4853" Text="I think this will definitely work...in my head I was so attached to the idea of a proper CDF that this didn't even occur to me. Thanks!" CreationDate="2010-11-23T23:46:16.413" UserId="2111" />
  <row Id="7206" PostId="4841" Text="A sum of $k$ exponentials with parameter $\lambda$ gives the erlang with parameters $k$ and $\lambda$. Thus, it is not a asymptotic result but a finitary one." CreationDate="2010-11-24T00:20:46.100" />
  <row Id="7207" PostId="4857" Text="Where can I find help to display the latex formulae!" CreationDate="2010-11-24T02:12:35.407" UserId="1709" />
  <row Id="7208" PostId="4857" Text="Surround latex with `$ .. $`" CreationDate="2010-11-24T02:16:52.837" />
  <row Id="7210" PostId="4808" Text="This combined with Log likelihood seems to work" CreationDate="2010-11-24T03:42:03.013" UserId="1951" />
  <row Id="7211" PostId="4857" Text="@Srikant Vadali $Thank\:  \, \, you$" CreationDate="2010-11-24T04:07:47.597" UserId="1709" />
  <row Id="7212" PostId="4859" Text="The only problem is, I have to be able to reproduce this with SPSS output. :( I'm still going to try it though! " CreationDate="2010-11-24T04:41:15.090" UserId="776" />
  <row Id="7213" PostId="1527" Score="1" Text="@Neil I can see how the proportion of heads in one flip can be 1.0 or 0.0, but I cannot see how it ever can be 0.50 (except in a Schrodinger Cat experiment, perhaps, but that's a different issue...)." CreationDate="2010-11-24T04:51:35.003" UserId="919" />
  <row Id="7214" PostId="4859" Text="Tried it out, can't seem to get predict.glmnet() to work. Is there any magic that needs to be happening when you set newx?" CreationDate="2010-11-24T05:09:05.783" UserId="776" />
  <row Id="7215" PostId="4856" Text="May I ask what kind of data do you want to model and for what usage you want to do it?" CreationDate="2010-11-24T06:35:21.417" UserId="1709" />
  <row Id="7217" PostId="4864" Text="Classification and Regression Trees? That is actually a second part to the assignment. After I've maximized the classification I have to classify based on probability deciles." CreationDate="2010-11-24T06:44:53.813" UserId="776" />
  <row Id="7218" PostId="3816" Text="Question: when citing a package, do you cite the vignette (if one exists) or the package itself?" CreationDate="2010-11-24T06:55:25.207" UserId="776" />
  <row Id="7219" PostId="4864" Text="Actually, someone helped me with producing the classification table in this related question: http://stats.stackexchange.com/questions/4832/logistic-regression-classification-tables-a-la-spss-in-r Thanks for the example with R, much appreciated, I found similar instruction on the quick-r website. Although, for this puprpose I'm forced to apply CHAID in SPSS. " CreationDate="2010-11-24T07:00:28.027" UserId="776" />
  <row Id="7220" PostId="4844" Text="For anyone interested, I found this based on Markov Chains... it uses the weather as an example. http://www.jstor.org/pss/2533067" CreationDate="2010-11-24T07:31:11.013" UserId="1784" />
  <row Id="7221" PostId="4861" Score="2" Text="@Andy (+1) Yes, there exist some penalized versions of logistic regression (or more generally, GLMs), see some references there: http://stats.stackexchange.com/questions/4272/when-to-use-regularization-methods-for-regression/4274#4274." CreationDate="2010-11-24T08:17:17.883" UserId="930" />
  <row Id="7224" PostId="4870" Text="@fabians (+1) Thanks for that. I know about the problem of asymmetry in the distribution of VC, and the difference between ML and REML, but I didn't know you developed that package. I look forward to trying it out." CreationDate="2010-11-24T09:51:16.240" UserId="930" />
  <row Id="7225" PostId="1527" Score="1" Text="@Neil : No you can't. It doesn't even make sense in regular english, let alone in statistics." CreationDate="2010-11-24T10:26:05.837" UserId="1124" />
  <row Id="7226" PostId="1559" Text="@Brian: proportions range from 0 to 1. Or from 0 to 100%. Like probabilities." CreationDate="2010-11-24T10:28:44.400" UserId="1124" />
  <row Id="7227" PostId="4872" Text="This seems off-topic." CreationDate="2010-11-24T11:06:24.317" UserId="930" />
  <row Id="7229" PostId="4834" Text="@Aniko I agree with you. My comment was not a critic, and reference to optimal scaling was mainly for the case where the assumption of an *a priori* continuous scale of measurement is not tenable--because in practice, we often rely on Likert items without making strong assumption on the underlying construct; I also agree for Mann-Whitney test with ordinal variables." CreationDate="2010-11-24T11:28:27.800" UserId="930" />
  <row Id="7230" PostId="4872" Text="It is.........." CreationDate="2010-11-24T11:42:33.893" UserId="88" />
  <row Id="7231" PostId="4845" Text="the solution to a problem in data analysis should never be &quot;throw away valid data points&quot; - you could try to use balanced training data sets in order to avoid these effects, but you should still evaluate the predictions on all of the data (i.e. all of the validation set)." CreationDate="2010-11-24T11:46:46.417" UserId="1979" />
  <row Id="7232" PostId="4845" Text="@fabians Not always -- sometimes it is obvious that the data is redundant and some algorithm don't have any weights to set. Not to mention this quite often leads to the removal of objects inside the algorithm." CreationDate="2010-11-24T12:15:16.250" UserId="88" />
  <row Id="7233" PostId="4864" Text="`predict` method for prediction, `table(originalClasses,predictedClasses)` for table construction. I tried RF (usually it has accuracy as in overfitted CART, but no overfit) and the result was not much better than `glm`." CreationDate="2010-11-24T12:20:17.777" UserId="88" />
  <row Id="7235" PostId="4871" Text="I think you have misunderstand the question (I believe it was Can I use CV to just make error approximation instead of the whole model selection?), but +1 for relevant remarks." CreationDate="2010-11-24T12:46:09.283" UserId="88" />
  <row Id="7236" PostId="4871" Text="I also had this interpretation in my mind, but I thought that this would be too easy ;), since this is exactly what cv does. Additionally, the whole &quot;all data&quot;-part confused me." CreationDate="2010-11-24T13:01:18.677" UserId="264" />
  <row Id="7237" PostId="4872" Score="2" Text="Seems like a good question to me.  If you only have a Bachelor's degree, you will be lucky to find a position, most likely as a SAS programmer.  Most advertised positions are looking for a Master's or above." CreationDate="2010-11-24T14:09:49.467" UserId="1920" />
  <row Id="7238" PostId="4876" Score="2" Text="It will help if you can tell us what you are trying to accomplish ultimately beyond the specific visualization difficulties you have at present." CreationDate="2010-11-24T14:23:50.610" />
  <row Id="7239" PostId="4879" Text="@chl hmm the table function is a useful one it seems, but it would be better if I could show the percentages of the share rather than the count of each OS. Is there a way to do that?" CreationDate="2010-11-24T14:44:44.407" UserId="2101" />
  <row Id="7240" PostId="4879" Text="@sfactor I've updated my response accordingly." CreationDate="2010-11-24T15:05:28.290" UserId="930" />
  <row Id="7241" PostId="4861" Text="@chl, thank you. I have updated to link to your previous answer." CreationDate="2010-11-24T15:07:20.127" UserId="1036" />
  <row Id="7242" PostId="4861" Text="@Andy Thx too. This was just a comment on your already excellent response." CreationDate="2010-11-24T15:09:09.783" UserId="930" />
  <row Id="7243" PostId="4879" Score="3" Text="@chl - You guys are after the function `prop.table()` which comes with base R, it's essentially a wrapper for what you've got above. With your example, you'd want something like `prop.table(table(x))`" CreationDate="2010-11-24T15:14:27.240" UserId="696" />
  <row Id="7244" PostId="4876" Text="When I look at the system performance metric I want to understand how each subsystem is contributing towards it so I can get an idea of how significant the difference is. Ultimately I'd want to work out the tolerance and raise a red flag when subsystems perform worse by more than that tolerance.  In a healthy system, I'd expect the the subsystems to be delivering approximately the same and therefore the subsystem values should be spread relatively closely around the system value." CreationDate="2010-11-24T15:15:39.933" UserId="2122" />
  <row Id="7245" PostId="4876" Text="In summary, I want to recognise when a subsystem is performing badly relative to its peers." CreationDate="2010-11-24T15:20:11.117" UserId="2122" />
  <row Id="7246" PostId="4877" Score="1" Text="@sfactor - Since you're new to R, I recommend checking out this [post](http://stats.stackexchange.com/questions/138/resources-for-learning-r) to find a good list of resources to help get you up and running doing some analysis." CreationDate="2010-11-24T15:23:39.677" UserId="696" />
  <row Id="7247" PostId="4879" Text="@Chase (+1) Huh, yes... That's it! I was thinking of even more complex ways with `addmargins()` or `margin.table()`." CreationDate="2010-11-24T15:33:00.533" UserId="930" />
  <row Id="7248" PostId="4877" Text="@Chase thanks for the link, yah i need to really learn the language, this was just something I need to do quickly." CreationDate="2010-11-24T15:35:58.523" UserId="2101" />
  <row Id="7249" PostId="4811" Text="Yes, it is the same. My appologize for overlooking that reponse." CreationDate="2010-11-24T15:41:12.860" />
  <row Id="7250" PostId="4884" Text="What is an AB-test? A link, a brief explanation etc may help." CreationDate="2010-11-24T16:27:44.840" />
  <row Id="7251" PostId="4873" Text="Thanks for trying, I played with the threshold as well and it provided little boost to the classification." CreationDate="2010-11-24T16:28:50.713" UserId="776" />
  <row Id="7252" PostId="4885" Text="(+1) especially for the insights in the interpretability of distances vs areas and slope vs height." CreationDate="2010-11-24T16:35:39.397" UserId="264" />
  <row Id="7253" PostId="4884" Text="@Srikant: Sorry, I added some links to the question." CreationDate="2010-11-24T16:40:02.187" UserId="264" />
  <row Id="7254" PostId="4871" Text="Thanks for your response, lots of relevant information here. Just to make sure we're on the same page let me state this another way:&#xA;&#xA;In some papers I've noticed that people set aside a certain fraction of the dataset for training purposes. Given a dataset of size 1000 they set aside 300 events for training and use 700 events for classification and never discuss the 300 events used to train the model (i.e. they don't train a separate model and then test those 300 events). Couldn't they just use cross validation technique to classify all of the data?" CreationDate="2010-11-24T16:46:13.817" UserId="2113" />
  <row Id="7255" PostId="4873" Score="1" Text="Although, it did do a better job of predicting defaulters. " CreationDate="2010-11-24T16:52:06.560" UserId="776" />
  <row Id="7256" PostId="4871" Text="@C.Reed: Not with the same ratios ! Sometimes you want to reduce the computation time and/or control the test-to-training-ratio to study the bias-variance tradeoff. I you have some time to spend, I recommend chapter 3 of Kohavis Phd Thesis discussing this topic extensively (http://robotics.stanford.edu/~ronnyk/teza.pdf). But all in all, the cv (from my point of view) is the favored method for error estimation (as mbq already said)" CreationDate="2010-11-24T16:55:08.647" UserId="264" />
  <row Id="7257" PostId="4888" Score="1" Text="+1 since `aggregate` is an overshoot here and plyr is slow." CreationDate="2010-11-24T18:15:57.843" UserId="88" />
  <row Id="7258" PostId="4871" Text="Thanks for the thesis reference, ch. 3 is quite helpful on this matter." CreationDate="2010-11-24T18:49:35.900" UserId="2113" />
  <row Id="7259" PostId="4891" Text="@onestop It's still an interesting point.  Thanks for bringing it up." CreationDate="2010-11-24T19:40:19.393" UserId="919" />
  <row Id="7260" PostId="4836" Text="So maybe I'm missing something - is the assumption of continuous distribution only needed for the approximation of the normal distribution of the statistic?  " CreationDate="2010-11-24T20:10:33.063" UserId="253" />
  <row Id="7261" PostId="4822" Text="Great extension and explanation Joris - thank you!" CreationDate="2010-11-24T20:12:07.090" UserId="253" />
  <row Id="7262" PostId="4822" Text="Actually - last question.  So if both are identical (and I think you might also need an &quot;absolute value&quot; around the b-c), then why do people go to the chi distribution instead of staying with the normal one?  Where's the advantage?" CreationDate="2010-11-24T20:14:44.297" UserId="253" />
  <row Id="7263" PostId="4836" Text="@Tal I believe that is the case.  In fact, the Wilcoxon (Mann-Whitney) rank sum test is actually required in some US state regulations due to its ability to handle certain kinds of non-continuous distributions resulting from data censoring: http://www.pabulletin.com/secure/data/vol27/27-33/1340c.html" CreationDate="2010-11-24T20:16:36.107" UserId="919" />
  <row Id="7265" PostId="4893" Text="@wok: oops, I haven't seen your last paragraph!" CreationDate="2010-11-24T20:30:25.570" UserId="1725" />
  <row Id="7267" PostId="4894" Text="@Ben (+1) Thanks and welcome to CrossValidated. No need to post it as a comment (which assumes you have enough rep); it's a response on its own. I always appreciate your input on [r-sig-mixed](https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models). Nice to see you there." CreationDate="2010-11-24T20:51:47.500" UserId="930" />
  <row Id="7268" PostId="4895" Text="This is correct and appealing.  However, in the end it appears only to rephrase the question without actually answering it: namely, why should we use the Euclidean (L2) distance?" CreationDate="2010-11-24T21:07:08.960" UserId="919" />
  <row Id="7269" PostId="4895" Text="That is indeed an excellent question, left unanswered. I used to feel strongly that the use of L2 is unfounded. After having studied a little statistics, I saw the analytic niceties, and since then have revised my viewpoint into &quot;if it really matters, you're probably in deep water already, and if not, easy is nice&quot;. I don't know measure theory yet, and worry that analysis rules there too - but I've noticed some new interest in combinatorics, so perhaps new niceties have been/will be found." CreationDate="2010-11-24T21:39:54.330" UserId="2456" />
  <row Id="7270" PostId="4741" Text="@Aniko: Yes, I am looking for something like a confidence interval for my counts per bin which propagate nicely when I project out parts of my 2D histogram into 1D." CreationDate="2010-11-24T21:42:12.113" UserId="56" />
  <row Id="7271" PostId="4895" Score="5" Text="@sesqu Standard deviations did not become commonplace until Gauss in 1809 derived his eponymous deviation using squared error, rather than absolute error, as a starting point.  However, what pushed them over the top (I believe) was Galton's regression theory (at which you hint) and the ability of ANOVA to decompose sums of squares--which amounts to a restatement of the Pythagorean Theorem, a relationship enjoyed only by the L2 norm.  Thus the SD became a natural omnibus measure of spread advocated in Fisher's 1925 &quot;Statistical Methods for Research Workers&quot; and here we are, 85 years later." CreationDate="2010-11-24T21:56:31.867" UserId="919" />
  <row Id="7272" PostId="4822" Text="@Tal: You don't want that absolute value around it. N(0,1) can take negative values as well. Why take the chi2? A matter of taste I guess? chi2 is one-tailed, so tests two-sided by default." CreationDate="2010-11-24T22:03:02.437" UserId="1124" />
  <row Id="7273" PostId="4836" Text="@Tal and @whuber: I was puzzled by the implication (and then by its apparent acceptance) that the test statistic for the Wilcoxon(or M-W) was normally distributed." CreationDate="2010-11-25T00:18:01.930" UserId="2129" />
  <row Id="7274" PostId="4836" Score="1" Text="@DWin For a continuous underlying distribution and sufficient sample sizes it is *approximately* normally distributed.  The exact distribution, however, is not normal and departs appreciably from normality when there are less than 10 values (in both datasets combined).  Tables of the distribution have been published to cover those small-data cases, but it is readily computed nowadays by brute force.  (E.g., with 10 values it takes little time to generate all the possible combinations.)" CreationDate="2010-11-25T02:49:38.557" UserId="919" />
  <row Id="7275" PostId="1516" Text="I like your answer. The sd is not always the best statistic." CreationDate="2010-11-25T03:03:17.077" UserId="1709" />
  <row Id="7276" PostId="4895" Score="3" Text="(+1) Continuing in @whuber's vein, I would bet that had Student published a paper in 1908 entitled, &quot;Probable Error of the Mean - Hey, Guys, Check Out That MAE in the Denominator!&quot; then statistics would have an entirely different face by now.  Of course, he didn't publish a paper like that, and of course he couldn't have, because the MAE doesn't boast all the nice properties that S^2 has. One of them (related to Student) is its independence of the mean (in the normal case), which of course is a restatement of orthogonality, which gets us right back to L2 and the inner product." CreationDate="2010-11-25T03:38:57.890" UserId="1108" />
  <row Id="7277" PostId="4900" Text="Thanks for the references and r function. I still prefer the interpretation of d-based measures (where they apply) over variance-explained-based measures. I find it clearer to think of the effect of an intervention in terms of a difference score." CreationDate="2010-11-25T04:40:42.377" UserId="183" />
  <row Id="7278" PostId="4822" Text="Dear Joris, you wrote &quot; chi2 is one-tailed, so tests two-sided by default&quot;, how is that? I thought that once you get to the P value, you can move back and forth from two tails to one tail with simply multiplying or dividing the value with 2." CreationDate="2010-11-25T06:25:44.537" UserId="253" />
  <row Id="7279" PostId="4888" Text="-1 That can't possibly work unless you make the components of the data frame visible (using say `with()`, or refer to the explicitly via, e.g., `df$Objects` where `df` is the op's data frame." CreationDate="2010-11-25T07:30:39.897" UserId="1390" />
  <row Id="7280" PostId="4888" Text="@mbq: why is `aggregate()` an &quot;overshoot here&quot;? It seems very natural as it returns for you a DF with the unique `SessionNo.` plus the sum for each. `tapply()` just gives the sums." CreationDate="2010-11-25T07:32:46.460" UserId="1390" />
  <row Id="7281" PostId="4896" Text="@DWin: if you are using `with()`, then go the whole hog and use `within()` here --- gets rid of more `$`. Your first line could be, `tst &lt;- within(tst, tmn &lt;- ave(Objects, SessionNo.))` whilst your second line could be `tst &lt;- within(tst, devmn &lt;- Objects - ave(Objects, SessionNo.))`. Note that the second line contains two modifications as the whole thing can be inside `with()` or `within()`." CreationDate="2010-11-25T07:39:54.520" UserId="1390" />
  <row Id="7282" PostId="4890" Text="Thank you very much for your response. However, I am not sure if I was able to explain my problem. I've edited the question and added a concrete example. If it is not too much hassle, could you check and (if necessary) revise your response ?" CreationDate="2010-11-25T07:46:43.937" UserId="264" />
  <row Id="7283" PostId="4903" Score="1" Text="sounds like a remark or is the answer &quot;think&quot; ? " CreationDate="2010-11-25T08:15:24.163" UserId="223" />
  <row Id="7284" PostId="4903" Text="@Robin - the latter. I find statistical modelling quite difficult (I'm an ecologist with little formal statistical training, most of what I've learned has been self-taught) but it is a lot easier if I think about the problem first, determine what is plausible, build that model, do my model diagnostics, try interactions where these make scientific sense." CreationDate="2010-11-25T08:25:34.090" UserId="1390" />
  <row Id="7285" PostId="4903" Text="@ucfagls (what is this name related to :) ?) I guess this is what statistician call modelling :)  injecting your apriori  knowledge (Beeing a self-learner here may be an advantage) into a formal description of what you observe. This is necessary in most case (i.e when the number of variables is not very small with respect to the number of observations). This introduces bias." CreationDate="2010-11-25T08:34:16.503" UserId="223" />
  <row Id="7286" PostId="4822" Score="1" Text="@Tal: You know R. plot the chi2 with one degree of freedom, you'll see." CreationDate="2010-11-25T09:02:23.250" UserId="1124" />
  <row Id="7287" PostId="4903" Text="@Robin: Oh, it is my old computer userid from when I was an undergraduate many moons ago. I keep forgetting to change to my real name here - done it on SO already..." CreationDate="2010-11-25T09:24:33.323" UserId="1390" />
  <row Id="7288" PostId="4903" Text="@Robin: There, that's better!" CreationDate="2010-11-25T09:26:14.040" UserId="1390" />
  <row Id="7289" PostId="4903" Text="@Gavin Thanks ! very interesting web page !  I'll try to understand what nitrogene cycle is ... I guess makes me understand your answer now :) " CreationDate="2010-11-25T10:03:58.313" UserId="223" />
  <row Id="7290" PostId="4901" Text="You could tell us a little bit about your data? size (cf my answer) and nature (cf Gavin's answer) " CreationDate="2010-11-25T10:09:58.307" UserId="223" />
  <row Id="7291" PostId="4888" Text="@Gavin: You´re right, i´ve edited my post." CreationDate="2010-11-25T10:36:17.957" UserId="1050" />
  <row Id="7292" PostId="4904" Score="1" Text="Some example data can be useful, so answerers could test their ideas." CreationDate="2010-11-25T11:22:57.167" UserId="88" />
  <row Id="7293" PostId="4872" Text="@Mike It is a good question, but not in the scope of this site." CreationDate="2010-11-25T11:24:40.753" UserId="88" />
  <row Id="7294" PostId="4888" Text="@EDi +1 following edit" CreationDate="2010-11-25T11:31:34.960" UserId="1390" />
  <row Id="7295" PostId="4901" Text="@Robin: Give him time to get out of bed, Brandon is in Toronto ;-)" CreationDate="2010-11-25T11:41:34.173" UserId="1390" />
  <row Id="7296" PostId="4746" Text="Thank you onestop for your detailed answer.&#xA;I ended up in doing the power analysis with the R function power.t.test.&#xA;&#xA;When I do the power test I get somewhat different numbers than you using the following parameters: power.t.test(n=10, power = 0.8, type =c(&quot;two.sample&quot;))&#xA;I get delta=1.32 which is higher than your 1.25 SD's. Am I missing here something?&#xA;&#xA;Unfortunately, I can not homogenise the soil before the experiment. I need undisturbed soil columns. &#xA;&#xA;I will take baseline measurements before I start the treatment. Any hints on ANCOVA are appreciated." CreationDate="2010-11-25T11:49:14.860" UserId="2063" />
  <row Id="7297" PostId="4904" Text="Will append example data later. Right now you can assume that if everybody plays fair, then the difference of v to the average v is roughly 0.7 times standard deviation. If somebody cheats (cheater must be excluded from calculating the stddev), his deviation from the fairplayer average v is noticably &gt; 1 times stddev, somewhere in the range between 2 and 3 times stddev normally, but this can go up to 8 times stddev. Right now I loop through the array of all players and calculate stddev, excluding the i-th player each time)" CreationDate="2010-11-25T11:51:22.900" UserId="2132" />
  <row Id="7298" PostId="4888" Text="I would have opted for by()." CreationDate="2010-11-25T11:58:34.007" UserId="144" />
  <row Id="7299" PostId="4888" Text="@Gavin Subjective thing; I like the simplicity of `tapply`, yet of course `subset` will be the way to go to make it the fastest possible in plain R." CreationDate="2010-11-25T12:08:48.690" UserId="88" />
  <row Id="7300" PostId="4888" Text="@Roman Add an answer, it will be a nice pool." CreationDate="2010-11-25T12:09:31.043" UserId="88" />
  <row Id="7301" PostId="4910" Text="For discrete distributions, there is no point in defining the ecdf at all in $\mathbb R$. It makes no sense to ask &quot;what proportion of families have at most 1.5 children&quot;, for example. As for sampling from the sample - I have never thought of that as a particularly good idea to begin with, either." CreationDate="2010-11-25T15:02:20.870" UserId="2456" />
  <row Id="7303" PostId="4906" Text="Is there a difference if i give the argument &quot;simplify = FALSE&quot; to tapply()?" CreationDate="2010-11-25T15:22:15.700" UserId="1050" />
  <row Id="7304" PostId="4901" Text="@Robin, I would prefer to keep it more general. If, in your response you're providing a method that requires an assumption about the size or nature of data, please state it. The problem I'm having spans a number of different modelling tasks, all with different data. So, in this case, I'm looking for general recommendation on identifying interaction effects. " CreationDate="2010-11-25T15:24:26.920" UserId="776" />
  <row Id="7305" PostId="4362" Score="1" Text="@Srikant: that prior cannot be formally defined. In any case, Prob(No of heads = X | experimenter is not faking) is always around zero when N=20000, no matter the value of X and no matter your prior. So your posterior for any number is also always close to 0. I don't see what this has to do with Bayes theorem. " CreationDate="2010-11-25T15:29:47.337" UserId="1124" />
  <row Id="7306" PostId="4903" Text="@Gavin, I agree thinking is important, but sometimes significant interaction effects exist that are not immediately explainable by what you know about the data. What I'm looking for here is a graphical or statistical way of identifying potential interactions. Some may agree that it is not always the case that interactions are easy to &quot;explain in words&quot;. " CreationDate="2010-11-25T15:50:25.867" UserId="776" />
  <row Id="7307" PostId="4902" Text="@Robin, thanks for the direction to Sobol indices. Again, I'd like to specify that I'm looking for a general rather than a specific answer here. I'm not asking about a specific set of data but rather trying to explain a problem I've been having with a number of different sets." CreationDate="2010-11-25T15:52:32.450" UserId="776" />
  <row Id="7308" PostId="4907" Text="Very simple and very useful. Thanks for the reference to Crawley's text as well!" CreationDate="2010-11-25T16:21:14.343" UserId="776" />
  <row Id="7309" PostId="4362" Text="All of this from a guy that was holed up trying to prove god existed. Elegant, really." CreationDate="2010-11-25T16:29:59.843" UserId="776" />
  <row Id="7310" PostId="4822" Text="Thanks Joris - I just got it (once you square it - it gives you the two sided test)" CreationDate="2010-11-25T16:30:16.737" UserId="253" />
  <row Id="7311" PostId="4907" Text="Be careful - you can't easily fit those kinds of interactions in say a linear model. The interactions occur only in one branch of the tree (or part of). You need a **lot** of data to use these sorts of tools in real world data." CreationDate="2010-11-25T17:38:44.573" UserId="1390" />
  <row Id="7312" PostId="4903" Text="@Brandon: If there is a missing interaction, there will be patterns in the residuals conditional upon values of the covariates. Plotting residuals against the covariates may help determine where an interaction might be appropriate." CreationDate="2010-11-25T17:41:17.440" UserId="1390" />
  <row Id="7313" PostId="4913" Text="Thanks! That's very good to get me started, I'll see how it goes and then mark it as correct answer when I'm capable to judge this. :)  _Note_ I also edited the question a little." CreationDate="2010-11-25T17:46:08.013" UserId="1785" />
  <row Id="7314" PostId="4903" Text="@Gavin, Is there a document, text, or article that describes the shape of these patterns?" CreationDate="2010-11-25T18:56:47.033" UserId="776" />
  <row Id="7315" PostId="4912" Text="X seems irrelevant to your question of whether A is more often positive than B. Unless perhaps you might wish to stratify by X, i.e. ask (1) whether A is more often positive than B when X is positive, then (2) whether A is more often positive than B when X is negative ??" CreationDate="2010-11-25T19:52:03.957" UserId="449" />
  <row Id="7316" PostId="4913" Score="1" Text="I think it's now clear that McNemar's test answers your original question. Fisher's exact test is not relevant here." CreationDate="2010-11-25T19:53:49.447" UserId="449" />
  <row Id="7317" PostId="4913" Text="Thanks onestop, I changed my answer accordingly." CreationDate="2010-11-25T20:11:03.317" UserId="1909" />
  <row Id="7318" PostId="4746" Text="No, you're not missing anything. I was doing the power calc in Stata using its -sampsi- command that uses the normal distribution rather than the *t*-distribution, so R's power.t.test() function is more accurate, especially when the sample size is very small as here. Power calculations are always approximate, so the difference is rarely important." CreationDate="2010-11-25T20:16:42.767" UserId="449" />
  <row Id="7319" PostId="4913" Text="I can use both tests to test the relationship between A and B, Fisher's test tells me whether these are independent indeed, McNemar's seems to be more appropriate when comparing the same measure before and after an event. What I intended to test is more whether A or B have a higher probability, e.g. if I toss two coins 350 times and one turns up heads 80% of the time and the other 20%, how can I calculate the significance of this difference. Sorry for my unclear question." CreationDate="2010-11-25T20:19:30.267" UserId="1785" />
  <row Id="7320" PostId="4912" Text="X seemed irrelevant to my question, but maybe it's not. Still, the question is more along the line of tossing two coins. If one turns up heads 80% of the time and the other 20% of the time, how can I asses this difference. With the coin I have an assumed probability of 0.5, so maybe this is solved differently, I don't have an assumed probability for my example, unless I deduce one from all samples, whether or not **X** is positive." CreationDate="2010-11-25T20:24:22.697" UserId="1785" />
  <row Id="7321" PostId="4746" Text="I guessed you wouldn't want to homogenise your soil columns. Nevertheless, if would help if you can take more soil columns and use baseline measures to select fairly homogeneous ones to use in the experiment, or find group of 2 or more soil columns that are similar on your three outcome measures and randomly allocate equal numbers within each group to each treatment, i.e. a form of blocking." CreationDate="2010-11-25T20:24:49.313" UserId="449" />
  <row Id="7323" PostId="4362" Text="@Joris I am talking about the OP's *subjective* beliefs about Prob(No of heads = 10k | experimenter is not faking) and not about the actual probabilities assuming a fair dice. I would imagine that the OP's subjective beliefs require that Prob(No of heads = 10k | experimenter is faking) be close to 1. Assuming prior probabilities vis-a-vis an experimenter being a liar or not (say Prob(fake experimenter) = 0.5) gives the desired posterior. Does that help clarify?" CreationDate="2010-11-25T22:16:07.317" />
  <row Id="7324" PostId="4890" Text="@steffen If you wish to deploy A if it is effective on a per day basis then you do aggregate to the day level. If on the other hand you do not care if it is inferior on some days then you aggregate it across days. Of course, if you aggregate it across days then as you said you will have one data point which means that you need to run the test many times across weeks/months to assess 'statistical effectiveness'. You need to decide first the answer to the qn: Under what conditions will you conclude that A is effective? and test if those conditions are met via experiments." CreationDate="2010-11-25T22:26:33.933" />
  <row Id="7326" PostId="4840" Text="thanks for your effort mbq&#xA;Yes, i want the mean level in 2006- 2007 or more if that is possible.Regarding data, i guess that is the only data that i got as of now,the measurement is once in a month because the data is available in monthly intervals.if more relevant info is needed then i will try to inquire them after two days.Do you mean the focusing can't be done without other data/info.&#xA;but you mean the focusing can't be done without other data/info." CreationDate="2010-11-25T22:43:27.503" UserId="2108" />
  <row Id="7327" PostId="4362" Text="@Srikant: Point is that getting _exactly_ 10000 out of 20000 is about just as likely as getting _exactly_ 10093 or 9781, being about 0.5^20000. So if you calculate it, P(cheating|10) appx P(not cheating|10) appx 0. In R both are 0 if you use P(10k|cheating) = 0.995, using Bayes theorem. This means by the way that cheaters almost always chose 10k, which is not the assumption OP makes. And on a more philosophical point, you could say that if you believe chance is big he's cheating when heads=10k, chance is bigger he's cheating. Bayes never meant his theorem to be used that way." CreationDate="2010-11-25T22:45:40.313" UserId="1124" />
  <row Id="7328" PostId="4915" Text="Thanks, but what is AIC? I just realized I asked a long winded question when the purpose is quite simple. I just need to get a &quot;normalized&quot; standard deviation. Someone suggested a t-student, but this gets quite cumbersome in excel. Am sure statisticians have found a way to eliminate differences in sample sizes and amplitudes to be able to compare standard-deviation-like measures? Thanks again " CreationDate="2010-11-25T23:02:14.140" UserId="2137" />
  <row Id="7329" PostId="4362" Text="@Joris No the probabilities are not *exactly* equal. You need to also account for the combinatorial coefficients. In any case, we may be talking past each other as you seem be talking about probabilities with a throw of fair dice whereas I am talking about subjective probabilities. " CreationDate="2010-11-25T23:02:35.137" />
  <row Id="7330" PostId="4914" Text=" just realized I asked a long winded question when the purpose is quite simple. I just need to get a &quot;normalized&quot; standard deviation. Someone suggested a t-student, but this gets quite cumbersome in excel. Am sure statisticians have found a way to eliminate differences in sample sizes and amplitudes to be able to compare standard-deviation-like measures?" CreationDate="2010-11-25T23:02:36.120" UserId="2137" />
  <row Id="7331" PostId="4362" Text="@Srikant : appx is not exactly. I am not talking about probabilities with fair dice, I'm calculating posteriors using bayes theorem and your subjective probabilities. Which still makes the posteriors to be essentially almost zero and even a bit smaller. But both posteriors are lower than the chance to win the lottery twice. It's not a philosophical point of view, it's basic math." CreationDate="2010-11-25T23:25:00.857" UserId="1124" />
  <row Id="7332" PostId="4856" Text="Turbulent velocity components (3D) collected by an HR Acoustic Doppler Current Profiler (HRCP) in an estuary." CreationDate="2010-11-26T00:41:06.087" UserId="1637" />
  <row Id="7333" PostId="4857" Text="Thanks fRed, first part is quite useful, but as this calculation is a part of much larger code, I cannot use dlm unless it can be linked with Mathematica Kernel." CreationDate="2010-11-26T00:43:30.033" UserId="1637" />
  <row Id="7334" PostId="4918" Score="1" Text="Calculate in what sense? " CreationDate="2010-11-26T01:02:13.797" UserId="88" />
  <row Id="7335" PostId="4921" Text="Well D itself is a another equation that is calculated by *other* variables: $D = \frac{n_1}{2} * \frac{N_2}{n_2}$. Does this still apply?" CreationDate="2010-11-26T01:25:52.083" UserId="1894" />
  <row Id="7336" PostId="4918" Text="because I want to sample those numbers based on how big they are, so I have to find a threshold, I want give the top 10% numbers a higher chance to be selected." CreationDate="2010-11-26T01:28:36.583" UserId="2141" />
  <row Id="7337" PostId="4857" Text="You can use http://www.scienceops.com/Rlink2.asp&#xA;But too bad you have to pay to use R!" CreationDate="2010-11-26T02:21:28.510" UserId="1709" />
  <row Id="7338" PostId="4915" Text="'what is AIC' is answered here: http://stats.stackexchange.com/questions/834/what-is-aic-looking-for-a-formal-but-intuitive-answer" CreationDate="2010-11-26T02:28:34.507" UserId="1381" />
  <row Id="7339" PostId="4915" Text="this question should also be useful  http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other" CreationDate="2010-11-26T02:38:57.170" UserId="1381" />
  <row Id="7340" PostId="4914" Score="1" Text="I would reccomend that you consider software designed for statisics rather than calculating your own statistics in Excel or using the opaque calculations in the analysis tool pack. Also see: http://stats.stackexchange.com/questions/3392/excel-as-a-statistics-workbench" CreationDate="2010-11-26T02:49:02.903" UserId="1381" />
  <row Id="7341" PostId="4917" Text="Thanks. the Scott B. Morris' article is just what I was looking for. And yes, I agree that I should provide an explanation of the calculation (perhaps I call it something like $\hat{d}$). " CreationDate="2010-11-26T03:01:59.617" UserId="183" />
  <row Id="7342" PostId="4921" Text="If you discard B or E and treat them as equivalent then you are implicitly asserting that V is all that really matters.  If that is the case, you would be better off retaining B in the model as its interpretation is clear.  Further, if you retain E, but D actually has limited variance, the validity of the interpretation of your results would even more suspect (than usual) for different values of D." CreationDate="2010-11-26T03:44:12.047" UserId="196" />
  <row Id="7343" PostId="4558" Text="Is there a standard article that can be cited to dissuade reviewers from demanding this all-too-common practice?" CreationDate="2010-11-26T03:56:18.223" UserId="196" />
  <row Id="7344" PostId="4919" Score="4" Text="Feel free to provide links to Wikipedia, but at least provide a sentence or two of explanation in your answer." CreationDate="2010-11-26T05:00:09.763" UserId="159" />
  <row Id="7345" PostId="4890" Text="@Srikant Thanks again for your response, especially for the question (I was too focused on statistical properties and hence lost the sight of the aim). First I was afraid that I am totally wrong, because it seems that hardly anyone is discussing this issue (at least on the web). Now I am afraid that most AB-Testers do not ask even this question." CreationDate="2010-11-26T07:45:50.260" UserId="264" />
  <row Id="7346" PostId="4926" Text="(+1) for the explanation of PCA." CreationDate="2010-11-26T08:16:29.437" UserId="264" />
  <row Id="7347" PostId="4918" Score="3" Text="Please reformulate the header of the question (more specific, less generic) and complete the last sentence to increase the long-term-value of this question for the site/audience. Thank you :)" CreationDate="2010-11-26T08:19:25.020" UserId="264" />
  <row Id="7348" PostId="4906" Text="Try it, I've provided a full working example. :) You will see the difference if you use str(df.out) on both instances." CreationDate="2010-11-26T08:55:55.973" UserId="144" />
  <row Id="7349" PostId="4928" Text="Sorry now that you have reformulated your question, my answer is not really appropriate" CreationDate="2010-11-26T09:40:03.347" UserId="1709" />
  <row Id="7350" PostId="4928" Text="That's ok! The first formulation was not clear enough, my bad." CreationDate="2010-11-26T11:22:13.900" UserId="1314" />
  <row Id="7351" PostId="4935" Text="@chl Thanks for the references. Incidentally, the paper by Friedman et al partially answers a related question I've just posted, about the relative performance of coordinate descent vs LARS." CreationDate="2010-11-26T11:33:31.097" UserId="439" />
  <row Id="7352" PostId="4926" Text="Thanks, this was a great explanation. I've heard and read about PCA, but this is for a final project for a &quot;regression&quot; graduate course I'm taking, and the professor just wants us to use LR. Regardless, I really appreciate the explanation of PCA and will probably use it myself for fun." CreationDate="2010-11-26T11:38:40.750" UserId="1894" />
  <row Id="7354" PostId="4935" Text="@chl &quot;A brief history of coordinate descent for the lasso&quot; on pp15-17 of the UseR! talk is a gem :-)" CreationDate="2010-11-26T13:14:57.703" UserId="439" />
  <row Id="7355" PostId="4938" Text="(+1) @ogrisel Thanks a lot! Since I'll probably end up having to code this up myself (need it in Java, and haven't seen any open-source Java implementations yet), which algorithm would you say is easier to implement?" CreationDate="2010-11-26T13:24:17.617" UserId="439" />
  <row Id="7356" PostId="4938" Text="both coordinate descent and SGD are easy to implement (check Leon Bottou's webpage for a nice intro to SGD). LARS is probably trickier to get right." CreationDate="2010-11-26T13:25:36.717" UserId="2150" />
  <row Id="7357" PostId="4938" Text="Superb, thanks! I'll check out Léon Bottou's site." CreationDate="2010-11-26T13:28:46.070" UserId="439" />
  <row Id="7358" PostId="4938" Text="@ogrisel (+1) Nice to see you there." CreationDate="2010-11-26T13:31:41.333" UserId="930" />
  <row Id="7359" PostId="4558" Text="The only critique I know is Blouin &amp; Riopelle (2005) but they don't get to the heart of the matter.  I generally don't insist on not showing them but doing something correct as in the effect graphs of Masson &amp; Loftus (2003, see figure 4, right panel... if they were removed from the left one you'd have done it right)." CreationDate="2010-11-26T13:41:01.367" UserId="601" />
  <row Id="7360" PostId="4558" Text="Just to be clear, the problem with those CI's is that they're purely used for inferential reasons with respect to differences among conditions and therefore are worse even than PLSD... in fact I prefer them.  At least they're honest." CreationDate="2010-11-26T13:42:01.457" UserId="601" />
  <row Id="7361" PostId="4938" Text="@ogrisel BTW `scikit-learn` looks very impressive indeed." CreationDate="2010-11-26T13:46:08.923" UserId="439" />
  <row Id="7362" PostId="4931" Text="hi chi, thanks for the answer. When you want you can also answer the identical question at SO and I would accept it there as well. =&gt; http://stackoverflow.com/questions/4280371/how-do-you-test-a-implementation-of-k-means" CreationDate="2010-11-26T13:56:34.980" UserId="2147" />
  <row Id="7363" PostId="4939" Text="If anyone wants me to, I can produce a similar graph as the one I was given and link to it here. This question came from a practice IEEE Certified Software Development Associate exam - the correct answer is apparently &quot;out of control caused by assignable causes&quot;. Unfortunately, I don't know why that's the answer - I said &quot;out of control caused by chance causes&quot; since there aren't a series of points out of control." CreationDate="2010-11-26T14:14:12.487" UserId="110" />
  <row Id="7364" PostId="4908" Text="How many contexts do you have and how many data points per context?" CreationDate="2010-11-26T15:19:05.767" />
  <row Id="7365" PostId="4942" Score="1" Text="$v_0=(M_1+M_2)/2$ seems to be a relatively good candidate (at least by symetry)? " CreationDate="2010-11-26T15:44:21.690" UserId="223" />
  <row Id="7366" PostId="4942" Text="@robin: although $v_0=(M_1+M_2)/2$ or $v_0=P_1M_1+P_2M_2$ (where $P_1$ and $P_2$ are a priori probabilities of classes) are good heuristic solutions, i need the optimum $v_0$. Is it possible to determine it?" CreationDate="2010-11-26T15:51:56.570" UserId="2148" />
  <row Id="7367" PostId="4910" Score="2" Text="@sesqu Hmm...  You ought to rethink that last statement, because it seems you have rejected out of hand some significant advances in statistical concepts and methodology.  The first statement seems to misunderstand what the ecdf is; it perhaps is related to an unusual conception of what a distribution function is.  Are you using a standard definition such as that given at http://mathworld.wolfram.com/DistributionFunction.html ?" CreationDate="2010-11-26T16:01:00.947" UserId="919" />
  <row Id="7368" PostId="4943" Text="I've only taken one course in Engineering Stats, but I seem to remember that you don't stop the process until you have 3 (maybe 2) points that are out of control. However, your second argument makes sense, where a process truly in control wouldn't have 4/15 observations outside of +/- 3 std dev. Unfortunately, I don't have my EngStats book at home to verify that. At least it's plausible. +1 for now, until I can research this some more. But at least it's a starting point." CreationDate="2010-11-26T16:07:35.603" UserId="110" />
  <row Id="7369" PostId="4943" Text="(+1) Good answer.  Alternatively, assuming the standard deviation was previously estimated from a very long series of data, one might wonder about the normality of the distribution.  Furthermore, these 15 points were not likely a random selection: they must have been chosen as a short sequence in which an unusual number of OOC measurements appeared.  The former suggests the chance of a single OOC may be quite a bit greater than 0.01 while the latter indicates that the binomial calculation is misleading. After all, it is virtually *certain* that such a sequence will eventually occur by chance!" CreationDate="2010-11-26T16:11:47.410" UserId="919" />
  <row Id="7370" PostId="4939" Text="Yes, the graph would be useful. As stated in my answer, the look of the chart is important as well, not only which points are outside the control limits." CreationDate="2010-11-26T16:12:23.353" UserId="666" />
  <row Id="7371" PostId="4932" Text="@sfactor: if it's really about programming in R, there are quite some plot people on http://stackoverflow.com that can help you out as well." CreationDate="2010-11-26T16:20:35.110" UserId="1124" />
  <row Id="7372" PostId="4939" Text="I just added a picture of the question, graph included. I marked the correct answer as well." CreationDate="2010-11-26T16:27:07.287" UserId="110" />
  <row Id="7373" PostId="4943" Text="I added a picture that includes the original question and graph." CreationDate="2010-11-26T16:28:00.813" UserId="110" />
  <row Id="7374" PostId="4944" Text="I added a picture that includes the original question and graph." CreationDate="2010-11-26T16:28:56.913" UserId="110" />
  <row Id="7375" PostId="4945" Score="4" Text="I think you'll get more useful help if you take a few steps back, and ask us what scientific question  you are trying to answer. Why are you looking for the smallest normalized SD? " CreationDate="2010-11-26T16:42:55.523" UserId="25" />
  <row Id="7376" PostId="4942" Score="1" Text="@Isaac What is missing from the explanation given at http://en.wikipedia.org/wiki/Linear_discriminant_analysis#Fisher.27s_linear_discriminant ?" CreationDate="2010-11-26T16:51:35.613" UserId="919" />
  <row Id="7377" PostId="4945" Text="I agree with Harvey." CreationDate="2010-11-26T17:04:57.817" UserId="439" />
  <row Id="7378" PostId="4919" Score="1" Text="@babelproofreader: please try to provide context to answers, not just links." CreationDate="2010-11-26T17:19:29.893" UserId="1390" />
  <row Id="7379" PostId="4942" Text="@whuber: then which one is better threshold? $v_0 = V^{T}(M_2+M_1)/2$ or $v_0 = V^{T}(P_1M_1+P_2M_2)$ ? (although generally none of them are optimum)" CreationDate="2010-11-26T17:21:34.763" UserId="2148" />
  <row Id="7380" PostId="4942" Score="1" Text="@Isaac The point of that article is &quot;there is no general rule for the threshold.&quot;  That is, the answer to your question is you cannot find an optimum $v_0$ analytically because there is nothing to be optimized in the general problem.  This situation invites you to examine your data more closely via &quot;analysis of the one-dimensional distribution.&quot;  In other words, *you need more information* in order to find a demonstrably best value for$v_0$." CreationDate="2010-11-26T17:27:32.853" UserId="919" />
  <row Id="7381" PostId="4750" Score="1" Text="I changed &quot;monotonous&quot; (= boring) to &quot;monotonic&quot; (= strictly increasing or strictly decreasing).  However, the second statement is not true: there are nonlinear f's that preserve the mean.  For example, when the RV is symmetric, then *any* f for which f(x - mu) = -f(mu - x) (for all x) will preserve the mean." CreationDate="2010-11-26T17:34:09.803" UserId="919" />
  <row Id="7382" PostId="4943" Score="3" Text="@Thomas It still looks like a bad question to me.  It attempts to measure two concepts (how to read a control chart and the distinction between &quot;assignable&quot; and &quot;chance&quot; causes), which is one mistake, and it punishes the thoughtful test-taker who knows that much more information is necessary to interpret the OOC points than is given here, which is the more egregious mistake." CreationDate="2010-11-26T17:40:26.973" UserId="919" />
  <row Id="7383" PostId="4926" Score="1" Text="In certain circumstances the recommendations in this answer would not work.  For example, what if the true relationship is Y = B + E = V/3000 + V*D?  Then the variables happen to have high correlation due to the ranges of V and D in the dataset--which is (or can be) pure accident--while throwing away either one of B or E will result in the wrong model.  In short, &quot;dependence&quot; is not in general a valid reason for removing some variables from a model; including strongly dependent variables does not necessarily &quot;weaken&quot; a model; PCA is not always the way out. " CreationDate="2010-11-26T17:49:19.910" UserId="919" />
  <row Id="7384" PostId="4938" Score="2" Text="@aix I have edited my answer to add some benchmarks on the current implementations in scikit-learn. Also checkout the [java version of liblinear](http://www.bwaldvogel.de/liblinear-java/) before implementing your own coordinate descent as it might be good enough for you (though you cannot have both L1 and L2 reg at the same time)." CreationDate="2010-11-26T17:49:32.500" UserId="2150" />
  <row Id="7385" PostId="4923" Text="+1: Not only is this suggestion a good approach to the problem in question, it shows that throwing away variables is not always the right (or even a good) approach to solving collinearity problems." CreationDate="2010-11-26T17:52:15.750" UserId="919" />
  <row Id="7386" PostId="4929" Text="+1 for the thoughtful considerations." CreationDate="2010-11-26T17:52:36.610" UserId="919" />
  <row Id="7387" PostId="4938" Text="@ogrisel Many thanks for both the pointer to `liblinear` (I'll check it out right away!) and the graphs. On the graphs, what is the significance of `alpha=0.01`? I take it `alpha` is the penalty parameter. Are we computing the entire regularization path, or only a part of it?" CreationDate="2010-11-26T17:58:28.283" UserId="439" />
  <row Id="7388" PostId="4938" Text="This benchmark is computing the LARS path down to a fixed alpha (usually called lambda, but lambda is a reserverd keyword in python). Here is the [source code of the benchmark](https://github.com/scikit-learn/scikit-learn/blob/544258d58bea1c8d1fe8a57105b206fcd4f23965/benchmarks/bench_lasso.py). In case of the Coordinate descent, alpha is fixed although it is possible to compute an approximate path using a logscaled grid of aphas and using warm restarts." CreationDate="2010-11-26T18:05:57.407" UserId="2150" />
  <row Id="7389" PostId="4942" Text="@whuber: For normal distributions, all we should know is the mean and variance. Then, if we assume the distribution of two classes are normal, it should be possible to determine the $v_0$. but how?" CreationDate="2010-11-26T18:23:43.500" UserId="2148" />
  <row Id="7390" PostId="4926" Text="@whuber, I am not sure I agree with your comments.  I would think &quot;dependence&quot; is in general a pretty valid reason for removing some variables from a regression model.  Otherwise, your regression coefficients can't be reliable.  In the example you use that would be problematic for regression, one simple solution is to use the entire expression (V/3000 + V*D) as a single variable.   " CreationDate="2010-11-26T18:35:27.067" UserId="1329" />
  <row Id="7391" PostId="4908" Text="The problem is that there are a lot of contexts and only one data point per context in common case." CreationDate="2010-11-26T18:37:47.883" UserId="1314" />
  <row Id="7392" PostId="4926" Score="1" Text="@Gaetan More generally, if the model is beta1*(V/3000) + beta2*(V*D) you can't do this: in other words, your suggestion presumes you know a linear constraint among the coefficients.  It is true that the regression coefficients can have *relatively* large VIFs or standard errors, but with sufficient amounts of data--or with well chosen observations--the estimates will be *reliable enough.*  So, we agree there is a problem and indeed I agree with your solution *as one of several alternatives to consider*.  I disagree that it is as general and necessary as you make it out to be." CreationDate="2010-11-26T19:00:32.323" UserId="919" />
  <row Id="7393" PostId="4942" Score="2" Text="@Isaac If I have understood the context correctly it seems we &quot;know&quot; nothing: all parameters are estimates from data.  Thus, at a minimum, we would want some information about the standard errors of these estimates.  If I'm wrong and you know the means and covariance matrix for certain (or have highly accurate estimates), then the problem is one-dimensional: it comes down to separating two normal distributions on the line, which is straightforward to do (this is the basis for power calculations when testing equivalence of means, for instance).  Would that be what you're looking for?" CreationDate="2010-11-26T19:05:32.273" UserId="919" />
  <row Id="7394" PostId="4942" Text="@whuber: we know underlying distribution parameters for certain. I think I got the idea. But what do you meant by _this is the basis for power calculations when testing equivalence of means, for instance_?" CreationDate="2010-11-26T19:33:14.260" UserId="2148" />
  <row Id="7395" PostId="4947" Text="Good point, and interesting links." CreationDate="2010-11-26T19:37:25.663" UserId="1381" />
  <row Id="7396" PostId="4945" Score="3" Text="It's less that the standard deviation is *sensitive* to scale, more that the standard deviation is a *measure of* the scale." CreationDate="2010-11-26T19:37:45.100" UserId="449" />
  <row Id="7397" PostId="4946" Text="@whuber (+1) Your comments on the possible ways to generate relevant synthetic data are very welcome. " CreationDate="2010-11-26T19:53:34.150" UserId="930" />
  <row Id="7398" PostId="4931" Text="@chl (+1) The first paragraph quickly gets to the heart of the matter." CreationDate="2010-11-26T20:07:56.853" UserId="919" />
  <row Id="7399" PostId="4949" Score="1" Text="Is this question the same as http://stats.stackexchange.com/q/4942/919 ?" CreationDate="2010-11-26T20:40:10.320" UserId="919" />
  <row Id="7400" PostId="4907" Score="1" Text="@gd047 As @Gavin said, one of the potential pitfalls is that decision trees need a large sample size and are quite unstable (which is one of the reason bagging and random forests were proposed as viable alternatives). Another problem is that it is not clear whether we seek for second- or higher-order interaction effects. In the former case, CARTs are not a solution. In any case, I will find very doubtful any interpretation of an interaction between 6 variables in any kind of study (observational or controlled)." CreationDate="2010-11-26T20:43:39.283" UserId="930" />
  <row Id="7401" PostId="4949" Text="@whuber Your answer suggests it is the case indeed." CreationDate="2010-11-26T20:47:29.560" UserId="930" />
  <row Id="7402" PostId="4949" Text="@whuber: Yes. i don't know this question suited to which one. I am waiting for a response for one to remove the other one. Is it against the rules?" CreationDate="2010-11-26T20:49:03.353" UserId="2148" />
  <row Id="7403" PostId="4949" Text="@Isaac It might be easier, and surely would be cleaner, to edit the original question.  However, sometimes a question is restarted as a new one when the earlier version collects too many comments that are made irrelevant by the edits, so it's a judgment call.  In any event it's helpful to place cross-references between closely related questions to help people connect them easily." CreationDate="2010-11-26T20:52:35.363" UserId="919" />
  <row Id="7405" PostId="4952" Text="+1 Good points.  Welcome to the site!" CreationDate="2010-11-26T21:09:37.563" UserId="919" />
  <row Id="7406" PostId="4950" Text="OK, thanks for the clarification: it resolves your original question. &quot;Assignable&quot; seems to mean &quot;not attributable to chance,&quot; which is consistent with the dichotomy in the question. What I am struggling with is the assumption that OOC events can *not* be due to chance. This is clearly mistaken, as @HairyBeast has noted. Another striking aspect of the study document is how informal, unquantitative, and *ad hoc* it seems, as in &quot;a number of points&quot; (how many?) and &quot;systematically&quot; (meaning what?). It seems to refer to CUSUM or runs charts without providing appropriate guidelines for their use." CreationDate="2010-11-26T21:15:34.003" UserId="919" />
  <row Id="7407" PostId="4950" Score="2" Text="@whuber I agree fully. Considering this is published and maintained by the IEEE, I expected a whole lot better. I'm just wondering if they are handwaving a bunch of stuff because it's a software engineering certification, and they don't want to get in too much depth in other things. But that's no excuse for some of this confusion." CreationDate="2010-11-26T21:30:04.567" UserId="110" />
  <row Id="7408" PostId="4827" Text="Update - I found that there is a relevant non parametric test - I added it as an answer." CreationDate="2010-11-27T08:01:52.390" UserId="253" />
  <row Id="7410" PostId="4903" Score="1" Text="@Brandon: This is standard model diagnostics and exploratory plotting skills. I would plot the residuals against one of the covariates I think might be a candidate for an interation, conditioned (in the ggplot2 or lattice way) on the values of the covariate I think is involved in the interaction. Stick a loess smoother through each panel to see if there are patterns. Depends on what type of variables your covariates are." CreationDate="2010-11-27T09:20:04.863" UserId="1390" />
  <row Id="7411" PostId="4948" Text="Great, thanks!!" CreationDate="2010-11-27T09:41:35.413" UserId="1785" />
  <row Id="7412" PostId="4918" Score="2" Text="@baboonWorksFine I've edited your title, to make it fit with other questions. As you may have noticed, questions are usually comprised of things like (a) a brief description of the problem at hand, (b) what are the available data actually, and (c) ... a motivated and understandable question. All three ingredients seem lacking there, which is the reason why you get downvoted. Would you pls make a little effort toward describing what your data set looks like, and take the time to finish your last sentence (nobody will ever complete it for you)?" CreationDate="2010-11-27T11:56:43.783" UserId="930" />
  <row Id="7413" PostId="4919" Text="@babelproofreader I've +1 because you addressed the above remarks from @Rob and @Gavin. The term 'software' seems, however, misleading here because the links you referred to have nothing to do with a possible instantiation in any statistical software (as [@Chase](http://stats.stackexchange.com/questions/4918/how-to-get-the-required-data/4925#4925) did), but this is no more than a remark." CreationDate="2010-11-27T12:04:17.800" UserId="930" />
  <row Id="7414" PostId="4960" Score="2" Text="+1 Nice answer!   " CreationDate="2010-11-27T17:10:22.080" UserId="919" />
  <row Id="7415" PostId="4960" Text="Thanks. A very neat way of answering the question (leading to the same answer) was also provided on math.stackexchange (link above in the question) a few minutes ago." CreationDate="2010-11-27T17:24:13.733" UserId="1934" />
  <row Id="7416" PostId="4961" Score="2" Text="An example is ridge regression, which is OLS with a bound on the sum of the squared coefficients. This will introduce bias into the model, but will reduce the variance of the coefficients, sometimes substantially. LASSO is another related method, but puts an L1 constraint on the size of the coefficients. It has the advantage of dropping coefficients. This is useful for p&gt;&gt;n situations Regularizing, in a way, means &quot;shrinking&quot; the model to avoid over-fitting (and to reduce coefficient variance), which usually improves the model's predictive performance." CreationDate="2010-11-27T18:30:09.193" UserId="2144" />
  <row Id="7417" PostId="4958" Text="I derived the posterior expression as the product of Normal likelihood function and Wishart prior. I sampled precision matrix from this posterior expression, but not getting the correct output. I am generating the samples from wishrnd command in matlab, by using the degree of freedom and the scale matrix of the posterior. " CreationDate="2010-11-27T19:13:35.950" UserId="2157" />
  <row Id="7418" PostId="4965" Score="2" Text="(+1) If you have the time and inclination you might strengthen this answer by expanding on your claim that including X1*X2 makes the effect of X1 on Y vary with X2.  Specifically, a model Y = b0 + b1*X1 + b2*X2 + b3*(X1*X2) + error can also be viewed as having the form Y = b0 + (b1 + b3*X2)*X1 + b2*X2 + error, showing precisely how the coefficient of X1--which equals b1 + b3*X2--varies with X2 (and, symmetrically, the coefficient of X2 varies with X1).  That's a simple, natural form of &quot;interaction.&quot;" CreationDate="2010-11-27T21:13:37.667" UserId="919" />
  <row Id="7419" PostId="4960" Score="1" Text="The math answer computes the integrals using linearity of expectation.  In some ways it's simpler.  But I like your solution because it exploits *statistical* knowledge: because you know a sum of independent Exponential variables has a Gamma distribution, you're done." CreationDate="2010-11-27T21:19:29.357" UserId="919" />
  <row Id="7420" PostId="4966" Score="1" Text="In the earlier question, he pointed out that the goal was to compare the SD of raw data with the SD of smoothed data with the SD of highly smoothed data. I don't think the F test (or Bartlett's or Levene's) tests will do any more than complete the circle. The whole point of smoothing is to reduce the variation, so tests that compare variation will tell you smoothed data are smoother. I don't see how this circular logic will get you anywhere." CreationDate="2010-11-27T21:22:33.277" UserId="25" />
  <row Id="7421" PostId="4965" Score="1" Text="@chl - Thanks for the response. The problem that I have is that I have a large `n` (11K) and am using MiniTab to do an Interactions Plot and it takes *forever* to calculate but doesn't show anything. I'm just not sure how I see *if* there is interaction with this dataset." CreationDate="2010-11-27T21:33:33.543" UserId="1894" />
  <row Id="7422" PostId="118" Text="Related question: http://stats.stackexchange.com/q/354/919 (&quot;Bias towards natural numbers in the case of least squares.&quot;)" CreationDate="2010-11-27T21:53:19.177" UserId="919" />
  <row Id="7423" PostId="2207" Score="1" Text="+1 for introducing the idea of loss.  (But aren't &quot;exponential&quot;, etc., *distributions*, not loss functions?) Historically linear loss was the first approach formally developed, in 1750, and there was a straightforward *geometric* solution available for it.  I believe Laplace established the relationship between this and the double-exponential distribution in an 1809 publication (for which the MLE will minimize absolute error, not squared error).  Thus squared loss is not uniquely distinguished by the criteria of having an MLE and being mathematically easy." CreationDate="2010-11-27T21:58:23.167" UserId="919" />
  <row Id="7424" PostId="4965" Score="3" Text="@TheCloudlessSky: One approach is to slice the data into bins according to values of X1.  Plot Y versus X2 bin by bin, looking for changes in slope as the bins vary.  Do the same with the roles of X1 and X2 reversed." CreationDate="2010-11-27T22:05:28.900" UserId="919" />
  <row Id="7425" PostId="4958" Text="It would be useful if you could update your question with the model setup, the assumptions about priors and the corresponding posteriors." CreationDate="2010-11-27T22:34:00.463" />
  <row Id="7426" PostId="1916" Text="I would agree with whuber though that the visualization of the variogram cloud would be really helpful in determining if you choice of model seems appropriate." CreationDate="2010-11-28T07:51:48.753" UserId="787" />
  <row Id="7427" PostId="4951" Text="After trying to find $v_0$ for this problem with a assumption of $\pi_1 = \pi_2 = 1/2$, i realized that $v_0 = -(\mu_1 + \mu_2)/2$ cannot give me the $v_0$ because the $\mu_i$'s are unknown (although $M_1$ and $M_2$ are known) and they directly depend on $v_0$ value. How i can determine $\mu_1$ and $\mu_2$ ? " CreationDate="2010-11-28T09:09:12.043" UserId="2148" />
  <row Id="7428" PostId="4965" Text="@whuber (+1) Thanks for both of your comments." CreationDate="2010-11-28T11:47:22.500" UserId="930" />
  <row Id="7429" PostId="4973" Text="@HairyBeast (+1) Nice points, especially the latest." CreationDate="2010-11-28T11:57:18.307" UserId="930" />
  <row Id="7430" PostId="4961" Text="@HairyBeast You should put your nice comment as an answer. If possible, try to add an illustrative example so that the OP can figure out how it translates to the problem at hand." CreationDate="2010-11-28T12:02:37.223" UserId="930" />
  <row Id="7431" PostId="4967" Text="@whuber Could you elaborate on how you propose to do a binary search in a sorted doubly-linked list?" CreationDate="2010-11-28T12:14:36.850" UserId="439" />
  <row Id="7432" PostId="4972" Score="2" Text="[This question](http://stackoverflow.com/questions/4290081/fitting-data-to-distributions/4290196#4290196) seems directly relevant. See my post for data vis tips to compare your data to other known distributions in base R. " CreationDate="2010-11-28T13:18:20.207" UserId="696" />
  <row Id="7433" PostId="4948" Text="In the denominator, isn't it $s^2/n$ instead? Also, do you need to assume independence of A and B for this test?" CreationDate="2010-11-28T14:35:34.680" UserId="1909" />
  <row Id="7434" PostId="4975" Text="You are right about the QQ plot being applicable to any distribution. I read the question to fast and (for whatever reason) assumed it was a QQ norm plot. One minor note: be careful about concluding a distribution from QQ plots. For example, data that has a t distribution with 20 df will still give you nice QQ norm plots." CreationDate="2010-11-28T15:25:23.290" UserId="2144" />
  <row Id="7435" PostId="4766" Text="I am not really familiar with R but it seems that there is a problem in your implementation: you should only keep the first (k + p) lines (or columns?) of q to actually truncate the svd and make it faster." CreationDate="2010-11-28T16:15:22.510" UserId="2150" />
  <row Id="7436" PostId="4362" Score="1" Text="Putting this in a more general perspective, the point, with which I agree, is that Bayes theorem is at work here.  Specifically there is are alternative likelihoods (corresponding to different generative processes) for cheating and for honest experimenters.  Establishing the cheating is posterior inference with respect to the intuitive and therefore woefully underspecified cheater process." CreationDate="2010-11-28T16:27:58.177" UserId="1739" />
  <row Id="7437" PostId="4362" Text="Happily one can use just this sort of problem to elicit what people really think is 'random' looks like.  There is often, it seems, a rather structured model available for things like a 'random coin flip sequence' (it's got negative auto-correlation, fewer runs, etc.)  Some fun research modelling both the structure of these intuitions and inferences based on them is here: http://cocosci.berkeley.edu/tom/papers/random.pdf and in later papers." CreationDate="2010-11-28T16:33:03.643" UserId="1739" />
  <row Id="7438" PostId="4967" Text="one 'link' allows you to traverse the list in sorted order; the other allows you to traverse in the order in which the elements appear. It is not clear how you would do this with pointers, though, as @aix questions." CreationDate="2010-11-28T17:02:03.763" UserId="795" />
  <row Id="7439" PostId="358" Text="So, if the errors are not normally distributed, but for instance according to another Lévy-stable distribution, it might pay off to use an exponent different from 2?" CreationDate="2010-11-28T18:28:22.097" UserId="2036" />
  <row Id="7440" PostId="4974" Text="Ripley's K is a global measure as well. But I agree with your first point, and one may be interested in simulating local autocorrelation as well as global autocorrelation. The ambiguity of the OP's question makes it difficult to give a more thorough or directly pertinent answer." CreationDate="2010-11-28T18:29:24.397" UserId="1036" />
  <row Id="7441" PostId="4766" Text="i will take a look into that." CreationDate="2010-11-28T19:01:13.553" UserId="2078" />
  <row Id="7442" PostId="4967" Text="@aix I think your intimation is correct; I would need an indexable skip list, not just a sorted doubly-linked list.  The idea is to have a data structure that permits the insertion of one element, the deletion of one element, and finding the median in expected O(log(n)) time (or better)." CreationDate="2010-11-28T19:17:32.337" UserId="919" />
  <row Id="7443" PostId="4948" Text="@caracal Yes, thanks for catching that error.  I have fixed it.  (The calculations in the example were nevertheless correct.)  Independence of A and B is asserted by the OP in the second sentence of the question." CreationDate="2010-11-28T19:19:15.797" UserId="919" />
  <row Id="7444" PostId="4965" Score="2" Text="@chl The trellis display is a nice illustration.  Slicing one variable at equal-interval quantiles is attractive.  There are other approaches.  E.g., Tukey recommended slicing by halving the tails: that is, slice the X2 values into halves at the median, then slice those halves by *their* medians, then slice the *lower* half of the lowest group at its median and the *upper* half of the highest group at its median, and so on, continuing for as long as the new groups have enough data." CreationDate="2010-11-28T19:25:41.377" UserId="919" />
  <row Id="7445" PostId="4951" Text="@Isaac When asked, you specifically said you &quot;know [the] underlying distribution parameters for certain&quot;!  When you don't know them, you'll have to guess.  One approach is to estimate v0, use that to estimate mu1 and mu2 from M1 and M2, then compute the &quot;optimal&quot; v0.  Iterate until the value of v0 does not change.  " CreationDate="2010-11-28T19:29:52.667" UserId="919" />
  <row Id="7446" PostId="4958" Text="There was a bug in my code, its working now." CreationDate="2010-11-28T19:31:12.043" UserId="2157" />
  <row Id="7447" PostId="4951" Text="@whuber: yes, i know the underlying distribution parameters that are $M_1$, $M_2$,$\Sigma_1$, and $\Sigma_2$. But $\mu_1$ $\mu_2$ are not parameters of data distribution, they are parameters of the distribution of $h(x)$ that is a mapping from data to 1-D space. And they depend on $v_0$ value. anyway, I believe a good solution is what you said: _iteration until finding a good $v_0$_. Then is there really no way to find best threshold value for Fisher criterion analytically?" CreationDate="2010-11-28T20:20:33.157" UserId="2148" />
  <row Id="7448" PostId="4972" Text="@Chase: +1 Actually yes :) I think I missed that one. I'll do that rightaway. Thanks a lot." CreationDate="2010-11-28T20:31:06.897" UserId="2164" />
  <row Id="7449" PostId="4973" Text="@HairyBeast: +1 for the suggestions. I updated my post with a negative binomial distribution as well. It looks like it will serve its purpose except that the third bar is not as expected. As for your final point, I heard that if the data does not come from any known distributions, I can use something like a kernel density estimation. Would you suggest this? If so, can you kindly give me a very short example on how to do this for discrete data using R? Would I still be looking at QQ-plots to verify my model?" CreationDate="2010-11-28T20:33:07.660" UserId="2164" />
  <row Id="7450" PostId="4975" Text="@onestop: +1 for the explanation as to why Cauchy does not make sense in this case. That would have been my next question if it were right :) If you get some time, could you kindly take a look at my comment above? In short, because my data need not come from a specific distribution, my readings yesterday revealed that a kernel density estimation technique can be used but am not really sure if this is the right approach and if it is, how one goes about doing it." CreationDate="2010-11-28T20:35:13.667" UserId="2164" />
  <row Id="7451" PostId="4972" Text="@Chase: So I tried out your approach there and obtained a bunch of plots and can confirm that the data does not fall into any of those three distributions that but I'll try the other ones as well and get back. I was just wondering if this is the defacto approach. Do I go through every possible distribution and look at the QQ-plots to figure out if my data fits into that distribution? " CreationDate="2010-11-28T20:45:05.657" UserId="2164" />
  <row Id="7452" PostId="4965" Text="@whuber That's again a good point. I will have a look at possible R implementation, or try it myself." CreationDate="2010-11-28T20:52:55.740" UserId="930" />
  <row Id="7453" PostId="4972" Score="2" Text="@Legend You can also try a [rootogram](http://rss.acs.unt.edu/Rdoc/library/vcd/html/rootogram.html) (don't know if it overlaps with @Chase's response on SO). Now, I don't understand why you want to try and fit every discrete distribution to your data. Either you have *a priori* knowledge or hypothesis about the law of your outcomes, or you don't. In the former case, you might want to explain why the observed data don't fit the model. In the latter case, you're left with exploratory data analysis (and, potentially, non-parametric density estimates, mixture models, etc.)" CreationDate="2010-11-28T21:12:47.680" UserId="930" />
  <row Id="7454" PostId="4972" Text="@chl: Thank you for the pointers. I will look into it now. I think my problem fits more into an exploratory domain because what I want to do is to see if I can get some model to fit this data so that I can use it in a discrete simulation. So for example, I want to evaluate a few scenarios for the case when the population is not really 100 but say 500. In that case, I am trying to find a model that would let me generate the numbers that I could get from the survey data. Please correct me if I am taking the wrong path though." CreationDate="2010-11-28T21:22:06.623" UserId="2164" />
  <row Id="7455" PostId="4972" Score="1" Text="@Legend 'Scenario' means that you already have some hypothesis, no? It's difficult to answer your question because you seek to fit the 'best' model (in the sense of goodness-of-fit) to your data, but it is not necessary the 'correct' model. After all, your data may be subjected to measurement error or any other sources of error. Finally, you can still work with your observed sample and use bootstrap to simulate new samples." CreationDate="2010-11-28T21:31:32.930" UserId="930" />
  <row Id="7456" PostId="4972" Text="@chl: Oh I think I am just using the wrong terminology here. I will try to correct myself as I am still reading some material. But for now, in my opinion, a scenario for me is to simulate the communication patterns inside a small city with a set number of people. I am trying to evaluate how much time it takes for some arbitrary piece of information to propagate for a population of size `X`. As this depends on the communication patterns of people, what I was trying to do was to build a model that could use the information from the 100 people I surveyed to give me patterns for the `X` number." CreationDate="2010-11-28T21:43:27.720" UserId="2164" />
  <row Id="7457" PostId="4972" Text="@chl: As an additional note, would you mind explaining a little more about bootstrapping to simulate new samples? Are you suggesting that I go ahead with the 'not-so-perfect' model as a starting point or are you suggesting something different? Thank You once again." CreationDate="2010-11-28T21:44:41.807" UserId="2164" />
  <row Id="7458" PostId="4951" Text="@Isaac Sorry, I completely misremembered what is going on here.  I should reread my own answers!  I defined mu1 and mu2 in the first sentence: if you know M1, M2, and the covariances, that determines V which determines mu1 and mu2." CreationDate="2010-11-28T22:22:33.830" UserId="919" />
  <row Id="7459" PostId="4979" Score="1" Text="(+1) That 2002 thread is a hell of an exchange of ideas." CreationDate="2010-11-28T22:50:49.563" UserId="919" />
  <row Id="7460" PostId="4362" Text="@Conjugate Prior: Yes, bayes theorem is at work here. But with Bayes theorem, as I calculated for you, both **posterior** probabilities mentioned are close to zero (smaller than R can cope with actually). So the posterior conclusion, based on Bayes theorem, when you have exactly 10k, is that the researcher is not cheating. Why talking philosophy if you have numbers? " CreationDate="2010-11-29T01:14:41.683" UserId="1124" />
  <row Id="7461" PostId="4362" Text="@Conjugate Prior. Plus, the paper you refer to mentions that people are likely to introduce less runs when they consider a &quot;random&quot; process (p=0.6 instead of 0.5). So it rejects intuition rather than confirm it. The paper also talks about sequences and patterns, but that is not what is discussed here. HTTHHT is more likely than HHHTTT according to that paper, but both have a p of exactly 0.5" CreationDate="2010-11-29T01:20:21.883" UserId="1124" />
  <row Id="7462" PostId="4987" Score="1" Text="Please do *not* cross-post simultaneously on SO and here. " CreationDate="2010-11-29T01:53:11.730" UserId="334" />
  <row Id="7463" PostId="4987" Score="1" Text="@Dirk Eddelbuettel: Deleted my other post. I wasn't sure if this comes under programming or pure statistics.. Anycase, thanks for pointing it out." CreationDate="2010-11-29T03:14:03.983" UserId="2164" />
  <row Id="7464" PostId="4908" Text="I am not sure that you can do statistics on the player's strategy without more data. Either you have enough data and you can train a machine learning algorithm, or you don't and you have to do assumptions on the underlying model you want to use." CreationDate="2010-11-29T03:17:40.347" UserId="1709" />
  <row Id="7465" PostId="4908" Text="Well you can try but I would not be very confident with the results. Do you want to invest real money into this strategy?" CreationDate="2010-11-29T03:19:09.300" UserId="1709" />
  <row Id="7466" PostId="4766" Text="the dimension of Q is ( N ) X ( K + P )" CreationDate="2010-11-29T04:32:58.047" UserId="2078" />
  <row Id="7467" PostId="4972" Score="1" Text="@Legend Bootstrap is useful to estimate, based on an observed sample, the variability of an estimator when you don't know (or don't want to assume) its law. But in your case, given the context you added, I would suggest to update your question so that people can have a better idea of what you really intend to do with (which is beyond simple distribution fitting, apparently)." CreationDate="2010-11-29T06:01:19.877" UserId="930" />
  <row Id="7468" PostId="4908" Text="@fRed I have machine learning algorithm, but I'm not sure about its reliability. No real money, this is an academic research." CreationDate="2010-11-29T06:07:14.863" UserId="1314" />
  <row Id="7469" PostId="4990" Text="@suncoolsu: Awesome! Thanks a lot for your time and explanation. I have accepted this as an answer but can you kindly explain how to interpret my original graph? " CreationDate="2010-11-29T06:52:43.410" UserId="2164" />
  <row Id="7470" PostId="4972" Text="@chl: Thank You for the suggestions. I updated my question to reflect the latest discussion." CreationDate="2010-11-29T06:58:32.767" UserId="2164" />
  <row Id="7471" PostId="4973" Text="@HairyBeast: I updated my question to explain more about what I am trying to achieve." CreationDate="2010-11-29T06:59:06.120" UserId="2164" />
  <row Id="7472" PostId="4975" Text="@onestop: I updated my question to explain more about what I am trying to achieve." CreationDate="2010-11-29T06:59:52.097" UserId="2164" />
  <row Id="7473" PostId="4990" Text="@Legend. As I pointed out, your graph plots the sorted data points (y in our case) vs the corresponding values of negative binomial random variable which correspond to the value of probabilities `ppoints(data)` in your case. Therefore, the last point in your graph is (175, 170) which is below the `abline(0,1)`. " CreationDate="2010-11-29T07:34:08.003" UserId="1307" />
  <row Id="7475" PostId="4991" Text="My question is a bit similar to  http://stats.stackexchange.com/questions/2149/what-is-the-difference-between-a-particle-filter-sequential-monte-carlo-and-a-k. I have re opened a question on purpose as the situation is a bit different and I would like different opinions. (The answer by gd047 was mainly focusing on unscented Kalman filter (UKF) )" CreationDate="2010-11-29T07:54:44.913" UserId="1709" />
  <row Id="7476" PostId="4988" Text="This made me discover there is some ML underground on TCS... Neh." CreationDate="2010-11-29T15:22:18.673" UserId="88" />
  <row Id="7477" PostId="4995" Text="@Mike Seems like you have two SE accounts, no?" CreationDate="2010-11-29T16:15:34.593" UserId="930" />
  <row Id="7478" PostId="4988" Text="@Yaroslav I am curious: why did you specify the &quot;graphical-model&quot; tag for this question?" CreationDate="2010-11-29T16:18:05.830" UserId="919" />
  <row Id="7479" PostId="4995" Text="@chl: I resolved this by merging the accounts." CreationDate="2010-11-29T16:21:32.887" UserId="5" />
  <row Id="7480" PostId="4979" Text="Very nice links, thanks" CreationDate="2010-11-29T17:07:57.063" UserId="2171" />
  <row Id="7481" PostId="4984" Text="Thanks. I didn't think that &quot;The logic of science&quot; talked about this stuff too, i'm definitely going to read that book." CreationDate="2010-11-29T17:15:08.090" UserId="2171" />
  <row Id="7482" PostId="4998" Score="2" Text="This question is not a very clear one... " CreationDate="2010-11-29T17:49:18.977" UserId="88" />
  <row Id="7483" PostId="4998" Text="@Raphael - what's the significance of the four categories? Do they have different critical values for `W`? How do you have a negative count? " CreationDate="2010-11-29T17:50:34.893" UserId="696" />
  <row Id="7485" PostId="4960" Score="1" Text="I enjoyed it quite a bit and I am by no means a statistician or a mathematician." CreationDate="2010-11-29T18:04:00.000" UserId="1673" />
  <row Id="7488" PostId="4988" Text="added clarification" CreationDate="2010-11-29T18:31:09.497" UserId="511" />
  <row Id="7489" PostId="5001" Text="When I initially asked this question I was planning to add this answer myself, besides alternatives I didn't know. Thanks for answering, I will accept this answer as the accepted as it is a 'standard' in the ASR world!" CreationDate="2010-11-29T19:12:35.613" UserId="190" />
  <row Id="7490" PostId="5004" Score="3" Text="Doesn't http://en.wikipedia.org/wiki/Kalman_filter#Unscented_Kalman_filter answer this question?" CreationDate="2010-11-29T19:26:17.420" UserId="919" />
  <row Id="7491" PostId="5003" Text="My understanding is that lme4, by default uses REML where glm uses ML.  They might be comparable if you made lmer use ML by setting REML = FALSE." CreationDate="2010-11-29T20:46:48.177" UserId="196" />
  <row Id="7492" PostId="5003" Text="In addition to your Gavin's comment, it also depends what do you want to do with the model. Is the model for prediction or Thomas is looking for parsimony ? (I think)" CreationDate="2010-11-29T20:52:14.553" UserId="1307" />
  <row Id="7493" PostId="5003" Score="1" Text="@drnexus: I don't think that is sufficient; you have to be sure that the same normalising constant is being applied in the log likelihood calculation." CreationDate="2010-11-29T20:52:20.247" UserId="1390" />
  <row Id="7494" PostId="5007" Score="2" Text="@drknexus - if you can share your data, that would be helpful, or point to a similar dataset within R that has the same shape as your data." CreationDate="2010-11-29T21:09:50.747" UserId="696" />
  <row Id="7495" PostId="5007" Text="@Chase: Added an example dataset." CreationDate="2010-11-29T21:30:44.347" UserId="196" />
  <row Id="7496" PostId="5009" Text="I think what I'm missing is a function like scale_color_discrete that ties into the &quot;group&quot; or &quot;lty&quot; specification in ggplot(data=ex.daata, aes(x=V4, y=DV, group=V3, lty=V3))" CreationDate="2010-11-29T21:31:59.447" UserId="196" />
  <row Id="7497" PostId="5007" Text="@drknexus - can you post the code as you currently have it?" CreationDate="2010-11-29T21:34:52.863" UserId="696" />
  <row Id="7498" PostId="5009" Score="1" Text="Ah I found it: scale_linetype_discrete(name=&quot;bob&quot;)" CreationDate="2010-11-29T21:35:34.700" UserId="196" />
  <row Id="7499" PostId="5009" Text="@Chase: Edit with the scale_linetype_discrete bit and I'll accept your answer." CreationDate="2010-11-29T21:36:10.673" UserId="196" />
  <row Id="7500" PostId="5009" Score="1" Text="@drknexus - the issue with that is in my working example, the appropriate command is `scale_colour_discrete()` and the code you are working with is obviously a bit different. I don't know what that code is..." CreationDate="2010-11-29T22:09:51.647" UserId="696" />
  <row Id="7501" PostId="5003" Text="@suncoolsu: I'm not trying to use this model for prediction.&#xA;&#xA;@Gavin: How would I know about normalising constants? I confess my understanding of what R is doing behind the scenes is minimal." CreationDate="2010-11-29T22:17:59.363" UserId="2182" />
  <row Id="7502" PostId="5003" Score="2" Text="@Thomas: for that you'd need to look at the code or speak to the person who wrote it to be sure. In general, assume the likelihoods aren't comparable across different software/packages/functions." CreationDate="2010-11-29T22:46:26.327" UserId="1390" />
  <row Id="7503" PostId="4996" Text="I think you are right, I may be asking the wrong question" CreationDate="2010-11-29T22:47:55.340" UserId="511" />
  <row Id="7505" PostId="5007" Text="Can someone with the editing powers hit that nifty 101010 button and format the code above?" CreationDate="2010-11-29T23:00:11.650" UserId="696" />
  <row Id="7506" PostId="5007" Text="@Chase The asker should be able to do that himself." CreationDate="2010-11-29T23:07:16.503" UserId="930" />
  <row Id="7507" PostId="4986" Text="I only have an implementation of the Bayesian LASSO for linear regression and the LASSO for logistic regression currently uploaded. Appologies." CreationDate="2010-11-29T23:12:38.550" UserId="530" />
  <row Id="7508" PostId="5009" Score="3" Text="@Chase Re: &quot;why we have to treat V3 as a factor&quot; `scale_linetype_discrete` expects a variable with discrete values (factor or character, from `plyr::is.discrete`), check out the src code on Github, http://j.mp/ejaRRT. Nice response (+1)." CreationDate="2010-11-29T23:29:35.103" UserId="930" />
  <row Id="7509" PostId="5003" Text="Is AIC the only thing that you have? Try using, may be, BIC. But again, going back to Gavin's comment, the comparison of likelihoods across package may not be &quot;consistent&quot;." CreationDate="2010-11-30T01:40:08.627" UserId="1307" />
  <row Id="7511" PostId="5017" Text="Thanks for the answer. These are significances of large sets of genome regions; typically a hypergeometric or binomial test is used in cases like these, and p-values on the order of 10^-50 to 100 are not at all uncommon (if you pass in a set of regions that turn out to all affect limb development, you get a pretty small p-value for a &quot;limb&quot; test). I initially dismissed Monte Carlo as too slow, but I don't care too much about speed so I will definitely experiment some with that." CreationDate="2010-11-30T06:02:35.643" UserId="2111" />
  <row Id="7512" PostId="4560" Text="It's fine if you use cross-validation, and don't extrapolate. Don't publish the p-values though, as they are meaningless. " CreationDate="2010-11-30T06:16:54.457" UserId="74" />
  <row Id="7513" PostId="5014" Score="2" Text="It's a little nitpicky but the CLT doesn't tell us that the 'sample' itself will be normally distributed." CreationDate="2010-11-30T06:16:55.343" UserId="1028" />
  <row Id="7515" PostId="2533" Text="Although I personally don't mind too much LMGTFY links (and I agree a Google search is all that's needed here), use of LMGTFY links is quite frowned upon around here. http://meta.stackoverflow.com/questions/15650/ban-lmgtfy-let-me-google-that-for-you-links" CreationDate="2010-11-30T06:37:59.183" UserId="582" />
  <row Id="7516" PostId="5017" Text="@bnaul: No idea how people work in your field, but contemplating differences on a 10^-50 or 10^-100 level?? Even with n=1000 I image that would be hard to back up with actual data. Hope you people can sleep well ;)" CreationDate="2010-11-30T06:41:14.363" UserId="56" />
  <row Id="7517" PostId="5015" Text="This in indeed a very interesting question, but maybe (only maybe) more appropriate for http://math.stackexchange.com/ ?" CreationDate="2010-11-30T07:08:17.360" UserId="264" />
  <row Id="7518" PostId="5012" Text="Why (2k+1)/n? I think it's k/n. For example, if my range is of length k=10 and my given point is 50, the range must fall on [41,50] or [42,51] or [43,52] or ... [50,60] in order to cover this point (10 options). The simplest case - the range is of length k=1 - it has only one option to cover the point (fall exactly on it)." CreationDate="2010-11-30T09:07:29.843" UserId="634" />
  <row Id="7519" PostId="5012" Text="And in the general case, when we look at the probability a randomly mapped range of length k covers a given range of length m on a circumference of length g, we get p=(k-m+1)/n. When m=1, as in the original post, this is indeed k/n." CreationDate="2010-11-30T09:12:10.213" UserId="634" />
  <row Id="7521" PostId="5023" Score="1" Text="'Non-independent' is a bit vague; there are many causes and models for dependence (clustered data, hierarchical/multilevel models, time series, ...). Could you be a bit more specific on your area of interest?" CreationDate="2010-11-30T10:31:42.940" UserId="449" />
  <row Id="7522" PostId="5026" Score="1" Text="This should be CW." CreationDate="2010-11-30T11:27:31.353" UserId="930" />
  <row Id="7523" PostId="5026" Text="@chl can't you do it with your rep? I can't do it in with the edit link." CreationDate="2010-11-30T11:30:46.923" UserId="2199" />
  <row Id="7524" PostId="5014" Text="@Chase In python, the AD test for normal, exponential, or Gumbel dist. is available in the [scipy](http://www.scipy.org/) toolkit (see `anderson()` in `morestats.py`)." CreationDate="2010-11-30T11:33:05.310" UserId="930" />
  <row Id="7525" PostId="5026" Text="@Olivier No. This has to be done by a mod. We can't set question as CW ourselves now that we're out of the beta stage :(" CreationDate="2010-11-30T11:35:42.297" UserId="930" />
  <row Id="7527" PostId="5026" Score="2" Text="(+1) - a very nice question" CreationDate="2010-11-30T11:51:43.107" UserId="264" />
  <row Id="7528" PostId="5003" Text="@suncoolsu: BIC is available (as is anything that's been written for R). But obviously I don't want to use the numbers it spits out unless I'm confident they're really comparable." CreationDate="2010-11-30T11:55:51.270" UserId="2182" />
  <row Id="7529" PostId="5023" Text="Yes, I'm mostly interested in dependencies between organisms. More specifically, a virus that would have evolved over time by means of mutations would lead to several different strains, which are all related in some way because they were derived from the same sequence. " CreationDate="2010-11-30T12:20:01.320" UserId="2197" />
  <row Id="7530" PostId="4987" Score="1" Text="@Legend - I came across [this article](http://cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf) regarding QQ plots and various distributions and thought you may appreciate reading through it." CreationDate="2010-11-30T12:58:42.083" UserId="696" />
  <row Id="7531" PostId="5028" Text="@mariana Here is a guide how to use the Markdown format: http://stats.stackexchange.com/editing-help" CreationDate="2010-11-30T13:29:10.487" UserId="88" />
  <row Id="7532" PostId="5026" Score="1" Text="fuzzy heterogeneous overlapping concepts cannot be distinguished by something else than a consensus. A consensus must involve a lot of people to be interesting. I don't see an interest in involbing a lot of people in that type of question. " CreationDate="2010-11-30T13:29:21.143" UserId="223" />
  <row Id="7533" PostId="4987" Text="@Chase: Awesome! Looks like it discusses a number of things. I'll read it right away. +1 Thank you very much. " CreationDate="2010-11-30T14:33:35.590" UserId="2164" />
  <row Id="7534" PostId="5023" Text="The mutations are still random and independent. It's the selection process that is not random and introduces dependencies (although which selection process is acting could be random). Or maybe you have more complex selection processes in mind where the specific proportion of genotypes can act as a selector as well, which would be an explicit dependence?" CreationDate="2010-11-30T14:44:50.293" UserId="2036" />
  <row Id="7535" PostId="5014" Text="@Dason - nice catch and good point. I modified my answer to more accurately reflect that subtlety. @chl - thanks for the reference. I don't have any experience with python, I'm still trying to grok R in my spare time!" CreationDate="2010-11-30T14:53:28.027" UserId="696" />
  <row Id="7537" PostId="5023" Text="No, what you describe seems to fit the situation, but an idea of what has been used so far to model that selection could still be helpful." CreationDate="2010-11-30T15:17:59.347" UserId="2197" />
  <row Id="7539" PostId="5023" Text="What do you know about the computational side of phylogenetic trees? There's a whole mathematical field flourishing around that type of problems. I'm no expert, but I know of a few papers using that approach. I'll try to dig them up and post them as a reply." CreationDate="2010-11-30T15:51:36.373" UserId="2036" />
  <row Id="7540" PostId="5012" Text="@David You're right, thank you.  I was (confusedly) equating &quot;length&quot; with &quot;radius.&quot;  But this does not render the explanation any less valid.  The question is whether the discrepancies you are seeing are as predicted (by the corrected formulas)." CreationDate="2010-11-30T15:58:01.030" UserId="919" />
  <row Id="7543" PostId="5015" Score="1" Text="I think this is a perfectly reasonable question on computation statistics." CreationDate="2010-11-30T16:36:24.897" UserId="8" />
  <row Id="7544" PostId="5016" Text="Where do the empirical estimates come from and in what form are they?  (In a brief test Mathematica computed the convolution of 1000 discrete RVs, each specified by ten parameters, in just a few seconds, so I don't understand why computing the exact distribution is so slow.)" CreationDate="2010-11-30T16:42:26.920" UserId="919" />
  <row Id="7545" PostId="4960" Text="very elegant answer." CreationDate="2010-11-30T16:44:37.633" UserId="96" />
  <row Id="7546" PostId="5014" Text="@Chase where is the modification you promised to @Dason?  Your answer still asserts that all &quot;samples&quot; of n &gt;= 30 will be &quot;normal,&quot; which is plainly wrong.  That's no &quot;nitpick&quot;: it's fundamental." CreationDate="2010-11-30T17:10:39.423" UserId="919" />
  <row Id="7547" PostId="5026" Score="3" Text="@Robin Girard I sympathize with your sentiment, but in some cases much can be revealed without arriving at a consensus at all--and I think this may be such a case.  This question has the potential to reveal the different ways that people think about and approach a set of seemingly related questions, potentially shedding light on all the disciplines with which this site is engaged." CreationDate="2010-11-30T17:14:48.950" UserId="919" />
  <row Id="7549" PostId="5005" Text="Whether the Poisson distribution is a &quot;good&quot; approximation depends on your purpose.  In this situation, a general rule of thumb sheds no light.  You have noted a discrepancy; one obvious place to look for an explanation lies in using the Poisson as an approximation to the correct distribution, which is Binomial (that's not in question, I hope).  Another place, indicated in your update, lies in using ranges of &quot;different sizes&quot;: now all bets are off.  You have a mixture of Binomials, which should create some overdispersion: http://en.wikipedia.org/wiki/Overdispersion" CreationDate="2010-11-30T17:26:47.693" UserId="919" />
  <row Id="7550" PostId="5014" Text="@whuber - Feel free to modify my answer as you see fit, you almost certainly have more expertise in formal statistics than I do. My answer above states that a sampled population will be &quot;approximately&quot; normal as `n` increases with `n == 30` as the generally accepted rule of thumb for minimum `n`. That language comes directly out of my stats 101 text book and class notes. I understand &quot;approximately&quot; is a bit of a vague term though unfortunately I don't have the experience or expertise to offer any further insight into defining &quot;approximately&quot;. " CreationDate="2010-11-30T17:26:56.787" UserId="696" />
  <row Id="7551" PostId="5014" Text="@Chase The potential for confusion here is that your stats textbook is referring to the distribution of the *sample mean,* not the sample data.  As you draw more and more data from *any* non-normal distribution, it will more and more clearly show itself to be non-normal! -- but, under mild assumptions, the *means* of many independent such samples will be almost normally distributed." CreationDate="2010-11-30T17:30:00.263" UserId="919" />
  <row Id="7552" PostId="5014" Text="It's not the *sample* itself that starts to resemble a normal distribution, but the distribution of certain *statistics* (like the mean) that will be approximately normally distributed." CreationDate="2010-11-30T17:31:50.787" UserId="1934" />
  <row Id="7553" PostId="5013" Score="1" Text="Out of curiosity, is the misspelling of &quot;theoretical&quot; in the plot built into the matplotlib and scipy.stats packages?" CreationDate="2010-11-30T17:31:55.287" UserId="919" />
  <row Id="7554" PostId="5014" Text="@whuber Got it, I understand the difference now - thanks for the clarification. If that's the case, then the CLT doesn't seem terribly applicable after all and I'm just babbling on confirming my idiocy in front of all of the interwebs...right?" CreationDate="2010-11-30T17:40:02.563" UserId="696" />
  <row Id="7555" PostId="5009" Text="@chl (+1) for the link to Github, very informative." CreationDate="2010-11-30T17:43:23.793" UserId="696" />
  <row Id="7557" PostId="5003" Text="@Thomas. It's great that you want to understand the numbers compared to just using it. AIC is known to overfit, but if the model is meant to be used for prediction, this is fine. However, if you think the true model lies in your present &quot;model space&quot;, BIC may give you better or parsimonious answers. " CreationDate="2010-11-30T17:48:25.230" UserId="1307" />
  <row Id="7558" PostId="5003" Text="@suncoolsu: Thanks :). This is whole-organism biology, so I think the 'true model' is probably not achievable." CreationDate="2010-11-30T17:51:38.253" UserId="2182" />
  <row Id="7559" PostId="5026" Score="1" Text="@whuber I sympathize with your compent too (also I sympathize with your simpathization)" CreationDate="2010-11-30T18:06:16.270" UserId="223" />
  <row Id="7560" PostId="5014" Score="1" Text="@Wolfgang Very nicely put!  In two lines you said what I could not express in 10.  @Chase Look at it this way: for the price of writing a few comments online and suffering the criticism of fools like me you have today learned something important that was somehow overlooked in an entire semester's course.  That's an efficient way to learn (and the price is right, too).  You will come across as quite knowledgeable once you fix that initial response :-)." CreationDate="2010-11-30T18:14:03.113" UserId="919" />
  <row Id="7561" PostId="5016" Text="@whuber They're stored as histograms, i.e. lists of frequencies. My problem is that I have to ~20000 of these computations in a reasonable time, so a few seconds is unfortunately not fast enough." CreationDate="2010-11-30T18:40:07.007" UserId="2111" />
  <row Id="7562" PostId="5005" Text="@whuber Indeed, I have a mixture of Binomials, and it seems sometimes the Poisson approximation is not very good. My question is what are my options for getting a p-value for an actual observation. One way would be to run many simulations and just get an empirical p-value. This is the simplest way I know, but it uses nothing of the info I have -- I do now it's a mixture of Binomials and I know all their p's -- isn't there a way to use this valuable information?" CreationDate="2010-11-30T19:13:44.810" UserId="634" />
  <row Id="7563" PostId="5016" Score="1" Text="@bnaul Summarize each histogram with a low-order truncation of its cumulant generating function and add those functions to obtain an Edgeworth-type expansion of the sum (http://en.wikipedia.org/wiki/Edgeworth_series ).  The computational effort is proportional to the total number of bars in all those histograms." CreationDate="2010-11-30T19:21:55.847" UserId="919" />
  <row Id="7564" PostId="5032" Score="1" Text="Thanks for your answer. It was very useful. Actually, I wasn't very precise in the formulation of the second part of the question. Imagine, that x-s and y-s are positive numbers, measured in the same units. The z-s (observed outcome) somehow measure the &quot;interaction&quot; in that sense that if there is no interaction the z-s should be (x+y)/2 (expected outcome). So from my point of view it was the same to use regression with the null hypothesis a=b=1/2 or to compare goodness of fit using Pearson's chi^2 statistics. Does this make any sense? Thanks!" CreationDate="2010-11-30T20:28:06.243" UserId="1215" />
  <row Id="7565" PostId="5034" Text="Wolfgang, thanks for your answer! " CreationDate="2010-11-30T20:45:51.780" UserId="1215" />
  <row Id="7566" PostId="5032" Score="1" Text="@Lan I think Wolfgang's answer nicely illustrates how to make the test you are proposing.  It is an example of what meant by testing a hypothesis &quot;in the usual way.&quot;" CreationDate="2010-11-30T20:56:56.467" UserId="919" />
  <row Id="7567" PostId="4893" Text="I was mistaken by the term 1. This is a linear separating rule in $\mathbb{R^{5}}$. :'(" CreationDate="2010-11-30T22:11:37.123" UserId="1351" />
  <row Id="7569" PostId="4889" Text="@Brandon - I think the colnames line above needs to be rewritten as `colnames(df)[1:2] &lt;- c(&quot;variable&quot;,&quot;value&quot;)`?" CreationDate="2010-11-30T22:53:43.337" UserId="696" />
  <row Id="7570" PostId="5013" Text="Good eye for details =)" CreationDate="2010-11-30T23:24:41.097" UserId="1934" />
  <row Id="7571" PostId="5013" Text="@whuber: That would by my speling of the title `;-)`" CreationDate="2010-11-30T23:26:24.557" UserId="2191" />
  <row Id="7572" PostId="5003" Text="@suncoolsu: OK in whole-ogranism biology I am sure BIC will give terrible answers. (I guess: a la &quot;big p small n&quot;)" CreationDate="2010-11-30T23:28:41.393" UserId="1307" />
  <row Id="7573" PostId="5039" Score="1" Text="@Wolfgang (+1) Good response. I would add that these are often particular contrasts that are interesting in DoE as analyzed through ANOVAs, and that the default ones (as returned by `summary.aov(lm(...))` or `anova(lm(...))` are not of particular interest." CreationDate="2010-12-01T00:17:44.793" UserId="930" />
  <row Id="7574" PostId="5039" Text="@chl: It's a good point.  Although, I think there is something to be said for actually constructing the contrasts for the hypotheses that you are interested in testing.  There is no fundamental distinction between ANOVA and regression.  Sometimes, ANOVA routines hide the hypotheses that are being tested.  At the same time, ANOVA procedures sometimes package things in exactly the way that you want to see the results.  " CreationDate="2010-12-01T01:47:21.767" UserId="485" />
  <row Id="7575" PostId="546" Text="Bananas are always tasty." CreationDate="2010-12-01T04:00:58.017" UserId="1577" />
  <row Id="7576" PostId="4750" Text="You are right, of course. I fixed my approach." CreationDate="2010-12-01T05:58:20.207" UserId="2456" />
  <row Id="7577" PostId="5038" Text="This is a near-duplicate of http://stats.stackexchange.com/questions/555/why-is-anova-taught-used-as-if-it-is-a-different-research-methodology-compared" CreationDate="2010-12-01T06:48:25.580" UserId="449" />
  <row Id="7578" PostId="5039" Text="Thank you sir :) So I would first look at the anova, if the factor made any difference at all, before I looked at the lm()-output to see _which_ factors made the difference? But doesn't the lm()-output already tell me if the different factors are significant (with the 1-3 stars behind them)?" CreationDate="2010-12-01T07:38:43.523" UserId="2091" />
  <row Id="7580" PostId="5039" Score="1" Text="Not quite. The output from lm() above tells you whether there is a significant difference between level A (the reference level) and levels B and C. It does not tell you whether levels B and C are significantly different from each other. Of course, you could make level B (or C) your reference (with the relevel() function), but suppose you have a factor with many levels. Checking all pairwise contrasts between levels will become tedious. Plus, you are conducting lots of tests. The alternative is to first inspect the omnibus test to check whether there is anything going on to begin with." CreationDate="2010-12-01T09:25:25.797" UserId="1934" />
  <row Id="7581" PostId="5039" Text="Ha! Thanks a lot. It seems like I didn't understand that lm() only tested the factors against the reference level." CreationDate="2010-12-01T10:23:52.460" UserId="2091" />
  <row Id="7582" PostId="5015" Text="@csgillespie: All I know about splines and interpolation I learned in numeric/math lectures. Hence I may be a little biased ;)." CreationDate="2010-12-01T11:10:05.563" UserId="264" />
  <row Id="7584" PostId="5049" Text="I think you made a slight mistake in the second paragraph: $|X|/|A| \approx m/n$. Otherwise, what you are doing is basically reinventing Monte Carlo integration, well, the subset version I have not encountered yet, but I'd not be surprised if it's done already." CreationDate="2010-12-01T12:12:02.703" UserId="2036" />
  <row Id="7585" PostId="432" Text="Definitively my favorite cartoon about Data Mining" CreationDate="2010-12-01T12:24:46.090" UserId="264" />
  <row Id="7586" PostId="5049" Text="Thanks, yes that was a mistake (in fact, there was a similar one later on also)." CreationDate="2010-12-01T12:26:10.457" UserId="386" />
  <row Id="7587" PostId="4629" Text="Take a look at the sin I have already proposed about testing. If you do the test and do not reject normality, it does not mean that you have a normal sample... it only means that you cannot say the sample is not normal. sin !  " CreationDate="2010-12-01T16:04:56.537" UserId="223" />
  <row Id="7588" PostId="5054" Text="Maybe you should ask [Johan Bollen](http://informatics.indiana.edu/jbollen/Home.html). It's his speciality." CreationDate="2010-12-01T16:31:33.113" UserId="2036" />
  <row Id="7589" PostId="5055" Text="great, I guess this is the answer ! I did not know Hansen-Hurwitz..." CreationDate="2010-12-01T16:43:54.273" UserId="223" />
  <row Id="7590" PostId="4801" Text="I wish to estimate a function. An example would be estimating the partition function of a Markov random field. &#xA;" CreationDate="2010-12-01T17:58:32.353" />
  <row Id="7591" PostId="5054" Score="2" Text="Isn't that what AR and MA are for (in ARMA)?  You might also look up &quot;long memory process&quot;, such as ARFIMA." CreationDate="2010-12-01T18:32:32.113" UserId="5" />
  <row Id="7593" PostId="4552" Text="indeed, inferring something from `A and B are correlated` usually only see `A causes B` but not `B causes A`...(and forget about `C` which causes `A` and `B`) " CreationDate="2010-12-01T20:13:59.237" UserId="961" />
  <row Id="7594" PostId="4574" Text="sorry to be ignorant, but what's wrong with a Neyman-Pearson construction for the analysis of (the outcome of) scientific experiments ?" CreationDate="2010-12-01T20:15:43.133" UserId="961" />
  <row Id="7595" PostId="5016" Text="Have you considered calculating the convolution by using (fast) Fourier transform (and its inverse) ?" CreationDate="2010-12-01T20:36:58.537" UserId="961" />
  <row Id="7596" PostId="5058" Text="Might you be able to remember *where* you 'read a few days back'?" CreationDate="2010-12-01T20:48:02.440" UserId="449" />
  <row Id="7598" PostId="5058" Text="@onestop: Sorry. I read that in some research paper but I found a quick link on wiki `http://en.wikipedia.org/wiki/Generalized_gamma_distribution` for a generalized gamma distribution. Please feel free to correct me though. " CreationDate="2010-12-01T21:04:41.400" UserId="2164" />
  <row Id="7599" PostId="5061" Text="@Wayne (+1) I particularly like the 2nd paragraph." CreationDate="2010-12-01T21:23:21.363" UserId="930" />
  <row Id="7600" PostId="5058" Score="1" Text="This probability plot strongly suggests a mixture of two distributions, one located between 5 and 60 and the other between 50 and 150+.  This casts doubt on any simpler characterization, like negative binomial." CreationDate="2010-12-01T21:27:36.137" UserId="919" />
  <row Id="7601" PostId="5060" Text="A pre-post, experimental-control group design is extremely common in psychology. I agree that few people seem to be aware of the comparatively complicated and strict assumptions. Mixed models often seem to be beyond the statistical horizon." CreationDate="2010-12-01T21:36:20.573" UserId="1909" />
  <row Id="7602" PostId="5061" Text="Machine learning and pattern recognition are not the same thing, machine learning is also interested in things like regression and causal inference etc.  Pattern recognition is only one of the problems of interest in machine learning.  Most of the machine learning people I know are in computer science departments." CreationDate="2010-12-02T00:04:02.240" UserId="887" />
  <row Id="7603" PostId="5058" Text="@whuber: That is interesting. Would you have any suggestions on how to proceed? Do I start looking for ways to separate the distributions?" CreationDate="2010-12-02T01:42:01.663" UserId="2164" />
  <row Id="7604" PostId="5067" Text="Does anyone know how LaTeX works here?  I tried the faq here and also Google searching, but that of course leads to questions on tex.stackexchange.com/ " CreationDate="2010-12-02T03:41:48.030" UserId="1815" />
  <row Id="7605" PostId="5067" Text="Okay I sort of got it working...but it's absurd how I have to keep doing the captchas due to my editing.  Can I get around this short of not editing?  " CreationDate="2010-12-02T03:51:36.400" UserId="1815" />
  <row Id="7606" PostId="5067" Text="Wow.  I am your biggest fan.  That is so awesome.  I got CVX, and moreover now have a couple hours of new homework to learn what the hell you did there.  : )" CreationDate="2010-12-02T04:28:01.283" UserId="2224" />
  <row Id="7607" PostId="5071" Text="Right, you already know the bias. Would this strategy also work for a dice where one face was heavier (again, you are aware that it is)?" CreationDate="2010-12-02T06:08:29.177" UserId="2226" />
  <row Id="7608" PostId="5072" Text="I'm not certain what 50:50 odds really means. Does the Kelly criterion require the bets to be dependent on the outcome of the previous bet?" CreationDate="2010-12-02T06:11:10.250" UserId="2226" />
  <row Id="7609" PostId="5071" Text="Yes, why would you think that it is different? you always play where the probability to win is the highest. Then you can also think about the sizing of your bet, but as you want to &quot;maximize our probability of winning&quot;, that's it I guess" CreationDate="2010-12-02T06:20:40.297" UserId="1709" />
  <row Id="7610" PostId="5073" Text="added the data-visualization tag" CreationDate="2010-12-02T06:44:14.967" UserId="696" />
  <row Id="7611" PostId="5074" Text="thanks about that but dont see how i can apply it before i draw the chart first" CreationDate="2010-12-02T07:09:20.973" UserId="201" />
  <row Id="7612" PostId="5074" Text="something like that? This is simple data visualization, I suggest you read a few tutorials" CreationDate="2010-12-02T07:34:02.107" UserId="1709" />
  <row Id="7613" PostId="5074" Text="@fRed (+1) Good update." CreationDate="2010-12-02T07:37:51.670" UserId="930" />
  <row Id="7614" PostId="5075" Text="@Chase +1 for the use of `qplot()` but maybe it would be interesting to also show the other way (stacked) and the use of color." CreationDate="2010-12-02T07:39:18.050" UserId="930" />
  <row Id="7615" PostId="5061" Text="@Dikran Agree but ML and PR are often aliased and presented under similar topics of data analysis. My preferred book is indeed *Pattern Recognition And Machine Learning*, from Christophe M Bishop. Here is a review by John MainDonald in the JSS, http://j.mp/etg3w1." CreationDate="2010-12-02T07:46:36.230" UserId="930" />
  <row Id="7616" PostId="5075" Text="@Chase +1 for the sorting" CreationDate="2010-12-02T07:48:52.313" UserId="1709" />
  <row Id="7617" PostId="5076" Text="Cleveland plot, of course!" CreationDate="2010-12-02T07:51:15.837" UserId="1709" />
  <row Id="7618" PostId="5076" Score="2" Text="I always seem to forget about dotcharts, even though everywhere you look Tufte talks about their superior data-ink ratio... I think this suggestion combined with the reordering of the data makes for an informative and easy to digest graph. For completeness sake, to change from bars to points in `ggplot2` simply needs the  `geom = &quot;point&quot;`: `qplot(x = names, y = freq, data = df, geom = &quot;point&quot;) + coord_flip()`" CreationDate="2010-12-02T08:09:44.393" UserId="696" />
  <row Id="7619" PostId="5063" Text="The errors I have are not that far away from the true circle, however they are shaped as a circle with greater radius. Therefore they have a big influence on the solution. The errors does not have a normal distribution." CreationDate="2010-12-02T08:46:00.960" UserId="2223" />
  <row Id="7620" PostId="5063" Text="The image processing approach is really a great idea. I will definately try it. Thanks a lot!" CreationDate="2010-12-02T08:48:47.807" UserId="2223" />
  <row Id="7621" PostId="5076" Text="@Chase Thanks for this (I also appreciated that you promptly update your response following my comment). Reordering would make sense iif there's no natural grouping among items (which is often the case in strutured questionnaire), but I use it too because it is very convenient to display potential ceiling/floor effects. Now, the point is that a stacked barchart rarely conveys effective information about binary items (it seems more appropriate for ordered response categories, like Likert-type items) whereas dotchart can cope with both type of items. Here, the limiting factor is the No. items." CreationDate="2010-12-02T09:26:14.267" UserId="930" />
  <row Id="7622" PostId="5056" Text="w = sum over i (ai ti xi). you will have to minimize the lagrangian to find the values of the multipliers, ai. i wonder how u got the support vectors ? the same process should also give u the value of ai's." CreationDate="2010-12-02T10:15:26.647" UserId="2071" />
  <row Id="7623" PostId="5072" Text="The Kelly criterion tells you how much to bet in order to maximize winnings without dipping to the negatives, depending on 1) payout 2) odds 3) current bankroll. In this case 50:50 payout meant that you're being paid as if the coin were fair, and you can take advantage of that." CreationDate="2010-12-02T10:20:02.313" UserId="2456" />
  <row Id="7624" PostId="5077" Score="1" Text="Never heard of this algorithm before, but it sounds pretty interesting. Could you provide a link to a explanatory source ? Thank you !" CreationDate="2010-12-02T10:35:07.253" UserId="264" />
  <row Id="7625" PostId="5063" Text="I discovered that this is really the Hough Circle Transform and have tested it now. It works really well!" CreationDate="2010-12-02T10:41:48.197" UserId="2223" />
  <row Id="7627" PostId="5077" Text="@steffen Here's a link to the [original paper](http://ioe.engin.umich.edu/techrprt/pdf/TR81-01.pdf) - well actually the technical report. I've never used this procedure either. " CreationDate="2010-12-02T11:20:51.397" UserId="8" />
  <row Id="7628" PostId="5082" Text="It seems that a disadvantage of this implementation is, that one cannot define a bn-architecture on his own." CreationDate="2010-12-02T11:59:01.083" UserId="264" />
  <row Id="7629" PostId="5077" Text="@csgillespie Thank you !" CreationDate="2010-12-02T11:59:38.697" UserId="264" />
  <row Id="7630" PostId="5078" Text="Thanks chl, the version number was a type (zeros were supposed to be nines), but your tip helped. plotLMER.fnc works again after the update. So, my immediate issues are solved. However, I think I'll leave the question here anyway (unless a mod thinks otherwise?) if anybody wants to discuss other ways of getting intervals outside of the plotLMER function." CreationDate="2010-12-02T12:28:49.593" UserId="2228" />
  <row Id="7631" PostId="5085" Text="Thanks! I tried the example on the FAQ and it provided a nice-looking plot." CreationDate="2010-12-02T12:41:29.453" UserId="2228" />
  <row Id="7632" PostId="5063" Text="@Björn Thank you for reminding me of the name: that provides a useful way for readers to learn more about the procedure." CreationDate="2010-12-02T13:37:25.007" UserId="919" />
  <row Id="7633" PostId="5069" Score="1" Text="This probability plot does not exhibit any outliers at all, but it does indicate a region of sparse probability density around 40-60.  This will not be detected by most GoF tests, which will be invalid anyway.  Distribution fitting is *exploratory*, not confirmatory, and so requires more flexibility and creativity than is afforded by parameter estimation and rote application of hypothesis tests.  If a mixture is suggested by theory--and it might be here--and if capturing that in a model might be important, then it is worthwhile considering despite the complications that ensue." CreationDate="2010-12-02T13:44:55.343" UserId="919" />
  <row Id="7634" PostId="5069" Text="(A GoF test, to be valid, must quantitatively account for how many possible distributions one has attempted to fit to the data.)" CreationDate="2010-12-02T13:46:43.953" UserId="919" />
  <row Id="7635" PostId="5058" Text="@Legend Not necessarily.  It depends on why you're doing this fitting.  At this stage it seems you are *exploring* the data.  Let the data behavior guide you.  Can you find any possible cause or explanation for a mixture?  Would there be a meaningful difference between values in the 0-60 range versus values in the 40-150+ range?  In short, what can you *learn* about your social experiment from this?" CreationDate="2010-12-02T13:50:16.057" UserId="919" />
  <row Id="7636" PostId="5084" Text="This is very helpful, thank you. I wonder could I ask though, I'm guessing statistical significance isn't that useful here, because one would expect there to be some differences in ranks- after all, they are not randomly drawn. So would it be best to look at Kendall's w value, do you think? With my data I get .474, which seems low-moderate when compared with other reliability co-efficients." CreationDate="2010-12-02T14:03:05.060" UserId="199" />
  <row Id="7637" PostId="5078" Text="@Richard Your question makes perfect sense to me, if we make abstraction of the R issues. Someone may have a good idea about how to derive relevant CIs in your case without resorting to any software consideration." CreationDate="2010-12-02T14:08:12.813" UserId="930" />
  <row Id="7638" PostId="5084" Score="2" Text="@Chris On the contrary, statistical significance is extremely useful here: it has *exactly* the same interpretation it would have for a huge dataset.  What you might be worried about is *power*: the ability of the data to discriminate between a null hypothesis and the alternative.  But you have no control over that.  The most you can hope for is to find a significant association, which would suggest that what you see in your scatterplots is not due to chance alone.  (Student, in his original paper of 1908, simulated datasets of just *four* values to demonstrate the t-test.)" CreationDate="2010-12-02T14:14:42.340" UserId="919" />
  <row Id="7639" PostId="5072" Text="Yes, sorry, I meant 1:1 odds, not 50:50 odds...I've changed the answer to reflect this" CreationDate="2010-12-02T14:21:36.233" UserId="2168" />
  <row Id="7640" PostId="5072" Score="2" Text="The Kelly criterion is not about Gambler's Ruin.  Simply observe that after many (N) plays, you expect to win about pN of them, multiplying your wealth by (1 + bx)^(pN), and lose about (1-p)N of them, multiplying your wealth by (1 - x)^((1-p)N).  To maximize this w.r.t. x, take logs.  Factoring out N yields the Kelly criterion.  In short, *if* you follow a system of investing a fixed proportion (x) of your wealth each time, and *arbitrarily small bets can be made,* and you can play for an indefinite time, then the Kelly Criterion gives the optimal proportion to choose." CreationDate="2010-12-02T14:31:32.183" UserId="919" />
  <row Id="7641" PostId="5072" Text="@user2168 1:1 odds and 50:50 odds are ways to express exactly the same thing; namely, 1/(1+1) = 50/(50+50) = 50% chance of winning." CreationDate="2010-12-02T14:32:29.767" UserId="919" />
  <row Id="7642" PostId="5072" Text="@whuber, x:y is usually written with no common denominator and was confusing as it was written, that's why I changed it.  Thank you for the explanation, that makes a lot of sense to me.  i.e. your winnings, $W_N$ at time $N$ will be $ W_N = (1 + b x)^{pN} (1 - p)^{(1-p)N} W_0 $ (with $W_0$ your initial savings), then just maximize.  The (easy to understand) assumption, which you've stated, is that you want to reinvest a constant proportion of your savings (assuming independence of trials)." CreationDate="2010-12-02T14:43:38.743" UserId="2168" />
  <row Id="7643" PostId="5072" Score="2" Text="@user2168 A subtlety is that your winnings are unlikely to be *exactly* this formula.  You really have to take the expectation.  But, as the Central Limit Theorem shows, when N is sufficiently large, the number of wins is highly likely to be within O(Sqrt(N)) of pN.  This chance of a relatively small deviation from the expected number of wins does not alter the optimal fraction." CreationDate="2010-12-02T15:36:31.863" UserId="919" />
  <row Id="7644" PostId="5088" Text="@Gavin : it's a start, but somehow I can't get the combined spline using these coefficients or a combination. Care to guide me on how I can go from the basis functions constructed with smoothCon to the final smoother, using only coefficient multiplications (basically doing it by hand)?" CreationDate="2010-12-02T16:29:55.373" UserId="1124" />
  <row Id="7645" PostId="5088" Text="@Gavin: Thx for the update, but indeed, they're not the same. Actually, some basis functions are turned upside down even. Thin plate regression splines are indeed a penalized analysis, and the mixed in gamm isn't making things easier either. I'm afraid I'll have to get back to the math as well." CreationDate="2010-12-02T16:50:20.177" UserId="1124" />
  <row Id="7646" PostId="5028" Text="Thanks a lot for formatting it, I really apreciate it" CreationDate="2010-12-03T00:12:43.437" UserId="1808" />
  <row Id="7647" PostId="5091" Text="Thank you for your explanation of ACF, I am looking more for the justification to do it on the wavelet coefficients, and the signification of it..." CreationDate="2010-12-03T01:26:24.493" UserId="1709" />
  <row Id="7649" PostId="5014" Text="@whuber - you are certainly on track with the price being right! I have thick skin and appreciate the time and thoughtful answers that others on this site provide. I hope that I can provide some useful knowledge back to this site as well. I'm quite confident my instructors did a very sufficient job at teaching these basic concepts and the issue is my inability to *apply* that information in an appropriate context. I was a little over zealous here and grasping at straws trying to fit a concept I vaguely understood to a question. As they say, when all you've got is a hammer...  " CreationDate="2010-12-03T03:50:03.087" UserId="696" />
  <row Id="7650" PostId="5095" Text="Thanks Paul for your detailed answer. I will up-vote it once I have enough points :) ! Features in HF is around 100, but NF is around 500. I have tried feature selection using information gain method. " CreationDate="2010-12-03T05:06:55.010" UserId="529" />
  <row Id="7651" PostId="5094" Text="Mbq: thanks for sharing your thoughts. I have tried a k-fold cross-validation and the results are average of the k-runs. HF is a subset of NF, but several features are merged as  one single feature. " CreationDate="2010-12-03T05:11:08.767" UserId="529" />
  <row Id="7652" PostId="5086" Text="I'll try. Which R-package do you recommend ?" CreationDate="2010-12-03T07:31:28.643" UserId="264" />
  <row Id="7653" PostId="4552" Score="3" Text="google makes $65B a year not caring about the difference..." CreationDate="2010-12-03T07:41:55.810" UserId="74" />
  <row Id="7654" PostId="4991" Text="Weird that my bounty doesn't help... Is my question badly formulated.... No one has a piece of answer? Or a question on my question?" CreationDate="2010-12-03T07:52:32.200" UserId="1709" />
  <row Id="7655" PostId="4552" Text="I agree with your points and they all are valid. But does Google's profit imply: correlation =&gt; causation?" CreationDate="2010-12-03T07:53:58.967" UserId="1307" />
  <row Id="7656" PostId="5095" Score="1" Text="@Khader RF importance is more reliable than IG." CreationDate="2010-12-03T09:25:47.407" UserId="88" />
  <row Id="7657" PostId="5095" Text="@Khader And a moderator tip: you can (and should) accept the best (in your view) answer by clicking the tick mark below voting arrows. This will give you 2 points and 15 extra points to the answerer. The accepted answer will be sticked to the top so that each new reader of a question would notice it first." CreationDate="2010-12-03T09:45:54.800" UserId="88" />
  <row Id="7658" PostId="5088" Text="In relation to Joris' 2nd comment above, my Update 2, shows that the difference is due to `um` in Joris' code not including the identifiability constraints in the basis functions. If you include these constraints into the basis function, `smoothCon()` returns the same thing as `Xp`" CreationDate="2010-12-03T11:08:57.223" UserId="1390" />
  <row Id="7659" PostId="823" Text="-1, see what @walkytalky and @Lèse majesté said." CreationDate="2010-12-03T12:00:52.483" UserId="1979" />
  <row Id="7660" PostId="5084" Text="That's that cleared up then! Thanks to both." CreationDate="2010-12-03T12:32:25.390" UserId="199" />
  <row Id="7661" PostId="5100" Text="What do you expect? If you want a typical Brownian motion, just generate one. If you want the mean behaviour, generate a lot and take the mean, like you did. Maybe you can add the variance as extra information. An idea would be to combine the three. Plot one realization against the mean and variances of many realization. You'll get 4 curves: the one from 1 realization, the mean curve, the mean+standard deviation and the mean-standard deviation." CreationDate="2010-12-03T13:24:20.927" UserId="2036" />
  <row Id="7662" PostId="5102" Text="I will look into the reference range. Best means approaching the values distribution as bes as possible. What I get are flat and boring curves, almost straight lines. The expectation is to get more wild curves with expononential growth/decline, sine wave like or something in between. Some amount of random noise has also to be thrown in for good measure." CreationDate="2010-12-03T13:54:54.483" UserId="2218" />
  <row Id="7663" PostId="5100" Text="The process could indeed have in theory some component of Brownian motion. The idea of the 4 curves sounds good to me. Although it seems that then you have to assume a normal distribution, which is certainly not the case. I was also thinking of using geometric mean, harmonic mean, median absolute deviation or something more exotic?" CreationDate="2010-12-03T13:59:53.747" UserId="2218" />
  <row Id="7664" PostId="2269" Text="@Andy W., with all due respect, your immediate dismissal of this question is what I think is inappropriate. Numerous questions on this forum have to do with specific R package issues and no one closes them. SAS is used widely in business settings (much much more than R; I use both, and Python) and it seems useful to allow software related questions, given that software and analysis are so integral these days. If we are going to have such obvious double standards then the forum should be renamed &quot;Statistical Analysis Using Open Source Software&quot;" CreationDate="2010-12-03T14:05:48.660" UserId="1080" />
  <row Id="7665" PostId="5100" Text="Sure, when you said random walk, I assumed Brownian motion because the term is sometimes used synonymously, although strictly speaking, you are right random can be refer to any type of randomness a priori. You could still use moving averages and variances over a certain time window if the process is more exotic. But you can avoid that by transforming to the proper scale sometimes. Say, you have a geometrical Brownian motion, just plot the log of it." CreationDate="2010-12-03T14:26:51.450" UserId="2036" />
  <row Id="7666" PostId="5103" Text="These plots look really spectacular. Looking at the first plot, I realize that what I want is to show the most probable paths, because there are certain paths that are just too improbable. The problem is how choose the most probable paths in an elegant manner and summarize them. Eventually I want to display a reference &quot;real&quot; curve of the measured observed data, next to the simulated ones." CreationDate="2010-12-03T14:44:49.840" UserId="2218" />
  <row Id="7667" PostId="2269" Text="@Josh Hemann, this topic was discussed on meta  in the question Shane and Srikant linked to. If you have comments it would be useful to bring them up there. This question was re-opened so if you have an answer feel free to contribute." CreationDate="2010-12-03T15:10:19.550" UserId="1036" />
  <row Id="7668" PostId="5103" Text="Perhabs you could make a histogramm after the 100 steps?" CreationDate="2010-12-03T15:52:01.990" UserId="1050" />
  <row Id="7669" PostId="5095" Text="@mbq: Thanks for your tip. I will wait for a day or two before chosing the final answer. " CreationDate="2010-12-03T16:14:32.990" UserId="529" />
  <row Id="7670" PostId="5099" Text="@mbq: Thanks for the insightful answer. I will try with SD of accuracy for further analysis. " CreationDate="2010-12-03T16:16:43.863" UserId="529" />
  <row Id="7671" PostId="5103" Text="@Navi: by definition, the most probably path is a horizontal line around zero. These random walks are just cumulative sums of Gaussian noise with mean 0 and unit variance" CreationDate="2010-12-03T17:13:01.100" UserId="1390" />
  <row Id="7673" PostId="2623" Text="If the time series is not stationary, often the 1st difference of the series will be stationary (for example, financial time series)." CreationDate="2010-12-03T22:18:19.247" UserId="1146" />
  <row Id="7675" PostId="813" Text="I don't know, you've seen many efficient citizens lately?" CreationDate="2010-12-03T23:26:37.490" UserId="2036" />
  <row Id="7676" PostId="878" Score="1" Text="Besides, Tzippy is misquoting Sagan, since Sagan never believed that. He in fact listed it among the [fallacies in his baloney detecion kit](http://en.wikiquote.org/wiki/Carl_Sagan#The_Demon-Haunted_World:_Science_as_a_Candle_in_the_Dark_.281995.29)." CreationDate="2010-12-03T23:31:58.870" UserId="2036" />
  <row Id="7678" PostId="5113" Text="Thanks! I agree Simulated Annealing might better handling such situation, but this is an assignment that requires to use Metropolitan-Hastings algorithms. So I am afraid I have not other choice." CreationDate="2010-12-04T00:05:07.413" UserId="1005" />
  <row Id="7679" PostId="5112" Text="Thanks! I agree Simulated Annealing might better handling such situation, but this is an assignment that requires to use Metropolitan-Hastings algorithms. So I am afraid I have not other choice." CreationDate="2010-12-04T00:05:30.033" UserId="1005" />
  <row Id="7680" PostId="5113" Text="@Tim As I wrote, MH is only a way of accepting steps -- SA is also MH. Yet you can hold the temperature constant to make it just 'plain MH'. " CreationDate="2010-12-04T00:08:26.740" UserId="88" />
  <row Id="7681" PostId="5113" Score="2" Text="Simulated annealing is an adaption of the Metropolis-Hastings algorithm (as mentioned in the Wikipedia link).  You could always ask whoever set the assignment whether SA was what he/she had in mind." CreationDate="2010-12-04T00:08:33.837" UserId="887" />
  <row Id="7682" PostId="5115" Text="Converted to community wiki." CreationDate="2010-12-04T00:14:08.750" UserId="88" />
  <row Id="7683" PostId="5115" Text="what is community wiki?" CreationDate="2010-12-04T00:53:59.723" UserId="1808" />
  <row Id="7685" PostId="5120" Text="Thank you for your reply.&#xA;In the original paper, one of the parameters that is estimated is $\sigma_{a}$, the variance of the technology shock. Unfortunately, in Dynare, while you are able to estimate transforms of parameters, you can't do so for the standard error of shocks, such as the technology shock in the model equation. Instead, you must assign a prior to the standard error of the shock directly. Given that the prior of the variance is inverse gamma I assumed that a suitable prior for the stderr would be the &quot;square root&quot; of this invgamma distribution. Does that make sense?" CreationDate="2010-12-04T03:10:24.333" UserId="2251" />
  <row Id="7686" PostId="5120" Text="Thank you so much, that's exactly what I was looking for! " CreationDate="2010-12-04T03:15:47.950" UserId="2251" />
  <row Id="7687" PostId="5115" Text="@Mariana: http://www.sharepointoverflow.com/questions/432/what-is-community-wiki" CreationDate="2010-12-04T04:32:47.227" UserId="1118" />
  <row Id="7688" PostId="5117" Score="1" Text="It's worth noting that Fisher is equally famous for his work as a biologist (evolutionary biology and agricultural science) as he is for his statistical work." CreationDate="2010-12-04T06:49:04.617" UserId="1679" />
  <row Id="7689" PostId="5117" Score="7" Text="*Even today, I occasionally meet geneticists who ask me whether it is true that the great geneticist R. A. Fisher was also an important statistician*&#xA;-- Leonard Savage &#xA;(Annals of Statistics, 1976 http://www.jstor.org/stable/2958221). " CreationDate="2010-12-04T08:06:48.953" UserId="449" />
  <row Id="7691" PostId="5098" Text="Right, power analysis. I think that perhaps I do indeed care about relative frequency, however. I'll try to read around that also.&#xA;&#xA;Without a clearly defined number of tests to run, I have been running 2% of the tests, selected uniformly randomly, on each of the datasets. 2% is arbitrary, but also tractable on the larger datasets. It means my sample size increases with respect to the population of tests on a dataset, which may lead to more tests than I _need_ on the larger datasets..." CreationDate="2010-12-04T10:54:26.267" UserId="2246" />
  <row Id="7692" PostId="5136" Score="1" Text="That seems rather a general question - would you like to take a look at http://en.wikipedia.org/wiki/Latent_variable and see if that helps, or if not, which bits you don't follow?" CreationDate="2010-12-04T12:17:29.563" UserId="449" />
  <row Id="7693" PostId="5135" Text="residuals are not so badly deviating from normality, why do you think so?" CreationDate="2010-12-04T13:14:18.493" UserId="582" />
  <row Id="7694" PostId="5135" Text="@nico: I think @Alexx Hardt was speaking hypothetically. I.e. once *could* use the five number summary to see if residuals were deviating from normal" CreationDate="2010-12-04T13:39:36.373" UserId="1390" />
  <row Id="7695" PostId="5138" Text="This will take some time and playing around with R to understand. Big thanks for now, I might follow up with some questions at some point :)" CreationDate="2010-12-04T13:42:01.740" UserId="2091" />
  <row Id="7696" PostId="5138" Score="2" Text="@Gavin (+1) Great response with nice illustrations!" CreationDate="2010-12-04T14:04:02.000" UserId="930" />
  <row Id="7698" PostId="5135" Text="@Gavin Simpson: you're right, I misread the sentence. Disregard my previous comment." CreationDate="2010-12-04T14:34:14.040" UserId="582" />
  <row Id="7699" PostId="5138" Score="1" Text="Nice job. One thing you might clarifiy, with regard to calculating t values: sqrt(diag(vcov(mod))) produces the SE of the estimates. These are the same SEs that are output in the model summary. Easier and clearer just to say that t = Estimate/SEestimate.  In that sense it is no different that any other t value." CreationDate="2010-12-04T14:49:48.403" UserId="485" />
  <row Id="7700" PostId="5138" Score="1" Text="(+1) This is great.  The only thing I'd add is that the $F$ value is the same as $t^2$ for the slope (which is why the p values are the same).  This - of course - isn't true with multiple explanatory variables. " CreationDate="2010-12-04T15:05:39.607" UserId="1108" />
  <row Id="7702" PostId="5138" Score="2" Text="@Jay; thanks. I thought about mentioning that equivalence too. Wasn't sure if it was too much detail or not? I'll ad something on this in a mo." CreationDate="2010-12-04T15:43:08.297" UserId="1390" />
  <row Id="7703" PostId="5136" Text="Can you fix this question? It gets answers because of the topic, but the content seems a pure word clutter to me." CreationDate="2010-12-04T15:48:47.357" UserId="88" />
  <row Id="7704" PostId="5138" Score="1" Text="@Brett; thanks. I've tried to clarify this a bit above as per your comment." CreationDate="2010-12-04T15:54:07.770" UserId="1390" />
  <row Id="7705" PostId="5115" Text="@Mariana The idea is that pools and list-ofs are converted to a form in which they can be easily managed (due to lower rep req to edit) and voted up/down without hurting participants' reputation (votes on CW posts does not give/take reputation)." CreationDate="2010-12-04T16:01:50.720" UserId="88" />
  <row Id="7706" PostId="5136" Text="i fix questions with examples..." CreationDate="2010-12-04T18:54:23.407" UserId="1154" />
  <row Id="7707" PostId="5139" Text="Thanks i see what you mean... But my goal is to have statistics rules. The complexity you talk about can be seen in econometrics problems which use classical approach." CreationDate="2010-12-04T18:56:29.413" UserId="1154" />
  <row Id="7708" PostId="5144" Text="@honitom It sounds like it is indeed the topic of design of experiments. However, your description lacks some information; e.g., what's the sample size, are all variables crossed, is this a repeated-measures design (i.e., data are collected on the same statistical units); how many levels have your variables, and most importantly, are you interested in a predictive or an explicative model (from what you described, your use of the term 'predict' is not obvious)?" CreationDate="2010-12-04T19:58:04.760" UserId="930" />
  <row Id="7709" PostId="5047" Text="@Adam: Do you know any good Matlab packages that can get the job done?" CreationDate="2010-12-04T20:00:35.780" UserId="1224" />
  <row Id="7710" PostId="5044" Text="@steffen: Do you know any good Matlab packages that can get the job done?" CreationDate="2010-12-04T20:01:15.590" UserId="1224" />
  <row Id="7711" PostId="5144" Text="@chl I'm honestly a bit lost with this, but I've updated the question with an example of what I'm trying to achieve. If that's not enough I'll try to expand it a bit more." CreationDate="2010-12-04T20:15:42.677" UserId="2261" />
  <row Id="7712" PostId="5144" Text="So, do you already assume a linear relationship between your variables (like the function you cooked up for $t$) or is that also part of what you want to determine?" CreationDate="2010-12-04T20:43:22.003" UserId="2036" />
  <row Id="7713" PostId="5143" Text="Great answer! Thanks Gavin!" CreationDate="2010-12-04T20:45:29.690" UserId="2241" />
  <row Id="7714" PostId="5144" Text="That is also part of what I want to determine. However I don't necessarily need extremely precise results - a linear approximation would probably be okay..." CreationDate="2010-12-04T20:48:10.030" UserId="2261" />
  <row Id="7715" PostId="5146" Text="Thanks for the detailed answer. I've heard that ANOVA's aren't practical for more then 3 independent variables. Is this true? If so, why?" CreationDate="2010-12-04T21:26:21.123" UserId="2261" />
  <row Id="7716" PostId="5146" Score="2" Text="@honitom It really depends on the context of your experiment and your sample size. Keep in mind that the reliability of your estimates (the regression coefficients summarizing the effect of your factors on your outcome) depends on the available sample size and associated variance. In DoE, interaction terms can be aliased with main effects, depending on the sample size; still we can estimate the main effects. Any decent textbook should cover this topic much well than me in a comment, so I'd suggest to have a look at ANOVAs and DoE." CreationDate="2010-12-04T21:34:05.670" UserId="930" />
  <row Id="7717" PostId="4991" Text="The way it's posed, this seems like a degenerate problem -- the errors could equally be attributed to the observation noise or the process noise. Are there more constraints? Is the state one dimensional?" CreationDate="2010-12-04T21:48:28.393" UserId="2077" />
  <row Id="7718" PostId="5148" Text="If (1) is mentioned in the lecture notes, it suggests that simulated annealing is the answer to your previous question on minimization of a function by Metropolis-Hastings algorithm as (1) would only be the mode for simulated annealing." CreationDate="2010-12-04T23:30:01.730" UserId="887" />
  <row Id="7719" PostId="5148" Text="The purpose of only considering the tail subsequence of the original samples is because there is time needed to wait for the instrumental markov chain to have its distribution converge to its stationary distribution, whose density has been designed to be the target function." CreationDate="2010-12-04T23:32:34.820" UserId="1005" />
  <row Id="7720" PostId="5148" Text="In the class note, the first choice was not just mentioned for simulated annealing, but also for Metropolitan-Hastings algorithms." CreationDate="2010-12-04T23:33:44.523" UserId="1005" />
  <row Id="7721" PostId="5148" Score="1" Text="After the chain has reached its stationary distribution, it will still be sampling from that distribution, so all you can say about the last sample is that it will be a sample from that distribution.  Of course the mode is the most probable value, but that doesn't even imply that the last sample is probably the mode! " CreationDate="2010-12-04T23:39:18.073" UserId="887" />
  <row Id="7722" PostId="5148" Text="Thanks! If I understand correctly, your opinion is to evaluate the function on every sample after the MC has been seen as reaching its stationary distribution. " CreationDate="2010-12-04T23:44:56.937" UserId="1005" />
  <row Id="7723" PostId="5148" Text="I would have thought that any sample, even before the burn in period is complete would be the mode (as it is only the maximum of f(x) that you seek).  However I could be wrong, but samples taken after reaching the stationary distribution would be fine, provided you had a sufficient number to be confident of adequately exploring the distribution. I suggest you speak to your instructor about the last sample issue as it seems there is some uncertainty about it." CreationDate="2010-12-04T23:52:16.877" UserId="887" />
  <row Id="7724" PostId="5148" Text="Thanks! I think I will ask him. I now see your point to consider the samples before reaching stationary distribution." CreationDate="2010-12-05T00:05:38.767" UserId="1005" />
  <row Id="7725" PostId="5148" Text="@Tim Please, Metropolis; this has nothing to do with cities, guy had just strange surname. Back to the topic -- if you degenerate MH to simple accept-only-if-better (very small T), with some luck at the end you'll get $x$ just sitting in the global minimum." CreationDate="2010-12-05T00:12:09.903" UserId="88" />
  <row Id="7726" PostId="5145" Score="2" Text="Maybe not the most famous statistician, but when you put it like that, definitely the most important! ;o)" CreationDate="2010-12-05T00:13:14.643" UserId="887" />
  <row Id="7727" PostId="5117" Text="@Michael Lew: He certainly is. He managed to get over the supposed disconnection between Mendelian genetics and Darwinian evolution, among other accomplishments. http://digital.library.adelaide.edu.au/coll/special//fisher/9.pdf" CreationDate="2010-12-05T00:18:50.350" UserId="1118" />
  <row Id="7728" PostId="5148" Text="@mbq: my mistake. By &quot;degenerate MH to simple accept-only-if-better&quot;, do you mean at each iteration evaluate the function immediately when a new sample is generated and compare to the previous best? Also why choose &quot;very small T&quot;, I assume T is fixed at some value, and if T is small, chance of getting stuck in a local minimum will be great, is it? " CreationDate="2010-12-05T00:34:01.140" UserId="1005" />
  <row Id="7729" PostId="5148" Score="1" Text="@Tim Yes -- this is sadly quite common. About getting stuck -- there is one more issue. Here we talk nothing about the next step generation, yet obviously making big steps can be a way to escape local minima (tunneling in physical termin.). Those are of course pathologies, but sometimes people use them &quot;because they work&quot;." CreationDate="2010-12-05T00:44:56.017" UserId="88" />
  <row Id="7731" PostId="5152" Text="How old are the subjects at the start of the study?" CreationDate="2010-12-05T10:48:03.980" UserId="449" />
  <row Id="7733" PostId="5162" Text="Shouldn't &quot;optimality&quot; take into account the cost of false positives versus false negatives? I can imagine a scenario where a false negative is far more costly than a false positive, which should call for a criterion biased towards capturing as much as the true distribution as possible. Also, I believe you mean &quot;sensitivity&quot; and not &quot;sensibility&quot; in your answer." CreationDate="2010-12-05T13:22:29.230" UserId="364" />
  <row Id="7734" PostId="5162" Text="@Mike Yes, your comment is really welcome and this was what I tried to point to in my last sentence (optimize FN/FP rates depending on the context). Thanks for noticing the typo (which was already addressed in a preceding post...)!" CreationDate="2010-12-05T13:31:45.803" UserId="930" />
  <row Id="7735" PostId="4991" Text="@lanS. All objects have indeed here only one dimension. Can you develop a bit more on the fact that the errors that can be either observation or noise. It is exactly what I would like to achieve. I would like to get a rolling estimation of the signal to noise ratio by estimating the sd of the 2 time varying noises....&#xA;" CreationDate="2010-12-05T15:34:36.663" UserId="1709" />
  <row Id="7736" PostId="4991" Text="Maybe should I start by fixing the sd of the process noise for a start and see how the sd of the observation noise reacts?" CreationDate="2010-12-05T15:36:28.847" UserId="1709" />
  <row Id="7737" PostId="3090" Text="We have also a dedicated Wiki page at http://grass.osgeo.org/wiki/R_statistics" CreationDate="2010-12-05T16:47:39.887" UserId="2267" />
  <row Id="7738" PostId="5166" Score="2" Text="This appears to be a duplicate of http://stats.stackexchange.com/questions/3324/why-use-a-lagged-dv-as-an-instrumental-variable" CreationDate="2010-12-05T17:42:26.860" UserId="449" />
  <row Id="7739" PostId="5162" Text="@chl So just to clarify, I have my data in the range of [0.6,1.2], then I calculate the corresponding ROC curve for a bunch of thresholds in that range. I find the cut-off points in each of those ROC curves and I try to pin-point the one that has the maximum Sensitivity/Specificity combination. Is that correct or have I completely misinterpreted you?" CreationDate="2010-12-05T19:48:32.487" UserId="1224" />
  <row Id="7740" PostId="5162" Text="@rohanbk Yes, that's basically the idea; e.g., compute Se/Sp and the four cells of your two-way table (TP, FN, etc.) in the range [0.6,1.2] (say with an increment of .05 or less -- it depends on the precision you can/want to attain in the future). Choose the compromise Se/(1-Sp) that best reflects your classification objective. Some references were provided to you in earlier threads. I don't really know if there are good MATLAB functions for ROC analysis, but I just found this page, http://j.mp/ekzg3m. HTH" CreationDate="2010-12-05T20:23:00.687" UserId="930" />
  <row Id="7741" PostId="5163" Text="@Conjugate Am I correct in assuming that when you talked of a parameter in place of one's location on a given latent trait you are indeed saying that subjects abilities are modeled without assuming any prior distribution (as is done in the marginal likelihood approach for estimating IRT model from the Rasch family)?" CreationDate="2010-12-05T20:27:49.200" UserId="930" />
  <row Id="7742" PostId="5158" Score="1" Text="You don't correct the mean!" CreationDate="2010-12-05T21:42:41.210" UserId="919" />
  <row Id="7743" PostId="5172" Text="do you know the sample sizes... the range of them?  Small sample sizes wreak havoc on the mean... moreso than the median." CreationDate="2010-12-06T00:40:56.477" UserId="601" />
  <row Id="7744" PostId="5168" Text="@chl Thank you. Actually I just needed pointers for the R package. I appreciate your helpfulness! By the way, the book is awesome. " CreationDate="2010-12-06T01:09:55.757" UserId="1307" />
  <row Id="7745" PostId="5172" Text="the sample sizes range from 1-100 million and the population size is upwards to 1 billion. But I have no knowledge of how disjoint the samples are from the total population." CreationDate="2010-12-06T03:33:15.570" UserId="2271" />
  <row Id="7746" PostId="5178" Text="ok, what I'm really interested is if it's statistically sound to include the median row in the summary table at http://en.wikipedia.org/wiki/Usage_share_of_web_browsers I can go into more details if you want." CreationDate="2010-12-06T04:52:09.327" UserId="2271" />
  <row Id="7747" PostId="5171" Text="Is each row the count of a different structure for the same conversation, or the count of the same structure for different periods of time?" CreationDate="2010-12-06T05:07:13.743" UserId="1709" />
  <row Id="7748" PostId="5077" Text="What is the advantage of the hit and run MCMC over other methods? Speed of convergence?" CreationDate="2010-12-06T05:14:53.223" UserId="1709" />
  <row Id="7749" PostId="5178" Text="In that case you should be okay. You can consider each of the samples A,B,C to be experimental units, and each of the categories a,b,c to be some measurement taken on each sample (with the restriction the measurements sum to 1.) This could be helpful when you have a lot of samples, not so much when you have 3 or 5." CreationDate="2010-12-06T05:40:02.133" UserId="2144" />
  <row Id="7750" PostId="5168" Text="@chl However, I will wait for someone to point me to an R package or tutorial that shows how to do HDP based modeling in R." CreationDate="2010-12-06T05:49:23.457" UserId="1307" />
  <row Id="7751" PostId="5044" Text="@rohanbk Sorry, no. It is a long time since I have used matlab." CreationDate="2010-12-06T07:42:58.960" UserId="264" />
  <row Id="7752" PostId="5168" Text="@suncoolsu I never tried those packages; so my contribution is just some googling (because I'm interested in using HDP in the future, but have no time for the moment -- this is the reason why I made my answer CW). I guess someone more used to those methods will provide insightful remarks." CreationDate="2010-12-06T07:48:01.620" UserId="930" />
  <row Id="7753" PostId="5169" Text="@whuber (+1) I like this answer. It took some time to me, but it seems I have been missing something in the question (*200 random samples* of size 25 each). " CreationDate="2010-12-06T07:57:24.543" UserId="930" />
  <row Id="7754" PostId="794" Text="I love this one. It is great !" CreationDate="2010-12-06T09:00:59.907" UserId="264" />
  <row Id="7755" PostId="1037" Text="-1: question's aim was to get a list of text books, not info on obscure papers about specialised sub-fields." CreationDate="2010-12-06T09:17:35.837" UserId="1979" />
  <row Id="7757" PostId="5135" Text="Minor quibble: You cannot say anything about normality or non-normality based on those 5 quantiles alone. All you can say based on that summary is whether the estimated residuals are approximately symmetric around zero.  You could divide the reported quantiles by the estimated residual standard error and compare these values to the respective quantiles of the N(0,1), but looking at a QQ-plot probably makes more sense." CreationDate="2010-12-06T09:29:00.867" UserId="1979" />
  <row Id="7758" PostId="2031" Score="1" Text="@xmjx: ... and yet it is commonly done.  One thing to point out, and yes this is a little pedantic, if you are using a single Likert-type item that isn't a Likert scale.  The difference is meaningful (though the question asker is talking about a Likert item and the ordinality is an issue). A Likert scale is a consequence of summing or averaging several Likert items.  This approach was developed specifically to offset the extent to which the ordinal data was actually ordinal and make it more reasonable to be treated as being on the interval scale." CreationDate="2010-12-06T09:31:45.500" UserId="196" />
  <row Id="7759" PostId="5179" Score="1" Text="Both assume continuous variables, so that the probability of ties within a sample is $0$. With the given data, there will be a lot of ties." CreationDate="2010-12-06T10:37:56.110" UserId="1909" />
  <row Id="7760" PostId="5179" Text="indeed ........" CreationDate="2010-12-06T11:00:11.490" UserId="1709" />
  <row Id="7761" PostId="5167" Text="1) Do you mean Wishart or Inverse Wishart?" CreationDate="2010-12-06T11:07:30.470" UserId="495" />
  <row Id="7762" PostId="5167" Text="2) What do you mean by &quot;maximum occurrence&quot;? Are you trying to find the mode (the point of highest density)?" CreationDate="2010-12-06T11:10:14.240" UserId="495" />
  <row Id="7764" PostId="5161" Text="**This response seems essentially wrong** as I forgot about the sampling issue." CreationDate="2010-12-06T12:15:32.990" UserId="930" />
  <row Id="7766" PostId="5181" Text="@Emile About using priors, I would rather suggest to look if there are some data available for country with small $n$ at the population level (e.g., through census or national surveys)." CreationDate="2010-12-06T12:41:56.800" UserId="930" />
  <row Id="7767" PostId="5163" Text="@chl Yes and no.  Yes, although they probably do come from some population we have chosen to ignore that in the model. If we make a 'prior' assumption about the population then we can ask for a posterior over the individual's location but not otherwise.  It's the panel model fixed effects/random effects distinction all over again.  But no, it's not like Marginal Maximum Likelihood (as in Bock and Aitkin) for IRT models.  This *does* assume a prior; that's what the quadrature machinery is busy integrating over.  *Joint* ML is all parameters, but is inconsistent as you add subjects (Anderson 70)" CreationDate="2010-12-06T14:01:47.573" UserId="1739" />
  <row Id="7768" PostId="5176" Text="Thanks Rob.  2 quick things though.  I converted by zoo object to ts using as.ts.  That didn't seem to do the trick as I got the same error.  I then backed all the way up created a new object from the raw CSV, converted to ts using ts(object).  Functions ets and auto.arima both found model, but I got an error when trying to plot the fitted auto.arima object (Error in xy.coords(x, y, xlabel, ylabel, log) : &#xA;  'x' and 'y' lengths differ).  Thanks again for your help! " CreationDate="2010-12-06T14:08:25.780" UserId="569" />
  <row Id="7769" PostId="5163" Text="@Conjugate (+1) Thanks for clarifying that point. I was unsure about (a) the use of 'parameters' because we often speak of person (resp. item) parameters when estimating IRT models (I don't speak of person-specific covariates, just locations on the latent trait), and (b) if there was any suggestion about differences between the marginal approach (where we assume that the $\theta_i$'s are random, with $\mathcal{N}(0;1)$ for prior distribution) and the conditional approach (where subject and item parameters are estimated separately, in line with Rasch's *specific objectivity*)." CreationDate="2010-12-06T14:16:26.530" UserId="930" />
  <row Id="7770" PostId="5176" Text="UPDATE.  Sorry, for the multiple comments. From the comment above, I didn't add the frequency portion, but that seemed to work for the auto.arima (and subsequently generated a great looking forecast) using the plot as found in your help.  You can probably tell I am new to R, but can you tell me why you add 4 in the start for April and not 5?" CreationDate="2010-12-06T14:33:22.363" UserId="569" />
  <row Id="7771" PostId="5184" Text="Never heard of Underwood's textbook, but this article seems to discuss the pros and cons of pooling: [Pragmatics of pooling in ANOVA tables](http://www.jstor.org/pss/2684424) (Hines, Am. Stat. 1996). Now, I seem to remember that Sokal &amp; Rohlf (1995) also recommend to consider very conservative values ($p\geq .25$); I need to check before posting an answer, unless better references come up." CreationDate="2010-12-06T14:35:35.517" UserId="930" />
  <row Id="7773" PostId="5186" Score="1" Text="Before the experiment, it was believed that the factor would be significant. However, it was swamped by the other two effects. I removed the factor because keeping it in didn't change the conclusions and only made the explanation harder." CreationDate="2010-12-06T15:03:05.850" UserId="8" />
  <row Id="7774" PostId="5184" Text="Just a comment.  A guideline based on $p \geq something$ smells like a misuse of a $p$-value, in that a non-significant $p$ value is not a measure of non-evidence. Since $p$-values are uniformly distributed under the null hypothesis, why not just flip a (biased) coin? The end result is the same, and at least it's honest about being dopey.  (OK, dopey is a bit strong, but you get the idea.)" CreationDate="2010-12-06T15:18:38.883" UserId="1108" />
  <row Id="7775" PostId="5184" Score="4" Text="That would be an interesting response to a referee: &quot;We thank the referee for their comments, but think they're a bit dopey&quot; ;) Good comment though." CreationDate="2010-12-06T15:28:27.310" UserId="8" />
  <row Id="7776" PostId="5185" Score="1" Text="+1 Nice answer: clear, to the point, insightful, and authoritative." CreationDate="2010-12-06T15:33:17.447" UserId="919" />
  <row Id="7777" PostId="5185" Text="gee, thanks @whuber" CreationDate="2010-12-06T15:53:57.587" UserId="1475" />
  <row Id="7778" PostId="5156" Text="The Pr(X &gt; 3) you provide is different than the question askers 0.0015, you might want to revise it." CreationDate="2010-12-06T16:33:50.740" UserId="196" />
  <row Id="7779" PostId="5186" Score="1" Text="Hmm, in that case i think i'd keep it in. I can't see why it makes the explanation harder, and as you've discovered it may be harder to explain why you dropped it than why you kept it in! " CreationDate="2010-12-06T17:12:14.180" UserId="449" />
  <row Id="7780" PostId="5187" Text="You mean something like http://www.people.fas.harvard.edu/~plam/teaching/methods/convergence/convergence_print.pdf ?" CreationDate="2010-12-06T17:32:58.093" UserId="919" />
  <row Id="7781" PostId="5188" Text="Thanks! That is just what I need.  I have &quot;Bayesian Data Analysis&quot; and I will take a closer look for the worked examples." CreationDate="2010-12-06T18:29:52.187" UserId="2274" />
  <row Id="7782" PostId="5187" Text="I looked through that exact link last night actually, which was what prompted me to convert my bugs output to CODA, but from what I can see, they aren't calling WinBUGS, they are using a built in function in R's MCMC package." CreationDate="2010-12-06T18:43:23.420" UserId="2274" />
  <row Id="7783" PostId="5077" Text="@fRed: That's why I offered the bounty. I've read about Hit and run, but don't know under what circumstances it would be useful." CreationDate="2010-12-06T18:46:49.900" UserId="8" />
  <row Id="7784" PostId="5188" Text="@fabians (+1) E.g. [Example of computation in R and Bugs](http://www.stat.columbia.edu/~gelman/bugsR/software.pdf) (Online Appendix C of BDA)." CreationDate="2010-12-06T18:52:13.237" UserId="930" />
  <row Id="7785" PostId="5189" Text="You are familiar with the tutorial at http://www.stat.pitt.edu/stoffer/tsa2/R_time_series_quick_fix.htm? Admittedly, it accompanies another package / book, but there may be some potential for carry over." CreationDate="2010-12-06T19:44:01.410" UserId="1909" />
  <row Id="7786" PostId="5007" Text="Everybody here seems very knowledgeable but I know for ggplot related questions I find their google group to be incredibly helpful.&#xA;&#xA;&#xA;http://groups.google.com/group/ggplot2" CreationDate="2010-12-06T19:48:47.210" UserId="1028" />
  <row Id="7788" PostId="5176" Text="@Rob: Weekly data is trouble anyhow, in my experience. Monthly data can have length-of-month issues and trading-day issues, but at least there are always 12 months in the year. And there are quite a few tools that aren't set up to handle weekly data, so you narrow your options considerably. I switched to monthly and saved myself a lot of grief." CreationDate="2010-12-06T20:20:08.557" UserId="1764" />
  <row Id="7789" PostId="2209" Text="Like Sivia, this is very nice if you have some physics background and can be rough if not.  Not a good guide to any kind of applied social statistics (for that use Gelman and Hill, or Gelman et al. above) but really *great* for prompting you to really think about the core issues." CreationDate="2010-12-06T21:12:25.640" UserId="1739" />
  <row Id="7791" PostId="5080" Text="Beautiful, exactly what I was looking for.&#xA;&#xA;The two lines:&#xA;&#xA;w &lt;- colSums(coef(svp)[[1]] * x[unlist(alphaindex(svp)),])&#xA;b &lt;- b(svp)&#xA;&#xA;were a godsend.  Thank you!" CreationDate="2010-12-06T21:54:00.193" UserId="2221" />
  <row Id="7792" PostId="5191" Text="Could you supply a reference for this definition of order of a kernel?" CreationDate="2010-12-06T22:01:57.360" UserId="919" />
  <row Id="7793" PostId="5176" Text="You need to drop the frequency component in the zoo object. So as.ts() won't work. There is no plot method for Arima objects. For the start argument, Jan 1 is 2009, so Jan 2 is 2009+1/365, etc." CreationDate="2010-12-06T22:08:27.793" UserId="159" />
  <row Id="7794" PostId="5191" Text="for example Tsybakov  &quot;Introduction to nonparametric estimation &quot;, definition 1.3 " CreationDate="2010-12-06T22:15:51.587" UserId="2189" />
  <row Id="7795" PostId="5189" Text="Thanks for the help.  I did see this and I think it is fantastic.  I did edit my question to just focus on how to read in a univariate time series.  Mainly, if I have the dates for my daily data, how can I construct a univariate time series in R dynamically?  I have tried a few methods but they seem to include the date as another variable when I would only like to use it as a reference for start/end dates.  Sorry, new to R and I am just trying to wrap my head around this." CreationDate="2010-12-06T22:16:03.373" UserId="569" />
  <row Id="7796" PostId="5191" Text="http://books.google.com/books?id=mwB8rUBsbqoC&amp;printsec=frontcover&amp;dq=tsybakov+++book+introduction+to+nonparametric+estimation&amp;source=bl&amp;ots=Z4wHSaMrEH&amp;sig=3VC9bW3JoOYOhwvZpVYiPgr4nmk&amp;hl=en&amp;ei=61_9TI7eK8-28QPPyojPCw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CBcQ6AEwAA#v=onepage&amp;q&amp;f=false" CreationDate="2010-12-06T22:16:21.557" UserId="2189" />
  <row Id="7797" PostId="5177" Score="1" Text="@Rob: auto.arima is in the forecast package, isn't it?" CreationDate="2010-12-06T22:19:06.690" UserId="1764" />
  <row Id="7798" PostId="5117" Text="If you want to ask a tough question, ask who the second most famous statistician is. There's no doubt that Fisher is # 1. You may not like the man or some of his ideas, but he is undoubtedly the creator of Statistics as we know it today." CreationDate="2010-12-06T22:41:13.103" UserId="666" />
  <row Id="7799" PostId="5178" Text="It's amusing that that site calculates the median (and goes to the trouble of labeling it as such and suggesting that they know the difference between the mean and the median) on just two points of data which results, actually, in the mean of the two values!" CreationDate="2010-12-06T23:33:47.537" UserId="485" />
  <row Id="7800" PostId="5177" Text="@Wayne. Yes. Sorry I omitted that information." CreationDate="2010-12-06T23:36:56.750" UserId="159" />
  <row Id="7801" PostId="5169" Text="@chl Thank you. This one's open to some interpretation. In light of a related question, the OP might be fishing for the use of a finite population correction term, which I didn't mention at all (http://stats.stackexchange.com/q/5158/919). What interests me most is the effect of rounding; everything else is routine. In some cases (not this one!) rounding has a pronounced effect (both as a bias and a variance inflator), so it's nice to be able to present a simple example of how one can evaluate that effect. Actually I'm amazed the effect is so small in this case." CreationDate="2010-12-07T00:08:31.050" UserId="919" />
  <row Id="7802" PostId="5195" Score="2" Text="@lokheart - I've got a few ideas about how to do and implement this, but would appreciate having some data to play with. Any ideas on that?" CreationDate="2010-12-07T01:53:05.997" UserId="696" />
  <row Id="7803" PostId="5195" Text="Yes, ggplot can easily draw a plot that is made up of points and lines ;) geom_smooth will get you 95% of the way - if you want more advice you'll need to provide more details." CreationDate="2010-12-07T02:05:40.573" UserId="46" />
  <row Id="7804" PostId="5171" Text="Each row is the total number of usages of the same structure by two different people talking to each other. So, for instance, if I were counting passive sentences, then the number in column A would be the number of passive sentences used by speaker A, and the number in column B the number of passive sentences used by speaker B. So each row is a different conversation. Each conversation has exactly two participants." CreationDate="2010-12-07T04:13:09.730" UserId="52" />
  <row Id="7805" PostId="5171" Text="I also have the same data calculated relative to number of sentences spoken by each speaker in each conversation, if that makes a difference." CreationDate="2010-12-07T04:15:42.287" UserId="52" />
  <row Id="7806" PostId="5179" Text="I did try this. But with so many 0s I am not sure what to make of it. I tried excluding any conversations in which either speaker used none of these constructions (i.e., the data point would fall along either axis), but the resulting Spearman correlation was not significant (and the coefficient was very small).." CreationDate="2010-12-07T04:20:25.560" UserId="52" />
  <row Id="7807" PostId="5167" Text="Sorry for being vague, I have modified the question." CreationDate="2010-12-07T04:26:49.983" UserId="2157" />
  <row Id="7808" PostId="5180" Text="You got it correctly, it is very easy to sample a variance from the distribution (by taking out the mode value). But I am not sure how to do that in multivariate case. I am using an Inverse Wishart distribution for sampling Covariance matrix." CreationDate="2010-12-07T04:31:15.410" UserId="2157" />
  <row Id="7809" PostId="5197" Score="2" Text="I may be completely wrong here, but all the example outputs seems like simple counting operations (additionally the word &quot;assumptions&quot; confuses me). Do you want to extract statistical measures or do you want to e.g. extract the combination of variable values which &quot;cause&quot; certain injuries and hence allow prediction?" CreationDate="2010-12-07T07:55:48.057" UserId="264" />
  <row Id="7810" PostId="5202" Score="2" Text="@downvoter: Why ?" CreationDate="2010-12-07T10:02:35.997" UserId="264" />
  <row Id="7811" PostId="5181" Text="chl: Actually I'm not *really* studying data on people's height, I just took that as a simplified example. My actual data has much more dimensions." CreationDate="2010-12-07T10:12:21.137" UserId="1737" />
  <row Id="7812" PostId="5186" Text="@onestop: I take your point, although I don't 100% agree with it. I could easily see another referee suggesting that you should remove the factor (that's what the bio-statisticians recommend that I've spoken too). As you mentioned, when it's a grey area an arbitrary rule is not the way to go. If we wanted to mislead we would never mention that the other factor was ever involved! Completely unethical, but I suspect it happens." CreationDate="2010-12-07T10:49:42.930" UserId="8" />
  <row Id="7813" PostId="5207" Text="In the [Singh-Maddala distribution](http://en.wikipedia.org/wiki/Singh-Maddala_distribution) link, the pdf has two parameters - {c, k}, yet the R function ``dsinmad`` takes three parameters or am I missing something." CreationDate="2010-12-07T11:10:11.260" UserId="8" />
  <row Id="7815" PostId="5207" Text="Sorry, the wikipedia link cites the wrong formula, it looked ok at first glance, when I was composing the question. I did not find a ready link, so I just put the formula in the question." CreationDate="2010-12-07T13:05:25.380" UserId="2116" />
  <row Id="7816" PostId="5211" Text="@Alex &quot;reliability of this measure&quot;: you mean of your Likert scale?" CreationDate="2010-12-07T14:05:40.847" UserId="930" />
  <row Id="7818" PostId="5212" Score="1" Text="Another reference for the comparison of binomial CIs is Brown LD, Cai TT, &amp; DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101-133. http://projecteuclid.org/euclid.ss/1009213286 (open access). R's `binom` package also has the Agresti-Coull CI." CreationDate="2010-12-07T15:48:48.367" UserId="1909" />
  <row Id="7819" PostId="5203" Text="Thank you very much. I picked up Durbin-Koopman today and it looks great." CreationDate="2010-12-07T15:55:09.080" UserId="2251" />
  <row Id="7821" PostId="5216" Text="(+1) I located and installed this package less than two months ago.  For more sophisticated work, consider using censored regression models." CreationDate="2010-12-07T17:10:35.940" UserId="919" />
  <row Id="7822" PostId="5195" Score="2" Text="This is not a funnel plot.  Instead, the lines evidently are constructed from estimates of standard errors based on the number of admissions.  They seem intended to enclose a specified *proportion* of data, which would make them *tolerance limits.*  They are likely of the form y = baseline + constant / Sqrt(# admissions * f(baseline)).  You could modify the code in the existing responses to graph the lines, but you likely would need to supply your own formula to compute them: the examples I have seen plot *confidence intervals* for the *fitted line itself*.  That's why they look so different." CreationDate="2010-12-07T17:32:19.790" UserId="919" />
  <row Id="7823" PostId="5216" Text="@whuber: I've done MCMC modelling on the data but the target audience is chemists so some simpler stats are useful to warm them up.  My worry is that NADA was withdrawn because it gave the wrong answers.  (I know it's calculations of medians was dodgy.)" CreationDate="2010-12-07T17:35:20.177" UserId="478" />
  <row Id="7824" PostId="5077" Text="Would you tell us why you want to use this instead of one of the more established MCMC-algorithms?" CreationDate="2010-12-07T18:09:03.210" UserId="1979" />
  <row Id="7825" PostId="5216" Text="@Richie That is extremely interesting to hear.  I didn't test NADA that far: I tried it out, quickly found it lacking in capabilities, and went immediately to survival analysis and censored regression models." CreationDate="2010-12-07T18:48:10.867" UserId="919" />
  <row Id="7826" PostId="5201" Text="@steffen, thanks for the response! I was first using SAS and could not get a different response variable. I fit the following in R as an example and see that a warning is issued about not being an integer reponse variable but the result did fit a different model. " CreationDate="2010-12-07T20:30:51.977" UserId="2040" />
  <row Id="7827" PostId="5201" Text="data(ToothGrowth)&#xA;attach(ToothGrowth)&#xA;&#xA;#1/0 coding&#xA;dep &lt;-ifelse(supp == &quot;VC&quot;, 1,0)&#xA;OneZeroModel &lt;-glm(dep~len,family=binomial)&#xA;OneZeroModel &#xA;predict(OneZeroModel)&#xA;&#xA;&#xA;#Platt coding&#xA;dep2 &lt;-ifelse(supp == &quot;VC&quot;, 31/32 ,1/32)&#xA;plattCodeModel &lt;-glm(dep2~len,family=binomial)&#xA;plattCodeModel&#xA;predict(plattCodeModel)&#xA;&#xA;compare&lt;-cbind(predict(OneZeroModel),predict(plattCodeModel))&#xA;plot(predict(OneZeroModel),predict(plattCodeModel))&#xA;" CreationDate="2010-12-07T20:32:15.673" UserId="2040" />
  <row Id="7828" PostId="5195" Text="@whuber (+1) That's a very good point, indeed. I hope this might provide a good starting point anyway (even if my R code isn't that optimized)." CreationDate="2010-12-07T20:34:30.570" UserId="930" />
  <row Id="7829" PostId="5171" Text="@Alan Sorry if I'm a bit slow, but I still have some difficulty to understand your data structure. Are there only 2 speakers, with 420 repeated measurements collected on each (I mean one row=one type of conversation, but the same outcome is recorded, e.g. No. passive sentences)? I ask this because you have some kind of matching or pairing (between subjects A and B), but in this latter case, you would also have to deal with repeated measurements, and this renders marginal models for matched pairs less relevant." CreationDate="2010-12-07T20:45:14.503" UserId="930" />
  <row Id="7830" PostId="5171" Text="No, these are all different speakers as well. This data was taken from a corpus of recorded telephone conversations. So speaker A in conversation 1 is not the same person as speaker A in conversation 2." CreationDate="2010-12-07T22:53:25.853" UserId="52" />
  <row Id="7831" PostId="5197" Text="I want to find where most incidents are in a particular facility if they can be grouped by attributes I have and I am asking about statistical apparatus which can help me to do it. I want to 'see' with statistics that people 'Fall' MOST OFTEN than they 'Burn' or have 'Wrong Medication' or I want to 'see' that there is nothing suspicious. Sorry for using non statistical language. It is not simple counting for sure.  " CreationDate="2010-12-07T23:41:47.297" UserId="2279" />
  <row Id="7832" PostId="5228" Text="BTW, since I am a beginner in R, I would appreciate any comments regarding my code, however brief, both on points of style and substance. In particular, I was hoping there might be a more elegant way to phrase the for loop." CreationDate="2010-12-08T03:05:39.730" UserId="795" />
  <row Id="7833" PostId="5178" Text="The table with 2 data sources, is one of the things I'm trying to correct. Some of the editors of the page are handling the data in a way that's statistically unsound, or at least I'm trying to prove that." CreationDate="2010-12-08T03:10:22.477" UserId="2271" />
  <row Id="7834" PostId="5226" Text="This is neither a quadratic nor a cubic form: it is *quartic*.  (Section 6.2.3 of the Matrix cookbook does not appear to be applicable.)" CreationDate="2010-12-08T04:03:06.330" UserId="919" />
  <row Id="7835" PostId="5234" Text="Hah, good point. Well, let's say I could break this down into about 400 different groups?" CreationDate="2010-12-08T05:47:08.907" UserId="52" />
  <row Id="7836" PostId="5234" Score="1" Text="Sure, that might work.  What you're doing then is tantamount to using the color value of a graphic to represent word density.  Color values are difficult to interpret accurately; graphical representations that map the variable of interest to *length* or *distance* usually work best.  That's exactly what a kernel density plot will do for you.  By varying the half-width you can range from extreme detail to a gross overview." CreationDate="2010-12-08T05:55:30.023" UserId="919" />
  <row Id="7837" PostId="5235" Score="3" Text="What is a &quot;criterion variable&quot;??" CreationDate="2010-12-08T05:59:48.523" UserId="919" />
  <row Id="7838" PostId="5234" Text="haha nice one, whuber" CreationDate="2010-12-08T06:02:03.937" UserId="1307" />
  <row Id="7839" PostId="5231" Text="@whuber - nice one!" CreationDate="2010-12-08T06:12:43.257" UserId="1307" />
  <row Id="7840" PostId="5240" Text="@ Matti Pastell: I have this book. It is very good indeed. My question is about the difference between the particule filter (which from what I understand is a sequential version of MCMC), and a MCMC on a rolling window (in the latter, we re run the optimisation process on a rolling window). &#xA;Which method should be prefered, and why?" CreationDate="2010-12-08T07:53:49.610" UserId="1709" />
  <row Id="7841" PostId="5240" Text="Also, I don't really find easy to build this time varying model with dlm. Honestly the package is very easy to use for non time varying models, but it starts to be more tricky for everything else. Edit: By more tricky I mean that there is no function to solve the problem. You need to code yourself the script." CreationDate="2010-12-08T07:56:24.460" UserId="1709" />
  <row Id="7842" PostId="5240" Text="OK, I also have the book but I haven't had time to read it yet. Sorry that it doesn't help with your problem. " CreationDate="2010-12-08T08:01:39.637" UserId="214" />
  <row Id="7843" PostId="5241" Score="1" Text="+1 for the Emacs point - I should take a look at that." CreationDate="2010-12-08T08:17:18.187" UserId="1390" />
  <row Id="7844" PostId="5201" Text="@user2040: 1. Yes ... is there a question or is this just a remark :) ? 2. why don't you add the code to the question ? No formatting is horrible." CreationDate="2010-12-08T08:21:12.520" UserId="264" />
  <row Id="7845" PostId="5229" Text="@Jeromy (+1) Thanks for these additional references." CreationDate="2010-12-08T08:21:58.697" UserId="930" />
  <row Id="7846" PostId="5228" Score="1" Text="@shabbychef: on the R style, you don't need `;` on the ends of your statements. You did right to pre-allocate, but no need to fill with `NA`, `kvals &lt;- numeric(length = n_trial)` would have sufficed. With `rep`, you only need arguments 1 and 3 from your call (e.g. `rep(NA, 10)`). In the `for` loop set-up, `1:n_trial` can be dangerous if programming; better is `seq_along(kvals)` or `seq_len(n_trial)` in this case. Finally, if you don't need to force printing, drop the `print()` round `summary()` - you'd only need it if you weren't working interactively with R. HTH." CreationDate="2010-12-08T08:28:44.730" UserId="1390" />
  <row Id="7847" PostId="5239" Text="With regards to reproducible research, if the file format is CSV, then the CSV file could easily be put under version control with commits when changes are made. Would this suffice for reproducible research? With regards to data entry, I often do or get others to do raw data entry in fix width format in order to maximise efficiency; however I prefer to store data in CSV or TSV (I find it more transparent); with regards to formatting a column as numeric. this information would presumably be lost on saving." CreationDate="2010-12-08T08:45:14.223" UserId="183" />
  <row Id="7848" PostId="5239" Text="with regards 12/3, I had an example of a multiple choice test where this represented the 12 divided by 3. However, I use it more as an illustrative example. When I have a large CSV file, I don't want even one inappropriate conversion. With regards to warnings, If I'm opening CSV files 10 times a day the warnings get tedious. They also make it less clear whether I have actually saved the CSV file or not. Sorry for the rant." CreationDate="2010-12-08T08:48:59.133" UserId="183" />
  <row Id="7849" PostId="5239" Text="@Jeromy yes, it would, if you were strict about making a single change (or single data processing step) that you immediately save and commit. I mean't to include a note about doing the processing in R (as it is my preferred language) using a script. That is how I work with data colleagues send me; I might have a quick look in a spreadsheet, then read the csv into R and write a script that includes all the data processing steps I need to apply. I then comment that script so I have a record of what I did and why I did it and I haven't changed the original data file at all." CreationDate="2010-12-08T08:50:23.133" UserId="1390" />
  <row Id="7850" PostId="5239" Text="@Jeromy: fair point on the warnings. I've long since filtered them out as noise so they don't bother me. I just tested Gnumeric and OpenOffice.org and they convert a csv with 12/3 to dates automagically - that is rubbish! So I see what you mean. The only way to stop that would be to store those data as text and force that data type on load/import." CreationDate="2010-12-08T08:58:33.440" UserId="1390" />
  <row Id="7851" PostId="5239" Text="@Jeromy: re: losing numeric formatting - yes, it will, unless you specify the column type as &quot;numeric&quot; (Gnumeric) or &quot;text&quot; in (OpenOffice.org) upon import. Better might be to store it as text (see my edited answer) to avoid the conversion - you'll still need to specify the data type upon import..." CreationDate="2010-12-08T09:05:28.973" UserId="1390" />
  <row Id="7852" PostId="5239" Text="@Gavin I agree about having a script that processes a supplied data file. I guess I'm thinking about some relatively ad hoc files that I create (for example, I might create a little csv file that lists the variable names and corresponding labels that I'll use when preparing an R table)." CreationDate="2010-12-08T09:05:32.217" UserId="183" />
  <row Id="7853" PostId="5243" Score="1" Text="following this question, I did a hunt: there is a Windows version http://projects.gnome.org/gnumeric/downloads.shtml" CreationDate="2010-12-08T09:11:07.450" UserId="183" />
  <row Id="7854" PostId="5224" Text="Sorry, but NN is not only ML. This use is a pure soft-computing, so I have to close it. This would fit on AI proposal on Area51, you can commit here: http://area51.stackexchange.com/proposals/6607/artificial-intelligence?referrer=K4nouwJjttuP1xPZSakPmQ2" CreationDate="2010-12-08T09:14:35.110" UserId="88" />
  <row Id="7855" PostId="5243" Text="@mbq; it does warn about only saving the current sheet though (which was one of Jeromy's bug-bears), and it has the annoying feature of converting properly saved 12/3 numerics to dates unless you manually specify they are numeric on import (Data &gt; Get External Data &gt; Import Text File...) not load." CreationDate="2010-12-08T09:31:36.547" UserId="1390" />
  <row Id="7856" PostId="5221" Text="Yes, mixture models are exactly what I need. " CreationDate="2010-12-08T09:31:39.843" UserId="2116" />
  <row Id="7857" PostId="5225" Text="For now flexmix is working very nicely, thank you very much for the hint" CreationDate="2010-12-08T09:32:09.390" UserId="2116" />
  <row Id="7858" PostId="4904" Text="@Quandary : be aware of the fact that 95% of your **fair** players are situated within 2 standard deviations from the mean as well. You need more than that to find cheaters." CreationDate="2010-12-08T09:49:44.020" UserId="1124" />
  <row Id="7859" PostId="5240" Text="Thank you anyway, it is a good book, it deserves to be cited here" CreationDate="2010-12-08T10:03:41.730" UserId="1709" />
  <row Id="7861" PostId="5239" Text="	&#xA;@Gavin Thanks for the suggestion about Data Interchange Format.  I've asked a separate question about this: http://stats.stackexchange.com/questions/5249/simple-reliable-open-and-interoperable-plain-text-format-for-storing-data" CreationDate="2010-12-08T10:44:32.513" UserId="183" />
  <row Id="7862" PostId="5249" Score="2" Text="I should add, R doesn't have a `write.DIF()` so it is a bit of a one-way street I am afraid." CreationDate="2010-12-08T11:01:00.837" UserId="1390" />
  <row Id="7863" PostId="5249" Score="1" Text="I do not understand the issue of csv and reliability. Do you mean that csv is not strict enough ? Strict means that if it the regulations for csv were strict enough, every tool following these definitions could load a file without the need of extra parameters." CreationDate="2010-12-08T12:20:51.963" UserId="264" />
  <row Id="7864" PostId="5249" Text="@steffen I mean things like: loading and saving a csv file in some programs changes the csv file; loading csv files can result in inappropriate conversion unless you are careful; csv files sometimes break when strange combinations of characters are added without proper escaping. Perhaps I'm confusing use of csv with the format itself, although I have heard people comment about the lack of an official standard. Of course, I realise that in many cases it works just fine." CreationDate="2010-12-08T12:36:18.053" UserId="183" />
  <row Id="7865" PostId="5222" Text="Wilson's macros are definitely the way to go if you want to use SPSS. You may also consider picking up a copy of &quot;Practical Meta-Analysis&quot; by Lipsey &amp; Wilson." CreationDate="2010-12-08T12:37:02.197" UserId="1934" />
  <row Id="7866" PostId="5253" Score="1" Text="I also recommend taking a look at `str()` as it provides other useful details about your object. Can often explain why a column isn't behaving as it should (factor instead of numeric, etc). " CreationDate="2010-12-08T13:45:39.647" UserId="696" />
  <row Id="7867" PostId="5249" Score="3" Text="@steffen: CSV doesn't store any information about the format or data-types of the data stored in the file. You can well open a CSV file in two different apps and have it interpret the data in the file in two different ways." CreationDate="2010-12-08T13:49:30.860" UserId="1390" />
  <row Id="7868" PostId="5249" Text="@JeromyAnglim, I think that changing of  the csv file depends on your software, not the csv format per se." CreationDate="2010-12-08T13:52:49.500" UserId="144" />
  <row Id="7869" PostId="5226" Text="@whuber: you are right. The applicable formula is on *8.2.4 Mean of Quartic Forms* of MatrixCookbook. And not surprisingly it gives $3\sum_{i}\lambda_i^2 + \sum_i \sum_j \lambda_i \lambda_j$" CreationDate="2010-12-08T14:27:46.980" UserId="2148" />
  <row Id="7870" PostId="5253" Score="2" Text="Please read the R guide of Owen first (http://cran.r-project.org/doc/contrib/Owen-TheRGuide.pdf), and if possible, Introduction to R (http://cran.r-project.org/doc/manuals/R-intro.pdf). Both are on the official website of R. You're incredibly lucky you actually get an answer. On the r-help list one would redirect you to the manual in less elegant terms. No offense meant." CreationDate="2010-12-08T15:19:06.880" UserId="1124" />
  <row Id="7872" PostId="5249" Text="@Jeromy,Gavin: I see. This is indeed an attribute I have seen only in tool-dependent formats (so far). All in all a very interesting question." CreationDate="2010-12-08T15:24:32.087" UserId="264" />
  <row Id="7873" PostId="5226" Text="@Isaac I believe the second (double) sum is restricted to i != j." CreationDate="2010-12-08T15:31:40.660" UserId="919" />
  <row Id="7874" PostId="5242" Text="+1 Nice example." CreationDate="2010-12-08T15:35:23.467" UserId="919" />
  <row Id="7875" PostId="5253" Text="@Joris - Point taken (without offence), but it was my impression that SE sites were designed to foster problem/solution learning in a way not afforded by manuals. Additionally, this question will now be available for other beginners. Thanks for the links though." CreationDate="2010-12-08T15:42:15.557" UserId="1950" />
  <row Id="7876" PostId="5244" Text="I tried installing it.  After a quick try it seemed like rubbish for data entry; maybe if I gave it a longer try I'd be able to figure out how to use it effectively, but I suspect not." CreationDate="2010-12-08T16:06:54.077" UserId="196" />
  <row Id="7877" PostId="5239" Text="I glanced at Wikipedia's page, and yes DIF files are plain text... but they also look more computer readable than human readable and seem like they'd also be difficult to do data entry with." CreationDate="2010-12-08T16:11:40.063" UserId="196" />
  <row Id="7878" PostId="5233" Text="Seems a little silly to post as an answer, but [wordle](http://www.wordle.net/) may give you something useful, or at least interesting to ponder..." CreationDate="2010-12-08T16:51:31.060" UserId="696" />
  <row Id="7879" PostId="5226" Text="@whuber: Yes you are completely right! ;)" CreationDate="2010-12-08T17:03:22.290" UserId="2148" />
  <row Id="7880" PostId="5253" Score="1" Text="If you're looking for pure code solutions, stackoverflow might be more appropriate. Although, all the R gurus present @ SO are also here (not counting myself). :)" CreationDate="2010-12-08T17:39:27.177" UserId="144" />
  <row Id="7881" PostId="5228" Text="thanks! definitely what I was looking for. I was running this from a file, so I needed the `print`. The args to `rep` were certainly erroneous." CreationDate="2010-12-08T17:46:21.007" UserId="795" />
  <row Id="7882" PostId="5242" Text="+1 for `set.seed`, which would help in examples like this. Is there a substantial reason for encapsulating in a function (_e.g._ the R interpreter will pre-compile functions, thus there is some speedup), or is stylistic (_e.g._ encapsulating functions, like broccoli, is good for you), or somewhere in between (_e.g._ there are, say, lots of operators in R which act on functions, so one should get used to functional programming)?" CreationDate="2010-12-08T17:49:21.407" UserId="795" />
  <row Id="7883" PostId="5245" Text="Thank you very much. Your explanation is very helpful to me." CreationDate="2010-12-08T17:50:04.013" UserId="400" />
  <row Id="7884" PostId="5232" Score="1" Text="The G2 estimator from `e1071` gives the 'standard' bias correction; see http://cran.r-project.org/web/packages/e1071/e1071.pdf . Using this estimator instead of g2, implemented by the `moments` package, had little effect, as I noted in the Q. The dependence on the eighth moment would indeed imply the sample size is too small here." CreationDate="2010-12-08T17:55:50.540" UserId="795" />
  <row Id="7885" PostId="5242" Text="@shabbychef: I think the main one is effort. You could run your code with the loop etc repeatedly and it would be fine, but you have to keep running all the lines of your code. By encapsulating it, you run 1 line of R code for each simulation you want to run. There is no speed up as R doesn't compile anything ahead of time IIRC." CreationDate="2010-12-08T18:07:35.020" UserId="1390" />
  <row Id="7886" PostId="5242" Score="1" Text="@Gavin Simpson, thanks for the clarification. Since this is all in a small file, it is one line anyway: `source('foo.r')` ;)" CreationDate="2010-12-08T18:10:02.193" UserId="795" />
  <row Id="7888" PostId="5272" Score="1" Text="@Daniel: if applying this, turn off the x-axis in the `plot()` call: `plot(x, y, type = &quot;l&quot;, xaxt = &quot;n&quot;)` and change the two `x` s so you don't overwrite the data you created to draw the plot." CreationDate="2010-12-08T19:12:45.340" UserId="1390" />
  <row Id="7889" PostId="5272" Text="This worked, except it plotted the new axis over the old one. How would I address this issue?" CreationDate="2010-12-08T19:16:12.437" UserId="1973" />
  <row Id="7890" PostId="5272" Text="@Gavin Thanks, you beat me to it!" CreationDate="2010-12-08T19:20:38.157" UserId="1973" />
  <row Id="7891" PostId="5238" Text="What kind of operations do you actually do on the files? In my mind, that dramatically affects the range of acceptable options. Also, it seems to me that you could edit TSV data in a word processor and line up tabs to preserve columns -- as long as your word processor can be set to not do &quot;smart&quot; conversions and can save as plaintext, which many can." CreationDate="2010-12-08T19:30:35.090" UserId="1764" />
  <row Id="7892" PostId="5273" Text="exactly. The error is in the first line of the question, where the OP defines $\hat{m}$ as the sum of the $X_i$ not the mean, as you do." CreationDate="2010-12-08T19:43:21.327" UserId="795" />
  <row Id="7893" PostId="5269" Text="Thanks. i fixed the question." CreationDate="2010-12-08T20:10:14.540" UserId="2148" />
  <row Id="7894" PostId="5273" Score="2" Text="The quantity $(N(0,\frac{\sigma^2}{n}))^2$ is a scaled Chi-square. As a check, the variance of a Chi-square with $k$ dof is $2k$, which gives the same result you get." CreationDate="2010-12-08T20:14:04.613" UserId="795" />
  <row Id="7895" PostId="5253" Score="1" Text="@Tom Wright I disagree with your assertion that this question will be helpful for other beginners, *especially* if they don't skim the manual.  They will just create a duplicate question." CreationDate="2010-12-08T21:01:34.220" UserId="1657" />
  <row Id="7897" PostId="5223" Text="If this answer is not satisfactory, could you explain why?" CreationDate="2010-12-08T22:11:15.703" UserId="1146" />
  <row Id="7898" PostId="5248" Text="Thank you! Just to add, I found in the sampleSelection R package, under the help page of the inverse mills function, the ability to extract inverse mill ratios for bivariate probit models which can then be used in the final outcome equation. This doesn't resolve the panel data issue, but it does address the multiple selection one. FWIW. " CreationDate="2010-12-08T22:35:18.340" UserId="2050" />
  <row Id="7899" PostId="5260" Score="1" Text="@Isaac The variance of the sample mean (&quot;and I know that...&quot;) scales as 1/N, not 1/N^2.  That's the only change you need." CreationDate="2010-12-08T22:55:59.210" UserId="919" />
  <row Id="7900" PostId="5201" Text="@steffen: Thank you." CreationDate="2010-12-09T01:08:02.200" UserId="2040" />
  <row Id="7901" PostId="5279" Score="1" Text="@Dirk RData certainly works well with R. It might be problematic with regards to version control. I suppose the R function `dput()` provides a plain text alternative that would work with version control. However, one of the appeals of csv/tsv is that when I share a repository with data (say for a journal article), people could take the data and reanalyse it easily using any software they like." CreationDate="2010-12-09T01:45:06.677" UserId="183" />
  <row Id="7902" PostId="5279" Score="1" Text="Yes, it is a hugely complicated matter. I think people have discussed this since the dawn of computing.  I had two more thoughts (and I could expand my answer): ProtocolBuffers are good for *efficiently* sharing with Python, Java, C++, ... and a host of other languages; Romain and I cover R.  The the new-ish site http://mldata.org covers this for research in Machine Learning -- they even have tools they make available to convert. That may be worth a look." CreationDate="2010-12-09T01:48:23.523" UserId="334" />
  <row Id="7903" PostId="5262" Text="@drknexus Thanks. It sounds like using a delimited file format with appropriate care may be the best option." CreationDate="2010-12-09T01:48:34.927" UserId="183" />
  <row Id="7904" PostId="5202" Text="the Finns need some representation!" CreationDate="2010-12-09T01:56:32.427" UserId="74" />
  <row Id="7905" PostId="5238" Text="@Wayne Good point. Because I've grown up with spreadsheet programs, there are many shortcuts that I'm familiar with (functions, copy and paste, adding an extra column, and much more). Note, I'm not talking here about data analysis, but just creating a simple tabular data file (e.g., some meta information needed to process my R code). While I could do all these basic table manipulations in R, it's not quite as intuitive for me. Probably over time, opening a csv in R, making a few little changes and saving it again will become my preferred option." CreationDate="2010-12-09T02:01:15.447" UserId="183" />
  <row Id="7906" PostId="5282" Text="Thank you Glen! Nice references." CreationDate="2010-12-09T02:33:04.593" UserId="1307" />
  <row Id="7907" PostId="5279" Text="Actually, SVN takes binary blobs such as pdf files etc without problem. I'd suspect git does too." CreationDate="2010-12-09T03:08:17.417" UserId="334" />
  <row Id="7908" PostId="5279" Text="@Dirk That's good to know about binary blobs. It would still be nice to be able to run diff on text files and get meaningful information about changes. Thanks also for link to mldata.org. That looks interesting." CreationDate="2010-12-09T03:27:35.123" UserId="183" />
  <row Id="7909" PostId="5279" Text="Pleasure. Sister site mloss.org is simply great, if hope they get traction for mldata.org.  The time is right for that." CreationDate="2010-12-09T03:32:35.970" UserId="334" />
  <row Id="7910" PostId="5274" Score="1" Text="You're on your way towards re-inventing Ripley's K function (http://www.stat.iastate.edu/preprint/articles/2001-18.pdf ).  Note, however, that much information is lost in this approach.  Suppose, for example, that 15 of the distances are zero and the other 15 equal 10,000.  Is that 15 clusters of two words or one cluster of 16 words and 14 isolated words?" CreationDate="2010-12-09T03:35:59.313" UserId="919" />
  <row Id="7911" PostId="5241" Text="Yep, and it turns out I already have `csv-mode.el` installed thanks to the `emacs-goodies-el` package in Debian/Ubuntu -- neat." CreationDate="2010-12-09T03:50:15.587" UserId="334" />
  <row Id="7912" PostId="5284" Text="Thanks, whuber! Could you recommend some books/tutorials that have some good cover on antithetic methods?" CreationDate="2010-12-09T05:02:07.487" UserId="1005" />
  <row Id="7913" PostId="5286" Text="@shabbychef (+1) Any thought about its performance compared to [NetCDF](http://www.unidata.ucar.edu/software/netcdf/)?" CreationDate="2010-12-09T06:39:50.913" UserId="930" />
  <row Id="7914" PostId="5202" Text="@Neil: Are you the downvoter ? I don't get it." CreationDate="2010-12-09T07:44:03.500" UserId="264" />
  <row Id="7916" PostId="5223" Text="I'm not sure about @fred, but when I offered the bounty I suppose I was wanting a bit more insight into hit-and-run MCMC. For example, what types of problems would it be best suited for. Of course, if there are no other answers, then this question would win the bounty." CreationDate="2010-12-09T09:49:17.370" UserId="8" />
  <row Id="7917" PostId="5260" Text="@whuber. yes you are right again :D" CreationDate="2010-12-09T10:19:03.773" UserId="2148" />
  <row Id="7918" PostId="5289" Score="3" Text="I'd add that the mean, sd, max and min provide some evidence regarding the issue of whether assuming the underlying distribution is symmetric is a reasonable assumption; if the distribution is symmetric then the population mean and median will be the same." CreationDate="2010-12-09T10:54:42.763" UserId="183" />
  <row Id="7919" PostId="5290" Score="2" Text="Your question is unclear at the moment - perhaps you could edit it to make it clearer?" CreationDate="2010-12-09T11:18:27.857" UserId="226" />
  <row Id="7920" PostId="5223" Score="1" Text="@csgillespie: I edited my answer to better accomodate your interest. Let it not be said that I was undeserving of the bounty. ;)" CreationDate="2010-12-09T11:50:48.890" UserId="1979" />
  <row Id="7921" PostId="3790" Text="If mean is unique minimizer, shouldn't it be &#xA;$E|X-\mu|\le E|X-m|$?" CreationDate="2010-12-09T13:09:09.680" UserId="2116" />
  <row Id="7922" PostId="5241" Text="+1, For point #1 all stat software I have ever worked (besides R, Stata, SPSS, and SAS) with provides this type of functionality. So your advice generalizes to whatever software someone wants to work with." CreationDate="2010-12-09T13:45:13.737" UserId="1036" />
  <row Id="7923" PostId="5289" Text="Thank you. Unfortunately my data does not conform to any particular distribution, so I will be unable to use these methods. It is still useful information to know, so thank you anyway." CreationDate="2010-12-09T14:50:59.457" UserId="261" />
  <row Id="7924" PostId="5289" Text="@robintw: If you guess the data does not fit to any particular distribution, the Normal distribution may be the best choice, especially if the values are mixture of several other values (Look at http://en.wikipedia.org/wiki/Central_limit_theorem )" CreationDate="2010-12-09T15:02:39.520" UserId="2148" />
  <row Id="7925" PostId="5292" Score="5" Text="I use Emacs + ESS so I guess I would not be of much help here. However, for your information, there will be a [special issue](http://www.jstatsoft.org/misc/CFP-JSS_SV-GUIs_in_R.pdf) on R GUIs (expected for mid 2011)." CreationDate="2010-12-09T15:08:41.043" UserId="930" />
  <row Id="7926" PostId="5299" Score="1" Text="[The Art of Data Augmentation](http://j.mp/hiZ5ED), by van Dyk and Meng may be a good start. I also quickly found some R examples on Bayesian statistics on [Peter M. Lee](http://j.mp/fslE35) and [Brian Neelon](http://j.mp/h1yJyQ)'s websites. But I guess you can find other good tutorials with Google, or a more experienced user will have some good recommendations to offer." CreationDate="2010-12-09T15:25:07.840" UserId="930" />
  <row Id="7927" PostId="5305" Text="Could you provide an example?" CreationDate="2010-12-09T16:23:33.723" UserId="1657" />
  <row Id="7928" PostId="5305" Score="1" Text="If there ever was a list of FAQ to xts, this would score highly. Please look around here, search for `[r] xts` and peruse the r-sig-finance archives." CreationDate="2010-12-09T16:24:21.273" UserId="334" />
  <row Id="7929" PostId="5299" Text="Great thanks.  Anyone know of a good book/textbook reference?" CreationDate="2010-12-09T16:35:00.393" UserId="2310" />
  <row Id="7930" PostId="5274" Text="If you hadn't let me know, I might've named it Wayne's Method, written a paper on it, and gotten lots of references... or maybe that only works in medical literature... ;-) My moment of glory gone. On a more serious note, I guess for this exploratory application, the usual worrying about appropriate bandwidths isn't really an issue?" CreationDate="2010-12-09T16:58:10.887" UserId="1764" />
  <row Id="7931" PostId="5286" Text="IIRC that is also the internal format chosen at http://mldata.org -- with a suite of tools which convert.  The converters may be worth a look.  I always had the feeling that R support for HDF5 was less that perfect. " CreationDate="2010-12-09T17:07:32.407" UserId="334" />
  <row Id="7932" PostId="5302" Score="1" Text="+1.  Given that this is truly &quot;quick and dirty,&quot; one might be inclined to emphasize that $\beta_4$ is the *average* marginal change in Y.  It could differ appreciably from the actual marginal changes.  This average will be meaningful only if the relationships among the independent variables and the dependent variable are reasonably linear." CreationDate="2010-12-09T17:13:08.987" UserId="919" />
  <row Id="7933" PostId="5286" Text="@chl I had vaguely thought that NetCDF used HDF5 internally, but that seems to be not quite accurate." CreationDate="2010-12-09T17:13:35.010" UserId="795" />
  <row Id="7934" PostId="5311" Text="+1 I agree completely.  Either command line or standard windows gui.  Learn the language, and leave all the bells and whistles for later." CreationDate="2010-12-09T17:14:11.327" UserId="5" />
  <row Id="7935" PostId="5274" Text="@Wayne (myself): just got back from lunch and realized that the usual bandwidth concerns are because you are using the kernel density to attempt to visualize an underlying population distribution from which you have sampled, while in this case the data is the entire population. Right?" CreationDate="2010-12-09T17:19:37.593" UserId="1764" />
  <row Id="7936" PostId="5307" Text="Ahh.  That answers the question.  However, it leads on to the next one - which one is preferred?" CreationDate="2010-12-09T17:37:12.800" UserId="1991" />
  <row Id="7937" PostId="5223" Score="1" Text="@fabians: Many thanks for the link. One of the reasons I placed the bounty was that my google searches turned up a few mathematical discussions of the method, but little in the way of practical applications. Please don't take it as a slight if I wait another 48 hours before awarding the bounty (it is a particularly generous bounty!)" CreationDate="2010-12-09T18:03:07.540" UserId="8" />
  <row Id="7938" PostId="5310" Text="There is only a small issue - if I have ..., 14:59, 15:00, ..., it will retrieve 14:59 and not 15:00 as I'd like. Is there a way to make it return 15:00? I tried something like `&quot;seconds&quot;, 3601`, but it doesn't work" CreationDate="2010-12-09T18:39:45.307" UserId="749" />
  <row Id="7939" PostId="5310" Text="Since the timestamps are irregular one can only make guarantees on the dt between data points. The actual points depend on the starting point of the series. If you fix your starting point to a round hour it will behave as you like... (assuming you have points every N minutes)" CreationDate="2010-12-09T18:46:12.197" UserId="300" />
  <row Id="7940" PostId="5316" Text="The analysis is being done on means of response times and accuracies that are collected into 8 bins based on an individual's ordered response times for each trial type. For this type of analysis, will nlme() work on means rather than the data points that compose the mean?" CreationDate="2010-12-09T18:59:31.887" UserId="2322" />
  <row Id="7941" PostId="5310" Text="This is exactly what I don't want - being dependent on the first point" CreationDate="2010-12-09T19:04:29.967" UserId="749" />
  <row Id="7942" PostId="5319" Text="Consider any time series - outside temperature. I want to know what's the temperature at 11:00 wall clock, at 12:00 wall clock, .... I'll check your example later to see if it does this" CreationDate="2010-12-09T19:05:35.707" UserId="749" />
  <row Id="7943" PostId="5316" Score="1" Text="Get the raw data.  The analysis works on the raw data the went into generating the means.  Otherwise, you'll get a result that looks suspiciously like your original ANOVA." CreationDate="2010-12-09T20:01:45.300" UserId="601" />
  <row Id="7944" PostId="5202" Text="nope, it was not me..." CreationDate="2010-12-09T20:51:36.403" UserId="74" />
  <row Id="7945" PostId="5289" Text="@Isaac The CLT is irrelevant and inapplicable: we seek the median of the *parent distribution*, not the median of the *sampling distribution of the mean*.  However, it (coincidentally) turns out that the maximum entropy solution (given the mean, sd, and upper and lower bounds for all values) is a doubly truncated Gaussian distribution.  One would numerically solve for its parameters to match the observed mean and sd and then compute its median (which will usually *not* be the same as its mean because the truncation can be asymmetric)." CreationDate="2010-12-09T20:52:36.177" UserId="919" />
  <row Id="7946" PostId="5289" Text="@Jeromy The only statistic that can be formed of those four values that provides any information about symmetry is the ratio (max - mean):(mean - min).  That's a (highly) non-robust statistic, though: I would entirely discount its use in a practical problem.  However, when data are *constrained* to lie between two limits, those limits (relative to the mean) do add useful information." CreationDate="2010-12-09T20:56:17.343" UserId="919" />
  <row Id="7947" PostId="4450" Score="1" Text="@Srikant Thank you.  I was thinking along similar lines when I posed the question and have since tested out a GLM (Poisson distribution with *linear* link) as well as some other models.  Unfortunately, it now looks like any model based solely on land cover type and proportion will not work well: a sample of these data suggests that population patterns depend on a larger spatial context.  At a minimum, then, we would need to include spatially lagged covariates in a linear model." CreationDate="2010-12-09T21:03:55.730" UserId="919" />
  <row Id="7948" PostId="5320" Text="Thanks! very helpful!" CreationDate="2010-12-09T21:43:23.963" UserId="1991" />
  <row Id="7949" PostId="5311" Text="+1 Un*x spirit..." CreationDate="2010-12-09T21:55:15.393" UserId="930" />
  <row Id="7950" PostId="4449" Score="1" Text="Thank you.  That work provides a pointer into a web of recent research on dasymetric mapping." CreationDate="2010-12-09T21:55:46.667" UserId="919" />
  <row Id="7951" PostId="5090" Text="I've been teaching myself DLM and have finally gotten a good model for some data I have, and as you found, the ModReg (for moving seasonality) prevents it from forecasting. I'm going to look into it tonight to see if it's an issue with getting the forecast X variables into the model, or if SS forecasting needs too much custom-tailoring for stochastic models, or if one of their core routines (Kalman, etc) simply don't like the additional JFF, etc, matrices. I am noticing that DLM has a nice front-end for making complex SS matrices, but the back end is a lot more dangerous/complex." CreationDate="2010-12-09T22:06:04.910" UserId="1764" />
  <row Id="7952" PostId="5211" Text="no sorry, I mean the combined measure of my likert scale items and the monetary item. In order to group them in one measure they have all been standardized." CreationDate="2010-12-09T22:06:17.023" UserId="2285" />
  <row Id="7953" PostId="5299" Text="@Glen Maybe you could add some info about its intended use and the target language ." CreationDate="2010-12-09T22:12:47.010" UserId="930" />
  <row Id="7954" PostId="5325" Text="Interesting, I'll try that tomorrow at work (don't have the data here with me now), and in the mean time I'll give you a +1 ;)" CreationDate="2010-12-09T22:21:14.287" UserId="582" />
  <row Id="7955" PostId="5301" Score="2" Text="+1 for Rcmdr. It's nice for when one is just learning R, and offers more of a gateway to a good statistics package than SPSS or Minitab, but the code it writes is needlessly verbose, often. A scatterplot produced in Rcmdr: scatterplot(tab~pct, reg.line=FALSE, smooth=FALSE, spread=FALSE, &#xA;  boxplots=FALSE, span=0.5, data=senate.race), when most of those parameters were defaults to begin with. It writes some very wordy code." CreationDate="2010-12-09T23:00:56.893" UserId="1118" />
  <row Id="7956" PostId="5301" Text="@Christopher I agree with you; sometimes the code is wordy.  It comes from Tcl/Tk and how Rcmdr is currently written: it would take a substantial rewrite to put in checks for whether the default is selected (and return an abbrev. version in that case).  But, yes, it would be an improvement.  A redeeming feature is that it's *good* code, in that it encourages good practice (FALSE instead of F, etc.).  This is important for beginners." CreationDate="2010-12-09T23:23:09.183" UserId="1108" />
  <row Id="7957" PostId="5296" Score="1" Text="Well done! The purpose of a GUI is not to learn the language, but to make learning the language unnecessary. (edit:) I didn't know that some GUI's had the features described by G. Kay Kerns. I suppose that would help you in the early stages of learning the language." CreationDate="2010-12-10T00:04:31.113" UserId="666" />
  <row Id="7958" PostId="5311" Score="2" Text="Honestly, I don't agree. Scripting is much more powerful. " CreationDate="2010-12-10T02:47:57.947" UserId="1709" />
  <row Id="7959" PostId="5301" Text="This is true. Another advantage to this is allowing beginners to see what parameters a function takes without using the ? command. Using FALSE instead of F is a great point. I once saw a program give an incorrect output because the programmer had earlier set the result of an ANOVA to a variable called &quot;F&quot;.  " CreationDate="2010-12-10T03:34:51.577" UserId="1118" />
  <row Id="7960" PostId="5289" Text="@whuber Would the SD have some relevance to variables like income and reaction time which are constrained just at the lower end?" CreationDate="2010-12-10T05:19:37.613" UserId="183" />
  <row Id="7961" PostId="5295" Text="@Andy (1) I would like to be able to open and close the data file as easily as I can open an Rdata, Excel, or SPSS data file. Spending time walking through a wizard works, but it's not quite the simple and reliable workflow that I'd ideally like. (2) Yep, I agree about using an irregular delimiter. In general Tab is sufficient for me most of the time; (3) I don't have huge problems with CSV/TSV. I have occasional problems that are easily resolved. However, I'd like not to have to think about the issues of delimiters and format conversion." CreationDate="2010-12-10T05:29:17.650" UserId="183" />
  <row Id="7962" PostId="5311" Text="@fRed Scripting in included in my operational definition of command line interface (-;" CreationDate="2010-12-10T07:09:04.110" UserId="88" />
  <row Id="7963" PostId="5337" Text="Thank you, a nice point, the problem with minimum distance in my case is that it is I have thousands of nodes." CreationDate="2010-12-10T07:55:22.090" UserId="2325" />
  <row Id="7964" PostId="5337" Text="I want to give you +1 but I need at least 15 points of reputation:(" CreationDate="2010-12-10T07:56:19.157" UserId="2325" />
  <row Id="7965" PostId="5333" Text="it would be helpful to see &lt;code&gt;summary(r.only.lmer)&lt;/code&gt;" CreationDate="2010-12-10T08:11:29.907" UserId="2116" />
  <row Id="7966" PostId="5338" Text="Thanks, but even if the data is not independent, when you want to calculate the EMA return from t-m to t, you would ideally not want to have the EMA at t include information before t-m. " CreationDate="2010-12-10T09:01:34.867" UserId="2316" />
  <row Id="7967" PostId="5280" Text="Is the first illustration produced using R?" CreationDate="2010-12-10T09:35:34.937" UserId="339" />
  <row Id="7968" PostId="5235" Text="@whuber I'm guessing Jfly is referring to the response/outcome/dependent/etc. variable. http://davidmlane.com/hyperstat/A101702.html It's interesting to see the many names such variables go by: http://en.wikipedia.org/wiki/Dependent_and_independent_variables#Alternative_terminology_in_statistics" CreationDate="2010-12-10T10:13:14.997" UserId="183" />
  <row Id="7969" PostId="5280" Text="@gd047 Yes. Should have the ugly source code somewhere if you want. The second one is done in Metapost." CreationDate="2010-12-10T10:29:15.637" UserId="930" />
  <row Id="7970" PostId="5333" Text="@Alan Is it related to the question here, http://stats.stackexchange.com/questions/5171/testing-paired-frequencies-for-independence/?" CreationDate="2010-12-10T10:45:21.303" UserId="930" />
  <row Id="7971" PostId="5268" Text="in addition to Galit: Do you want primarily to find correlations and hence clustering is just the way, not the goal, or do you want to find clusters of users to discover some knowledge in the data (with respect to correlation) ?" CreationDate="2010-12-10T10:53:53.980" UserId="264" />
  <row Id="7972" PostId="5343" Text="What do you mean by &quot;n some cases I can observe ...&quot;. Does that man that neither y1 nor y2 can be observed all the time (i.e. for all the x)?" CreationDate="2010-12-10T11:02:14.550" UserId="264" />
  <row Id="7973" PostId="5348" Score="2" Text="The question mentions that each $X_i$ has different success probability. The sum of such $X_i$ will not have binomial distribution." CreationDate="2010-12-10T11:38:30.397" UserId="2116" />
  <row Id="7974" PostId="5348" Text="@mpikta: Sorry I missed that, I'll update my answer." CreationDate="2010-12-10T11:44:21.127" UserId="8" />
  <row Id="7976" PostId="5280" Text="@chl I'd love to see the code. If you don't think it's appropriate to post it here, you could e-mail it to gd047@mineknowledge.com. Thank you!" CreationDate="2010-12-10T11:51:31.313" UserId="339" />
  <row Id="7977" PostId="5345" Score="1" Text="@ 2) more precisely, in `lme4` you can either specify a diagonal covariance structure (i.e., independent random effects) or unstructured covariance matrices (i.e. all correlations have to be estimated) or partially diagonal, partially unstructured covariance matrices for the random effects. I'd also add a third difference in capabilities that may be more relevant for many longitudinal data situations:  `nlme` let's you specify variance-covariance structures for the residuals (i.e. spatial or temporal autocorrelation or heteroskedasticity),  `lme4` doesn't." CreationDate="2010-12-10T11:55:51.977" UserId="1979" />
  <row Id="7978" PostId="5348" Text="I should also note that `p_i`s are sometimes very small (~0.001). The actual distribution often resembles Poisson." CreationDate="2010-12-10T12:00:18.717" UserId="634" />
  <row Id="7979" PostId="5345" Text="@fabians (+1) Ah, thanks! Didn't realize `lme4` allows to choose different VC structures. It would be better that you add it in your own response, together with other ideas you may have. I will upvote. BTW, I also realized that `lmList()` is available in `lme4` too. I seem to remember some discussion about that on R-sig-ME." CreationDate="2010-12-10T12:09:26.517" UserId="930" />
  <row Id="7981" PostId="5345" Text="@chl: what's the etiquette here - should I delete my comment now?" CreationDate="2010-12-10T13:19:16.070" UserId="1979" />
  <row Id="7982" PostId="5295" Text="@Jeromy Anglim, for point #1, I would guess you normally only have to do this once (unless you migrate between two different environments frequently that can not read or output the others files). For point #3, fixed text files fix that problem. I have never come across a situation where SPSS formatted a different file type incorrectly. If you don't need to disseminate the files this whole question is mute, if you can get the file to save correctly in whatever environment you will be working in there is no more need for conversion/storage." CreationDate="2010-12-10T13:40:35.893" UserId="1036" />
  <row Id="7983" PostId="5280" Score="1" Text="@gd047 All right, as usual it's always when we look for old code that we cannot found it (despite my best effort with grep/find), so I rewrote a quick (still ugly) [R script](https://gist.github.com/736212) for that. I've also put an example of the [MP code](https://gist.github.com/736216)." CreationDate="2010-12-10T13:54:21.740" UserId="930" />
  <row Id="7984" PostId="5345" Text="@fabians I was thinking that your comments would in themselves constitute a response, in addition to other ideas that you may have (because it seems you have some good experience with GLMM). Feel free to leave them as is or to develop them in a new post (that I will vote up because I appreciated your remarks). Sorry if I was unclear or if my preceding comment looks offending or whatever." CreationDate="2010-12-10T14:04:45.167" UserId="930" />
  <row Id="7985" PostId="5345" Text="@fabians Ah... I just realized that you answered in the meantime. Forget about that." CreationDate="2010-12-10T14:12:55.117" UserId="930" />
  <row Id="7986" PostId="5327" Text="You should not be using truncated regression just because you have skewed or bounded data. It is specifically for situations when values below a threshold (eg negative values) are possible, but would not be observed for some reason. Is that the situation you have?" CreationDate="2010-12-10T15:28:13.053" UserId="279" />
  <row Id="7987" PostId="5348" Text="@David This is an important point, because it indicates the Central Limit Theorem might not apply." CreationDate="2010-12-10T16:13:48.883" UserId="919" />
  <row Id="7988" PostId="5349" Score="2" Text="This is not true when the p_i approach zero as i increases.  Otherwise, you have just proven that the Poisson distribution is Normal!" CreationDate="2010-12-10T16:14:38.633" UserId="919" />
  <row Id="7989" PostId="5348" Text="@csgillespie Nice idea, but unfortunately this distribution will be *overdispersed*: the variation in the p_i can have an important influence on the distribution, invalidating the use of p-bar as an approximation." CreationDate="2010-12-10T16:15:54.980" UserId="919" />
  <row Id="7990" PostId="5327" Text="@Aniko, negative values of the dependent variable don't really make sense (it would mean getting paid to receive a service), but I'd heard that Wooldridge (in *Econometric Analysis of Cross Section and Panel Data*, 2002) had recommended truncated or censored regression models instead of OLS when $P(Y=0)&gt;0$ yet $Y$ is a continuous random variable over positive values." CreationDate="2010-12-10T16:27:34.857" UserId="1583" />
  <row Id="7991" PostId="5347" Text="Are your p_i fixed throughout the modeling exercise or can they change from one calculation to the next?" CreationDate="2010-12-10T16:30:42.473" UserId="919" />
  <row Id="7992" PostId="5235" Text="@Jeromy Thanks!  I had guessed that was the meaning but wasn't sure.  That's a new term to me--and to Wikipedia, evidently." CreationDate="2010-12-10T16:32:44.810" UserId="919" />
  <row Id="7993" PostId="5252" Score="1" Text="+1 This is exactly the right analysis.  But why don't you finish the job and answer the question?  The OP asks whether this correlation is &quot;high&quot; and what it might *mean*." CreationDate="2010-12-10T16:33:54.123" UserId="919" />
  <row Id="7994" PostId="5289" Text="@Jeromy Yes.  But note that we're talking about the *constraints*, not the observed min and max.  In fact, the maxent solution for a lower constraint, given the mean and SD, is a shifted exponential distribution: exactly the example @Isaac uses!" CreationDate="2010-12-10T16:36:07.523" UserId="919" />
  <row Id="7995" PostId="5343" Text="@James And what do you mean when you call y2 a &quot;prediction&quot;?  Isn't it functioning here as a *predictor*?" CreationDate="2010-12-10T16:39:20.093" UserId="919" />
  <row Id="7996" PostId="5340" Score="2" Text="@Gaetan Please vote up *any* and *all* reasonable answers: let's encourage people to contribute their ideas!  Remember, you have the additional ability to award an additional 15 points to the answer you eventually think is best.  Overall you will gain far more by *awarding* points than by *withholding* points." CreationDate="2010-12-10T16:42:12.727" UserId="919" />
  <row Id="7997" PostId="5340" Text="@whuber, I so agree with you on all counts.  That is why I actually expressed the concept within my question.  I feel that many participants in this forum do not quite get this concept as clearly as you do.  Thanks for your comment." CreationDate="2010-12-10T16:56:29.157" UserId="1329" />
  <row Id="7998" PostId="5355" Text="Would the ridiculously small residual deviance support the idea of perfect separation? When doing a lack-of-fit test, you expect this value to follow a chi-square distribution with 2291291 degrees of freedom, right?" CreationDate="2010-12-10T16:57:23.603" UserId="1973" />
  <row Id="7999" PostId="5355" Text="@Daniel; yes, given the Null device you have perfectly explained the response. Notice too the large standard errors, so that even though you predict the data perfectly, the Wald test is suggesting that the covariate is not significant. I believe the latter phenomenon is known as the Hauck Donner effect after the authors who noted this issue with the binomial GLM." CreationDate="2010-12-10T17:02:51.867" UserId="1390" />
  <row Id="8000" PostId="5355" Text="@Conjugat Prior Both of your thoughts were correct, but your second thought gets at the real problem. I should have seen it when you told me the first time, but seeing it again from Gavin made me look closer. The formula should be `BigDelay ~ AirTime`. Silly mistake." CreationDate="2010-12-10T17:07:30.110" UserId="1973" />
  <row Id="8001" PostId="5354" Score="1" Text="@Daniel; not sure I deserve the &quot;accept&quot;. @Conjugate Prior's answer explained what was wrong with your model. I thought it worth explaining the warning you mentioned in terms of the algorithm." CreationDate="2010-12-10T17:11:51.610" UserId="1390" />
  <row Id="8002" PostId="5342" Score="1" Text="+1 This is correct.  Just for fun, how does Excel's answer differ from R's when you use 5000 as an argument instead of 5?" CreationDate="2010-12-10T17:13:03.917" UserId="919" />
  <row Id="8003" PostId="5340" Text="@Gaetan Check out @chl's voting: *there* is someone who gets it!" CreationDate="2010-12-10T17:14:15.913" UserId="919" />
  <row Id="8004" PostId="5354" Score="1" Text="@Daniel If you have the actual delay times, you are likely to get better information by modeling them, rather than reducing them to a binary variable." CreationDate="2010-12-10T17:18:46.050" UserId="919" />
  <row Id="8005" PostId="5355" Text="@Conjugate Nice catch!" CreationDate="2010-12-10T17:19:42.883" UserId="919" />
  <row Id="8006" PostId="5333" Text="@chl: Yes, it's the same data and basically the same hypothesis." CreationDate="2010-12-10T17:19:53.950" UserId="52" />
  <row Id="8007" PostId="5343" Text="@steffen - that's correct, sometimes I have estimates of x independent of any y's. " CreationDate="2010-12-10T17:24:46.507" UserId="2328" />
  <row Id="8008" PostId="5337" Text="Thanks a lot sirus anyway, I am glad you found it helpfull" CreationDate="2010-12-10T17:55:22.430" UserId="1808" />
  <row Id="8009" PostId="5349" Score="1" Text="That is why it must be $B_n\to\infty$. If $p_i$ approach zero at rate faster than $1/i$, $\lim B_n&lt;\infty$." CreationDate="2010-12-10T17:58:11.103" UserId="2116" />
  <row Id="8010" PostId="5353" Text="I don't think I have the funding to purchase anything from SSI.  Just in case I find $250 under the couch, did you have a particular product in mind?" CreationDate="2010-12-10T17:59:05.820" UserId="196" />
  <row Id="8011" PostId="5353" Text="Thanks for the pointers.  I guess JAGS MCMC it is; thankfully I can be very confident in the priors so hopefully something shakes out.  I'm a little concerned because though I have lots of observations on the items I have relatively few subjects.  I'll start throwing models at it and see how it shapes out." CreationDate="2010-12-10T18:00:43.540" UserId="196" />
  <row Id="8012" PostId="5258" Text="Wow, this looks really useful. I will acquire a copy of this book as soon as I can, since everyone seems to be citing it. In the meantime, just a naive question: can these models deal with an arbitrary number of random effects? I think I need 3 in my model." CreationDate="2010-12-10T18:03:00.793" UserId="52" />
  <row Id="8013" PostId="5349" Text="@mpiktas is right.  The analogy to the Poisson distribution doesn't quite fit, here." CreationDate="2010-12-10T18:12:53.410" UserId="1108" />
  <row Id="8014" PostId="5333" Text="@mpiktas: I have edited the question so that it includes the random effects only model." CreationDate="2010-12-10T18:16:12.287" UserId="52" />
  <row Id="8015" PostId="5348" Score="1" Text="@whuber overdispersion only plays a role if the $p_i$'s are random, which the original question doesn't specify (it reads like they are fixed, but maybe not)?  Yes, I see that you asked the OP to clarify.  My point is that if the $p_i$'s are fixed then @csgillespie's  $\overline{p}$ is right, and the final answer is essentially the same as what @mpiktas said, below. But you are right that the variance will be messed up if the $p_i$'s are random (yet the mean will be unchanged)." CreationDate="2010-12-10T18:16:21.583" UserId="1108" />
  <row Id="8016" PostId="5349" Text="By the way, I didn't actually check that monstrous condition in the second paragraph." CreationDate="2010-12-10T18:22:31.443" UserId="1108" />
  <row Id="8018" PostId="5359" Text="What's the source for that quote?" CreationDate="2010-12-10T18:31:40.920" UserId="5" />
  <row Id="8019" PostId="5359" Text="'Pattern Recognition and Machine Learning' by C.M.Bishop" CreationDate="2010-12-10T18:32:33.470" UserId="1371" />
  <row Id="8020" PostId="5349" Text="@G. Jay Kerns I agree that the analogy to the Poisson is imperfect, but I think it gives good guidance.  Imagine a sequence of p's, p_i = 10^{-j}, where j is the order of magnitude of i (equal to 1 for i &lt;= 10, to 2 for i &lt;= 100, etc.).  When n = 10^k, 90% of the p's equal 10^{-k} and their sum looks Poisson with expectation 0.9.  Another 9% equal 10^{1-k} and *their* sum looks Poisson (with the same expectation).  Thus the distribution looks approximately like a sum of k Poisson variates.  It's obviously nowhere near Normal.  Whence the need for the &quot;monstrous condition.&quot;" CreationDate="2010-12-10T18:53:44.430" UserId="919" />
  <row Id="8021" PostId="5358" Text="+1 Not a bad idea.  It's likely that a small mixture of Poissons would do a good job, depending on how the question is clarified." CreationDate="2010-12-10T18:55:10.937" UserId="919" />
  <row Id="8022" PostId="5348" Score="1" Text="@G. Jay Kerns Thank you for clarifying the issue about overdispersion.  I was sloppy in not laying out my assumptions.  Yes, when the p_i are fixed the $\bar{p}$ is correct, and so is the variance, *but the distribution is unlikely to be Binomial.*" CreationDate="2010-12-10T18:57:22.133" UserId="919" />
  <row Id="8023" PostId="5358" Text="I did think about suggesting a negative binomial distribution, which arises as a Gamma-Poisson mixture, but that has a variance larger than its mean, while this problem has a variance smaller than its mean. Based on that, i'm not sure if any mixture of Poissons will work, as surely any such mixture will have variance larger than its mean??" CreationDate="2010-12-10T19:25:29.770" UserId="449" />
  <row Id="8024" PostId="5342" Text="Both give 1. Though I agree with your answer below, I would not trust Excel with anything." CreationDate="2010-12-10T19:57:33.423" UserId="2116" />
  <row Id="8025" PostId="5342" Text="Hmmm... I get 1 - 2.81997E-14 in Excel.  I'm using an older version, Excel 2002 SP3.  The correct value is 1 - 2.79823E-14.  Regardless, 1 = 1-0 is *way* off.  (Remember, the quantity of interest is usually the *tail* value, not the cumulative value.)" CreationDate="2010-12-10T20:25:32.993" UserId="919" />
  <row Id="8026" PostId="5258" Text="@Alan Be aware that we (@caracal and me) are not citing the same book. CDA is the more complete one, but Laura Thompson's textbook already includes about 10 pages of summary for each chapter. Look at her textbook first. ICDA is available on Google books I think." CreationDate="2010-12-10T20:38:56.483" UserId="930" />
  <row Id="8027" PostId="5342" Text="If I subtract the result with 5000 from 1, in Excel I get 2.79776E-14 and in R: 2.797762e-14. I use R 2.11.1 on Mac OS X 10.6.5" CreationDate="2010-12-10T20:55:12.393" UserId="2116" />
  <row Id="8028" PostId="5348" Text="@whuber I can certainly agree with that!" CreationDate="2010-12-10T21:00:22.053" UserId="1108" />
  <row Id="8029" PostId="5361" Score="1" Text="@whuber +1 for highlighting the importance of model interpretation. I won't add anything about the uninformed ML approach (or ensemble methods) with more complex cross-validation schemes, because I feel you already said what's really matter here: (1) feature selection through resampling is hardly interpretable in isolation (i.e., by comparing one result after the other), and (2) it all depends if we are seeking for a predictive or an explanatory model." CreationDate="2010-12-10T21:03:26.437" UserId="930" />
  <row Id="8030" PostId="5349" Text="@whuber, well said." CreationDate="2010-12-10T21:05:17.183" UserId="1108" />
  <row Id="8031" PostId="5350" Text="With the possible exception (as you pointed out) of being able to incorporate temporal autocorrelation in nlme but not lme4.  If the data set is big enough, and if the data have this sort of structure, that could be a big advantage of nlme." CreationDate="2010-12-10T21:10:50.873" UserId="2126" />
  <row Id="8032" PostId="5362" Text="@Tal (+1) I always think that the takeaway messages when teaching ANOVA is: (1) we have an F-ratio which reflects the relative importance of the variance accounted for by our factor of interest wrt. total variance (or MSB/MSW where MSW=MST-MSB), (2) the differences between the group means is a variance, and (3) we explicitly test $H_0:~\mu_1=\mu_2=\ldots=\mu_k$ *vs.* $H_1:~\exists\ i,j~|~\mu_i\neq\mu_j$ ($H_1\equiv\neg~H_0$). If you're able to convey those ideas into a graphical display--which seems to be the case here--, then I think you're almost done." CreationDate="2010-12-10T21:21:14.077" UserId="930" />
  <row Id="8033" PostId="5343" Text="Do you have observed values of x at least sometimes matched with observed values of y1 and/or y2?" CreationDate="2010-12-10T21:28:04.020" UserId="1583" />
  <row Id="8034" PostId="5361" Text="Thanks for your insight.  I have done some pre-screening to narrow my search space and simply want to find the best model for prediction with the fewest variables.  I am only throwing 7 predictors into the model, which as I understand it, should be ok.  I understand the idea of sticking with a sample, but on the flip side, my model was fundamentally different and shows that the results are entirely sample-dependent, which made me pause." CreationDate="2010-12-10T21:30:30.120" UserId="569" />
  <row Id="8036" PostId="5367" Text="Thanks, Alex. I see a problem with this though: what if over the loops the solutions generated never satisfy the constraint? That could happen if k means were set to run with no cluster size constraint.  I'd love a solution that avoids this. (The nature of the application is such that I really need to ensure clusters are of a minimum size.)" CreationDate="2010-12-10T21:50:26.300" UserId="96" />
  <row Id="8037" PostId="5326" Text="Note that `FORMAT x y 17.12` will show both `x` and `y` as 17 characters long (including the decimal and, when present, the negative sign), and 12 of those characters will be digits after the decimal point. Numbers with more characters or greater decimal precision may not display correctly." CreationDate="2010-12-10T21:57:37.027" UserId="1583" />
  <row Id="8038" PostId="5361" Text="@Btibert3 Right: when the results vary among random subsets of your data, you can take that as evidence that the independent variables are not strong or consistent predictors of the independent variable." CreationDate="2010-12-10T22:03:15.237" UserId="919" />
  <row Id="8039" PostId="5365" Score="2" Text="I'm not familiar with Bishop's notation, but IIRC, the loading $\lambda_{ij}$ in matrix $W$ equals the correlation of observed variable $i$ with latent factor $j$ only in the orthogonal-factor model. In a model with correlated factors, matrix $W \cdot C_f$ contains the variable-factor correlations if $C_f$ is the correlation matrix of the factors (not equal to the identity matrix)." CreationDate="2010-12-10T22:08:41.067" UserId="1909" />
  <row Id="8040" PostId="5342" Text="@mpiktas Interesting; it looks like some improvements were made in more recent versions of Excel.  It's also interesting that R and Mathematica differ.  I suspect R is using a high-precision approximation to the integral (there are many good ones) whereas Mathematica is numerically integrating to whatever precision is requested (I used 6 to 20 sig figs).  That doesn't mean R is bad; it only means we need to pay attention to numerical precision and accuracy when computing extreme probabilities with *any* software." CreationDate="2010-12-10T22:08:42.907" UserId="919" />
  <row Id="8041" PostId="5365" Text="@caracal (+1) Good point. Yes, Bishop considers an orthogonal solution." CreationDate="2010-12-10T22:20:06.547" UserId="930" />
  <row Id="8042" PostId="5322" Text="@Aniko: Thanks! What is a good approach if it's not linear in the logit scale? " CreationDate="2010-12-10T23:52:55.953" UserId="1991" />
  <row Id="8043" PostId="5343" Text="@Firefeather - Yes I do. I'd like to know what's strictly right versus what's pragmatic to do - maybe modeling these as Gaussians isn't right, so adding or multiplying is the theoretical right answer but not practical. " CreationDate="2010-12-11T00:54:41.767" UserId="2328" />
  <row Id="8044" PostId="5358" Text="@onestop Where was it said that the variance is less than the mean?  I missed that statement." CreationDate="2010-12-11T07:02:49.883" UserId="919" />
  <row Id="8045" PostId="5362" Text="Hi chl, thank you for the positive feedback (and for your previous detailed answer)! I think some of the biggest take home massages I got from preparing the material for this class are:  1) How to describe the transformation of the original data so to get te MSB and MSW variances measures.  2) How the test statistic of the MSB/MSW is actually a one sided (not a two sided) test where the H0 is that MSB&lt;=MSW. Lastly, I just thought to note that it is true that SSW=SST-SSB (but I don't see how it's true for MSW=MST-MSB).  " CreationDate="2010-12-11T08:25:11.763" UserId="253" />
  <row Id="8046" PostId="5369" Score="2" Text="Well, you can do FA on the factors determined by an FA method, its called hierarchical factor analysis and is done all the time in psychology. So in that edge case, it is useful. On the original question, i really dont see why you would even want to do that. " CreationDate="2010-12-11T09:33:52.250" UserId="656" />
  <row Id="8047" PostId="5358" Text="Sorry whuber, that was a bit cryptic but these comments don't allow that much elaboration. mpiktas's $B_n = \sum p_i(1-p_i)$ is the variance, which is less than the mean, $\sum p_i$. Only slightly less if the $p_i$'s are on average very small though, so standard Poisson might be a good enough approx. Maybe I should expand my answer above.. but then the conversational thread gets confusing." CreationDate="2010-12-11T10:40:33.597" UserId="449" />
  <row Id="8048" PostId="5374" Text="@onestop (+1) Nice catch!" CreationDate="2010-12-11T12:11:36.380" UserId="930" />
  <row Id="8050" PostId="5362" Text="@Tal Yep, sorry I wrote quickly. I meant: Consider the model $y_{ij}=\mu + \alpha_i + \varepsilon_{ij}$ or $y_{ij}=\mu_i + \varepsilon_{ij}$ and highlight graphically the following decomposition: $y_{ij}=\bar{y_i}+\varepsilon_{ij}=\bar{y}+(\bar{y}_i-\bar{y})+(y_{ij}-\bar{y}_i)$, that is obs. are expressed as deviations from the grand mean + group mean + fluctuations around group mean. Then, we have $(y_{ij}-\bar{y})=(\bar{y}_i-\bar{y})+(y_{ij}-\bar{y}_i)$ or total variation = between-group variation + within-group variation (which is basically your first picture)." CreationDate="2010-12-11T12:30:00.023" UserId="930" />
  <row Id="8051" PostId="5369" Score="2" Text="@richiemorrisroe: higher-order factor analysis is done on the basis of **correlated** lower-order factors. As a sidenote, I wouldn't count the fact that some method is routinely used in some psychological disciplines as evidence for its usefulness." CreationDate="2010-12-11T12:39:12.960" UserId="1909" />
  <row Id="8052" PostId="5363" Text="NN is rather an obsolete technology, so I'm afraid you won't get an answer because no one here is using them..." CreationDate="2010-12-11T12:55:40.160" UserId="88" />
  <row Id="8053" PostId="5347" Text="The `p_i`s are fixed." CreationDate="2010-12-11T13:18:02.770" UserId="634" />
  <row Id="8054" PostId="5348" Text="The `p_i`s are indeed fixed. Sometimes the `p_i` values are very small, and sometimes the number of RVs is smaller than 15-40k. I noticed that many times the histogram of of Y values across simulations is much more similar to a Poissonian rather than to a Gaussian." CreationDate="2010-12-11T13:22:26.030" UserId="634" />
  <row Id="8055" PostId="5358" Text="What do you mean by $\sum X_i$? How do I get $X_i$ values?" CreationDate="2010-12-11T13:24:43.190" UserId="634" />
  <row Id="8056" PostId="5358" Text="Good point! That should be $\sum p_i$. I'll change it. " CreationDate="2010-12-11T13:56:41.137" UserId="449" />
  <row Id="8057" PostId="5375" Text="@EDi (+1) I remind me of `plot.design()` (but yours in an enhanced version :-)" CreationDate="2010-12-11T14:41:44.953" UserId="930" />
  <row Id="8058" PostId="5352" Text="As for a general tutorial, I always like to refer to [A visual guide to item response theory](http://www.metheval.uni-jena.de/irt/VisualIRT.pdf) (it does not cover the MCMC approach)." CreationDate="2010-12-11T14:43:56.373" UserId="930" />
  <row Id="8059" PostId="5353" Text="@Conjugate (+1) A recent tutorial with BUGS/R can be found here: [The Use of R and WinBUGS in Fitting Item Response Theory Models](http://idecide.mskcc.org/stats/multi/irt_winbugs.php)." CreationDate="2010-12-11T14:45:06.573" UserId="930" />
  <row Id="8060" PostId="5361" Text="+1, but I had to add my few remarks." CreationDate="2010-12-11T18:07:30.683" UserId="88" />
  <row Id="8061" PostId="5378" Score="1" Text="@gd047 (+1) I gave a link to Robert Pruzek's article, but I didn't know it was available in R." CreationDate="2010-12-11T18:25:00.263" UserId="930" />
  <row Id="8062" PostId="5374" Text="Ah, I get it. Thank you!" CreationDate="2010-12-11T18:47:20.367" UserId="52" />
  <row Id="8063" PostId="5380" Text="+1 for the pointer to regularized logistic regression.  It's unclear how one could formally &quot;look at patterns&quot; when resampling the &quot;data many times,&quot; though.  That sounds a lot like data snooping and therefore seems likely to lead to frustration and error." CreationDate="2010-12-11T19:07:17.033" UserId="919" />
  <row Id="8064" PostId="5379" Text="+1 I worry that lots of resampling will only be confusing and misleading.  Resampling in a controlled way, via cross validation or a hold-out sample for verification, obviously is not problematic." CreationDate="2010-12-11T19:08:34.550" UserId="919" />
  <row Id="8065" PostId="5374" Text="You're welcome! BTW, once you're happy that each of your questions has been resolved to your satisfaction you can indicate this by 'accepting' the best answer. This helps to indicate that the question has been resolved in the listing on the front page and elsewhere." CreationDate="2010-12-11T19:53:30.987" UserId="449" />
  <row Id="8066" PostId="5322" Text="You can try transformations (e.g. using user2040's suggestions), or splines." CreationDate="2010-12-11T20:02:28.763" UserId="279" />
  <row Id="8067" PostId="5374" Text="Ah, I see! Thanks." CreationDate="2010-12-11T20:22:04.270" UserId="52" />
  <row Id="8068" PostId="5382" Score="1" Text="What exactly do you mean by &quot;effective rating&quot;?  If it reflects expertise, why would this have anything to do with an amount of work allotted, which has little or nothing to do with expertise?  What would it mean to rate a user &quot;relative to the group&quot;?" CreationDate="2010-12-11T20:32:38.830" UserId="919" />
  <row Id="8069" PostId="5380" Score="3" Text="Feature selection when the selection is unstable will always be a recipe for frustration and error. Using only one sample cuts down on the frustration, but increases the likelihood of error as it encourages you to draw inferences about the relevant features for the problem based on what works best on the particular sample you look at - which is a form of over-fitting.  The re-sampling gives you an idea of the uncertainty in the feature selection - which is often just as important.  In this case we shouldn't draw any strong conclusions about the relevant features as there isn't enough data. " CreationDate="2010-12-11T20:42:30.493" UserId="887" />
  <row Id="8071" PostId="5382" Text="I was with you up to, but not including, the sentence starting &quot;Now, we want to...&quot;. Could you try explaining what you want to achieve again? " CreationDate="2010-12-11T20:54:56.330" UserId="449" />
  <row Id="8072" PostId="5377" Text="Why do you say &quot;the code isn't really appropriate if your $p$s are too small&quot;? Seems to work ok to me, e.g. with shape1=1, shape2=999, giving a mean $p$ of 0.001." CreationDate="2010-12-11T22:07:32.453" UserId="449" />
  <row Id="8073" PostId="5378" Text="Thanks gd047, I remembered seeing this package, but didn't remember it's name :)" CreationDate="2010-12-11T22:52:39.670" UserId="253" />
  <row Id="8074" PostId="5387" Text="Isn't it software dependent?" CreationDate="2010-12-11T23:14:55.230" UserId="88" />
  <row Id="8075" PostId="5387" Score="1" Text="Different software may allow different options, but there should be a principled answered, no?" CreationDate="2010-12-11T23:16:07.263" UserId="82" />
  <row Id="8076" PostId="5380" Text="@Dikran Good point; I hate when people only count mean from resampling, it's such a waste." CreationDate="2010-12-11T23:38:55.780" UserId="88" />
  <row Id="8077" PostId="5388" Text="Are you saing it's fine to treat ordinal data as interval data in the model? If so, how can I intepret the coefficient if I don't know the &quot;distance&quot; between the levels?" CreationDate="2010-12-11T23:49:02.697" UserId="82" />
  <row Id="8078" PostId="5358" Text="I wish I could vote up this answer again in light of the edit: that's great information.  I'll post a response to elaborate on why." CreationDate="2010-12-11T23:52:20.790" UserId="919" />
  <row Id="8079" PostId="5347" Text="In light of the current responses, could you share estimates of (a) the sum of the p's and (b) the sum of their squares?  These values determine your options." CreationDate="2010-12-11T23:53:49.183" UserId="919" />
  <row Id="8080" PostId="5387" Text="Ok, stupid comment, sorry." CreationDate="2010-12-12T01:53:10.770" UserId="88" />
  <row Id="8081" PostId="5377" Text="@onestop what I meant was the specific choice of (1,10) written above doesn't give values of $p$ that are very small, to the point that the normal approximation looks pretty good.  If a person wanted the Poisson to come out then they would need to try something else;  it sounds like your choice of (1,999) does a good job, yes?  I had also thought to make $\alpha &lt; 1$, say, 0.25, but I haven't tried that." CreationDate="2010-12-12T02:55:02.737" UserId="1108" />
  <row Id="8082" PostId="5393" Text="To some purists this may not be Bayesian. This is actually empirical Bayesian, but it is a quick way to simulate your probabilities in R, without resorting to hyper prior mumbo jumbo." CreationDate="2010-12-12T03:41:05.133" UserId="1307" />
  <row Id="8083" PostId="5382" Text="Ok, I'll try to explain. Users think they are expert to some level in every component. They can rate themselves pretty good numbers. But not everyone has actually the same level of knowledge. This is what we have to find out with the help of amount of actual work done data. The ranking of a user should be higher if he has done some work in the component than the one who has not - even though the latter might have rated himself higher. This is a comparison of expertise of users based on their perception of possession of skill and actual work done. Is this clear now? Please write back if not." CreationDate="2010-12-12T04:01:13.650" UserId="2344" />
  <row Id="8084" PostId="5358" Text="...and @onestop for the win.  That's really, really cool." CreationDate="2010-12-12T04:19:54.180" UserId="1108" />
  <row Id="8085" PostId="5389" Text="good stuff, @whuber." CreationDate="2010-12-12T04:21:13.943" UserId="1108" />
  <row Id="8086" PostId="5388" Score="2" Text="Yes. The coefficient reflects the change in log odds for each increment of change in the ordinal predictor. This (very common) model specification assumes the the predictor has a linear impact across its increments. To test the assumption, you can compare a model in which you use the ordinal variable as a single predictor to one in which you discretize the responses and treat them as multiple predictors (as you would if the variable were nominal); if the latter model doesn't result in a significantly better fit, then treating each increment as having a linear effect is reasonable." CreationDate="2010-12-12T05:21:16.283" UserId="2347" />
  <row Id="8087" PostId="5358" Text="+1 Great! Is there an elaboration somewhere on what `pi`s are considered reasonably small enough?" CreationDate="2010-12-12T08:33:30.483" UserId="634" />
  <row Id="8088" PostId="5392" Text="Don't you mean $Var(w_1 A + w_2 B)$, since you weigh them?" CreationDate="2010-12-12T09:20:46.730" UserId="2036" />
  <row Id="8089" PostId="5392" Text="@Raskolnikov: Thank you for pointing that out. I have edited it." CreationDate="2010-12-12T10:41:41.237" UserId="1636" />
  <row Id="8091" PostId="5363" Text="@mbq: I do not doubt your words, but how do you come to the conclusion that NN are &quot;obsolete technology&quot;?" CreationDate="2010-12-12T11:15:47.833" UserId="264" />
  <row Id="8092" PostId="5388" Text="@dmk38 I give you +1 because this response sounds fine, and I guess @fgregg would have do the same (with enough rep)." CreationDate="2010-12-12T11:30:14.553" UserId="930" />
  <row Id="8093" PostId="5397" Text="@steffen (+1) Thanks for the link." CreationDate="2010-12-12T11:42:13.230" UserId="930" />
  <row Id="8094" PostId="5278" Text="@Tal Given the way responses and comments are going, I added the 'data-viz' tag; hope you don't mind (feel free to remove if it's not appropriate here)." CreationDate="2010-12-12T11:45:23.507" UserId="930" />
  <row Id="8095" PostId="5399" Score="1" Text="@asd123 1) when you write $\Sigma$ it suggests that $x_1$ and $x_2$ are vectors, in which case the product $x_{1}x_{2}$ isn't defined as written (unless $n=1$).  Do you mean $x_{1}^{\mathrm{T}}x_{2}$? If not, what do you mean? 2) Without independence it is not necessarily true that $(x_{1},x_{2})$ is jointly normal, so it would seem that you would need more information about their joint distribution (and/or variance/covariance matrix) before you could say anything definitive.     " CreationDate="2010-12-12T12:31:49.003" UserId="1108" />
  <row Id="8096" PostId="5399" Text="Yes, I did mean that $x_1$ and $x_2$ are vectors. I also know that they are jointly Gaussian. Does that help?" CreationDate="2010-12-12T12:50:12.677" UserId="2353" />
  <row Id="8097" PostId="5400" Text="If the MATLAB looks wonky, I apologize; I haven't touched MATLAB in years... XD" CreationDate="2010-12-12T13:08:48.600" UserId="830" />
  <row Id="8098" PostId="5357" Text="I read your answer in full, but I'd vote it up even if all you wrote was the first paragraph: &quot;I mistrust all but the lowest-level functions in Excel...&quot; :D" CreationDate="2010-12-12T13:12:01.967" UserId="830" />
  <row Id="8099" PostId="5342" Text="@whuber, @mpiktas: Is there anything in the R documentation that discusses what's under the hood? My hazy memory of *Mathematica* 's implementation when I managed to take a peek (don't ask how) is that the error function routine uses an optimized minimax rational approximant for machine precision, and series (power or asymptotic) and continued fractions for arbitrary precision (definitely not numerical quadrature! ;) ), at least in the real case. That may be one reason for the accuracy difference." CreationDate="2010-12-12T13:17:51.120" UserId="830" />
  <row Id="8100" PostId="5342" Text="Also, there is also the possibility that the environment did not provide for both $\mathrm{erf}$ and $\mathrm{erfc}$ ; trying to compute one in terms of the other at extreme probabilities is bound to end disastrously." CreationDate="2010-12-12T13:20:00.197" UserId="830" />
  <row Id="8101" PostId="5337" Text="An other question, do the data need to be positive or normalized, because the representation of no-path can be many values, like &quot;inf&quot; or a negative value." CreationDate="2010-12-12T13:55:00.277" UserId="2325" />
  <row Id="8102" PostId="4994" Text="nothing at all? only a simple elucidation could help a bit.." CreationDate="2010-12-12T14:12:19.827" UserId="2046" />
  <row Id="8103" PostId="5399" Score="1" Text="@asd123 partly, yes, because then $x_1$ and $x_2$ will be independent if and only if they are uncorrelated (look at the variance/covariance matrix of $x^{\mathrm{T}}=(x_{1}^{\mathrm{T}},x_{2}^{\mathrm{T}})$.  If the off-diagonal block matrices are zero then they are uncorrelated).  If they are independent then you can just write out the dot product above, take expected values and you will be all set.  If they aren't independent then do you know anything about the off-diagonal block entries?" CreationDate="2010-12-12T14:42:49.500" UserId="1108" />
  <row Id="8104" PostId="5399" Text="By the way, if the above is truly what you mean then I recommend you change the title to &quot;Expectation of dot-product of Gaussian random vectors&quot;." CreationDate="2010-12-12T14:50:01.650" UserId="1108" />
  <row Id="8105" PostId="5399" Text="Sorry, I intended to transpose the other variable. So the result is a matrix. I.e. (Mx1) x (1xM) = (MxM)" CreationDate="2010-12-12T15:03:46.263" UserId="2353" />
  <row Id="8106" PostId="5399" Text="Also, I don't have the covariance matrix of $x_1$ and $x_2$, though I do know that one of them is has diagonal covariance." CreationDate="2010-12-12T15:22:58.703" UserId="2353" />
  <row Id="8107" PostId="5399" Text="It's a special case of random Gram(ian) matrices." CreationDate="2010-12-12T15:55:55.300" UserId="2036" />
  <row Id="8108" PostId="5370" Text="Thanks, Marianna.  I would prefer a solution that relies less heavily on (typically, unjustifiable) parametric models, but will definitely look into it." CreationDate="2010-12-12T15:59:17.907" UserId="96" />
  <row Id="8109" PostId="5371" Text="Thanks, Manuel.  This actually sounds like a very intriguing possibility.  I need to think about whether the hierarchical partitioning would impose certain constraints that would prevent the algorithm from achieving the optimal cluster partitioning directly under the size constraint.   But intuitively, I can see that this might work." CreationDate="2010-12-12T16:01:20.307" UserId="96" />
  <row Id="8110" PostId="5398" Text="Awesome answer. Thanks!" CreationDate="2010-12-12T17:18:57.753" UserId="82" />
  <row Id="8111" PostId="5358" Text="@David B: I've added a bit more to my answer (EDIT 2) partly in response to that. " CreationDate="2010-12-12T17:21:26.937" UserId="449" />
  <row Id="8112" PostId="4362" Text="@Joris Meys Your calculations are off, *way* off.  The chance of getting *exactly* 10,000 heads is 0.564%.  The chance of getting *exactly* 10,093 heads is 0.238%.  (These assume *any* reasonably diffuse and uninformative prior.)  Both are astronomically larger than 2^(-20000) and neither is so close to zero as to be entirely surprising *even if it were predicted in advance* (concerning which see a comment by @G. Jay Kerns following his answer)." CreationDate="2010-12-12T18:12:44.570" UserId="919" />
  <row Id="8113" PostId="4924" Text="+1, excellent points." CreationDate="2010-12-12T18:15:29.387" UserId="919" />
  <row Id="8114" PostId="4916" Text="+1, But I'm not convinced.  What's special about 10,000 is that it exactly equals the expected number of heads under the hypothesis that the coin is fair.  This fact is independent of any psychology or system of number representation.  The analysis in this response might provide some insight into a situation where, say, 20,005 coins were flipped and 10,000 heads (and therefore 10,005 tails) were noted and somebody's &quot;intuition&quot; suggested fakery took place." CreationDate="2010-12-12T18:18:42.723" UserId="919" />
  <row Id="8115" PostId="5402" Text="Where did you hear this statement?" CreationDate="2010-12-12T18:20:24.360" UserId="919" />
  <row Id="8116" PostId="5114" Score="1" Text="In your question you have $P(X = 2) = 0$ and $P(X = 8) = 0$.  Do you mean: $P(X &lt; 2) = 0$ and $P(X &gt; 8) = 0$??" CreationDate="2010-12-12T18:32:26.010" UserId="1499" />
  <row Id="8117" PostId="5278" Text="Sure chl - thanks." CreationDate="2010-12-12T18:34:34.120" UserId="253" />
  <row Id="8118" PostId="5395" Text="I thought of using Bayesian methods, but I'm very new to such techniques and still trying to figure out factors and variables. But your suggestions conforms that Bayesian methods may hold the answer. I'm trying :-/" CreationDate="2010-12-12T18:56:54.313" UserId="2344" />
  <row Id="8119" PostId="5401" Text="About self-rating, they can not be completely ignored. A person might not get much work, but has great knowledge of component will lose the ranks significantly if self-rating is not considered." CreationDate="2010-12-12T19:01:42.967" UserId="2344" />
  <row Id="8121" PostId="5398" Score="2" Text="(+1) very thorough answer. I'd only add that it is possible to obtain tests that assume only monotonicity of the dose-response relationship without assigning scores to the ordinal variable using isotonic regression., although simulation or permutation methods are usually needed to obtain the distribution of the test statistic under the null. See Salanti &amp; Ulm 2003 http://dx.doi.org/10.1002/bimj.200390012 . " CreationDate="2010-12-12T19:16:46.370" UserId="449" />
  <row Id="8122" PostId="5401" Text="@Nayan: I'm afraid you're in a corner then. You can't exclude the bad variance from the self-rating and keep just the good variance.  In absence of actual performance information you have no source of data other than their self-rating and you say you don't trust the self-rating. The method I outline above doesn't penalize those who don't get much work, but it does somewhat penalize those who get very low amounts of work or no work at all.  One thing you might do is just acknowledge you don't have sufficient evidence to judge some of the people." CreationDate="2010-12-12T20:33:22.617" UserId="196" />
  <row Id="8123" PostId="5398" Text="@onestop Thanks for this reference. I'll look at it." CreationDate="2010-12-12T21:06:34.210" UserId="930" />
  <row Id="8124" PostId="5020" Text="Would you break these out into separate options?" CreationDate="2010-12-12T21:27:42.730" UserId="196" />
  <row Id="8125" PostId="4362" Score="1" Text="@Srikant @whuber: the combinatorials... you're right. I started from an uniform probability, which is off course nonsense in this case. My bad" CreationDate="2010-12-12T21:34:30.403" UserId="1124" />
  <row Id="8126" PostId="5410" Text="+1 Good finds.  However, most of the differential geometry applications to statistics do not use symplectic geometry.  I think our difficulty in finding references is testimony to how obscure and rare are the applications of symplectic geometry to statistics." CreationDate="2010-12-12T21:45:08.240" UserId="919" />
  <row Id="8127" PostId="5349" Text="@mpiktas The convergence to normality can be slow.  For example, with p_i = 1/(i+1) (a sequence for which B_n diverges) and n = 2^16 = about 33000, the skewness is still 0.26 and a normal approximation assigns 0.1% probability to negative totals, an impossibility.  Clearly we're starting to approach normality but we're not there yet." CreationDate="2010-12-12T22:04:14.043" UserId="919" />
  <row Id="8128" PostId="5114" Text="@M. Tibbits: yes, sorry for the error." CreationDate="2010-12-12T22:39:46.030" UserId="1381" />
  <row Id="8129" PostId="5393" Text="@suncoolsu Why do you need priors when the p_i are given?" CreationDate="2010-12-12T22:55:38.990" UserId="919" />
  <row Id="8132" PostId="4592" Text="+1  I made sure to read this book shortly after it came out.  I get plenty of opportunities to make statistical mistakes so I'm always grateful to have them pointed out *before* I make them!" CreationDate="2010-12-12T23:02:01.107" UserId="919" />
  <row Id="8133" PostId="4574" Text="@Andre I think this remark may be closely related to another one offered by @Michael Lew elsewhere in this thread (http://stats.stackexchange.com/questions/4551/what-are-common-statistical-sins/4567#4567 )." CreationDate="2010-12-12T23:04:48.967" UserId="919" />
  <row Id="8134" PostId="5416" Score="3" Text="@Andriyev This vague formulation will require potential respondents to guess at your meaning.  Please clarify what you mean by RMS &quot;normalization&quot; and, if possible, provide a reference or link.  Also, because this is potentially a broad subject, it would help if you had a particular application in mind that you could share with us." CreationDate="2010-12-12T23:06:34.190" UserId="919" />
  <row Id="8138" PostId="4924" Text="I think this is a very good answer, but I disagree with your second paragraph. I don't think Srikant's original conditional probability is counter-intuitive, and simply because it is a hard question to answer is not an argument against it. I also don't think your uniform probability of lying within 9900 to 10100 makes any sense at all, although it is useful for demonstration purposes." CreationDate="2010-12-12T23:47:37.370" UserId="1036" />
  <row Id="8140" PostId="5393" Text="@whuber. Thanks, you are right. I missed the fixed part. I thought David is just using the value to be $p_i$ as (q-x)/c and is not fixed. I will edit my answer." CreationDate="2010-12-12T23:58:43.960" UserId="1307" />
  <row Id="8141" PostId="5398" Text="@chl. Great answer." CreationDate="2010-12-13T00:17:27.033" UserId="1307" />
  <row Id="8142" PostId="5351" Score="1" Text="Question is not clear. What do you mean by &quot;standard error for a vector&quot;? Typically we want the standard error of a parameter estimate. What parameter are you estimating?  The code you provided produces the Newey West estimate of the squared standard error of the mean. Is that what you want?" CreationDate="2010-12-13T00:23:41.270" UserId="96" />
  <row Id="8143" PostId="152" Text="I was considering asking &quot;How can I do an MCMC model of the output on lmer for models with random slopes?&quot; but I'm wondering whether that question is redundant with this one.  That is, is the &quot;label switching issue&quot; when using MCMC to estimate mixture models the same sort of problem take makes it so that pvals.fnc() in languageR is able to MCMC intercept models but not models with slopes?  If not, please let me know and I'll go back to asking my initial question." CreationDate="2010-12-13T01:14:44.917" UserId="196" />
  <row Id="8144" PostId="5419" Text="@Ramnath. Thanks. Why do you recommend it?" CreationDate="2010-12-13T01:48:45.673" UserId="183" />
  <row Id="8145" PostId="5421" Text="@Dirk. Great points. I was planning to explore myself. And of course, exploration is half the fun. And I agree different packages  teach different things. I agree that many packages would be good starting points. I'll update my question to reflect the idea that multiple packages would satisfy the criteria. But having some concrete suggestions might save myself and others new to R packages some time. I also think that established developers of R packages are in a better position to comment about the merits of a given package for the purposes of learning." CreationDate="2010-12-13T03:14:27.330" UserId="183" />
  <row Id="8146" PostId="5401" Text="@drknexus: Hmm.. you are right. Still we can experiment with the different models until we reach the one which suits best. Frankly, since I'm not into statistics and maths, I find it bit hard to formulate your idea. Sorry, but can you help by showing the formula or pointing to some system that I can study? Thx!" CreationDate="2010-12-13T03:18:00.247" UserId="2344" />
  <row Id="8147" PostId="5401" Text="@Nayan: I need to determine where we can start in terms of common tools/knowledge.  Do you use SPSS or R?  Do you know how to do regression?  Do you know how to calculate a Z-score?" CreationDate="2010-12-13T05:22:12.677" UserId="196" />
  <row Id="8148" PostId="5363" Text="@steffen By observation; I mean it is obvious that no one significant from NN community will come out and say &quot;Hey guys, let's drop our life work and play with something better!&quot;, but we have tools that achieve same or better accuracy without all this ambivalence and never-ending training. And people do drop NN in favor of them." CreationDate="2010-12-13T09:18:04.653" UserId="88" />
  <row Id="8149" PostId="5351" Text="@Cyrus -- By &quot;vector&quot; I mean not an `lm` object. I frequently have a vector (let's say a series of stock returns) that I don't want to involve in any regressions (because I don't care about it's projection, other than on 1), but for which I still want the HAC standard error. In this case the parameter estimate is the stock return. The answer above does that, but requires calculating the `lm` object, which I really don't need. So I'm wondering if there's a routine in R that does this without creating an `lm` object." CreationDate="2010-12-13T11:07:52.410" UserId="1445" />
  <row Id="8150" PostId="5428" Text="@mpiktas -- Thanks! There may not be a more efficient way for me to code this than forming the `lm` object." CreationDate="2010-12-13T11:11:11.973" UserId="1445" />
  <row Id="8152" PostId="5383" Text="I planned to estimate lambda for each store/SKU separately.   (1) What did you have in mind regarding mixed models, and what are mixed models :)? I must mention that I'm dealing with very different stores (from 50 sq m to 1500 sq m) with very different turnovers. this is reason why I looked each store separately. (2) I planned to adress this problem with modelling sales per hour. You asked me about my statistical background Well, I have some basic knowledge about mathematical statistics (common distributions, statistical tests, etc..)." CreationDate="2010-12-13T11:30:42.877" UserId="2341" />
  <row Id="8153" PostId="5351" Text="Sorry, still not clear: &quot;In this case the parameter estimate is the stock return.&quot; By that, do you mean the &quot;average of the stock returns in the series&quot;? If yes, then what you've got is perfectly fine." CreationDate="2010-12-13T12:59:30.083" UserId="96" />
  <row Id="8154" PostId="5430" Text="@Theodor This doesn't look like a gaussian at all. Not only do you need to make abstraction of values &lt; 4m, but also of the truncation at zero, the long right tail, and the bimodal shape (at approx. 3 and 7m)." CreationDate="2010-12-13T13:38:39.017" UserId="930" />
  <row Id="8155" PostId="5430" Text="Is the peak at around 7m the *only* object? What about the peak around 4m? Is that an artefact you wish to filter out, or can you just ignore it?" CreationDate="2010-12-13T13:46:04.577" UserId="449" />
  <row Id="8156" PostId="5429" Score="1" Text="@radek (+1) Nice follow-up." CreationDate="2010-12-13T13:56:25.920" UserId="930" />
  <row Id="8157" PostId="5430" Text="@onestop - Yes, the peak aroun 7m is the only object. The peak around 4 m is noise. Yes it's noisy." CreationDate="2010-12-13T14:14:40.920" UserId="1411" />
  <row Id="8158" PostId="5430" Text="@chl - I'll insert an image with the log-normal fit." CreationDate="2010-12-13T14:15:20.623" UserId="1411" />
  <row Id="8159" PostId="5430" Text="Are there known harmonics for your equipment?  That is, is there a reason to expect that when a signal occurs at 7m that you'd get a sort of echo at 4m? " CreationDate="2010-12-13T14:29:51.310" UserId="196" />
  <row Id="8160" PostId="5430" Text="@drknexus - no, the noise curve looks the same without the object present." CreationDate="2010-12-13T14:53:45.647" UserId="1411" />
  <row Id="8161" PostId="5351" Text="@Cyrus -- I know that what I have works, but I was hoping that there is a way to calculate the SEs without passing through the `lm` object for the case of a single vector. I guess not. Thanks for helping me clarify my question!" CreationDate="2010-12-13T15:54:28.437" UserId="1445" />
  <row Id="8162" PostId="4916" Text="I fully agree that - as you point out in your answer - it all depends on the a-priori definition of the hypothesis: if you define in advance that by &quot;faking the experiment&quot; you mean &quot;achieving a result for number of heads that is close to the expected value&quot;, then that's a basis for a statistical test with &quot;number of heads&quot; as a test statistic. However, without such a-priori clarification, the meaning of &quot;faking&quot; and &quot;special value for number of heads&quot; remain cloudy, and it's not clear what they have to do with each other." CreationDate="2010-12-13T16:06:17.147" UserId="1909" />
  <row Id="8163" PostId="5383" Text="If you've got enough data, then estimating lambda for each store/SKU separately should be OK. Mixed models (multilevel, hierarchical) are models that assume that the lambdas for SKUs are themselves random variables, drawn from a distribution. See e.g. Gelman and Hill (2009? I forget the title etc -- something about multilevel regression modeling)" CreationDate="2010-12-13T16:12:24.740" UserId="2126" />
  <row Id="8164" PostId="5430" Score="4" Text="@Theodor; couldn't you just subtract the curve when the object is not present from the one when it is present and start from there?" CreationDate="2010-12-13T17:16:21.237" UserId="1390" />
  <row Id="8165" PostId="5436" Text="I've got plenty of data (almost 300k data points), but I am not convinced there are significant interactions between each predictor. I wanted to explore this possibility, but I guess I bit off more than I could chew." CreationDate="2010-12-13T17:39:20.587" UserId="1973" />
  <row Id="8166" PostId="5436" Text="@Daniel by all means then give it a go, but you will need to supply some starting points for the algorithm" CreationDate="2010-12-13T17:47:19.697" UserId="1390" />
  <row Id="8167" PostId="5437" Score="2" Text="slightly simpler would be: `Response ~ .^2 -1, data = DF`, but excellent point!" CreationDate="2010-12-13T18:00:01.120" UserId="1390" />
  <row Id="8168" PostId="5123" Text="was he a statistician? I mean, did he wrote things about estimation? " CreationDate="2010-12-13T18:26:37.463" UserId="223" />
  <row Id="8169" PostId="5431" Text="You should give some details about the test you want to do. This would give more adapted and focused answers. " CreationDate="2010-12-13T18:48:44.887" UserId="223" />
  <row Id="8170" PostId="5437" Text="Thanks for this suggestion!" CreationDate="2010-12-13T20:45:06.400" UserId="1973" />
  <row Id="8171" PostId="5437" Text="Ok, ran the code based on this suggestion and now I have the error &quot;Warning while fitting theta: alternation limit reached&quot;. What is the alternation limit and how do I address this?" CreationDate="2010-12-13T20:55:32.380" UserId="1973" />
  <row Id="8172" PostId="5437" Text="As described on the `?glm.nb` help page: &quot;An alternating iteration process is used. For given theta the GLM is fitted using the same process as used by glm(). For fixed means the theta parameter is estimated using score and information iterations. The two are alternated until convergence of both.&quot; Your model does not converge (you reached the maximum of the theta iterations). You could try setting the `trace` parameter to see what's going on. Or you could use a different package - I believe `VGAM` might use a different algorithm. " CreationDate="2010-12-13T21:04:07.443" UserId="279" />
  <row Id="8173" PostId="5445" Text="@whuber thanks for your answer - this is exactly what I was looking for. I have fixed the error in my code (rescaled by $\alpha/\beta$. I could rescale the distribution except that to do so would require adding a special case to code that is generalized to use any distribution that is supported by JAGS." CreationDate="2010-12-13T21:10:05.210" UserId="1381" />
  <row Id="8174" PostId="5430" Text="@Gavin - Well, the curve is a histogram, accumulated data from &gt;50k samples. Unfortunately the sample rate is way too low to do the subtraction you mention. I think a probabilistic approach might be the only way to go." CreationDate="2010-12-13T21:30:50.580" UserId="1411" />
  <row Id="8175" PostId="5192" Score="1" Text="While looking for other names in Wikipedia, I stumbled on the fact that Box was a son-in-law of Fisher's." CreationDate="2010-12-13T22:05:22.627" UserId="1764" />
  <row Id="8176" PostId="4589" Text="Not to mention not labeling them at all http://xkcd.com/833/" CreationDate="2010-12-13T22:18:00.607" UserId="22" />
  <row Id="8177" PostId="5446" Score="1" Text="This story is identical to one told about John Milnor.  In Milnor's case there's at least one paper (co-authored with his professor when Milnor was still an undergraduate) to give the story credence.  Have you ever found a reference to the paper(s) Dantzig wrote giving his solutions?" CreationDate="2010-12-13T23:17:07.367" UserId="919" />
  <row Id="8178" PostId="5445" Text="@whuber is it appropriate to assume that X*a/b also has an F distribution?" CreationDate="2010-12-13T23:49:47.520" UserId="1381" />
  <row Id="8179" PostId="5450" Score="2" Text="@Jen you could probably get a better answer if you provided more details about your experimental design, research question, and statistical model." CreationDate="2010-12-13T23:52:15.460" UserId="1381" />
  <row Id="8180" PostId="5450" Text="I have survey data, v1 and v2 predict the outcome, as I expected; however, the interaction between v1 (dichotomous) and v2 (5 groups) is not significant -- and (my question) it makes my v1 and v2 direct effects non-significant too. I can't find an example on reporting this in the literature. " CreationDate="2010-12-14T00:03:33.020" UserId="2367" />
  <row Id="8181" PostId="5450" Text="If the v1:v2 interaction is not significant, do you need to have it included in the model?" CreationDate="2010-12-14T01:40:08.247" UserId="1118" />
  <row Id="8183" PostId="5454" Text="Thanks. The devtools wiki, in particular, has lots of great ideas." CreationDate="2010-12-14T02:29:11.977" UserId="183" />
  <row Id="8184" PostId="5454" Text="@Jeromy - Having read your recent posts on reproducible research etc (uber uber helpful so thank you!) means that you'll take to roxygen very quickly. I should have also mentioned that Eclipse supports roxygen syntax and made it a pretty easy transition. The most difficult task I've been overcoming is writing vignettes that are useful and productive. I imagine you're starting from a much higher jumping off point than I am in that regard. " CreationDate="2010-12-14T02:30:30.527" UserId="696" />
  <row Id="8185" PostId="5401" Text="I'm so afraid that I do not know much. But I'm willing to learn. Any guidance will be highly appreciated. I've just started studying R." CreationDate="2010-12-14T02:52:09.460" UserId="2344" />
  <row Id="8186" PostId="5430" Score="1" Text="@Theodor Does the noise come from the environement, or the radar? Is there a rational (theory based) to have such distribution of noise? I think that you should first dig in that direction before removing data. " CreationDate="2010-12-14T02:52:22.410" UserId="1709" />
  <row Id="8188" PostId="5450" Text="Maybe this question is relevant?  http://stats.stackexchange.com/questions/5184/removing-factors-from-a-3-way-anova-table" CreationDate="2010-12-14T03:00:55.870" UserId="2310" />
  <row Id="8189" PostId="5457" Text="I threw on the careers tag. I think that fits somewhat. Also thinking this should be CW." CreationDate="2010-12-14T03:48:47.607" UserId="1118" />
  <row Id="8190" PostId="5446" Text="According to Wikipedia, the first paper was with/by Neyman and the second result was enough to get Dantzig a co-authorship with Abraham Wald. No actual references given, though." CreationDate="2010-12-14T04:09:51.950" UserId="1764" />
  <row Id="8191" PostId="152" Text="@drknexus I do not know R to comment in your question. Perhaps, you should just post your question with a comment that your qn may be lined to this one." CreationDate="2010-12-14T04:10:28.170" />
  <row Id="8192" PostId="5455" Text="Thanks Ben! I'll let you know what works out.  " CreationDate="2010-12-14T04:24:19.547" UserId="2367" />
  <row Id="8195" PostId="5343" Text="@James, why wouldn't a regression model work, then?" CreationDate="2010-12-14T04:38:03.280" UserId="1583" />
  <row Id="8196" PostId="5453" Text="How do you get that $Dr_t=\frac{1}{2}+V$? It is clear that $Er_t=\sin t$, since $\varepsilon\sim N(0,V)$, then $Dr_t=E(r_t-Er_t)^2=E(r_t-\sin t)=E\varepsilon^2=D\varepsilon=V$." CreationDate="2010-12-14T05:00:51.790" UserId="2116" />
  <row Id="8198" PostId="5453" Text="I believe 'unconditional' here means that $t$ is uniform on $[0,2\pi]$ and independent of $\epsilon$." CreationDate="2010-12-14T06:16:35.800" UserId="795" />
  <row Id="8199" PostId="5465" Text="please make this CW!" CreationDate="2010-12-14T06:21:04.050" UserId="795" />
  <row Id="8200" PostId="5459" Score="5" Text="Your last sentence - brilliant!" CreationDate="2010-12-14T07:02:06.323" UserId="253" />
  <row Id="8201" PostId="5453" Text="correct, E(r) = 0 when taken over t" CreationDate="2010-12-14T07:07:55.863" UserId="2379" />
  <row Id="8202" PostId="5347" Text="@whuber: these greatly vary between cases. It's not a one-time module I'm creating (unfortunately)." CreationDate="2010-12-14T07:09:04.663" UserId="634" />
  <row Id="8203" PostId="5458" Text="i don't follow on how p(v|r) = 0, wont the epsilon noise term in r mean that there is uncertainty in v even when you know r and m? even more specifically wont p(v|r) be exactly and only the noise term in r and thus p(v|r) ~ N(0,V)?" CreationDate="2010-12-14T07:13:04.390" UserId="2379" />
  <row Id="8204" PostId="5443" Score="1" Text="If x1 &lt; .5 AND x2 &lt; .1, then y=&quot;b&quot;. Is that the intended behaviour? (i.e., is the order of lines 7 and 8 significant?)." CreationDate="2010-12-14T08:27:41.083" UserId="892" />
  <row Id="8205" PostId="5457" Score="2" Text="&quot;Statisticians are people who try to come up with meaningful conclusions from messy data.&quot; - unknown" CreationDate="2010-12-14T08:54:18.220" UserId="830" />
  <row Id="8206" PostId="5430" Score="1" Text="@fRed - The noise is from the environment. I'm sorry that I cannot disclose more details about the experimental setup." CreationDate="2010-12-14T09:06:37.947" UserId="1411" />
  <row Id="8207" PostId="5457" Score="1" Text="I think it is two different questions. The first is the one you asked, the second is what to tell your friends. It depends on friends really. I think saying that you will do consulting work, where you consult firms on solving problems with their data, will be enough for the majority of your friends. At least for me it works :)" CreationDate="2010-12-14T09:19:58.290" UserId="2116" />
  <row Id="8208" PostId="5383" Text="Thanks for directions. Just one more thing, does using this mixed models have sense when dealing with high differences between demand, or should one first group stores and then use it on these groups?" CreationDate="2010-12-14T09:23:11.857" UserId="2341" />
  <row Id="8209" PostId="5452" Text="You can write in LaTeX on this site, just put the code in `$ $`." CreationDate="2010-12-14T09:38:44.987" UserId="88" />
  <row Id="8210" PostId="5453" Score="1" Text="If $t$ is uniform, then $p(v|r)=p(\varepsilon)$ is Gaussian, but $p(r)$ is definitely not. Maybe that is your problem." CreationDate="2010-12-14T09:51:17.627" UserId="2116" />
  <row Id="8211" PostId="5470" Score="2" Text="(+1): I cannot count the times I thought I've understood something, but then I failed to explain it to someone else in easy words. Example: p-value ;)" CreationDate="2010-12-14T10:21:38.893" UserId="264" />
  <row Id="8212" PostId="5465" Text="One example: http://stats.stackexchange.com/questions/4768/amoeba-interview-question" CreationDate="2010-12-14T12:01:50.907" UserId="22" />
  <row Id="8213" PostId="5443" Text="Hi F Tusell - it is significant since this order creates a behavior were a is dependent on x1 and x2, while b only depends on x2." CreationDate="2010-12-14T12:21:03.913" UserId="253" />
  <row Id="8214" PostId="5430" Score="2" Text="@Theodor: &quot;I'm sorry that I cannot disclose more details about the experimental setup.&quot; - then that would make your problem insoluble as it stands. I would suggest that your firm hire and pay for a professional statistician." CreationDate="2010-12-14T13:07:20.217" UserId="830" />
  <row Id="8215" PostId="5383" Text="With some programs (such as lme4) you can fit *crossed effects*, i.e. fitting a parameter for each store (deviation from average demand), for each SKU (deviation from average demand), and for each store:SKU combination (deviation from expectation based on store and SKU)." CreationDate="2010-12-14T13:07:20.597" UserId="2126" />
  <row Id="8216" PostId="5470" Score="1" Text="&quot;If you can't explain it to a six-year-old, then you probably don't understand it yourself&quot; - Albert Einstein. Maybe not that extreme, but you get the point... :)" CreationDate="2010-12-14T13:09:37.643" UserId="830" />
  <row Id="8217" PostId="5474" Text="Yes, I forgot the log on the left hand side, or rather I added unnecessary log on the right hand side. Motivation for $p_i =1/(1+\exp(f(x_i)))$s comes from the fact, that $f(x_i)$ is unbounded, but $p_i$ is constrained to interval $(0,1)$, so we need to transform $f(x_i)$ in order to succesfully model $p_i$. This transformation does exactly that. Look for more explanation in wikipedia under logit function." CreationDate="2010-12-14T13:24:34.817" UserId="2116" />
  <row Id="8218" PostId="5467" Score="4" Text="This statement is only true if you assume that the other methods of survey recruitment contain less bias than an all internet based sampling approach. The fact is that all sampling methods contain some amount of bias, and it is incumbent upon the researcher to be cognizant of what those biases may be and to account for them to the extent possible. Where accounting for that bias is not possible, documenting the limitations of your findings is critical. There is research that shows internet based sampling methods induce less or equal bias than other &quot;traditional&quot; methods for certain surveys." CreationDate="2010-12-14T14:01:03.337" UserId="696" />
  <row Id="8219" PostId="5479" Score="1" Text="As a side note, I'll say that the DLM approach to time-series modeling is nice, but it reminds me of C programming: you can do anything, but you can cut your hand off easily as well. The dlm package makes it easy to assemble your matrices, but you've got to figure out what each resulting column means and if you change something, you can easily get crazy results because the columns changed." CreationDate="2010-12-14T14:08:16.820" UserId="1764" />
  <row Id="8220" PostId="5476" Text="This would probably be subsumed under Chris's answer." CreationDate="2010-12-14T14:13:43.700" UserId="830" />
  <row Id="8221" PostId="5479" Text="Could you please add what are you trying to model? " CreationDate="2010-12-14T14:35:30.327" UserId="2116" />
  <row Id="8222" PostId="5347" Text="@David But can't you give some guidance, such as typical ranges?  For instance, if the sum of the p's ranges between 1 and 100 that's useful information and suggests some efficient solutions, but if it can get up to 10,000 that could exclude some approaches." CreationDate="2010-12-14T14:48:47.757" UserId="919" />
  <row Id="8223" PostId="5445" Text="@David That's what you stipulated!  The notation you use (and your sample code) imply alpha and beta are constants.  If they are not, then you need to tell us what their distributions are." CreationDate="2010-12-14T14:51:13.013" UserId="919" />
  <row Id="8224" PostId="5453" Text="@Jesse &quot;This product will itself be a Gaussian with total variance T = 1/(1/V + 1/(0.5+V)).&quot;  Where does this come from?  If V and 0.5+V are variances, then T should be their *sum*.  If V and 0.5+V are precisions, then this formula gives the *precision,&quot; which--of course--goes down." CreationDate="2010-12-14T15:00:10.273" UserId="919" />
  <row Id="8225" PostId="5383" Text="thx Ben. you have been of great help.. I think that i will post some questions about mixed modelling when i study it a bit :)" CreationDate="2010-12-14T15:25:22.900" UserId="2341" />
  <row Id="8226" PostId="5481" Text="Hmmm. This all seems plausible but more complex than my solution (the comments on the original question suggest that the predictors are both categorical). But as usual, the answer is &quot;look at the data&quot; (or the residuals)." CreationDate="2010-12-14T15:34:31.483" UserId="2126" />
  <row Id="8227" PostId="5476" Score="3" Text="Does the correct answer to this one include knowing that there is a controversy about whether fixed marginals make sense and having an informed opinion on the subject?" CreationDate="2010-12-14T15:37:05.537" UserId="2126" />
  <row Id="8228" PostId="3816" Score="2" Text="@Brandon: if the package author cares enough to guide you, then they have given the answer in a form that will be picked up by citation(&quot;some_package&quot;)" CreationDate="2010-12-14T15:38:46.690" UserId="2126" />
  <row Id="8229" PostId="5479" Text="@mpiktas: If you mean what the data is, it's the weekly average attendance (by month) at a theater. If you mean what I want to learn, it's the overall trend of attendance and the seasonal effects, to help the management understand those dynamics and in preparation for forecasting. Hope that helps." CreationDate="2010-12-14T15:44:04.463" UserId="1764" />
  <row Id="8230" PostId="5481" Text="@Ben I agree but I don't understand where the perception of &quot;more complex&quot; comes from, because analysis of univariate distributions and post-hoc analysis of residuals are essential in any regression exercise.  The only extra work required here is to think about what these analyses mean." CreationDate="2010-12-14T15:53:13.067" UserId="919" />
  <row Id="8231" PostId="5445" Text="@whuber sorry, my previous question was poorly written. Maybe I should have asked: if X~F() is kX~F()? I was wondering if there are general rules about transforming continuous random variables by a constant; it seems that a variable X transformed by k would maintain its shape if its pdf is scaled by k&gt;0, but that this would not hold for k=0 or k&lt;0. (and is this somewhere in my intro probability text by S. Ross? - I cant find it)." CreationDate="2010-12-14T16:11:56.073" UserId="1381" />
  <row Id="8232" PostId="5481" Text="Perhaps by &quot;more complex&quot; I just mean &quot;In my experience, I have seen the issues I referred to in my answer (contrast coding) arise more frequently than those you referred to (non-additivity)&quot; -- but this is really a statement about the kinds of data/people I work with rather than about the world." CreationDate="2010-12-14T16:23:41.157" UserId="2126" />
  <row Id="8233" PostId="5458" Text="Sorry, got my notation messed up. Fixed now." CreationDate="2010-12-14T16:28:17.267" UserId="2067" />
  <row Id="8234" PostId="5349" Text="@mpiktas +1 Thank you for quantifying the approximation error.  It's good to see this kind of analysis." CreationDate="2010-12-14T16:54:46.353" UserId="919" />
  <row Id="8235" PostId="5482" Text="I suspect an R expert could supply efficient code for this algorithm in a half dozen lines, using the p_i and z for input and outputting the distribution f_n.  Any takers?" CreationDate="2010-12-14T17:07:16.967" UserId="919" />
  <row Id="8236" PostId="3084" Text="An old topic, but a key point for clarification in the OP: when they say that they have &quot;rain or not&quot; data against which to compare the prediction, do they mean &quot;at my house&quot;, or do they mean &quot;within the prediction area&quot;?" CreationDate="2010-12-14T17:34:24.573" UserId="1764" />
  <row Id="8237" PostId="5478" Text="This is a good point. I have found in the past that it is very difficult to tell whether a given candidate will work out, unless you have worked with them in the past." CreationDate="2010-12-14T17:34:37.497" UserId="795" />
  <row Id="8238" PostId="5470" Score="1" Text="I like &quot;Explain a p-value&quot;, with or without the &quot;to a novice&quot; part." CreationDate="2010-12-14T17:36:06.637" UserId="795" />
  <row Id="8239" PostId="5485" Text="How do I calculate this? " CreationDate="2010-12-14T17:47:58.290" UserId="2380" />
  <row Id="8240" PostId="5485" Text="This looks like a homework question.  What are the rules about homework questions on SE?  I don't see anything in the FAQ -- but I would probably be more willing to contribute if you could convince me this is **not** homework. PS.  If you want to know how to calculate bounds using Chebyshev's inequality, just follow the link posted!  There is an example in the middle of that page ..." CreationDate="2010-12-14T18:12:16.577" UserId="2126" />
  <row Id="8241" PostId="5486" Text="That reference is merely an exercise in completing the square.  Its &quot;product&quot; of Gaussians multiplies two functions *of a single variable* (x)--it is neither a product of random variables nor a convolution.  I don't see how it's relevant.  What's really confusing is the OP's assertion that &quot;The unconditional variance of r is 0.5 + V&quot;, because this seems to imply that t is a random variable.  What is its distribution?  Regardless, when t is random it's unlikely that r has a Gaussian distribution at all." CreationDate="2010-12-14T18:14:13.460" UserId="919" />
  <row Id="8242" PostId="5486" Text="Note that it's best to carry on conversations in comments, not by augmenting the list of answers." CreationDate="2010-12-14T18:14:55.333" UserId="919" />
  <row Id="8243" PostId="5485" Text="@Ben: visit http://meta.stats.stackexchange.com/search?q=homework .  Note that most homework questions are both artificial and specific.  The vagueness of this one suggests to me it originates from a personal interest only." CreationDate="2010-12-14T18:17:50.447" UserId="919" />
  <row Id="8244" PostId="5461" Text="What do you want to know?  I don't see a clear question here." CreationDate="2010-12-14T18:21:29.367" UserId="919" />
  <row Id="8245" PostId="5485" Text="Believe me, it is purely interest, not homework. But believe what you want." CreationDate="2010-12-14T18:25:52.530" UserId="2380" />
  <row Id="8246" PostId="5461" Text="I'm trying to figure out which of my sample groups displays different statistical properties (in this case by taking slightly longer) from the rest. There is exactly one group for which this is known to be the case, all other groups come from the same distribution." CreationDate="2010-12-14T18:28:42.033" UserId="1750" />
  <row Id="8247" PostId="5453" Text="@mpiktas You should post your comment as a solution.  (There are other problems with the calculation in question, but this one alone demonstrates it is invalid.)" CreationDate="2010-12-14T18:32:26.037" UserId="919" />
  <row Id="8248" PostId="5487" Score="1" Text="Surely one of the applets at http://statpages.org/#Power will solve your problem." CreationDate="2010-12-14T18:35:19.217" UserId="919" />
  <row Id="8249" PostId="5461" Text="@Paul &quot;Statistical properties&quot; in what sense?  What role does the timing play: is it an auxiliary variable or is it the variable of interest?  I would find it *much* easier to think about your question if you were to reformulate it without the extensive speculation about types of tests and simply describe as clearly as possible what your experiment consists of and what you want to learn from it." CreationDate="2010-12-14T18:57:11.460" UserId="919" />
  <row Id="8250" PostId="5349" Text="You're welcome :) I lifted the approximation from Petrov's book: http://www.amazon.co.uk/Limit-Theorems-Probability-Theory-Independent/dp/019853499X/ref=sr_1_1?ie=UTF8&amp;s=books&amp;qid=1292354039&amp;sr=8-1" CreationDate="2010-12-14T19:14:19.370" UserId="2116" />
  <row Id="8252" PostId="5461" Text="I have a process which takes a certain amount of time to run depending on the input value. I have 16 potential inputs. Of these, one will make the process run for a slightly longer period of time than the others. I need to identify this input. I am attempting to do so by running many tests, and recording the time my process takes for each test." CreationDate="2010-12-14T19:44:34.543" UserId="1750" />
  <row Id="8253" PostId="5461" Text="Unfortunately, my data is noisy. This noise manifests itself (in one of many ways) by periodically causing all samples (regardless of input) to take longer to process. Since I am sampling in a round-robin fashion, it seems likely that I may be able to correct for this." CreationDate="2010-12-14T19:49:45.877" UserId="1750" />
  <row Id="8254" PostId="5468" Score="1" Text="Its also worth noting that as soon as you changed the model because of the modification indices, you were no longer doing CFA, rather you have moved back into exploratory mode. " CreationDate="2010-12-14T19:50:29.043" UserId="656" />
  <row Id="8256" PostId="5461" Text="Part of the reason that the question is long is that I wanted to show that I've done my homework. It seems rude to just present the problem and ask &quot;which test should I use?&quot;" CreationDate="2010-12-14T19:53:45.780" UserId="1750" />
  <row Id="8257" PostId="5483" Score="1" Text="The most information usually hides in the data itself, not the description of it (-;" CreationDate="2010-12-14T19:59:12.550" UserId="88" />
  <row Id="8259" PostId="5453" Text="@jdav I have merged your both users, so you could add comments to answers of your questions." CreationDate="2010-12-14T20:07:36.990" UserId="88" />
  <row Id="8260" PostId="5458" Text="luispedro, I believe your answer is correct. my misunderstanding was in thinking that p(v|r) was a Gaussian, but it is not, as you point out, and so the rest of my &quot;solution&quot; is invalid. Thank you very much for the help." CreationDate="2010-12-14T20:22:35.907" UserId="2379" />
  <row Id="8261" PostId="5486" Text="whuber, thank you. you point out a problem that mpitkas also pointed out which is that r is not Gaussian. the simplified version of the problem, where r = c + epsilon, epsilon ~ N(0,V) does have r as Gaussian, and still would have fallen prey to the rest of my argument. It looks like there is a second root problem, however, which luispedro points out which I believe to be the ultimate cause of my misunderstanding. although clearly, I had plenty of misunderstanding to go around..." CreationDate="2010-12-14T20:28:35.220" UserId="2379" />
  <row Id="8262" PostId="5492" Text="Thanks for your quick reply. Yes I know this straight forward computation. But I am not sure is this the lower bound of the entropy or upper bound.                                                                   " CreationDate="2010-12-14T20:44:42.030" UserId="2384" />
  <row Id="8263" PostId="5492" Text="Bound in what sense?  You need a context here.  What set of values are you trying to bound and how is it defined?  Read the Wikipedia segment I linked to: it has some good examples.  For instance, if you model the image as one realization of a set of possible images, then its entropy equals -1*log(1) = 0.  You can't do better than that!" CreationDate="2010-12-14T20:47:55.777" UserId="919" />
  <row Id="8264" PostId="5493" Score="2" Text="(+1), excellent program. To spare the reader an extra click, you could link to the new version (G*Power3) which has a GUI for Windows and MacOS X: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/" CreationDate="2010-12-14T20:50:07.290" UserId="1909" />
  <row Id="8265" PostId="5492" Text="Let us take an example, we have a sensor filed of M*N dimension and sensors are uniformly distributed in grid architecture. let say they are sensing the temperature in the agriculture field. and it is obvious they have spatial correlated data. Assume the they are sensing 35,40,45 and 50 degree C. Now I want to calculate the minimum entropy, is this follow the same formula as you said earlier in your post? " CreationDate="2010-12-14T20:50:57.267" UserId="2384" />
  <row Id="8266" PostId="5492" Text="Nope: This is a different probability model.  You need to specify the nature of the spatial correlation and estimate its parameters." CreationDate="2010-12-14T21:13:07.410" UserId="919" />
  <row Id="8267" PostId="5495" Text="Dave, wouldn't glmmADMB work for this problem (as a start)?" CreationDate="2010-12-14T22:42:11.840" UserId="2126" />
  <row Id="8268" PostId="5471" Text="@mpiktas: Thanks! I did not get the connection that f(x) was the logit transform. Did you happen to have insight into the terminal node -- should that be self explanatory to me? It looks like GBM uses a regression tree for classification (versus the classification algorithm shown in Elements of Statistical Learning. Is that your understanding as well? Thanks again!" CreationDate="2010-12-14T23:14:31.983" UserId="2040" />
  <row Id="8269" PostId="5464" Text="Thanks you so much for the suggestions." CreationDate="2010-12-15T01:43:12.913" UserId="2371" />
  <row Id="8270" PostId="1370" Text="How about multiplying only $X$ by a constant $\alpha$? And what happens if $\alpha=0$ ?" CreationDate="2010-12-15T01:44:13.520" UserId="2148" />
  <row Id="8271" PostId="5493" Text="Oops - thanks for that, caracal.  I just grabbed the link without noticing the content of the page." CreationDate="2010-12-15T02:34:12.453" UserId="71" />
  <row Id="8272" PostId="5491" Text="Thank you for your answer. Where can I learn how to do SMC and which R package would you recommend?" CreationDate="2010-12-15T03:05:43.873" UserId="1709" />
  <row Id="8273" PostId="5467" Text="OK. interesting. I guess also that it depends on the type of survey you want to do." CreationDate="2010-12-15T03:07:12.950" UserId="1709" />
  <row Id="8274" PostId="5463" Text="@idclark If you want to open an SPSS file without paying for software check out: PSPP: http://www.gnu.org/software/pspp/ or read.spss in R http://stat.ethz.ch/R-manual/R-devel/library/foreign/html/read.spss.html" CreationDate="2010-12-15T04:23:12.637" UserId="183" />
  <row Id="8275" PostId="5503" Text="I am asking this here because of its use in Reproducible Research and if I remember correctly this quote was in a presentation related to reproducible research. Apologies in advance if it wasn't meant to be here." CreationDate="2010-12-15T04:40:50.250" UserId="1307" />
  <row Id="8276" PostId="5463" Text="@Jeromy: I had trouble opening an SPSS sav file in the past, but was unable to solve them via PSPP. Here's my blog post about it: http://blog.bzst.com/2008/09/data-conversion-and-open-source.html" CreationDate="2010-12-15T04:59:51.080" UserId="1945" />
  <row Id="8277" PostId="5463" Text="@Galit Your post is from September 2008. It might be worth trying again. Alternatively, use the read.spss() function in R." CreationDate="2010-12-15T05:07:52.443" UserId="183" />
  <row Id="8278" PostId="5463" Text="@jeromy  Ah yes, i completely forgot! I haven't used PSPP, but I use read.dta several times a day. thanks for the reminder. " CreationDate="2010-12-15T05:31:07.483" UserId="2262" />
  <row Id="8279" PostId="5471" Text="I am not very familiar with machine learning. For what I gathered from the vignette of GBM, yes GBM uses regression tree for classification, but regression trees are just one of classification algorithms. The terminal node formula is used to calculated the addition to regression function, during the learning process. It is used everytime after classification. I suggest to formulate separate question to get a better answer." CreationDate="2010-12-15T07:16:09.787" UserId="2116" />
  <row Id="8281" PostId="5117" Text="@Carlos &quot;undoubtedly the creator&quot; are you a member of FA (Fisher's Adorators) ? Anyway Gauss introduced least square almost a hundred year before Fisher was born, and Fisher as well as Gauss were inspired by a lot of other very inspired people... it is a long and laborious story and unfortunatly for those who like THX surround movies I don't really think it has its Guru." CreationDate="2010-12-15T07:30:19.340" UserId="223" />
  <row Id="8283" PostId="5140" Text="Can you tell us more than &quot;his work in machine learning&quot; which is a bit vague. " CreationDate="2010-12-15T07:37:55.637" UserId="223" />
  <row Id="8284" PostId="5503" Score="1" Text="I don't really see the point made by this quote. Is there a computational philosophy ? computational poetry ? ... It seems that it would be more correct to say : &quot;For every field &quot;x&quot; that has a &quot;computational x&quot; there is a &quot;computational x&quot;. :) " CreationDate="2010-12-15T08:04:13.200" UserId="223" />
  <row Id="8285" PostId="5504" Score="1" Text="The same question here http://math.stackexchange.com/questions/14387/normal-distribution-probability" CreationDate="2010-12-15T08:13:24.473" UserId="2116" />
  <row Id="8286" PostId="5508" Text="Really 21st century 2000 and onward? Not 20th, 1900 and onward? Could you please cite these references. I think this is a misquote." CreationDate="2010-12-15T08:24:20.767" UserId="2116" />
  <row Id="8288" PostId="5123" Text="he was a mathematician that was very important to statistics. Half of the men on here lived before statistics was an &quot;official&quot; field of study ." CreationDate="2010-12-15T08:48:24.790" UserId="74" />
  <row Id="8289" PostId="5507" Score="4" Text="Note that you don't need to evaluate the probabilities to answer the question, you only need that –3/4 &lt; –2/3." CreationDate="2010-12-15T08:54:45.483" UserId="449" />
  <row Id="8291" PostId="4734" Text="@chl, i don't see how the related question is about EFA at all. the accepted answer is about confirming hypothesized factors, not finding them. " CreationDate="2010-12-15T09:12:42.327" UserId="74" />
  <row Id="8292" PostId="5122" Text="+1 for Francis Galton played a very important role in giving importance to the concept of correlation. However, I found a bit strong the formulation &quot;creating correlation&quot;. I would quote Galton itself: &lt;&lt; &quot;Co-relation or correlation of structure&quot; is a phrase much used in biology, and not least in that branch of it which refers to heridity, and the idea is even more frequently present than the phrase; but I am not aware of any attempt to define it clearly &gt;&gt; In : &quot;Co-relation and their Measurment&quot; (see here http://galton.org/galton/index.html)" CreationDate="2010-12-15T09:19:36.523" UserId="223" />
  <row Id="8293" PostId="5509" Text="@chl. Thanks. You are a great resource. Did you google this? If yes, can you please enlighten me with your google-fu skills? And yes, I did check these two articles but didn't find the quotation." CreationDate="2010-12-15T09:21:42.013" UserId="1307" />
  <row Id="8294" PostId="5503" Text="@Robin. I agree with you, but it sounds good for a presentation on reproducible-research and I am about to give one." CreationDate="2010-12-15T09:23:24.757" UserId="1307" />
  <row Id="8295" PostId="5508" Text="@mpiktas 1900 and onward seems unlikely as the standard probit model dates from 1934/5. What's computationally tractable depends on the available algorithms as well as the available computing power, so 2000 onward seems believable to me." CreationDate="2010-12-15T09:35:23.250" UserId="449" />
  <row Id="8297" PostId="4734" Text="@Neil Ok, maybe I missed something and in this case I'll let @Jeromy add support to his claims, but from what I've read it's mainly about PCA vs. EFA, where in the latter we *assume* a 'reflective' measurement model; I agree that when he said &quot;if you assume or wish to test  (...)&quot; we can easily switch to any CFA framework. But in this case, usual PCA has strictly nothing to do with CFA, so the question becomes ill-posed. Do you know this paper from Denny Borsboom, [The Theoretical Status of Latent Variables](http://j.mp/h0KhJo)? Its focus is on LV models, not really PCA, but it is a good paper" CreationDate="2010-12-15T10:09:43.330" UserId="930" />
  <row Id="8298" PostId="5509" Text="@suncoolsu Yes, I was lucky enough with `reproducible research de leeuw &quot;for every field&quot;` in Google, but as I know you have some background in genetics, I was initially expecting to get a hit from Bioconductor tutorials instead..." CreationDate="2010-12-15T10:13:32.157" UserId="930" />
  <row Id="8300" PostId="5492" Text="Entropy calculations for fully speciﬁed data have been used to get a theoretical bound on how much that data can be compressed. My specified filed is the data from sensor filed. And I want to calculate the minimum entropy (assuming lossless compression) for entire sensor field not for individual nodes. " CreationDate="2010-12-15T13:02:20.707" UserId="2384" />
  <row Id="8301" PostId="5495" Text="It might require some tweaking.  glmmADMB is intended to fit a Negative Binomial mixed model. It would be fairly simple to modify it for just a Negative Binomial regression." CreationDate="2010-12-15T14:16:50.560" UserId="1585" />
  <row Id="8302" PostId="2293" Text="I don't think NIPS is a really a data mining conference, it is a machine learning and (decreasingly) computational neuroscience conference.  " CreationDate="2010-12-15T14:42:44.327" UserId="887" />
  <row Id="8303" PostId="5401" Text="To Z score a variable use the scale function.  If you scale G1 G2 etc... you'll get a sense of how people rate their expertise compared to others.  If you also scale WD then both stated expertise and work done will be put on the same scale and you can simply sum the scores to get some combined metric." CreationDate="2010-12-15T16:11:19.477" UserId="196" />
  <row Id="8304" PostId="5401" Text="My approach is something like: scale(resid(lm(WD.G1 ~ W.G1)))+scale(G1).  You'll (of course) have to restructure the data some to make that all work.  Good luck.  If you want clarification on any of these techniques, consider asking a new question." CreationDate="2010-12-15T16:13:07.510" UserId="196" />
  <row Id="8305" PostId="5503" Score="1" Text="Computational poetry (PDF): http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.4337&amp;rep=rep1&amp;type=pdf" CreationDate="2010-12-15T16:15:23.357" UserId="71" />
  <row Id="8306" PostId="5517" Score="3" Text="(1) I'm surprised that pvals.fnc still works at all; I thought mcmcsamp (which pvals.fnc relies on) had been non-functional in lme4 for quite a while. What version of lme4 are you using? (2) There is no conceptual reason why having random slopes should break whatever one is doing to get a significance test (3) Combining significance testing with MCMC is a little bit incoherent, statistically, although I understand the urge to do so (getting confidence intervals is more supportable) (4) the only relationship between this Q &amp; the other is 'MCMC' (i.e. none, practically)" CreationDate="2010-12-15T16:42:26.313" UserId="2126" />
  <row Id="8307" PostId="5418" Score="1" Text="Another way to look at this is to look for particular package *authors*; certain authors are very good at following best practices and writing clear code, in which case you can study all of their materials." CreationDate="2010-12-15T16:52:47.603" UserId="5" />
  <row Id="8308" PostId="5114" Score="1" Text="@David After reading this question over many, many times I still don't understand it.  Are you using the two observations and the scientist's opinion to estimate a prior for a Bayesian analysis?  Is your prior going to be based on the scientist's opinion only and then you want to update it with the observations?  What is the distinction between &quot;unsurprising&quot; and having zero probability?" CreationDate="2010-12-15T17:24:27.373" UserId="919" />
  <row Id="8309" PostId="5114" Text="@whuber I was originally trying to state the prior using the expert opinion and two data points by summarizing the two data points as a central tendency. But the idea of setting a prior with the expert opinion and updating it with may be a better idea since this approach does not rely on my own choice of a measure of central tendency. &quot;Unsurprising&quot; can be interpreted as &quot;small but nonzero probability&quot;, e.g. $P(X&gt;5)&lt;0.05$ and $P(X&gt;6)&lt;0.01$" CreationDate="2010-12-15T17:32:24.967" UserId="1381" />
  <row Id="8310" PostId="5114" Text="@David Why not just use a &quot;maximally uninformative&quot; prior or even a uniform prior then?  (Both are scaled beta distributions.)  You could even fit a maximal entropy solution to the expert opinion information (although I would be disinclined to do that, given that experts notoriously mis-estimate probabilities)." CreationDate="2010-12-15T17:34:50.097" UserId="919" />
  <row Id="8311" PostId="5438" Text="Why the downvote?" CreationDate="2010-12-15T17:36:52.963" UserId="919" />
  <row Id="8312" PostId="5446" Text="@Wayne Thanks.  It is interesting how such a story can recur and how it might actually be true in both cases!" CreationDate="2010-12-15T17:39:48.860" UserId="919" />
  <row Id="8313" PostId="5114" Text="@whuber I'd like to stay away from the uninformed priors, and my question to the scientist attempted to take into account the difficulty of estimating probabilities, for example by following &quot;what is your 95%CI&quot; by &quot;What values of X are inconceivable?&quot; and &quot;What values of X are conceivable but unlikely&quot; as well as &quot;Is X=6 at all possible, even if you would not expect to see it?&quot;. I tried to summarize this information in the question. I want to be conservative but not exclude available information." CreationDate="2010-12-15T17:40:39.720" UserId="1381" />
  <row Id="8314" PostId="5509" Text="@chl, suncoolsu: The brute-force obvious search on 'For every field &quot;x&quot; there exists a field &quot;computational x&quot;' turns up this thread as hit #1 and Peng's tutorial as hit #2." CreationDate="2010-12-15T17:44:19.713" UserId="919" />
  <row Id="8315" PostId="5114" Score="1" Text="@David I appreciate your care in eliciting the information.  But assigning specific values of 0.05 and 0.01 to those probabilities is questionable.  That's not your fault; it's just how things are.  We can't expect people to pin down probabilities that well (that's what data are for).  Maybe you would like to represent those with hyperpriors :-)?" CreationDate="2010-12-15T17:54:23.557" UserId="919" />
  <row Id="8316" PostId="5123" Text="@Neil  good point ! I guess I meant that even before it was official, some mathematicians (like Gauss) concidered statistical problems while I don't really know if Markov did concider statistical problem himself.  " CreationDate="2010-12-15T18:01:36.157" UserId="223" />
  <row Id="8317" PostId="5438" Text="@whuber thanks, you saved my favorite :) " CreationDate="2010-12-15T18:06:24.903" UserId="223" />
  <row Id="8318" PostId="5358" Text="@onestop, @David B: I compared the Poisson approximation to the exact results for the sequence p_i = 1/(i+1), i = 1, ..., 2^15. The Poisson parameter is the mean of the p_i, 9.97447. The approximation of individual probabilities is within 5% for sums between 6 and 15 inclusive: that is, within one to two SDs of the mean.  The relative errors increase exponentially outside this interval. The *absolute* errors are all between -0.0026 and +0.0043. Errors in the CDF range from -0.0090 to +0.0074: just under 1%. By any of these measures, simulation with 10,000 trials would be at least as accurate." CreationDate="2010-12-15T18:18:11.277" UserId="919" />
  <row Id="8319" PostId="5348" Text="@David B It's guaranteed not to look Gaussian when the sum of the p_i is small.  &quot;Small&quot; means less than 10 or so: at this point, a Normal approximation starts projecting noticeable probability onto negative sums.  Whether a Poisson approximation is worthwhile depends on how much accuracy you need.  Unless all the p_i are equal or all of them keep away from 0 and 1, neither approximation will come close to the &quot;exact probability&quot; you ask for." CreationDate="2010-12-15T18:27:00.190" UserId="919" />
  <row Id="8320" PostId="5497" Text="+1 That's certainly a lively way to put it!" CreationDate="2010-12-15T18:36:59.530" UserId="1118" />
  <row Id="8321" PostId="5114" Text="@whuber thanks for the suggestion, I can't justify hyperpriors, but it would be sufficient to inflate the variance and overestimate the probabilities in the tails; isn't this done by giving an upper bound rather than an exact value, e.g., to $P(X=5) &lt; 0.05$ ?" CreationDate="2010-12-15T18:39:24.370" UserId="1381" />
  <row Id="8323" PostId="5114" Text="@David I don't see how an upper bound allows you to determine the distribution unless you specify an objective function to be optimized subject to that upper bound as a constraint.  For example, you might want to maximize the variance or the entropy subject to these constraints.  (The latter has more justification.)  Until you provide such a criterion, there seems to be no basis to choose among the many distributional forms suggested by existing responses." CreationDate="2010-12-15T18:55:04.707" UserId="919" />
  <row Id="8324" PostId="5122" Text="@robin: You could say that Galton did not create the idea of correlation, but did create the first statistical representation of it. Google &quot;francis galton's account of the invention of correlation&quot;" CreationDate="2010-12-15T19:03:39.697" UserId="74" />
  <row Id="8325" PostId="5118" Text="I'm really surprised that Tukey is beating Pearson right now! :)" CreationDate="2010-12-15T19:04:14.767" UserId="74" />
  <row Id="8326" PostId="5472" Score="3" Text="In more academic contexts one may also ask: 'have a look at this model output in this paper *that you (co-)authored*.  Tell me what it means.'  Underwhelming answers are then fatal because there are no unfamiliarity excuses available, yet dismayingly common." CreationDate="2010-12-15T19:27:49.403" UserId="1739" />
  <row Id="8327" PostId="5358" Text="That's interesting, whuber. I assume you mean the *sum* of the p_i is 9.97447 ? I suspect the relatively few large probabilities affect things - is the approx much better if you start at i=10 or 100 instead of i=1? David B hasn't told us how large his largest $p_i$ is but i suspect it's considerably smaller than 0.5. Starting at i=1 the sum of the squares of the $p_i$s is 0.645 so Le Cam's inequality says the sum of the absolute differences is no more than 1.29. If you start at i=10 that would be 0.19, and for i=100 it would be 0.02." CreationDate="2010-12-15T19:28:49.937" UserId="449" />
  <row Id="8328" PostId="5114" Score="1" Text="@whuber Thank you for following up. Now I understand the concept of the maximum entropy solution that you previously suggested, although it would require some learning on my end before I would want to apply it. Your alternative, maximizing variance, sounds sufficiently consistent with my objective and straightforward for me to implement. Thanks again." CreationDate="2010-12-15T19:37:00.303" UserId="1381" />
  <row Id="8329" PostId="5522" Text="@Conjugate (+1) Nice catch!" CreationDate="2010-12-15T19:38:59.297" UserId="930" />
  <row Id="8330" PostId="5521" Text="@Conjugate +1 but what would be the rationale for considering a rotation when the GRM actually assumes an unidimensional scale?" CreationDate="2010-12-15T20:00:37.850" UserId="930" />
  <row Id="8332" PostId="5470" Text="this is why cross-validated is great. lots of &quot;layman&quot; questions and answers. " CreationDate="2010-12-15T20:17:04.217" UserId="74" />
  <row Id="8334" PostId="592" Text="only 3 reviews on Amazon...sketchy" CreationDate="2010-12-15T20:32:33.867" UserId="74" />
  <row Id="8335" PostId="5487" Text="@Roland How could you compute a sample size in a power study from a non-inferiority trial if you don't know the expected difference (or equivalently, the % of patients responding to the new treatment compared to the actual one)?" CreationDate="2010-12-15T20:35:09.287" UserId="930" />
  <row Id="8336" PostId="5114" Score="1" Text="@David Be aware that a problem with maximizing variance is that the solution will be a discrete distribution.  I expect it to concentrate 0.01 probability at X=8, 0.04 at X=6, 157/300 at X=2, and the rest (32/75) at X=5.  (This variance equals 2.59.)" CreationDate="2010-12-15T20:38:41.280" UserId="919" />
  <row Id="8337" PostId="5494" Text="Should we add Hadley Wickham for making GoG possible in R?" CreationDate="2010-12-15T20:41:09.000" UserId="930" />
  <row Id="8338" PostId="5507" Text="Haha, yes of course, noted." CreationDate="2010-12-15T21:19:16.573" UserId="2387" />
  <row Id="8339" PostId="5526" Text="Thank you. In the meanwhile I realized that I also need the number of observations (n). One way is this: dim(model.matrix(model))[1] Can you suggest another way ? " CreationDate="2010-12-15T21:28:32.217" UserId="632" />
  <row Id="8340" PostId="5531" Score="1" Text="Doesn't multiple regression with $N$ observations require an $N$-dimensional TinkerToy set though?" CreationDate="2010-12-15T21:31:05.197" UserId="449" />
  <row Id="8343" PostId="5522" Text="Wooo! Hats off to your googling skills :-( Me sux." CreationDate="2010-12-15T22:21:21.003" UserId="1307" />
  <row Id="8344" PostId="5522" Text="@Conjugate and Chl. You guys have put me in a dilemma: whose answer should I accept?" CreationDate="2010-12-15T22:22:01.893" UserId="1307" />
  <row Id="8345" PostId="5521" Text="@chl The thought was that some people rotate to simple structure to be able to say things like 'indicators 1-4 measure one thing and indicators 5-11 measure something else' on the basis of the rotated loadings.  The related but not quite identical thought with the IRT approach would be to say things like: 'this Mokken procedure tells me there's a scale underlying indicators 1-4 and another one underneath 5-11 so I'll apply my graded response model to each subset separately'.  Hope that makes better sense." CreationDate="2010-12-15T22:27:37.213" UserId="1739" />
  <row Id="8346" PostId="5522" Score="2" Text="@suncoolsu No googling involved - I look through de Leeuw's mammoth preprint list once every 6 months to see what goodies have newly arrived.  If you can bear the fierce matrix algebra and super terse presentation, there's always something interesting to read." CreationDate="2010-12-15T22:30:25.760" UserId="1739" />
  <row Id="8347" PostId="5509" Text="@whuber I was adding an extra Jan de Leeuw to your search and I guess it messed up the results. " CreationDate="2010-12-15T22:35:06.873" UserId="1307" />
  <row Id="8348" PostId="5505" Score="1" Text="(Sorry I could not include graphs,  I don't have permission to share the data.) So there can be a Kalman filter settling time. That makes total sense, looking at the Big Bash (moving seasonality) effect: the first year has movement in the &quot;baseline&quot;: in the one case it wanders a bit, and in the second case it has that horrible spike. After that, the baseline (Big Bash indicator 0) runs nearly flat around 0.95, while the spikes (Big Bash indicator 1) go up to 1.10-1.25 (multiplicative effect), which is what I'd expect. The 0.95 baseline I think due to me not centering the 0/1 Big Bash series." CreationDate="2010-12-15T22:46:50.467" UserId="1764" />
  <row Id="8349" PostId="5498" Text="Yes, point well-taken. I was doing things the hard way to learn DLM's, mainly, since one of the touted benefits of the SS/DLM approach is that you model each element explicitly so your results are more open to understanding/explanation than a more black-box approach like ARIMA. In some ways true, and in some ways not. Especially if a combination of tools (`stl` and `arima`) can give you similar results with less pain and on more familiar territory." CreationDate="2010-12-15T22:55:31.707" UserId="1764" />
  <row Id="8350" PostId="5505" Text="In my previous note I used &quot;spike&quot; in two different ways: 1) the Bad Spike that appeared evidently due to filter settling, and 2) the Good Spikes which correspond to the once-a-year Big Bash indicator being 1. Sorry for any confusion." CreationDate="2010-12-15T22:57:29.740" UserId="1764" />
  <row Id="8351" PostId="5498" Text="(Not to mention that the `dlm` package won't forecast if you include a `dlmModReg` element, in the current version anyhow. As a follow-on exercise, I am going to see if I can modify their `dlmForecast` function (which is completely in R, thankfully) to handle this specific case.)" CreationDate="2010-12-15T23:00:53.840" UserId="1764" />
  <row Id="8352" PostId="5505" Text="@F Tusell: several good hints in one reply, thanks!" CreationDate="2010-12-15T23:05:03.087" UserId="1764" />
  <row Id="8353" PostId="5522" Text="@suncoolsu I like Jan's work (I was at this UseR also!), and actually I think you really should accept this answer." CreationDate="2010-12-15T23:44:19.053" UserId="930" />
  <row Id="8354" PostId="5509" Score="1" Text="@suncoolsu In terms of &quot;Google fu&quot;, a good approach seems to be to start with broad searches and then narrow them in response to the hits you are getting.  Starting with an extremely narrow search risks overlooking good stuff.  In this case even my search perhaps was too aggressive.  When you consider, though, that every word in that quote can have many meanings unrelated to your intended topic, using the entire quotation seemed to be a reasonable starting point." CreationDate="2010-12-15T23:46:56.233" UserId="919" />
  <row Id="8355" PostId="5521" Text="@Conjugate Yes, indeed. Thanks for clarifying what I've extrapolated from your last sentence. Still we have no way of linking each latent trait if it happens they are truly correlated (unless looking at MIRT)." CreationDate="2010-12-15T23:48:44.990" UserId="930" />
  <row Id="8356" PostId="5526" Score="1" Text="@Brani Try `nrow(na.omit(cbind(y,x1,x2)))` (following your notation) or `length(lm.fit$model$y)`, to get the number of observations used when estimating model parameters. BTW, you're better off making a data.frame, e.g. `df &lt;- data.frame(y,x1,x2)`, and then use `lm(y ~ ., data=df)`. " CreationDate="2010-12-15T23:55:17.307" UserId="930" />
  <row Id="8357" PostId="5541" Text="+1 It's a straightforward sampling problem and this is a straightforward solution. " CreationDate="2010-12-16T02:36:52.477" UserId="96" />
  <row Id="8358" PostId="5418" Text="@Shane Good point. Any suggestions on which package authors might be good to study when first learning to write packages?" CreationDate="2010-12-16T03:20:48.183" UserId="183" />
  <row Id="8360" PostId="5517" Text="The version of lme4 I use depends on the computer I am sitting at. This console has lme4_0.999375-32, but I seldom use this one for analysis. I noticed several months ago that pvals.fnc() was ripping apart the lme4 results after analysis - I built a work around for it at the time, but it doesn't seem to be an issue anymore.  I'll have to ask another question on your 3rd point in the near future." CreationDate="2010-12-16T04:12:01.880" UserId="196" />
  <row Id="8361" PostId="5509" Text="@whuber. Thank you. You are a great resource." CreationDate="2010-12-16T05:55:14.253" UserId="1307" />
  <row Id="8364" PostId="5552" Text="What does NIPS stand for?" CreationDate="2010-12-16T08:23:14.067" UserId="449" />
  <row Id="8365" PostId="5552" Text="@onestop link added. NIPS=Neural Information Processing Systems . It is a community (not a system :) ) . But pisk is talking about the conference NIPS 2010. " CreationDate="2010-12-16T08:40:29.463" UserId="223" />
  <row Id="8366" PostId="5516" Text="Wow, I didn't even think about the question from this perspective.  Thanks!" CreationDate="2010-12-16T09:03:09.557" UserId="2385" />
  <row Id="8367" PostId="5506" Text="Thanks for the help!" CreationDate="2010-12-16T09:03:26.390" UserId="2385" />
  <row Id="8368" PostId="5507" Text="Thanks for the help!" CreationDate="2010-12-16T09:05:15.483" UserId="2385" />
  <row Id="8369" PostId="5554" Text="Ok but i know that the Bonferroni is very conservative. As an exemple, if i'm doing 10 comparaisons, does that mean that for a p &lt; .05 (for a single comparaison) i'll need to find a p &lt; .05/10 on the 10 comparaisons?" CreationDate="2010-12-16T09:22:23.770" UserId="2402" />
  <row Id="8370" PostId="5554" Score="4" Text="Yes. You may try this http://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method. It is less conservative and at least in regression setting for me it gave reasonable results. " CreationDate="2010-12-16T09:32:15.317" UserId="2116" />
  <row Id="8372" PostId="5554" Score="6" Text="@Richard There's generally a trade-off between FPs and FNs in MCP, the Bonferroni's method being prone to increase FNs, while step-down methods (like the one suggested to you by @mpiktas) will perform better in most cases. As an alternative, you might want to look at permutation-based methods which offer a good compromise for the simultaneous control of FP and FN rates. @mpiktas (+1) In R, relevant methods are found in the base function `p.adjust()`, and the [multcomp](http://j.mp/eVflBe) and [multtest](http://j.mp/hirQrK) packages have great vignettes describing all of this." CreationDate="2010-12-16T10:13:09.383" UserId="930" />
  <row Id="8373" PostId="5433" Text="Thanks. Great answer." CreationDate="2010-12-16T10:32:12.357" UserId="183" />
  <row Id="8374" PostId="5553" Text="+1 Apparently, it's &quot;lapply(xcomb...&quot;" CreationDate="2010-12-16T13:18:20.263" UserId="339" />
  <row Id="8375" PostId="5553" Text="@gd047, thanks for pointing out. " CreationDate="2010-12-16T13:21:45.720" UserId="2116" />
  <row Id="8376" PostId="5559" Score="3" Text="@slotishtype Could you provide additional context about this? E.g., do you work with ordinal data such as Likert-type item or discrete scale scores that are are highly skewed, where you want to find some kind of meaningful quantiles on the underlying scale? What's the purpose of this discretization?" CreationDate="2010-12-16T13:25:43.063" UserId="930" />
  <row Id="8377" PostId="5140" Text="http://en.wikipedia.org/wiki/Leo_Breiman " CreationDate="2010-12-16T14:24:41.730" UserId="88" />
  <row Id="8378" PostId="5559" Text="Thanks for the reply, Let me re-work the question...I am not sure of the correct way to communicate the question and the correct vocab to use..." CreationDate="2010-12-16T14:29:58.810" UserId="2405" />
  <row Id="8379" PostId="5561" Text="@chl +1. Did not know about this Formula feature. Thanks" CreationDate="2010-12-16T15:16:30.320" UserId="2116" />
  <row Id="8380" PostId="5563" Score="2" Text="I would suggest using a symbolic computation package such as Maple or Mathematica." CreationDate="2010-12-16T15:24:38.713" UserId="229" />
  <row Id="8381" PostId="5563" Text="@Baha What would be the use of such a function, i.e. do you want to make symbolic calculus in R?" CreationDate="2010-12-16T15:24:51.747" UserId="930" />
  <row Id="8382" PostId="5446" Text="@whuber Yes, it is! My skept-o-meter says that the details of multiple stories may not be quite right, though. Perhaps both of them solved open problems, but maybe one of them knew the problem was open and the other mistook it for a homework problem. The stories would match in terms of &quot;naive young student solves open problem, not having realized how hard it really was&quot;, but when the story became legend and someone told it as &quot;some student ...&quot;, someone else chimed in, &quot;oh, that was Dantzig&quot;, or &quot;oh, that sounds like Milnor&quot; and the stories converged." CreationDate="2010-12-16T15:44:21.533" UserId="1764" />
  <row Id="8383" PostId="5520" Text="	&#xA;For two balls with probabilities p and 1-p and two draws the expectation equals 4p(1-p), not 1. With n &gt; 2 draws the expectation is np(1-p)(p^(n-2) + (1-p)^(n-2)) because you cannot have two unique balls and the only way to have one unique ball is to draw n-1 of the other color." CreationDate="2010-12-16T16:21:20.147" UserId="919" />
  <row Id="8384" PostId="5566" Score="2" Text="Haven't read the Wikipedia entry (which one?), but: this problem is not fully specified. In the absence of more information I would say you should calculate P(H&gt;=14 OR H&lt;=6), i.e. a two-tailed test (the probability, under the null hypothesis (H=10) that the value deviates by as much or more than observed **in either direction** from the expected value under the null)." CreationDate="2010-12-16T16:31:22.490" UserId="2126" />
  <row Id="8385" PostId="5562" Text="Buy a ticket on the lottery?" CreationDate="2010-12-16T16:47:28.330" UserId="1124" />
  <row Id="8386" PostId="5559" Score="1" Text="@slotishtype The question as it stands is clear enough (although I am wondering how you can characterize merely &quot;ordinal&quot; data as &quot;skewed&quot;).  What we need is a statement of your objectives and how to determine how well any particular binning will meet them.  That is, *why* are you binning?  *How* can that influence the decision or action you ultimately will take based on the data?  It can also help us to understand something about the data generation process: are these data a random sample of something?  A convenience sample?  An entire population?" CreationDate="2010-12-16T16:48:50.887" UserId="919" />
  <row Id="8387" PostId="5545" Score="1" Text="The answer depends (*strongly*) on whether these repeated correlations are independent.  In a common case you have a set of *m* related variables and you test all m(m-1)/2 mutual correlations, which obviously are not independent." CreationDate="2010-12-16T16:53:01.147" UserId="919" />
  <row Id="8388" PostId="5564" Text="Great answer - thanks. This will save me a lot of brain power, time, and - most importantly - from potential errors." CreationDate="2010-12-16T16:54:37.150" UserId="1381" />
  <row Id="8389" PostId="5342" Text="@J. M. I don't think Excel or R are *quite* that bad!  (A standard initial move in any algorithm for erf or erfc is to reduce the problem to a tail probability, subtracting it from 1 afterwards if need be.)  BTW, by &quot;numerically integrating&quot; I didn't mean &quot;numerical quadrature&quot;; I merely meant any method that is a numerical approximation, however that is arrived at. " CreationDate="2010-12-16T17:02:23.230" UserId="919" />
  <row Id="8390" PostId="5568" Text="Perfect - I wasn't sure if something so simple existed or not." CreationDate="2010-12-16T17:30:08.677" UserId="2395" />
  <row Id="8391" PostId="5551" Text="Good analysis, but it leads to computationally intractable algorithms even for moderate numbers of colors and balls.  In some cases that's ok when the approach suggests approximations or asymptotic formulas, but that does not seem to be the case here.  It turns out the essence of this problem concerns the *additivity* of *f*: f(R,G,B,Y) = f(R,G,B) + some function of Y.  If *f* were not additive but could still be computed recursively as f(R,G,B,Y) = g(f(R,G,B), Y) then there might be a dynamic programming solution of complexity O(n*N), which is still pretty efficient." CreationDate="2010-12-16T17:40:19.407" UserId="919" />
  <row Id="8392" PostId="5548" Text="@John nice answer. I can see where you are going with this and I think that it is the 'correct' answer. However, I have chosen not to do this (yet) because of the time it would require since I have developed the model (itself heirarchical) to be very flexible and accept a variety of types of data already, and would need to build this generality in to the extended model as well. So your answer is correct although I would really like to know how to calculate a prior as in a discrete step." CreationDate="2010-12-16T18:16:47.877" UserId="1381" />
  <row Id="8393" PostId="5551" Text="@whuber, I think it is possible to use recursion here. Maybe it is even possible to get analytic solution. I would be pretty hopeful that there is some combinatorical trick which provides it." CreationDate="2010-12-16T19:27:01.123" UserId="2116" />
  <row Id="8394" PostId="5568" Text="@whuber (+1), nice one." CreationDate="2010-12-16T19:28:33.090" UserId="2116" />
  <row Id="8395" PostId="5342" Text="@J. M.: R is open source, so you can find the code pretty easily. " CreationDate="2010-12-16T19:31:12.273" UserId="2116" />
  <row Id="8396" PostId="5551" Text="@mpiktas Yes, recursion works: it leads to the dynamic programming solution.  You won't get a solution that's any simpler than the description of the urn, which involves the array (p_i) of individual probabilities. " CreationDate="2010-12-16T19:48:05.070" UserId="919" />
  <row Id="8397" PostId="5562" Score="1" Text="Can you improve the clarity of the question? It seems you assume everyone knows in detail what your problem is. Otherwise I'm going to close it." CreationDate="2010-12-16T19:58:58.570" UserId="88" />
  <row Id="8398" PostId="5576" Text="I've seen that paper and I don't like it because the measure seems dependent on parameterization, ie, it may rank your models differently depending on whether you use regular or logarithmic units for data" CreationDate="2010-12-16T20:24:04.010" UserId="511" />
  <row Id="8399" PostId="5571" Text="You want to compare the models' *fit* using a statistical test, right? What kind of hypothesis would you like to test?" CreationDate="2010-12-16T20:56:07.783" UserId="1583" />
  <row Id="8400" PostId="5573" Score="1" Text="added R tag for appropriate syntax highlighting.." CreationDate="2010-12-16T20:58:32.027" UserId="696" />
  <row Id="8401" PostId="5571" Text="@Firefeather For example, I would like to test whether the fit of `model.nb.inter` is _significantly_ better than that of `model.pois.inter`. Yes, the AIC is lower, but how much lower constitutes _significantly better_?" CreationDate="2010-12-16T21:00:14.590" UserId="1973" />
  <row Id="8402" PostId="5554" Text="First thanks to all of you for your answers. I think i'm going to try what @mpiktas suggested with the p.adjust() function and holm method in r." CreationDate="2010-12-16T21:03:41.283" UserId="2402" />
  <row Id="8403" PostId="5571" Text="Note: the answer to this question need not actually include the AIC." CreationDate="2010-12-16T21:07:01.390" UserId="1973" />
  <row Id="8404" PostId="5571" Text="I don't know the answer to this question, but I can give a start. I know you can use an $F$ test to compare `model.pois` against `model.pois.inter` (and similarly compare `model.nb` against `model.nb.inter`), but I can't guarantee that comparisons between a Poisson model and a negative binomial model would work. I wonder if an $F$ test to compare the variances of each pair would be reliable." CreationDate="2010-12-16T21:16:51.833" UserId="1583" />
  <row Id="8405" PostId="5571" Text="@Daniel, note that if you're doing more than one comparison between pairs of models, then you'll need to adjust for multiple comparisons (e.g., use [Scheffé's method](http://en.wikipedia.org/wiki/Scheffé's_method))." CreationDate="2010-12-16T21:19:43.500" UserId="1583" />
  <row Id="8406" PostId="5571" Score="1" Text="@Firefeather, yes I'm aware of the need to control the familywise confidence level. Would Scheffe be more appropriate here than, say, Bonferroni?" CreationDate="2010-12-16T21:40:04.240" UserId="1973" />
  <row Id="8407" PostId="5571" Text="@Daniel, I'd have to do some digging to remind myself whether Scheffé can be applied to Poisson or negative binomial models. However, I do know that other methods are typically preferred over the Bonferroni correction, because it is commonly regarded as too conservative (you give up more power than you should have to)." CreationDate="2010-12-16T21:48:28.563" UserId="1583" />
  <row Id="8408" PostId="5578" Text="@shabbychef (+1) Good to know, thanks. I should look again in Michel Tenenhaus's work -- I'll let you know if I find sth interesting." CreationDate="2010-12-16T22:01:28.173" UserId="930" />
  <row Id="8409" PostId="5577" Score="1" Text="@Chase (+1) Shouldn't it be `cut2(x, ...)`? Also, I don't think you need to assign a value to `m`: The only constraint in the question was about the No. groups." CreationDate="2010-12-16T22:08:16.717" UserId="930" />
  <row Id="8410" PostId="5575" Text="@mbq (+1) Huh, `Hmisc`... a really great package!" CreationDate="2010-12-16T22:10:01.220" UserId="930" />
  <row Id="8411" PostId="5570" Text="@whuber (+1) Thanks. A very interesting answer." CreationDate="2010-12-16T22:13:09.720" UserId="930" />
  <row Id="8412" PostId="5549" Text="Just to clarify: by &quot;P(1 Blue)&quot; you must mean &quot;probability of exactly one blue and *no other color appears exactly once*,&quot; etc.  Otherwise the equation is not correct.  Similarly in the second sentence you must restrict to combinations of all other colors where none of them appear exactly once." CreationDate="2010-12-16T22:16:29.133" UserId="919" />
  <row Id="8413" PostId="5572" Score="1" Text="Is treatment randomly allocated? Is allocation concealed? Are the participants blinded to the treatment allocation? How long do you expect it will take for the treatment to have an effect?" CreationDate="2010-12-16T22:24:16.367" UserId="449" />
  <row Id="8414" PostId="5572" Text="@onestop, some good questions. I see that random allocation would have consequences for choosing a model, so let's assume random allocation, though if you have an model which uses self-selection (or ITT) into the treatment, I am very interested in that answer. Otherwise, say allocation is concealed, the participants know whether they are assigned to the control or treatment group, and the treatment should have an effect within several days, if not sooner." CreationDate="2010-12-16T23:05:09.203" UserId="743" />
  <row Id="8415" PostId="5566" Text="I suggest you have a look at &quot;L. D. Brown, T. T. Cai and A. DasGupta, Interval Estimation for a Binomial Proportion, Statistical Science, Vol. 16, pp. 101-117, 2001&quot;. It includes a very nice discussion of interval estimation for the Binomial distribution.&#xA;" CreationDate="2010-12-16T23:20:24.160" UserId="530" />
  <row Id="8416" PostId="5581" Text="+1 Well done!  I'm shocked this hasn't been mentioned yet." CreationDate="2010-12-16T23:28:00.023" UserId="919" />
  <row Id="8417" PostId="5577" Text="@chl - doh! You are right on both accounts. I'm not sure why I thought it was necessary to pass arguments to both `g` and `m`. Code modified accordingly." CreationDate="2010-12-16T23:29:15.117" UserId="696" />
  <row Id="8419" PostId="5548" Text="Makes sense. Might want to add a bit about that to description, so others don't think I have answered the question you're asking." CreationDate="2010-12-16T23:41:49.860" UserId="1146" />
  <row Id="8420" PostId="5573" Text="@Chase - I didn't realize the tag did that. Good to know!" CreationDate="2010-12-17T00:08:13.273" UserId="1583" />
  <row Id="8421" PostId="5560" Text="The terminology you are using in your question is a little bit nonstandard: for example, your title asks for a &quot;likelihood&quot; but your problem description asks for a probability distribution on $(\nu_{1},\nu_{2})$.  In short, a Bayesian solution would start with a (conjugate) Dirichlet prior on $(\nu_{1},\nu_{2})$, then you would observe a multinomial likelihood with $(n_{1},n_{2})$, then the posterior would be Dirichlet again, equivalent to your $F(n_{1},n_{2},n_{3})$, above.  This gives you a density, but it isn't entirely clear from your description that this is what you want." CreationDate="2010-12-17T01:59:27.050" UserId="1108" />
  <row Id="8422" PostId="5560" Text="By the way, the solution outlined above is written out in more detail at the bottom of this other problem: http://stats.stackexchange.com/questions/3194/how-can-i-test-the-fairness-of-a-d20" CreationDate="2010-12-17T02:04:31.287" UserId="1108" />
  <row Id="8423" PostId="5557" Text="So in this particular case, I should calculate a z-score for probabilities of .99 and .01  (i.e. 2.326), then multiply that number by the sampling standard deviation and add or subtract to the mean to get the upper and lower limits?  " CreationDate="2010-12-17T03:28:16.503" UserId="2385" />
  <row Id="8424" PostId="5563" Text="chl, yes i think so." CreationDate="2010-12-17T04:38:58.587" UserId="1700" />
  <row Id="8425" PostId="5563" Text="thank you, james! " CreationDate="2010-12-17T04:40:23.250" UserId="1700" />
  <row Id="8426" PostId="4595" Text="I'd say any model that involves lots of sines/cosines or decaying exponentials would make for badly-behaved problems." CreationDate="2010-12-17T05:27:30.520" UserId="830" />
  <row Id="8427" PostId="5582" Text="Did you remember to subtract out linear trends before FFT'ing your data?" CreationDate="2010-12-17T05:28:47.340" UserId="830" />
  <row Id="8428" PostId="5582" Text="If we're looking at average daily temperature (or average hourly), there couldn't be any linear trends, right?" CreationDate="2010-12-17T06:15:41.320" UserId="1566" />
  <row Id="8429" PostId="5508" Text="@mpiktas, @onestop has it more or less right. The particular example is from a 1993 paper I was reading which used the explicit phrase, &quot;intractible.&quot; I've since closed the PDF, but the history of computation intuitively seems to support my characterization." CreationDate="2010-12-17T07:14:02.440" UserId="53" />
  <row Id="8430" PostId="5586" Score="1" Text="Are you fitting all three models to one data set? Why would youbwant to average those parameters?" CreationDate="2010-12-17T07:14:44.313" UserId="25" />
  <row Id="8431" PostId="5540" Text="Thanks, that's helpful. I'd combine this with @onestop's response for a full answer -- looking into the GHK algorithm provides the math behind what you're explaining. That is, while multidimensional integrals are hard, it doesn't necessarily seem to me that they'd cause extreme issue: we tend to simulate these things anyway, so adding another dimension is just generating one more set of random draws. Looking at how GHK smooths this process, though, helps demonstrate why it takes some time (I think)." CreationDate="2010-12-17T07:18:04.573" UserId="53" />
  <row Id="8432" PostId="5589" Text="Thanks Mike.  That's what I thought.  It's just in common practice in econometrics literature to present several models to show that your result is robust across specifications. Your point about omitted variables is well taken." CreationDate="2010-12-17T07:18:27.473" UserId="2414" />
  <row Id="8433" PostId="5589" Score="1" Text="@Thomas, if it is the same data for all three models, then I do not suggest averaging. The common practice you mention is used  to show that there is no omited variable problem. But since the same data is used to estimate all three models, then you cannot assume independence. If there is no omitted variable bias, then the average will be unbiased, but the standard error of such average cannot be calculated easily, since the coefficients will be clearly correlated." CreationDate="2010-12-17T07:36:21.993" UserId="2116" />
  <row Id="8434" PostId="5583" Score="1" Text="That's why I asked if the treatment is randomly allocated - if it is then the groups would be expected to be balanced for all known and unknown covariates and matching or propensity score approaches aren't likely to be of use." CreationDate="2010-12-17T07:44:08.013" UserId="449" />
  <row Id="8435" PostId="5580" Text="@vqv (+1) Very handy." CreationDate="2010-12-17T07:47:31.027" UserId="930" />
  <row Id="8436" PostId="4138" Text="@gregor if you are not interested in an analytical approach it should be necessary to fix $\Sigma$ and $\mu$  ?" CreationDate="2010-12-17T07:52:51.657" UserId="223" />
  <row Id="8437" PostId="4837" Text="there must be a condition such $&lt;a,b&gt;=0$ missing ? for example if n=2 $a_i=b_i=1$ $X_1+X_2$  and $X_1+X_2$ are not independant." CreationDate="2010-12-17T08:38:41.850" UserId="223" />
  <row Id="8438" PostId="5582" Score="2" Text="Personally I think Fourier analysis is quite an elegant approach and see nothing inelegant about using the first few Fourier terms, but I guess elegance is in the eye of the beholder.." CreationDate="2010-12-17T09:13:34.223" UserId="449" />
  <row Id="8440" PostId="5557" Text="@user2385 You got the idea, but don't forget that the 1% are distributed in both tails so that you have to work with .005 ($Z$ will be $&lt;0$, $X$ will be $&lt;68$) and .995 ($Z$ will be $&gt;0$, $X$ will be $&gt;68$), for the total area exceeding the limit equals 1%. No need to substract to the mean, you already know that $X=Z\mu +\sigma$. Just use the correct $\sigma$ to go back from the std normal to the sampling distribution of your empirical mean." CreationDate="2010-12-17T10:48:04.140" UserId="930" />
  <row Id="8441" PostId="5591" Text="You would still have to model your experiment somehow to be able to compute the likelihood-function." CreationDate="2010-12-17T11:06:02.340" UserId="2036" />
  <row Id="8443" PostId="5592" Text="Thanks for your explanation." CreationDate="2010-12-17T12:30:03.337" UserId="1973" />
  <row Id="8444" PostId="5428" Text="@mpiktas -- I guess the `lm` object is the way to go! Thanks for a great summary... sometimes in the application I get too far from the theory." CreationDate="2010-12-17T13:33:48.930" UserId="1445" />
  <row Id="8445" PostId="5591" Score="5" Text="Pete Dixon wrote an article back in 1998 called &quot;Why scientists value p-values&quot; (http://www.psychonomic.org/backissues/1631/R382.pdf) that might be an informative read. A good follow-up would be Glover &amp; Dixon's 2004 paper on the likelihood ratio as a replacement metric (http://pbr.psychonomic-journals.org/content/11/5/791.full.pdf)." CreationDate="2010-12-17T14:18:39.503" UserId="364" />
  <row Id="8446" PostId="5140" Text="@mbq so we could say something like &quot;for is work on the use of trees in statistics&quot;" CreationDate="2010-12-17T14:31:35.830" UserId="223" />
  <row Id="8447" PostId="5592" Score="3" Text="Actually, most coins are actually very close to fair, and it's hard to come up with a physically plausible way to bias them very much -- see e.g. http://www.stat.columbia.edu/~gelman/research/published/diceRev2.pdf" CreationDate="2010-12-17T14:32:11.207" UserId="2126" />
  <row Id="8448" PostId="5592" Score="2" Text="Being very close to fair is not the same thing as being exactly fair, which is the null hypothesis. I was pointing out one of the idiosyncrasies of hypothesis testing, namely that we often know that the null hypothesis is false, but use it anyway.  A more practical test would aim to detect whether there is evidence that the coin is significantly biased, rather than significant evidence that the coin is biased." CreationDate="2010-12-17T15:13:41.473" UserId="887" />
  <row Id="8449" PostId="5600" Score="2" Text="@csgillespie Thanks for sharing an honest perspective." CreationDate="2010-12-17T15:27:31.490" UserId="1973" />
  <row Id="8450" PostId="5592" Text="BTW thanks for the link to the paper - I'm sure it will be useful.  " CreationDate="2010-12-17T15:29:12.023" UserId="887" />
  <row Id="8451" PostId="5600" Score="2" Text="I'd just like to reinforce the point about the money (for projects where there is little novelty in the statistics)! The other point I'd make is that for projects that do advance the statistics, make sure you put as much effort into helping your collaborator in publishing his paper in stats journals that you would want from him in publishing the primary result in a biology journal. I know from painfull experience that this does not always happen - I have had a couple of projects where the stats paper went unpublished as my collaborator lost interest :-( " CreationDate="2010-12-17T15:55:49.317" UserId="887" />
  <row Id="8452" PostId="5591" Score="2" Text="Mike, that looks suspiciously like a good answer to me.  What's it doing in the comments?" CreationDate="2010-12-17T16:20:05.863" UserId="71" />
  <row Id="8453" PostId="2432" Text="This may be a moot/foolish question, but you say &quot;timing&quot;: is this for use with time series? I believe most of the answers below assume this (&quot;changepoint, etc&quot;), though LOESS can be applied in non-time-series situations, with discontinuities. I think." CreationDate="2010-12-17T16:52:41.593" UserId="1764" />
  <row Id="8454" PostId="5582" Text="I'm not sure of the question. Are you essentially asking for domain-specific knowledge that &quot;Oh yeah, temperatures are modeled as ___&quot;? Or are you asking for help in defining &quot;simple&quot; or &quot;elegant&quot;? In the former case, this may not be the right place. In the latter, it's really your call about whether 3 Fourier terms is &quot;elegant&quot;. (Or if the very simple sine wave with period 1 is &quot;accurate enough&quot;, which is also your call, depending on your intended use.)" CreationDate="2010-12-17T17:02:54.920" UserId="1764" />
  <row Id="8455" PostId="5582" Text="link seems broken" CreationDate="2010-12-17T17:21:02.980" UserId="582" />
  <row Id="8456" PostId="5602" Text="example: get a list of  flickr images with the according metadata. someone sorts them in to two groups (the ones he likes and the one hi does not like so much). then i want to find out if there is possible metadata upon which i could have guessed his choice. " CreationDate="2010-12-17T17:24:12.280" UserId="2423" />
  <row Id="8457" PostId="5604" Score="8" Text="The best thing I think for you to do right now is explain what you're trying to do in general (forget about removing outliers explain what your overall goal is). I highly encourage against removing data unless you know it was somehow recorded incorrectly." CreationDate="2010-12-17T19:11:12.193" UserId="1028" />
  <row Id="8458" PostId="5548" Text="@John I have updated my question, and have proposed an answer. Having pondered this further, I can see why my approach is not ideal, e.g. I would be loosing information by assuming a distribution rather than using the full posterior mcmc chain; on the other hand, my model is relatively insensitive to the choice of priors so it is difficult for me to justify using the alternative approach." CreationDate="2010-12-17T20:11:37.143" UserId="1381" />
  <row Id="8461" PostId="5607" Text="&quot;For the continuous parts where Xi∉D, the flat sections of F correspond to intervals where Xi has 0 probability so they don’t really matter when considering F−1(Yi).&quot;&#xA;That is it! That was a beautiful answer. Thank you." CreationDate="2010-12-17T20:41:44.150" UserId="2399" />
  <row Id="8462" PostId="5500" Text="Good point - smoothing the data is the first problem, since I already have some methods that work for the second part. I'll give that a go and see how it does." CreationDate="2010-12-17T21:06:28.460" UserId="1750" />
  <row Id="8463" PostId="5584" Text="This is along the lines of what I have been thinking. I am considering running a separate  multiple regression model for each subject using all of their pre-treatment data, then putting the result in as covariates in the ANCOVA. But, this seemed a lot like a multilevel model to me. I just can't think of how to structure the model." CreationDate="2010-12-17T21:44:13.360" UserId="743" />
  <row Id="8464" PostId="5572" Text="@mbq Originally, the question was also tagged panel-data and mixed-model. The reason that I included these tags was because the data clearly falls into these categories. I was hoping that someone with experience using those models would have some suggestion. Economists, especially, often work with time-series panel-data data that changes as the result of some event." CreationDate="2010-12-17T21:49:03.743" UserId="743" />
  <row Id="8465" PostId="5572" Text="@Seth Sorry, fixed." CreationDate="2010-12-17T21:51:09.310" UserId="88" />
  <row Id="8466" PostId="5610" Text="This needs correction: the lhs, because it is an expectation, is a number, whereas the rhs is a random variable.  Don't you want to take the expectation of the rhs?" CreationDate="2010-12-17T22:52:12.967" UserId="919" />
  <row Id="8467" PostId="5597" Text="This sounds like CW material." CreationDate="2010-12-17T22:57:44.693" UserId="919" />
  <row Id="8468" PostId="4837" Text="@robin good catch.  I've been puzzling over the implicit quantifiers, too.  Unfortunately, all I have access to is that (titillating) quotation from the review, not the book.  It would be fun to find it in a library and browse through it..." CreationDate="2010-12-17T23:03:37.137" UserId="919" />
  <row Id="8469" PostId="5610" Text="Thanks. I corrected the typo." CreationDate="2010-12-17T23:25:28.680" UserId="1670" />
  <row Id="8470" PostId="5557" Text="Ah I see.  Thanks for all the help, I really appreciate it!" CreationDate="2010-12-17T23:31:41.010" UserId="2385" />
  <row Id="8471" PostId="5582" Text="I believe there is some fundamental simple underlying function here, not just a sum of sine waves. In other words, even if we get Fourier terms, I believe they'll reveal that these temperatures follow a simple, physically sensible formula." CreationDate="2010-12-18T02:00:33.137" UserId="1566" />
  <row Id="8472" PostId="5599" Text="Who was it that said &quot;calling a statistician after an experiment has been completed is like calling a doctor for an autopsy&quot; or something like that?" CreationDate="2010-12-18T08:48:45.893" UserId="1795" />
  <row Id="8473" PostId="5588" Text="Thanks for the derivation of likelihood function. It is clear that intercept term cancels out from the numerator and denominator.  Cheers" CreationDate="2010-12-18T13:37:33.717" UserId="2264" />
  <row Id="8474" PostId="5618" Text="Thanks Dason, that helped.  Also, after reading your reply, it suddenly became clear to me that I am not fully sure how this generalizes in case we are having more factors.  Could you advise?  Thanks again.  Tal" CreationDate="2010-12-18T14:21:25.903" UserId="253" />
  <row Id="8475" PostId="5618" Score="2" Text="You can test multiple contrasts simultaneously.  So for example if A had three levels and B had 2 we could use the two contrasts: C1 = (A1B1 - A2B1) - (A2B1 - A2B2) and C2 = (A2B1 - A2B2) - (A3B1 - A3B2) and use a 2 degree of freedom test to simultaneously test if C1 = C2 = 0.  It's also interesting to note that C2 could equally have been (A1B1 - A1B2) - (A3B1 - A3B2) and we would come up with the same thing." CreationDate="2010-12-18T14:23:50.153" UserId="1028" />
  <row Id="8476" PostId="5617" Score="1" Text="I don't have the reputation to allow me to edit, but I think you want $H_0 = \mu_{A1}=\mu_{A2}$ (or $\mu_{A_1}$ if you want a double subscript) [oops, it has automatically tex-ified that:  `H_0 = \mu_{A1}=\mu_{A2}` or `\mu_{A_1}`]" CreationDate="2010-12-18T15:12:04.290" UserId="2126" />
  <row Id="8477" PostId="5617" Text="Thanks Ben - I see chl already solved it." CreationDate="2010-12-18T15:41:18.987" UserId="253" />
  <row Id="8478" PostId="5617" Text="@Tal Oups, didn't see that you are using capital letters to denote factor name *and* their levels --  fix it (following @Ben notation)." CreationDate="2010-12-18T16:12:34.120" UserId="930" />
  <row Id="8479" PostId="5621" Text="gotcha, thanks for clarifying." CreationDate="2010-12-18T16:16:27.747" UserId="1212" />
  <row Id="8480" PostId="5619" Text="@clare Interpretation of ESs generally have no mathematical/statistical justification." CreationDate="2010-12-18T18:01:53.980" UserId="930" />
  <row Id="8482" PostId="5599" Score="1" Text="@cespinoza : Ronald Fisher : To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of.&#xA;" CreationDate="2010-12-18T19:36:56.380" UserId="1124" />
  <row Id="8483" PostId="5599" Score="1" Text="+1 apart from money, the first and foremost thing I want a researcher to bring me is a hypothesis! and not -again- some garbled dataset and the question : can you get something out of this? Yes I can. No, it won't mean a bloody damn thing." CreationDate="2010-12-18T19:37:55.660" UserId="1124" />
  <row Id="8484" PostId="5612" Score="2" Text="This answer should have been made as a comment to Michael's answer or as an edit to your initial question.  No big deal - just an FYI." CreationDate="2010-12-18T19:40:21.557" UserId="196" />
  <row Id="8485" PostId="5604" Text="Do you expect your 'bad data' to be from those who visited the site many times or those who viewed it few times.  Do you have actual access times?  Can you use those to tell the difference between good data and bad data." CreationDate="2010-12-18T19:41:34.783" UserId="196" />
  <row Id="8486" PostId="5625" Text="+1 Great advice, especially #1.  Concerning #2, be careful not to overestimate what your potential client might know!  (I must have had hundreds of professional conversations in which the client said &quot;I took *x* statistics courses in graduate school but have forgotten everything&quot;; *x* typically is 3 or 4 and the clients were anywhere from zero to 40 years out of school.)" CreationDate="2010-12-18T21:21:28.177" UserId="919" />
  <row Id="8487" PostId="5619" Text="@chl It is possible your remark could be misinterpreted.  I *think* you are saying that one should seek non-mathematical justification of the interpretation of a particular ES, and *not* that you are saying interpreting an ES is altogether unjustified!" CreationDate="2010-12-18T21:30:17.950" UserId="919" />
  <row Id="8488" PostId="5619" Text="@whuber Indeed, I originally meant that interpreting ESs is highly task or domain-dependent (with some general rules of thumb for e.g., Cohen's d or Fleiss' Kappa), and that a good literature review is generally welcome. " CreationDate="2010-12-18T21:52:25.123" UserId="930" />
  <row Id="8489" PostId="5582" Score="1" Text="&quot;Physically sensible&quot; and &quot;simple&quot; may not necessarily go hand in hand. Especially not when describing relatively complicated phenomena like weather. If there was a simple formula for predicting the weather, I'm pretty sure you could get quite rich by discovering it." CreationDate="2010-12-18T22:14:09.880" UserId="2443" />
  <row Id="8490" PostId="5625" Text="@Peter Thanks for the thorough response!" CreationDate="2010-12-19T00:41:00.923" UserId="1973" />
  <row Id="8491" PostId="711" Text="How much wider are robust standard errors than regular standard errors when the assumptions of OLS are not being violated?" CreationDate="2010-12-19T01:47:04.343" UserId="196" />
  <row Id="8492" PostId="5589" Text="+1.  But what would the interpretation of the average of the $\beta_1$ be?  You might just as well average the price of bananas in Costa Rica with the answers to question #1 in the latest Gallup poll.  And the &quot;CI&quot; would require careful interpretation indeed; it certainly doesn't reveal anything about the relationship between $X_1$ and $Y$!" CreationDate="2010-12-19T06:27:41.223" UserId="919" />
  <row Id="8493" PostId="711" Text="Not always wider at all - in fact they can sometimes be narrower. See the latest post on the blog for Angrist &amp; Pischke's book :&#xA;http://www.mostlyharmlesseconometrics.com/2010/12/heteroskedasticity-and-standard-errors-big-and-small/" CreationDate="2010-12-19T07:44:32.550" UserId="449" />
  <row Id="8494" PostId="5547" Text="Thanks &quot;user&quot;.  This doesn't answer my question - but it is a valuable reference for me - thank you." CreationDate="2010-12-19T08:57:42.537" UserId="253" />
  <row Id="8496" PostId="5605" Text="Hmm, two down votes.  If the answer is so bad it would be nice have some commentary. " CreationDate="2010-12-19T10:57:51.740" UserId="1739" />
  <row Id="8497" PostId="4687" Score="1" Text="I may be misunderstanding this question, but it seems that you're asking how to specify a prior on a node value when the node has parents.  But you don't need to.  Only nodes without parents need priors.  You may want priors on some parameters that define the *mapping* from parent(s) to child.  But these are priors on parameters not priors on node values not on values." CreationDate="2010-12-19T13:45:44.123" UserId="1739" />
  <row Id="8499" PostId="5635" Text="I had to munge the data so it was in columns rather than rows, but that's on it's way to working. Thanks." CreationDate="2010-12-19T16:26:54.410" UserId="1531" />
  <row Id="8500" PostId="5635" Text="@sprugman: There should be a setting to choose wether data is in columns or rows" CreationDate="2010-12-19T16:39:36.350" UserId="582" />
  <row Id="8501" PostId="5635" Text="@nico No setting is needed: Select the range of cells in the two rows, invoke the Chart Wizard, specify &quot;XY (Scatter)&quot;, and you have it." CreationDate="2010-12-19T17:02:15.140" UserId="919" />
  <row Id="8502" PostId="5593" Text="The Lebesgue measure of this set is irrelevant.  You need to be looking at the probability that the CDF of a uniform distribution on Y does not agree with the CDF of X.  This probability is still zero.  Take a closer look at the Web page I referenced and think harder about the Bernoulli case: once you understand this (the simplest possible situation) all should be clear." CreationDate="2010-12-19T17:11:19.047" UserId="919" />
  <row Id="8503" PostId="5607" Score="1" Text="+1 for the rigor and for addressing the second question about the EDFs." CreationDate="2010-12-19T17:16:37.560" UserId="919" />
  <row Id="8504" PostId="5635" Text="@whuber: I remember to be able to select rows or columns (well, at least OpenOffice has that option). Of course Excel will choose automagically the longest direction, but that's not always what you want! :)" CreationDate="2010-12-19T17:33:51.180" UserId="582" />
  <row Id="8505" PostId="5543" Score="1" Text="@David Not really an anwer, but I found this cheat sheet useful, [Some Useful Distributions in Bayesian Analysis with Models from Educational Measurement](http://faculty.ksu.edu.sa/69424/Primary/Distributions.pdf) (R.J. Mislevy, 2001) -- it mainly covers BUGS distributions." CreationDate="2010-12-19T21:21:11.660" UserId="930" />
  <row Id="8506" PostId="4837" Text="This feels like a generalization of G. Jay Kerns' (currently #1) answer." CreationDate="2010-12-19T21:53:05.980" UserId="1670" />
  <row Id="8508" PostId="1268" Text="@Amro it would also help to know _why_ the series are of different lengths.  For example, if they represent the trajectory of a pencil during a handwriting task, say the X displacement while writing out a digit, then you might want to align the time series so that they are the same length.  It is also important to know what type of variation you are interested in retaining, and what you are not." CreationDate="2010-12-19T22:02:56.880" UserId="1670" />
  <row Id="8509" PostId="5622" Text="A really impressive answer Caracal - thank you." CreationDate="2010-12-19T22:05:45.640" UserId="253" />
  <row Id="8510" PostId="2723" Text="Yes!  Look before you leap.  Please, look at the data." CreationDate="2010-12-19T22:09:36.317" UserId="1670" />
  <row Id="8511" PostId="5124" Text="But was he _really_ a statistician?" CreationDate="2010-12-19T22:27:36.527" UserId="1670" />
  <row Id="8512" PostId="4727" Score="1" Text="I think that to get good advice you will need to describe your model for us." CreationDate="2010-12-19T22:42:47.683" UserId="1585" />
  <row Id="8514" PostId="2944" Text="See here for a recently developed method for approximating the eigenvectors of a large matrices: http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.ejs/1291126498" CreationDate="2010-12-20T00:09:35.853" UserId="1670" />
  <row Id="8515" PostId="5630" Text="@whuber Well, it was just &quot;Is there a programming language with built-in overloadable operators so I could implement small error propagation?&quot;, so I have moved it to StackOverflow and that's why it was deleted here -- also the OP edited it on SO, so I think it should stay deleted here." CreationDate="2010-12-20T00:28:29.947" UserId="88" />
  <row Id="8516" PostId="904" Text="It’s not just that it is defined on an ordered set.  Otherwise, how would you distinguish time series analysis from functional data analysis?  I agree with @user549 in that it boils down to the types of questions that are asked.  They are specific to the structure of the data." CreationDate="2010-12-20T04:04:22.090" UserId="1670" />
  <row Id="8517" PostId="5639" Text="@Chase - thanks good solution. Actually, I was also thinking along the same lines. I was curious if there is a solution using lapply or (for my liking) using plyr's functions (summarize,etc). (I know lapply won't necessarily improve the speed)" CreationDate="2010-12-20T04:06:59.000" UserId="1307" />
  <row Id="8518" PostId="812" Text="Smoothing _is_ a big part of functional data analysis, and it can be converted into a vector mean estimation problem by projection onto an appropriate basis (e.g. Fourier or wavelet), but there are other problems in functional data analysis depending on the functional structure that don’t translate as easily.  Take for example functional regression where you are interested in predicting a functional response from covariates." CreationDate="2010-12-20T04:11:15.893" UserId="1670" />
  <row Id="8519" PostId="5639" Score="2" Text="@suncoolsu - Have you seen `count()` from `plyr`? I wasn't aware of it previously, but your comment turned me down a path to finding it. Could be another viable alternative for you." CreationDate="2010-12-20T04:17:41.587" UserId="696" />
  <row Id="8521" PostId="5603" Text="Your example still sounds like a first order Markov chain." CreationDate="2010-12-20T04:20:26.343" UserId="1670" />
  <row Id="8522" PostId="967" Text="This needs to be corrected.  It was Hal Varian that said it. http://www.nytimes.com/2009/08/06/technology/06stats.html" CreationDate="2010-12-20T04:23:04.797" UserId="1670" />
  <row Id="8524" PostId="2698" Text="+1 for the geometric explanation." CreationDate="2010-12-20T04:49:14.567" UserId="1670" />
  <row Id="8526" PostId="1150" Score="2" Text="+1 The extremely pathological case where $X = Z$ highlights this further.  $Y = b_0 + b_1 X + b_2 Z + e$ and $Y = b_0 + (b_1 + b_2) X + 0 Z + e$ would be indistinguishable." CreationDate="2010-12-20T06:41:04.350" UserId="1670" />
  <row Id="8527" PostId="5401" Text="Thanks for the comments! I am studying it. But please also have a look at my answer!" CreationDate="2010-12-20T07:27:07.073" UserId="2344" />
  <row Id="8528" PostId="5395" Text="@Jeromy: Please comment on my Bayesian equation. Thanks!" CreationDate="2010-12-20T07:28:20.223" UserId="2344" />
  <row Id="8530" PostId="5591" Text="John D Cook posted an excellent answer to a question of mine, which i think you would find interesting: http://stats.stackexchange.com/questions/1164/why-havent-robust-and-resistant-statistics-replaced-classical-techniques" CreationDate="2010-12-20T08:12:15.477" UserId="438" />
  <row Id="8531" PostId="5628" Text="Thanks a lot Ben.  This looks like it will point me in the right direction.  I just need to spend some time reading through the help and related articles for MCMCglmm to see if I can wrap my head around what is happening." CreationDate="2010-12-20T08:26:35.490" UserId="196" />
  <row Id="8532" PostId="4359" Text="Geary, R. C. (1936), “The Distribution of ‘Student’s’ Ratio for Non-Normal Samples,” Journal of the Royal Statistical Society, Suppl. 3, 178–184." CreationDate="2010-12-20T09:11:23.793" UserId="1670" />
  <row Id="8533" PostId="5622" Text="@caracal +1 - good and detailed explanation" CreationDate="2010-12-20T10:38:42.493" UserId="1307" />
  <row Id="8534" PostId="5639" Score="2" Text="Well, `sort` inside `unique` is redundant; removing it gives me 3x speedup of `table2` on this data." CreationDate="2010-12-20T12:49:06.297" UserId="88" />
  <row Id="8535" PostId="5648" Text="@mbq I will try that. Many thanks" CreationDate="2010-12-20T13:03:04.070" UserId="2264" />
  <row Id="8536" PostId="5639" Text="@mbq - you are right, we can `sort` outside `unique`. We do need sort though as `unique` would return `5 1 4 3 2` for this data so would not give the right results without sorting them...though it's obviously much cheaper to sort 5 values vis a vis 1M values. Nice catch." CreationDate="2010-12-20T13:10:31.707" UserId="696" />
  <row Id="8537" PostId="5639" Text="@Chase good point that it is indeed required for integers; for factors, you just have to copy `levels` to the output of `tabulate`. " CreationDate="2010-12-20T13:18:42.887" UserId="88" />
  <row Id="8538" PostId="5648" Score="3" Text="@mbq `extractAIC()` seems to work too (e.g., `extractAIC(m)[2]`)." CreationDate="2010-12-20T14:29:57.597" UserId="930" />
  <row Id="8539" PostId="1286" Text="I'm sure you have some reason for doing this in R, but wouldn't a low-level language like C be more appropriate if you are looking for speed? It would be nastier to code, but would run faster, no doubt." CreationDate="2010-12-20T14:45:26.710" UserId="1118" />
  <row Id="8540" PostId="5630" Text="@mbq Fair enough.  Although the question was not well formulated, between it and its title it appeared there was interest in error propagation, which would seem to fit with the interests of a stats site." CreationDate="2010-12-20T15:17:30.000" UserId="919" />
  <row Id="8541" PostId="5382" Text="@Nayan I cannot find any information in your question that relates &quot;work&quot; to &quot;rating&quot;.  As far as I can tell, &quot;work&quot; is something &quot;allotted&quot; to a worker and &quot;work done&quot; could mean--well, anything.  Without such a connection it's impossible to justify any mechanism to use the &quot;W&quot; and &quot;WD&quot; values to modify the self-rating data." CreationDate="2010-12-20T15:23:09.693" UserId="919" />
  <row Id="8542" PostId="5644" Score="1" Text="I cannot tell whether it's right because I find it meaningless: although you call this formula &quot;Bayesian&quot; you haven't provided a probability model, you haven't defined a &quot;team&quot;, and you haven't (yet) formulated a clear question." CreationDate="2010-12-20T15:24:16.950" UserId="919" />
  <row Id="8543" PostId="5635" Text="Numbers is pretty limited (at least the ~08 version I was using). I basically had to put the data in one row like: 5, 20, 3, 30, 2, 10, 5, 5. It was dumb, but I got the result I wanted." CreationDate="2010-12-20T17:03:51.930" UserId="1531" />
  <row Id="8544" PostId="5124" Text="That's a tough one...when did statistics become a real field? Many of the fathers of stats were not statisticians. " CreationDate="2010-12-20T18:23:52.967" UserId="74" />
  <row Id="8545" PostId="5433" Text="Where can one find this package?" CreationDate="2010-12-20T19:28:07.463" UserId="1084" />
  <row Id="8546" PostId="5433" Text="@Adam: I added a link to zoo in my answer.  That page also has a link to zoo's R-forge project page." CreationDate="2010-12-20T19:51:38.893" UserId="1657" />
  <row Id="8547" PostId="3405" Text="@Matt its only infuriating if you are &quot;storing&quot; data in Excel. Its a great feature when using Excel purely as a front end for reporting and chart creation." CreationDate="2010-12-20T22:02:03.123" UserId="1577" />
  <row Id="8548" PostId="3399" Text="@Gaetan, great points, good to hear someone using it the right way, and benefiting." CreationDate="2010-12-20T22:09:10.433" UserId="1577" />
  <row Id="8549" PostId="5650" Score="2" Text="What is a &quot;LOESS/ARIMA model&quot;? Do you meant a nonlinear time trend with ARIMA errors where the nonlinear trend is estimated using LOESS?" CreationDate="2010-12-20T22:33:35.577" UserId="159" />
  <row Id="8550" PostId="711" Text="+1, with @onestop's caveat in comment above that robust se's could very well be bigger or smaller, though typically we expect them to be bigger and thus &quot;conservative&quot; wrt type I error. And yes, I always use either heteroskedastic robust or cluster robust se's in my work, as does everyone I know. " CreationDate="2010-12-20T22:39:45.990" UserId="96" />
  <row Id="8551" PostId="5654" Score="2" Text="It's a great question.  The optimal solution will depend on some additional things, including (a) the principal elements of the cost, including the cost of including one farm and the cost of measuring one subject; and (b) the experimental design.  E.g., will both the treatment and control solutions be applied at each farm (a good choice, but not without its potential problems) or will you be able to apply only one solution at each farm?  Will subjects be clustered (physically) within farms or treated and sampled truly at random?" CreationDate="2010-12-20T22:47:06.967" UserId="919" />
  <row Id="8552" PostId="4687" Text="I think I'm missing a subtlety in the last part of your comment, but also think you may be close to making sense out of my question and resolving my confusion.  With a fully discrete network, my understanding is that the prior probability of a child node is specified in terms of its parents' combined states.  How this notion of dependence carries forward to a network with continuous variables is what confuses me. Does that give you enough to elaborate?  Is there a simple two node (parent -&gt; child) network that exemplifies this kind of 'mapping'?" CreationDate="2010-12-20T23:06:58.403" UserId="1474" />
  <row Id="8553" PostId="5652" Text="@gregr: Thanks! That helps a lot." CreationDate="2010-12-21T00:51:00.930" UserId="2040" />
  <row Id="8555" PostId="5657" Text="Thanks for the observation, John. I have changed it a little to make it a factor. All I did was remake the original vector from the table though, so if there's a way to skip that step, this will be faster." CreationDate="2010-12-21T03:11:24.053" UserId="1118" />
  <row Id="8556" PostId="5633" Text="if i understand correctly what you wrote, wouldn't $D_n^{(r)}$ be the same for all $r$? also - i can see how to get a monte-carlo estimated critical value for $D_n$; but how about for $D_n^{(r)}$?" CreationDate="2010-12-21T03:36:37.917" UserId="1112" />
  <row Id="8557" PostId="5592" Text="Hi there, maybe I am mistaken but I thought in science, you can never say that the alternative hypothesis is true, you can only say that the null hypothesis is rejected and you accept the alternative hypothesis. To me the p value reflects the chance you will make a type 1 error, i.e. that you will reject the alternative hypothesis and accept the null hypothesis (say p=.05 or 5% of the time. It is important to distinguish between type 1 error and type 2 error, and the role that power plays in your modelling of events. " CreationDate="2010-12-21T07:06:47.437" UserId="2238" />
  <row Id="8558" PostId="5613" Text="Brett, this is interesting and a great example. The model here to me seems to be that people expect the order of heads and tails to occur in a random fashion. For example, if I see 5 heads in a row, I infer that this is an example of a non-random process. In fact, and I may be wrong here, the probability of a toin coss (assuming randomness) is 50% heads and 50% tails, and this is completely independent of the previous result. The point is that if we threw a coin 50000 times, and the first 25000 were heads, provided the remaining 25000 were tails, this still reflects a lack of bias   " CreationDate="2010-12-21T07:26:32.933" UserId="2238" />
  <row Id="8559" PostId="5522" Text="@chl can you pls tell which useR was this?" CreationDate="2010-12-21T08:39:08.157" UserId="1307" />
  <row Id="8560" PostId="5522" Text="@suncoolsu [UseR! 2006](http://www.r-project.org/useR-2006/)." CreationDate="2010-12-21T09:00:30.340" UserId="930" />
  <row Id="8561" PostId="5644" Text="Then can you suggest how to clearly declare such things properly and what is missing in my question which is not clear? Thx!" CreationDate="2010-12-21T10:11:10.193" UserId="2344" />
  <row Id="8562" PostId="5382" Text="@Whuber: Though I think I explained it the best I could, I'll try again to define relationship between W and WD. W is work given to a team member, WD is the work done by him. W &gt;= WD &gt;= 0 (in a given component). I'm trying to formulate a method which can take this info in account and use his self-rating to find out his (normalized) rating in the team." CreationDate="2010-12-21T10:14:47.943" UserId="2344" />
  <row Id="8564" PostId="5662" Text="@CHI Thanks. I have studied the caret package and have used it to tune meta parameters. very useful!." CreationDate="2010-12-21T13:03:16.817" UserId="2040" />
  <row Id="8565" PostId="5662" Text="@chl +1, nice one. I wrote my function solely because code a[levels(a) %in% lf] &lt;-&quot;Other&quot; does not work, so I assumed that factor level change is complicated affair. As usual it turned out, that R is not complicated, I am :)" CreationDate="2010-12-21T13:04:19.987" UserId="2116" />
  <row Id="8566" PostId="5650" Text="sorry I mean LOESS or ARIMA model. e.g I use LOESS to find the the residuals as:   residuals(loess(x ~ time)).  because the data x is vector with 1000 values, which updates every 15 minutes.  How can I efficiently get the residuals, but not rerun the whole datset as input everytime?  as only 1 value update everytime, the other 999 values is still same as last time.&#xA;" CreationDate="2010-12-21T13:16:40.080" UserId="2454" />
  <row Id="8567" PostId="5665" Text="In particular, 10%, 20%, 30% etc. are called Deciles." CreationDate="2010-12-21T13:41:18.463" UserId="582" />
  <row Id="8568" PostId="5665" Text="I've read about percentiles, quartiles and deciles, but they did not quite seem to fit the bill. Maybe I've misunderstood? I thought that a percentile was a specific value, like that the median is really the 2-quantile. I'm looking for a name for a certain range of values. What I'm looking for is to tell people something like &quot;in the ordered list of mean scores for all the talks, your talk ended up in this 'box'&quot;." CreationDate="2010-12-21T14:01:27.003" UserId="2470" />
  <row Id="8569" PostId="5657" Text="Thanks to everyone who responded. My R is weak but the ability to do this with so few lines of code is testament to how powerful it is and makes me want to learn. " CreationDate="2010-12-21T14:16:25.880" UserId="2040" />
  <row Id="8570" PostId="5665" Text="Deciles are then what you need. Households for example are divided into deciles according to their income. Poorest are in the first, richest in tenth. " CreationDate="2010-12-21T14:20:47.933" UserId="2116" />
  <row Id="8571" PostId="5633" Text="@ronaf Could you provide more detail about $D_n^{(r)}$?  What is R?  I don't see that permuting the categories does anything at all: notice that no permutation will change the sum of absolute differences of their counts." CreationDate="2010-12-21T14:30:13.353" UserId="919" />
  <row Id="8572" PostId="5666" Text="@Mike Anderson Good start (+1).  The formula needs some fixing.  The variance of the difference of the proportions equals p0(1-p0)/n0 + p1(1-p1)/n1.  With n0 = n1 = n and p0 = .5, p1 = .2, that equals 0.41/n, implying n = 41 z^2.  Note, too, that this is a one-sided test, so z = 1.65 works fine.  (Some precision is needed here because the result is sensitive to the squaring of z.)  Regardless, these calculations establish that approximately 10^2 *independent* subjects will need to be tested *if* this model is correct.  (I do *not* expect the bleach or the new formula to have &quot;random&quot; effects.)" CreationDate="2010-12-21T14:39:05.797" UserId="919" />
  <row Id="8573" PostId="5665" Score="2" Text="@mpiktas Right.  So to give a clear, final answer to the original question, we might say the first category is the &quot;top (i.e., tenth) decile,&quot; the next category consists of the &quot;eighth and ninth deciles,&quot; the third category is the &quot;fifth through seventh deciles,&quot; and the last category contains the &quot;lowest four deciles&quot; (or, to be more circumspect, is &quot;not in the top six deciles&quot; or even &quot;did not make it in the top 60%&quot;)." CreationDate="2010-12-21T14:52:48.250" UserId="919" />
  <row Id="8574" PostId="5666" Text="Mike Anderson and Whuber, thank you for your suggestions.  You asked good questions, which I'll try to answer.  The poultry experimental units will be random, not from clusters.  As of now, cost is not a consideration." CreationDate="2010-12-21T14:55:55.090" UserId="2473" />
  <row Id="8575" PostId="5659" Text="... and then, of course, we conclude $|\rho| \le 1$.  This illustrates the answer provided by @vqv.  @ronaf: why didn't you upvote @vqv's answer then??" CreationDate="2010-12-21T15:15:03.940" UserId="919" />
  <row Id="8576" PostId="5662" Text="@mpiktas Thx. You can work at the vector level with e.g., `a[as.character(a) %in% lf] &lt;- lf[1]; a &lt;- factor(droplevels(a), labels=c(&quot;Other&quot;,LETTERS[3:5]))`." CreationDate="2010-12-21T15:27:38.873" UserId="930" />
  <row Id="8577" PostId="5645" Text="@vqv (+1) Thanks for this, especially Friedman's paper. I'll come back later with additional comments." CreationDate="2010-12-21T15:30:05.797" UserId="930" />
  <row Id="8579" PostId="5630" Text="@mbq A clarification posted on the SO version makes it abundantly clear that the OP is about error propagation.  There is no longer any question that this is relevant to stats.  Apparently there's no longer any option for us to vote to migrate it back, but it definitely belongs here and should be re-opened." CreationDate="2010-12-21T16:12:30.660" UserId="919" />
  <row Id="8580" PostId="1286" Text="@Christopher: Yes, it probably would be faster in C but since the outputs are being used in R it seems clearer from a reproducible research standpoint to use only R code.  Moreover, (the real answer) I'm more comfortable in R than C and haven't learned how to call C code from R yet." CreationDate="2010-12-21T17:02:54.573" UserId="196" />
  <row Id="8581" PostId="5054" Text="... or maybe the question is asking whether models of heteroscedasticity exist, such as GARCH." CreationDate="2010-12-21T17:28:30.517" UserId="919" />
  <row Id="8582" PostId="5664" Text="Please pay close attention to dmk38's comments, below. *Who* is rating each speaker, and how much variability there is in each speakers' ratings make a difference. That 4.5-rated person has a higher mean than someone rated, say 4.3, but depending on whether their audiences had significantly different personal grading criteria and how much variation in grades there was around each mean, there may be no real-life difference between the two. The numbers will look authoritative, since you ran them through a computer, but the question is: are they saying what you think they're saying?" CreationDate="2010-12-21T17:32:42.900" UserId="1764" />
  <row Id="8583" PostId="4172" Text="I'm just a learner around here, but I noticed that you say that you matched up the &quot;closest geographically&quot; point from setup 2 to setup 1, seemingly without knowing that the two devices actually were at the same location when the respective measurements were taken. That seems to my naive eye to be cheating in favor of setup 2. I defer to my statistical superiors on that question." CreationDate="2010-12-21T17:46:43.577" UserId="1764" />
  <row Id="8585" PostId="5644" Text="@Nayan: Yes: please provide a probability model and define terms like &quot;team&quot; and &quot;effective rating.&quot;  If we have to guess at the meanings our chances of answering the intended question go down.  (Normally it's too much to ask for an explicit probability model, but since you've already done a probability calculation you must have one!)" CreationDate="2010-12-21T18:31:37.020" UserId="919" />
  <row Id="8586" PostId="5662" Text="+1. a[levels(a) %in% lf] &lt;-&quot;Other&quot; sure saves a ton of code lines. Clever and efficient!" CreationDate="2010-12-21T18:40:09.760" UserId="1118" />
  <row Id="8587" PostId="5630" Text="@whuber I still think it is more about operator overloading than about error propagation (not to mention what I think about error propagation), but you have it opened again. " CreationDate="2010-12-21T18:41:42.583" UserId="88" />
  <row Id="8589" PostId="5630" Text="@mbq Thank you!" CreationDate="2010-12-21T18:43:25.473" UserId="919" />
  <row Id="8590" PostId="5629" Text="@mankoff On the SO version of this question (which I can no longer find), you had a link to an &quot;error propagation&quot; site.  Could you provide that here?" CreationDate="2010-12-21T18:46:18.983" UserId="919" />
  <row Id="8592" PostId="5259" Text="This is practically identical to http://stats.stackexchange.com/questions/726/famous-statistician-quotes/2044#2044 but has been attributed to a different person!  Who's right?" CreationDate="2010-12-21T18:49:57.643" UserId="919" />
  <row Id="8593" PostId="5259" Text="Google searches suggest, by 20 to 1, that Easterbrook originated this quotation, but he didn't really start writing until after Coase was quoted in print.  The best evidence I can find concerning this (and it's still not very good) is Coase's Wikipedia page, http://en.wikipedia.org/wiki/Ronald_Coase ." CreationDate="2010-12-21T18:58:58.760" UserId="919" />
  <row Id="8594" PostId="5629" Text="Found the SO version!  http://stackoverflow.com/questions/4483266/implementing-error-propagation" CreationDate="2010-12-21T19:03:36.400" UserId="919" />
  <row Id="8595" PostId="5670" Text="I only have a german text-book as a reference for it, but the authors state that a) Spearman's $r_{S}$ systematically overestimates true $\rho$ for normally-distributed data, and that b) $r_{S}$ is difficult to interpret when not near 0, 1 or -1. (Büning H &amp; Trenkler G. 1994. Nichtparametrische statistische Methoden. 2. Aufl. Berlin: de Gruyter. With regards to a), they also state that typically, Kendall's $|\tau| &lt;$ Spearman's $|r_{S}|$." CreationDate="2010-12-21T19:15:05.250" UserId="1909" />
  <row Id="8596" PostId="5670" Text="I edited too slowly... With regards to a), B&amp;T cite Walter (1963) and also state that $r_{S}$ is only nonparametric under independency. Walter E. 1963. Rangkorrelation und Quadrantenkorrelation. Züchter Sonderhefte 6: Die Frühdiagnose in der Züchtung und Züchtungsforschung II, 7-11." CreationDate="2010-12-21T19:24:17.833" UserId="1909" />
  <row Id="8597" PostId="5665" Text="But the wikipedia article states that a decile is &quot;any of the nine values that divide the sorted data into ten equal parts, so that each part represents 1/10 of the sample or population&quot; - thus a single value, not a range. The R function ´quantile´ also returns a single value. Is it not more correct to say for instance &quot;between the 8th and 9th decile&quot;?" CreationDate="2010-12-21T19:29:57.490" UserId="2470" />
  <row Id="8598" PostId="5115" Text="If it weren't CW it would have to be closed as subjective and argumentative!" CreationDate="2010-12-21T19:37:00.287" UserId="919" />
  <row Id="8599" PostId="5668" Text="The conference had almost 500 participants, at all time divided into three tracks. Not all of our participants gave us ratings, of course, not all participants were at a talk at all times, and they were not evenly distributed among the tracks. I'd say we have about 50% feedback rate, based on the numbers. Some tracks has as much as ~89 feedbacks, some as low as ~22. Since I do not have full access to the raw data, I cannot do stuff like eliminate outliers etc. When will the data be unsafe to draw conclutions from?" CreationDate="2010-12-21T19:45:17.407" UserId="2470" />
  <row Id="8600" PostId="5676" Text="ok thanks, stating the statistic was a mistake. I meant z or t table." CreationDate="2010-12-21T19:46:53.867" UserId="1395" />
  <row Id="8601" PostId="5675" Text="Is this what you're looking for? http://www.weibull.com/DOEWeb/confidence_intervals_in_simple_linear_regression.htm" CreationDate="2010-12-21T19:51:02.257" UserId="1118" />
  <row Id="8602" PostId="5629" Text="Copied SO version; I'll ask mmyers to remove it from SO." CreationDate="2010-12-21T20:03:01.953" UserId="88" />
  <row Id="8603" PostId="5629" Text="K​a​b​l​a​m​m​o." CreationDate="2010-12-21T20:39:37.457" UserId="877" />
  <row Id="8604" PostId="5668" Text="I don't grasp the layout fully. But if you had groups of evaluators judging multiple speakers, you should perform an appropriate inter-rater reliability test (e.g., ICC) for each group. If that test shows sufficiently high agreement among evaluators--and you believe they were competent to judge (presumably; why else ask? still you are assuming the validity of your measures)--compute the mean scores &amp; standard errors for each speaker. You must use judgment at that point to decide how much imprecision you are comfortable with in making your rankings--much as you would when grading on a curve." CreationDate="2010-12-21T20:42:52.650" UserId="2347" />
  <row Id="8605" PostId="5630" Text="@whuber and others... Thank you. Sorry for the poorly worded question/title. I debated here v. SO, as the question is both about error propagation AND software implementation of it. Anyway, this is an excellent answer. Thanks. " CreationDate="2010-12-21T20:44:34.300" UserId="957" />
  <row Id="8606" PostId="5671" Text="@Mohit, I assume that you meant the steps reversed. But if this is so simple why do we need Gibbs sampling? " CreationDate="2010-12-21T20:52:05.417" UserId="2116" />
  <row Id="8607" PostId="5677" Score="1" Text="It seems like a contradiction..." CreationDate="2010-12-21T20:58:10.147" UserId="88" />
  <row Id="8608" PostId="5665" Text="@mranders, decile means both the single value and the range. So you can say both, between 8th and 9th decile or in 9th decile." CreationDate="2010-12-21T20:58:30.887" UserId="2116" />
  <row Id="8609" PostId="5662" Text="But note that a[a==&quot;a&quot;] &lt;- &quot;Other&quot; will not work, which for me is quite natural to assume that it should. Especially since a[a==&quot;a&quot;] is perfectly valid." CreationDate="2010-12-21T21:04:01.210" UserId="2116" />
  <row Id="8610" PostId="5662" Text="@Christopher, a[levels(a) %in% lf] &lt;-&quot;Other&quot; will not work, levels(a)[levels(a) %in% lf] &lt;-&quot;Other&quot; on the other hand will." CreationDate="2010-12-21T21:06:06.777" UserId="2116" />
  <row Id="8611" PostId="5677" Score="1" Text="@Firefeather Don't you mean to say &quot;**do** use&quot; rather than &quot;don't use&quot;??" CreationDate="2010-12-21T21:07:05.717" UserId="919" />
  <row Id="8612" PostId="5677" Score="1" Text="It definitely seems like the poster made a mistake here." CreationDate="2010-12-21T21:07:41.810" UserId="1028" />
  <row Id="8613" PostId="5677" Text="@whuber: Oops! Thanks for the catch. (Darn it, my reputation is headed the wrong direction now!)" CreationDate="2010-12-21T21:19:37.047" UserId="1583" />
  <row Id="8614" PostId="5677" Text="@Firefeather I just sent it back in the correct direction ;-)." CreationDate="2010-12-21T21:23:14.047" UserId="919" />
  <row Id="8615" PostId="5677" Text="@whuber: Much obliged. It's quite embarassing that I have been able to contribute little to Cross Validated (compared to other Stack Exchange sites), considering I got my undergrad degree in the university's department of statistics. ;-)" CreationDate="2010-12-21T21:27:14.153" UserId="1583" />
  <row Id="8616" PostId="5677" Text="@Firefeather Be patient. Simple mistakes are no problem; as you've seen, you can edit them ;-).  But sooner or later you'll find some questions where it's obvious you have something valuable and special to contribute.  I look forward to seeing them." CreationDate="2010-12-21T21:49:10.720" UserId="919" />
  <row Id="8617" PostId="5679" Text="(+1) It seems to me this advice is complementary rather than countervailing.  After all, using Student's t matters when the degrees of freedom are small and that's usually where you can't detect heteroscedasticity anyway.  So with lots of data you can check assumptions and protect yourself somewhat with robust SEs and, furthermore, you won't really care whether you're using t tables or z tables." CreationDate="2010-12-21T21:54:16.813" UserId="919" />
  <row Id="8618" PostId="5680" Score="1" Text="Could you link to some of those threads?  My gut instinct is &quot;NOOO no no no&quot;, but I'm hardly an expert and I'd be interested in reading some of those arguments." CreationDate="2010-12-21T21:54:31.573" UserId="71" />
  <row Id="8619" PostId="5680" Score="1" Text="You sure can't trust any p-values derived from F distributions with those kinds of data!" CreationDate="2010-12-21T21:56:42.740" UserId="919" />
  <row Id="8620" PostId="5677" Text="@whuber: Why, thank you! It may come easier once I start back at school again to get a PhD in Statistics, too." CreationDate="2010-12-21T22:15:52.380" UserId="1583" />
  <row Id="8621" PostId="5680" Score="2" Text="Many cite the ANOVA's robustness as justification for using it with non-normal data. IMHO, robustness is not a general attribute of a test, but you have to precisely state a) against which violations of its assumptions a test is robust (normality, sphericity, ...), b) to what degree these violations have no big effect, c) what the prerequisites are for the test to show robustness (large &amp; equal cell size ...). In your split-plot design, I'd love to have somebody state the precise assumptions of sphericity &amp; equality of covariance matrices. It's already mind-boggling in the 2-factorial case." CreationDate="2010-12-21T22:26:45.443" UserId="1909" />
  <row Id="8622" PostId="5670" Text="@caracal I guess Siegel's book is also a good reference. I also know that Kendall's $\tau$ has better statistical properties (but this has already been discussed in previous Q&amp;A). But it seems to me that the question has to do with the interpretation of an ES, and although we're not informed of the sample size, it is likely to be small. Anyway, your proposed transformation should work fine and there exist transformation for converting an $r$-based measure into a Cohen's $d$ one. But as you didn't address directly the question, I wasn't sure of where you want to go. Now, I've +1 your response." CreationDate="2010-12-21T22:32:57.727" UserId="930" />
  <row Id="8623" PostId="5662" Text="@mpiktas `a[a==&quot;a&quot;]` is _subsetting_; `a[a==&quot;A&quot;] &lt;- &quot;B&quot;` would be valid because &quot;B&quot; is one of the level of `a`, but not &quot;Other&quot;. `a[as.character(a) %in% lf]` is _indexing_ (we create a boolean vector of length `length(a)`). This is the vector view. Now, using `levels()` or `relevel()` allows to directly work with attributes of `a` (check `attr(a, &quot;levels&quot;)`)." CreationDate="2010-12-21T22:44:26.387" UserId="930" />
  <row Id="8624" PostId="5670" Text="Thanks! Actually, I deliberately tried to answer only the first part of the question (significance test of 0.33), since I wasn't sure about using $\rho$ as an ES measure." CreationDate="2010-12-21T22:48:05.973" UserId="1909" />
  <row Id="8625" PostId="5670" Text="@caracal Ok, I mostly focused on the other point. The references I gave are mainly opposing the Hypothesis Testing approach *vs.* interpretation of ES measures. I still prefer giving confidence intervals for $\hat r$ or $\hat\rho$, but anyway I hope the asker would find good directions for further investigations." CreationDate="2010-12-21T22:55:26.573" UserId="930" />
  <row Id="8626" PostId="5680" Text="Some thoughts: is there any way you can aggregate any of your groups (e.g. averaging individual values) and still do meaningful analyses? [Murtaugh, Paul A. 2007. Simplicity and Complexity in Ecological Data Analysis. Ecology 88, no. 1: 56-62. http://www.esajournals.org/doi/abs/10.1890/0012-9658%282007%2988%5B56%3ASACIED%5D2.0.CO%3B2] I would be quite worried about a situation with 90% of observations equal to zero ..." CreationDate="2010-12-21T22:56:33.883" UserId="2126" />
  <row Id="8627" PostId="5662" Text="Put differently, you can't change the levels of an existing factor by changing its (vector) elements.[1] I think factors are a bit harder to grasp precisely because attributes usually don't play such a big role in working with R, but with factors, they often need to be explicitly manipulated. [1] Ok, maybe `drop=TRUE` for subsetting counts." CreationDate="2010-12-22T00:39:54.143" UserId="1909" />
  <row Id="8628" PostId="5681" Score="1" Text="Are you asking for multivariate hypothesis tests? For your first question, if you're looking for a multivariate equivalent to a t-test, you might consider Hotelling's T-Squared. For the second question, are you trying to compare k groups? Is MANOVA appropriate?" CreationDate="2010-12-22T03:08:51.987" UserId="1118" />
  <row Id="8629" PostId="5659" Text="@whuber - on further consideration, i realize that my answer - to be useful - requires knowing the covariances between the coordinates of $x_1$ and $x_2$ [$\Sigma_{12}$, in @vqv's notation]. $\Sigma_{12}$ is not mentioned specifically in the OP's question - so perhaps it should not be thought of as part of the known 'data' for the problem [a point that managed to escape my notice, i'm afraid]. in that case, @vqv's answer - and yours - are certainly more germane. i am upvoting both of your answers - and thanx for making me stare a bit harder at the issues involved." CreationDate="2010-12-22T04:23:49.250" UserId="1112" />
  <row Id="8631" PostId="54" Text="For cross-reference: it was also [asked](http://math.stackexchange.com/questions/15098) @ m.SE ..." CreationDate="2010-12-22T07:00:40.410" UserId="830" />
  <row Id="8632" PostId="5686" Score="3" Text="I'd say NA is more of a &quot;placeholder&quot;; NaN is for (IEEE) arithmetic purposes." CreationDate="2010-12-22T07:04:17.460" UserId="830" />
  <row Id="8633" PostId="5686" Text="@JM. good way to summarize. " CreationDate="2010-12-22T07:09:59.047" UserId="1307" />
  <row Id="8634" PostId="5662" Text="@chl,  do not follow, how a==&quot;a&quot; is different from as.character(a) %in% &quot;a&quot;? They both create boolean vectors of length length(a)." CreationDate="2010-12-22T07:20:56.040" UserId="2116" />
  <row Id="8635" PostId="5686" Text="and there is also Inf, which stands for expressions like for instance 1/0..." CreationDate="2010-12-22T08:43:34.067" UserId="573" />
  <row Id="8636" PostId="5592" Score="1" Text="For frequentist tests, I would use an even weaker statement, which is that you either &quot;reject the null hypothesis&quot; or you &quot;fail to reject the null hypothesis&quot;, and don't accept anything.  The key point being that (as in the case of the biased coin) sometimes you know a-priori that the null hypothesis is not true, you just don't have enough data to demonstrate that it isn't true; in which case it would be odd to &quot;accept&quot; it.  Frequentist tests have type-I and type-II error rates, but that doesn't mean that they can talk of the probability of a particular hypothesis being true, as in the OP. " CreationDate="2010-12-22T08:50:08.897" UserId="887" />
  <row Id="8638" PostId="5662" Text="@mpiktas Right. Wrote to quickly (neither is of length `length(a)`): `a[a==&quot;a&quot;]` is _subsetting_, `a==&quot;a&quot;` is _indexing_. The point was about assigning an unknown level to `a` which is a factor. The `%in%` operator or simply `intersect()` is cool when you want to match more than one case, e.g. `as.character(a) %in% c(&quot;a&quot;,&quot;c&quot;)` is equivalent to `a==&quot;a&quot; | a==&quot;c&quot;`." CreationDate="2010-12-22T09:52:13.793" UserId="930" />
  <row Id="8639" PostId="5665" Text="The decile value is just the border of the range. Values higher are within this range (until the next decile)." CreationDate="2010-12-22T09:52:19.127" UserId="144" />
  <row Id="8641" PostId="5689" Text="@Brett Is it a duplicate: http://stats.stackexchange.com/users/485/brett-magill?" CreationDate="2010-12-22T10:54:58.893" UserId="930" />
  <row Id="8642" PostId="5674" Text="@Tom I've merged your accounts; now you can accept (click the tick mark next to it) whuber's answer. As chl mentioned, you can register with your Gmail account so that you won't loose your account any more; it can be done here: http://stats.stackexchange.com/users/login" CreationDate="2010-12-22T10:58:58.990" UserId="88" />
  <row Id="8643" PostId="5654" Text="I have integrated your extension answer into the question." CreationDate="2010-12-22T11:05:49.980" UserId="88" />
  <row Id="8644" PostId="5691" Score="1" Text="@Tal An answer to the last comment in your code: `library(reshape); untable(data.frame(X1=gl(2,1),X2=gl(2,2)), c(12, 5, 7, 7))`." CreationDate="2010-12-22T11:31:25.010" UserId="930" />
  <row Id="8645" PostId="5691" Text="great - thanks chl!" CreationDate="2010-12-22T11:37:44.067" UserId="253" />
  <row Id="8646" PostId="5646" Text="Thx. But both bootstrap estimates have a normal distribution and I'd really like to give a joint probaility for the two occuring at the same time. " CreationDate="2010-12-22T13:18:29.787" UserId="1291" />
  <row Id="8647" PostId="5671" Score="1" Text="No, i meant steps 1 and 2 in the order I wrote. After all, the distribution of y is specified conditional on X, so you must generate an X before Y. As for Gibbs sampling, that is a more complicated solution meant for more complicated problems. Yours, as you describe it, is pretty straighforward, IMHO." CreationDate="2010-12-22T14:15:25.387" UserId="2472" />
  <row Id="8648" PostId="5695" Text="@CP I like your Bayesian approach. You also justify you name!" CreationDate="2010-12-22T14:27:37.543" UserId="1307" />
  <row Id="8649" PostId="5680" Text="@whuber I have since tried a bunch of transformations. A square root transformation results in a distribution with a skew of 2.77 and kurtosis of 6.21. Would ANOVA results be more trustworthy from this distribution? Are there any guidelines for how far a distribution can deviate from normality and still work with ANOVA?" CreationDate="2010-12-22T14:43:52.083" UserId="2322" />
  <row Id="8650" PostId="5689" Text="@Brett I'm not sure what you're getting at. I know it's not an ANOVA. My question is about an analysis I performed using ANOVA.I put in the FYI so people would see that I'm aware that there are other ways to analyze this data." CreationDate="2010-12-22T14:45:54.330" UserId="2322" />
  <row Id="8651" PostId="5680" Score="2" Text="@Matt It sounds like 90% of the residuals are zero.  If that's the case, no transformation is going to make the residuals remotely close to normal.  Simulation studies have shown that p-values from F-tests are highly sensitive to deviations from normality.  (In your case it's fairly likely that some denominators in the F-tests will be zero: a sharp indicator of how far things can go wrong.)  You need a different approach.  What to do depends on *why* so many residuals are zero.  Lack of sufficient precision in measurements?" CreationDate="2010-12-22T15:11:49.190" UserId="919" />
  <row Id="8652" PostId="3174" Score="1" Text="A valuable resource for learning about generalized linear mixed models (GLMMs) and how to fit and interpret them is maintained by Ben Bolker and others at http://glmm.wikidot.com/ " CreationDate="2010-12-22T16:28:32.137" UserId="1979" />
  <row Id="8653" PostId="5691" Score="1" Text="@Tal Do you need to reproduce any of the marginals in the table?" CreationDate="2010-12-22T16:32:00.587" UserId="919" />
  <row Id="8654" PostId="5680" Text="@whuber I guess you could say that the 0s resulted from lack of precision. Quite simply, it was an easy task with high accuracy that pushed the variance of interest into response times. Another approach I've taken is to use GEEs with a negative binomial distribution. I was not able to get a model that was equivalent to the ANOVA to converge, but I was able to make multiple reduced models with key variables of interest. Does this sound like a reasonable approach? " CreationDate="2010-12-22T16:35:13.283" UserId="2322" />
  <row Id="8655" PostId="5680" Score="1" Text="@Matt that's sounding more appropriate, assuming your data are counts.  Another attractive consideration is a zero inflated negative binomial response (http://www.ats.ucla.edu/stat/r/dae/zinbreg.htm )." CreationDate="2010-12-22T16:42:37.240" UserId="919" />
  <row Id="8656" PostId="5650" Text="one possible method, maybe use the first 1000 values to predict the next 1000 values(although LOESS only support predict 4 values), then calculate the residual as the difference between actual value and corresponding predict value.Then retrain the model every 1000 values  However, this is not the original LOESS model I want at all :-(" CreationDate="2010-12-22T16:59:26.460" UserId="2454" />
  <row Id="8657" PostId="5693" Text="OK, you're right in my misuse of terminology.  I've been incorrectly calling them priors or sometimes 'conditional priors' because I specify the conditional probability table as part of modeling the network. I'll update my question so hopefully future travelers aren't as confused.  " CreationDate="2010-12-22T17:06:29.787" UserId="1474" />
  <row Id="8658" PostId="5693" Text="That's nice. Does the other part of the answer helps you, too ? Or does it need some refinement ? " CreationDate="2010-12-22T17:31:18.213" UserId="264" />
  <row Id="8659" PostId="5700" Score="2" Text="What is the policy on cross-posting? The exact same question was asked on math.stackexchange.com: http://math.stackexchange.com/questions/15214/finding-the-change-point-in-data-from-a-piecewise-linear-function" CreationDate="2010-12-22T17:58:31.557" UserId="2116" />
  <row Id="8660" PostId="5697" Score="1" Text="@Silvina, because then only coefficient on player will be random, but the intercept will be fixed." CreationDate="2010-12-22T18:03:37.717" UserId="2116" />
  <row Id="8661" PostId="5704" Score="1" Text="You've accidentally asked this question in the meta site. The **meta** site deals with questions about the site. For example, what type of questions are allowed.&#xA;&#xA;Would you re-ask your question on the [main site](http://stats.stackexchange.com/)." CreationDate="2010-12-17T10:34:32.723" UserId="8" />
  <row Id="8662" PostId="5693" Text="Yes, I think you've nailed it, but I haven't had time to fully digest it yet.  I'll come back and mark accepted when I've had time to digest." CreationDate="2010-12-22T18:22:25.503" UserId="1474" />
  <row Id="8663" PostId="5704" Score="2" Text="Maximum Likelihood Estimation (MLE) is a way to find good parameter values in a model.  Hierarchical Mixture of Experts (HME) is a model.  Since some of us (or me, at least) do not known the Haykin book, could you clarify which of these two things you have a question about?  " CreationDate="2010-12-22T18:33:55.520" UserId="1739" />
  <row Id="8664" PostId="3174" Text="@fabians Thanks for this." CreationDate="2010-12-22T19:03:06.233" UserId="930" />
  <row Id="8665" PostId="5696" Text="It looks like situation where functional data analysis methods can be applied. Did you consider them?" CreationDate="2010-12-22T19:57:29.070" UserId="2116" />
  <row Id="8666" PostId="5708" Score="3" Text="Not to mention that every individual rater probably creates their own scale, with their own variations on the differences between levels... this stuff is absolutely loathsome. Gary King's papers on [anchoring vignettes in survey research](http://gking.harvard.edu/category/research-interests/methods/survey-research) provide a good overview of the problems here, and an interesting solution (that can't be applied to this particular case)." CreationDate="2010-12-22T20:08:07.280" UserId="71" />
  <row Id="8667" PostId="5706" Text="You can use LaTeX for formulas, enclose them in dollar signs as in normal tex document. What is the reasoning of your second-last equation. As far as I understood $S_2$ is the number of trains with non-failed units, so your equation does not make sense then. " CreationDate="2010-12-22T20:14:49.960" UserId="2116" />
  <row Id="8668" PostId="5708" Text="Holy crap, this just gets worse... Unfortunately, due to the way we counted up the scores from the feedback forms, I do not have the data for how many of each score the speakers received. I think I will include a disclaimer in the feedback email, detailing some of the uncertainties of the data interpretation." CreationDate="2010-12-22T20:21:53.237" UserId="2470" />
  <row Id="8669" PostId="5664" Text="Thank you for all the help, people. I really appreciate it! Also, thanks for pointing out the drawbacks with this way of measurement. I have lots to learn about this stuff." CreationDate="2010-12-22T20:27:28.523" UserId="2470" />
  <row Id="8670" PostId="5684" Text="Thank, I'll read up on that!" CreationDate="2010-12-22T20:28:01.487" UserId="2470" />
  <row Id="8671" PostId="5703" Score="2" Text="Only knowing $\rho_{1}, \rho_{2}$, the interval containing $\rho(x, y_{1} + y_{2})$ must include $\rho_{1}$ and $\rho_{2}$: for each $y_{1}, y_{2}$ could have very small values (while having any rank-order), and thus simply &quot;jitter&quot; the values in $y_{1}$ when added to $y_{1}$. Thus the rank-order of $y_{1}$ wouldn't be affected. I don't know if the interval can exceed the $\rho_{i}$." CreationDate="2010-12-22T20:36:33.967" UserId="1909" />
  <row Id="8672" PostId="5706" Text="Thanks for TeXing that up for me. This is my first time on a TeX-enabled SE, and the WYSIWYG for TeX wasn't working for me, so I left it out. Also, I fixed the last two equations, in which I accidentally put in $S_2$ where I meant $S_0$." CreationDate="2010-12-22T20:39:10.920" UserId="2491" />
  <row Id="8673" PostId="5703" Score="2" Text="@caracal Good observations.  The interval definitely can be wider than the $\rho_i$: just consider the case where both correlations are zero.  The correlation with the sum can easily be nonzero--it can range all the way from -1 to 1.  E.g., x = (1,2,3,4,5); y1 = (3,-10,2,10,1); y2 = (-8,9,-2,-9,4); y1+y2 = (-5,-1,0,1,5) has $\rho_1=\rho_2=0$ but $\rho=1$." CreationDate="2010-12-22T20:57:32.963" UserId="919" />
  <row Id="8674" PostId="5703" Text="@whuber: this seems to imply only trivial bounds exist (i.e. $l = -1, u = 1$). Perhaps I have to throw another constraint at the problem." CreationDate="2010-12-22T21:13:35.290" UserId="795" />
  <row Id="8675" PostId="5703" Text="@shabbychef No, you have posted a nice problem: it's not trivial.  In case $\rho_1 = \rho_2 = 1$, for instance, the *only* possibility is $\rho = 1$.  I suspect the bounds are nontrivial except when $\rho_1 = \rho_2 = 0$; they must get narrower as $\rho_1$ and $\rho_2$ approach $\pm 1$." CreationDate="2010-12-22T21:21:29.073" UserId="919" />
  <row Id="8676" PostId="5703" Text="@whuber Neat idea. Geometrically, I think I understand your principle in terms of moving points into the desired quadrants of the centered scatter-plot. However, I think there's another issue: Spearman's $\rho$ only assumes ordinality, thus sums or differences like $y_{1} + y_{2}$ are not a-priori meaningful." CreationDate="2010-12-22T21:22:46.420" UserId="1909" />
  <row Id="8677" PostId="5703" Text="@Caracal There's nothing faulty about @shabbychef's formulation.  Spearman's rho can be computed for so-called &quot;continuous&quot; data just fine.  You have actually put your finger on the entire difficulty, which is characterizing the interaction between addition and ranking.  One potential application is this: suppose $y_1$ is a response variable and $-y_2$ is a vector of unknown errors.  What can we say about the Spearman correlation between $x$ and the *true* values $y_1 - y_2$ given a measured correlation with $y_1$ and an assumed correlation with $-y_2$?" CreationDate="2010-12-22T21:28:16.120" UserId="919" />
  <row Id="8678" PostId="5709" Text="Thanks! In my case, the true value of $n$ is on the order of 30, and the worst case I've seen is $S_2$ around 20 and $S_1$ around 10, making $\hat{S_0}/(S_1 + S_2)$ on the order of a few percent. Does that qualify as $S_2$ being small enough to make $\hat{S_0}$ very biased? I'm not sure how to best express my loss function, but it's negligible for $(S_0 - \hat{S_0})/n$ of, say 5% or less, but rises nonlinearly as that error gets away from 5%. The bottom line is that if the method of moments estimator is &quot;close enough,&quot; I'd be biased toward using it rather than something more sophisitcated." CreationDate="2010-12-22T21:32:56.593" UserId="2491" />
  <row Id="8679" PostId="5692" Text="Hi Suncoolsu - you got it right, that's what I was wondering on how to do.  And my question was - is it a reasonable solution for that question.  From your answer, I deduce that you believe it is valid to do it this way (Also chl directed me on where to look for implementing this within R, which is great).  Thanks for you input.  Tal" CreationDate="2010-12-22T21:59:09.853" UserId="253" />
  <row Id="8680" PostId="5709" Score="1" Text="@Isaac Simulations suggest the bias in your case isn't bad at all: around 0.5 on average.  It also seems the variance of this MM estimator is approximately the square of the estimate itself.  For your data this means (if the model is correct!) that you can be confident $S_0$ is between 0 and 4 or so." CreationDate="2010-12-22T21:59:28.760" UserId="919" />
  <row Id="8681" PostId="5694" Text="thanks mpiktas. chl suggestion is also a good direction on how to handle such data." CreationDate="2010-12-22T22:00:00.130" UserId="253" />
  <row Id="8682" PostId="5691" Text="Whuber - That was a question I didn't think about.  Let's say we wish to create a CI for the chi square statistic using bootstrapping, should we then preserve the marginals of the bootstrapped table to be the same as the ones we had?" CreationDate="2010-12-22T22:17:18.410" UserId="253" />
  <row Id="8683" PostId="5703" Score="1" Text="Here’s another pathological case.  Suppose that $x = y_1$ and $y_1 = -y_2$. Then $\rho(x, y_1 + y_2) = 0$, but $\rho_1 = 1$ and $\rho_2 = −1$.  &#xA;&#xA;It might be enlightening to think about a  simpler,  probabilistic version of the problem.&#xA;Let $X$, $Y_1$, and $Y_2$ be random variables, each with marginally Uniform distributions.  Now let $G$ be the CDF of $Y_1 + Y_2$.  What can we say about $Cov(X, G(Y_1 + Y_2))$ based on $Cov(X,Y_1)$ and $Cov(X,Y_2)$?" CreationDate="2010-12-22T23:56:50.410" UserId="1670" />
  <row Id="8684" PostId="5703" Text="One more note.  Since $G$ is a non-decreasing function, it feels like there might be a subadditive relationship between $Cov(X, G(Y_1 + Y_2))$ and $Cov(X,Y_1)$ and $Cov(X, Y_2)$.  The relationship would have to take into account the signs of the the latter two covariances." CreationDate="2010-12-23T00:10:28.500" UserId="1670" />
  <row Id="8685" PostId="4687" Text="Pick some function f(p1,p2,c), and define conditional probability as f(p1,p2,c)/sum_c f(p1,p2,c). Replace sum with integral when c is continuous" CreationDate="2010-12-23T00:12:19.447" UserId="511" />
  <row Id="8686" PostId="5703" Text="@vqv: this is a good point. One of the copula inequalities might be relevant here." CreationDate="2010-12-23T00:32:22.637" UserId="795" />
  <row Id="8687" PostId="5691" Score="1" Text="@whuber - I was wondering why did you ask this question. I am curious, can you pls tell. The only reason I can think of is bootstrapping might not produce the same marginals. Am I thinking correctly." CreationDate="2010-12-23T00:37:46.000" UserId="1307" />
  <row Id="8688" PostId="5713" Text="What can I say, all your points are valid, so it is possible that the authors of the articles in question are doing the wrong thing. Without more context it is impossible to say anything concrete." CreationDate="2010-12-23T07:19:41.880" UserId="2116" />
  <row Id="8689" PostId="5646" Text="@Misha, why do bootstrap estimates should have normal distribution? Usually bootstrap is used when the distribution is unknown. Or you know that the distribution is normal and you use bootstrap to estimate the covariance matrix?" CreationDate="2010-12-23T10:38:31.373" UserId="2116" />
  <row Id="8690" PostId="5646" Text="I was uncertain of the distribution, but as it happens both A &amp; B turn out normally distributed. However what I really would like to know is to estimate the probability of both A &amp; B occuring at the same time." CreationDate="2010-12-23T11:09:44.180" UserId="1291" />
  <row Id="8691" PostId="5712" Text="You made a (critical) typo. &quot;Na*N* is 'Not a Number'&quot;" CreationDate="2010-12-23T11:22:54.047" UserId="190" />
  <row Id="8693" PostId="5640" Text="This is helpful. I've added more details in the question description. Any thoughts?" CreationDate="2010-12-23T15:49:09.393" UserId="446" />
  <row Id="8694" PostId="5696" Text="I had thought about functional data analysis, but it's not an area that I'm very familiar with. I'm working thru Ramsay and Silverman's book now.  But I don't immediately see how to deal with a multinomial outcome variable...?" CreationDate="2010-12-23T16:58:08.457" UserId="2484" />
  <row Id="8695" PostId="5712" Text="@Peter Smit: ouch, thanks. my X server is lagging keystrokes..." CreationDate="2010-12-23T17:23:56.437" UserId="795" />
  <row Id="8696" PostId="5712" Text="Several language have equivalent constructs. For instance PHP and Javascript have null and NaN. " CreationDate="2010-12-23T17:55:50.470" UserId="582" />
  <row Id="8697" PostId="5720" Score="2" Text="Unfortunately, you have to have at least 50 reputation in order to post comments.  The purpose of that restriction has never been clear to me, as commenting seems like a pretty low-cost way for new users to start participating." CreationDate="2010-12-23T17:56:48.360" UserId="71" />
  <row Id="8698" PostId="5687" Text="@suncoolsu Of note, `is.na()` returns `TRUE` for both NA and NaN, which differs from `is.nan()` e.g. `is.na(c(0/0,NA))` *vs.* `is.nan(c(0/0,NA))`." CreationDate="2010-12-23T18:04:40.117" UserId="930" />
  <row Id="8699" PostId="5717" Score="1" Text="Perhaps we have different takes on partialling out? In my mind it would involve i) e1 &lt;- resid(lm(y ~ Sex)), ii) e2 &lt;- resid(lm(X ~ Sex)), and finally iii) lm(e1 ~ e2) . i) residualises y with respect to Sex, ii) residualises the other covariates (X) with respect to Sex, iii) fits the partial regression. In that case it doesn't matter how one codes Sex. In the above, we aren't really interested in the effect of Sex nor the interpretation of the coefficients. If we are model building, i.e. controlling for Sex as a Null, then how we parametrise the model *is* an important consideration, however." CreationDate="2010-12-23T18:38:43.347" UserId="1390" />
  <row Id="8700" PostId="5705" Text="Yes, that helps, thanks. " CreationDate="2010-12-23T19:29:22.023" UserId="1474" />
  <row Id="8701" PostId="5722" Text="But the real problem is that ranks don't add.  See my comment to the question." CreationDate="2010-12-23T19:59:20.577" UserId="1670" />
  <row Id="8702" PostId="5722" Text="@vqv but if $y_1$ and $y_2$ are permutations of the integers $1\ldots n$ then they are exactly the same as their ranks." CreationDate="2010-12-23T20:52:56.087" UserId="449" />
  <row Id="8703" PostId="5723" Text="Good info. But you may want to update that link to Wicklin's book =) For example, to: http://support.sas.com/publishing/authors/wicklin.html" CreationDate="2010-12-23T21:02:39.000" UserId="1934" />
  <row Id="8704" PostId="5722" Text="@onestop: half the sum of permutations need not be a permutation; But this is very close, and answers the question for Pearson, I believe." CreationDate="2010-12-23T21:23:34.157" UserId="795" />
  <row Id="8705" PostId="5696" Text="@dschulman Is there a gold standard available for those measurements (i.e., do you know when the event(s) of interest occur(s))? How many coders are included in this study? How many distinct events can we expect?" CreationDate="2010-12-23T21:37:27.713" UserId="930" />
  <row Id="8706" PostId="5690" Text="@Misha Could you pls give some indication about sample sizes, time frame, % survival, etc. so that we get a better idea of the design of your study?" CreationDate="2010-12-23T21:40:50.050" UserId="930" />
  <row Id="8707" PostId="4752" Text="...recapitulating prior work by Finch and Chater (1992, 1994), Schütze (1993), and others. HAL, LSA, and other prior art work by generating a similarity measure for words by computing their contextual similarity.  (This is Shephard's 'second order' similarity: 'first order' similarity is when word a occurs near word b; 'second order' similarity is that word a occurs near the same sorts of words as word b does)." CreationDate="2010-12-23T23:07:24.413" UserId="1739" />
  <row Id="8708" PostId="4752" Text="Comparing and contrasting: For LSA the context is the complete document. For HAL and others it is a textual window surrounding the target word. LSA measures distance in a linear subspace extracted via SVD/PCA, and the other deal with distances in the original space of surrounding word counts." CreationDate="2010-12-23T23:09:12.307" UserId="1739" />
  <row Id="8709" PostId="5202" Score="1" Text="and maybe also Bishop, Svensen and Williams for putting SOM on a clean probabilistic footing via the Generative Topographic Mapping.  (And I didn't down vote anything either...) " CreationDate="2010-12-23T23:13:30.263" UserId="1739" />
  <row Id="8710" PostId="5687" Text="@chl Thanks. I should have mentioned that." CreationDate="2010-12-24T02:39:20.500" UserId="1307" />
  <row Id="8711" PostId="5720" Text="I was somewhat frustrated with that restriction myself.  I remember having something along the lines of 49 reputation and wanted to make a comment and I couldn't but I definitely knew it wasn't appropriate to be a standalone answer..." CreationDate="2010-12-24T05:35:20.887" UserId="1028" />
  <row Id="8712" PostId="5727" Text="I know MAD (http://en.wikipedia.org/wiki/Median_absolute_deviation) is used a lot for wavelet shrinkage. " CreationDate="2010-12-24T06:31:31.627" UserId="223" />
  <row Id="8713" PostId="5728" Text="Could you please add information what method are you using?" CreationDate="2010-12-24T09:54:55.740" UserId="88" />
  <row Id="8714" PostId="5728" Text="And please add googlable names of the software or at least links to their sites." CreationDate="2010-12-24T09:57:14.303" UserId="88" />
  <row Id="8715" PostId="5731" Text="Hi, thank you very much for the reply. A(t) is time varying coefficient associated with state variable X(t). For some textbooks, when they deal with time varying coefficient, they assume the coefficients is a function of constant variable lamda, however, in this particular question, I don't know the function. Should I treat A(t) as another state variable which follows AR(1) process? Thank you." CreationDate="2010-12-24T13:05:33.003" UserId="2510" />
  <row Id="8716" PostId="5731" Text="@user2510: You may include in the state whatever you please, but you cannot (within the framework of the standard Kalman filter) include non-linear functions of the state in the observation equation; and $A(t)X(t)$ would be non-linear if both $A(t)$ and $X(t)$ are components of the state." CreationDate="2010-12-24T13:59:47.977" UserId="892" />
  <row Id="8717" PostId="5708" Text="That's too bad. If it's any consolation, Amazon uses the average of 5-star ratings to rank their data (as one option), so you've got good company." CreationDate="2010-12-24T16:23:23.600" UserId="2489" />
  <row Id="8719" PostId="5730" Text="I think for practical purposes what I said about above and below the diagonal can be ignored.  My problem isn't really a classification problem, but in terms of reducing the matrix to a rectangular matrix for other purposes your approach may be useful." CreationDate="2010-12-24T17:19:43.093" UserId="196" />
  <row Id="8720" PostId="4372" Text="Naively, it appears that Y_(t-1) could be moved into an expanded theta_t and with an expanded Phi (include F, essentially), could keep updating the Y_(t-1), as a seasonal ARIMA would work when translated into DLM... Except that it would not include e_t, while the actual Y_t does. :-(" CreationDate="2010-12-24T17:51:19.310" UserId="1764" />
  <row Id="8721" PostId="5124" Text="I know my choice is kind of arbitrary, because many are important but this is my favourite one, and his method allowed me to do lots of things." CreationDate="2010-12-24T18:35:22.593" UserId="1808" />
  <row Id="8722" PostId="5742" Text="I think you have your 'quadratic' and your 'cubic' models mixed up." CreationDate="2010-12-24T19:36:50.053" UserId="1028" />
  <row Id="8723" PostId="5742" Text="@David If the $g_i$ are the models and $Y$ is the data, your question already is Bayesian, because it requires a prior and Bayes' theorem to compute $\Pr(g_i|Y)$." CreationDate="2010-12-24T19:41:29.160" UserId="919" />
  <row Id="8724" PostId="5742" Score="2" Text="@David It's legitimate to use AIC for *nested* model comparison (as in null - linear - quadratic - cubic), but not (AFAIK) for completely different models, such as &quot;monod&quot; versus &quot;log&quot;." CreationDate="2010-12-24T19:42:48.927" UserId="919" />
  <row Id="8725" PostId="5742" Text="@Dason thanks for catching the error" CreationDate="2010-12-24T19:47:33.417" UserId="1381" />
  <row Id="8727" PostId="5742" Text="@whuber sorry, I got these backwards since my recent work has been in a Bayesian framework that considers data fixed and models variable. But the underlying point- that I would prefer to retain the original likelihood-based approach, remains. " CreationDate="2010-12-24T19:56:17.127" UserId="1381" />
  <row Id="8728" PostId="5742" Text="@David With the fix it looks like Bayes' Theorem but you have omitted the (requisite) prior probabilities for the models.  Once you specify that prior you can obtain (in the usual way) a posterior distribution for the models based on the data." CreationDate="2010-12-24T19:58:11.020" UserId="919" />
  <row Id="8729" PostId="5742" Text="@whuber I think that is exactly my problem. I was just hoping that there was a more direct way to estimate these probabilities from AIC (or another penalized likelihood like BIC if that is more appropriate for this set of models). I think that Aikake weights provide something along these lines, but I am still not sure of the interpretation. I'll post an answer describing this in more detail." CreationDate="2010-12-24T20:28:39.653" UserId="1381" />
  <row Id="8730" PostId="5740" Text="(+1) There's some code available in the Scipy [Sandbox](http://projects.scipy.org/scipy/browser/trunk/Lib/sandbox?rev=2844) which is based on Fortran code available on [Netlib](http://www.netlib.org/)." CreationDate="2010-12-24T21:41:56.333" UserId="930" />
  <row Id="8731" PostId="5740" Text="Cool, thanks. I bet it comes from the same original work." CreationDate="2010-12-24T22:00:35.133" UserId="334" />
  <row Id="8732" PostId="5736" Text="I'm interested in sampling possible returns for a given asset.  Since the distribution isn't normal AND the assets are correlated over time, that creates a challenge in choosing a distribution.    I'm exploring monte carlo methods for portfolio optimization." CreationDate="2010-12-24T22:05:32.213" UserId="2566" />
  <row Id="8733" PostId="5743" Text="This approach seems implicitly to place a uniform prior on the models you are using.  Whether that's appropriate or not depends on the application. " CreationDate="2010-12-24T22:59:23.563" UserId="919" />
  <row Id="8734" PostId="5743" Text="@whuber That is pretty much what I am going for; I have plenty of data for informative priors on these models but that would be another paper altogether. Thanks for the feedback." CreationDate="2010-12-24T23:06:26.630" UserId="1381" />
  <row Id="8735" PostId="5596" Text="hadn't heard of him before, but I'll check him out. Thanks for the intro!" CreationDate="2010-12-24T23:59:42.247" UserId="74" />
  <row Id="8736" PostId="5690" Text="@misha are there censored values in the data - other than for the largest values?" CreationDate="2010-12-25T04:31:42.477" UserId="1112" />
  <row Id="8737" PostId="5685" Text="@mbq thanks for having the patience to correct my lazy typing." CreationDate="2010-12-25T04:39:36.560" UserId="1112" />
  <row Id="8738" PostId="5685" Text="and what if the distribution is not multivariate normal? Also, the vector might have 0 in certain locations (not necessarily the same for all subjects e.g. X1=0, and X7=0 for one subject and X3=0 for another etc.). Is it still ok to use MANOVA?" CreationDate="2010-12-25T05:03:29.853" UserId="2478" />
  <row Id="8739" PostId="5742" Text="I'm going to agree with whuber. If you put a non informative discrete uniform prior on the models, then the posterior likelihood of model k given data is proportional to the likelihood of the data under the model k. This seems much  simpler then trying to derive an approximation of posterior model likelihoods from AIC. " CreationDate="2010-12-25T05:31:36.920" UserId="2144" />
  <row Id="8740" PostId="5736" Text="@Noah,  have you considered various stochastic volatility models, such as GARCH?" CreationDate="2010-12-25T14:44:30.263" UserId="2116" />
  <row Id="8741" PostId="5745" Text="That python library seems perfect. Thanks." CreationDate="2010-12-25T18:56:09.163" UserId="957" />
  <row Id="8742" PostId="5748" Score="1" Text="@Nice. I think you mean &quot;A and **B** are not correlated&quot; in the very last part of your last sentence." CreationDate="2010-12-25T21:05:10.417" UserId="1307" />
  <row Id="8743" PostId="5749" Score="1" Text="Taking the means of the x's is probably not the smartest thing to do if you have high skew. I would choose to regress y on x instead of y on mean(x). This will allow you to determine outliers during your residual analysis. " CreationDate="2010-12-25T21:54:25.523" UserId="1118" />
  <row Id="8744" PostId="5752" Text="The reason why I wanted to do it like this was to have an open mind towards understanding my data, so maybe in a way I am fishing for correlations, which I did not think of before, for the purpose of getting enlightened. I am certainly not doing this to satisfy my boss or something abitrary. I would rather not get into the specifics of the data, as I want a general answer to this question, so I can use it in all situations in the future. " CreationDate="2010-12-25T22:43:11.960" UserId="888" />
  <row Id="8745" PostId="5751" Text="Unfortunately, I am not an R user.  So, the codes above mean less to me than they mean to you. " CreationDate="2010-12-25T23:07:06.773" UserId="1329" />
  <row Id="8746" PostId="5748" Text="Yes, Nico with suncoolsu correction... this is a reasonably good explanation.  You are partially describing Path Analysis. " CreationDate="2010-12-25T23:10:46.970" UserId="1329" />
  <row Id="8747" PostId="5747" Score="13" Text="An example is to take $A=X$, $B=Y$, and $C=X+Y$.  We can take $X$ and $Y$ to be independent, yet both $A$ and $B$ are correlated (positively, Pearson) with $C$." CreationDate="2010-12-25T23:51:49.903" UserId="1108" />
  <row Id="8748" PostId="5747" Text="Thanks, that's actually a great comment.  Short, but it captures the essence of the reason why it is so." CreationDate="2010-12-26T00:07:08.530" UserId="1329" />
  <row Id="8749" PostId="5754" Text="Thanks for the excellent answer!" CreationDate="2010-12-26T00:18:50.060" UserId="888" />
  <row Id="8750" PostId="5753" Text="whuber, what are &quot;multinormal variables&quot;?" CreationDate="2010-12-26T00:19:14.987" UserId="1329" />
  <row Id="8751" PostId="5753" Text="http://en.wikipedia.org/wiki/Multivariate_normal_distribution" CreationDate="2010-12-26T00:20:25.823" UserId="919" />
  <row Id="8752" PostId="5746" Text="What exactly is different about your problem?  The solution to the Secretary problem does not depend on the underlying distribution.  (It only requires that any possible ties can be resolved.)  If there is lack of independence among successive observations, you will need to quantify this (or at least bound it somehow) in order to make any progress." CreationDate="2010-12-26T00:24:08.973" UserId="919" />
  <row Id="8753" PostId="5751" Score="1" Text="@Gaetan Lion: in this code, $x_1$ and $x_2$ are independent root normals, and $y = 3x_1 + 2x_2$ plus a normal noise term with standard deviation of 0.3. Clearly $y$ is positively correlated to $x_1$ and $x_2$, which are independent." CreationDate="2010-12-26T06:08:51.307" UserId="795" />
  <row Id="8754" PostId="5722" Text="@onestop The ranked values of $y_1 + y_2$ are in general a nonlinear function of $y_1 + y_2$ — even if $y_1$ and $y_2$ are each a permutation of the integers $1,\ldots,n$.  Here’s an example: $y_1 = (1,2,3,4)$ and $y_2 = (2,3,1,4)$. Then $y_1+y_2 = (3,5,4,8)$ and $rank(y_1+y_2) = (1,3,2,4)$.  Plot $y_1+y_2$ against $rank(y_1+y_2)$ and you’ll see that there is no linear relationship between the two.  The above assertion that $\rho(x,y_1+y_2) = Cov(x,y_1+y_2) / \cdots$ **is in general false**, even under the assumption that $y_1$ and $y_2$ are permutations of the integers." CreationDate="2010-12-26T06:16:14.497" UserId="1670" />
  <row Id="8755" PostId="5748" Text="Yes, sorry, I got mixed up with the letters ;)" CreationDate="2010-12-26T08:17:19.517" UserId="582" />
  <row Id="8756" PostId="5750" Score="2" Text="This is a great example of why one needs multiple hypothesis testing." CreationDate="2010-12-26T13:44:44.447" UserId="88" />
  <row Id="8757" PostId="5754" Text="@whuber (+1) Nice answer! " CreationDate="2010-12-26T14:08:23.353" UserId="930" />
  <row Id="8759" PostId="5757" Text="@FreshCode What do you call &quot;every experiment&quot;? Do you have several outcomes measured at the same time (for the control and the active group), or do you have repeated measurements for both groups at different dates (like in a sequential trial)? What kind of outcome is it: continuous or categorical?" CreationDate="2010-12-26T14:39:23.987" UserId="930" />
  <row Id="8761" PostId="5746" Text="Sorry, I wasn't clear, have tried to clarify." CreationDate="2010-12-26T15:18:14.340" UserId="557" />
  <row Id="8762" PostId="5757" Text="@chl, I meant every study. Assume the simplest case for my question. I don't know the difference between &quot;continuous&quot; and &quot;categorical&quot;." CreationDate="2010-12-26T15:22:38.863" UserId="1497" />
  <row Id="8763" PostId="5760" Text="+1 for modelling unpredictable deviations as random variables, the tickets from a box analogy and critical region references. I've updated my question with a more concrete (though hypothetical) example of what I'm interested in. I'll check out those books. I was hoping a widely accepted method exists for flagging significant results in a set of data based on prior data." CreationDate="2010-12-26T16:49:00.580" UserId="1497" />
  <row Id="8764" PostId="5690" Text="There are indeed censored values in the data and the total population is approx 1500, median overall survival is 18 months (range 300-600 days)... the time frame is the period 2000-2007. " CreationDate="2010-12-26T17:10:25.537" UserId="1291" />
  <row Id="8765" PostId="5729" Text="Yes, this can be done if the classification algorithm cannot utilize weights directly (e.g. Naive Bayes is able to). However it is important to note that a mindful validation is absolutely necessary, since this approach may induce a Sampling Bias into the model. " CreationDate="2010-12-26T20:55:23.917" UserId="264" />
  <row Id="8766" PostId="5763" Text="Thank you very much! It's just that my supervisor specifically instructed me to use a logistic regression, sorry I forgot to add that there are three groups involved autism, language impaired and typically developing kids and I need to look at whether langauge scores of the autistic kids and language impaired kids are also a signifcant predictor of scoring low on a social attribution task in later life. Please can you help with this?? " CreationDate="2010-12-26T22:29:30.613" UserId="2528" />
  <row Id="8767" PostId="5710" Text="Is it churlish to wince when I read &quot;data is&quot;?" CreationDate="2010-12-26T23:06:44.780" UserId="199" />
  <row Id="8768" PostId="5756" Text="+1 correlation in terms of an angle between multi-dimensional vectors is intuitive for me." CreationDate="2010-12-26T23:55:58.960" UserId="1497" />
  <row Id="8769" PostId="5736" Text="I have looked at GARCH models.  However, GARCH wouldn't solve this problem.  I'm looking at generating correlated random samples from multiple time series.  Random multi-variate norm sample work, but they require the assumption that returns are normally distributed and  in this case, they are not." CreationDate="2010-12-27T09:02:37.307" UserId="2566" />
  <row Id="8770" PostId="5765" Text="It would be useful to know more about the scale these values come from - is the data continuous or categorical? Are differences meaningful, or are the measurements just ordinal? A dispersion measure for, e.g., unordered categorical needs to be different from such a measure for continous interval data." CreationDate="2010-12-27T14:29:47.030" UserId="1909" />
  <row Id="8771" PostId="5761" Text="@James  Good points (+1).  I just want to clarify a possible misunderstanding concerning an earlier comment of mine.  In your example *there is no single underlying distribution:* if there is a trend over time, then *a fortiori* the distribution is changing.  When the distribution does *not* change in any way over time, when there is serial independence of the observations, and the distribution is continuous, then the answer to the Secretary problem remains the same no matter what shape this distribution has." CreationDate="2010-12-27T17:22:23.120" UserId="919" />
  <row Id="8772" PostId="5765" Text="The simplest possible one is the *range*: 40 - 0 = 40.  The mean density of the numbers can be estimated as the quantity divided by the range; e.g., 5/40 = 1/8 (one value for every difference of eight)." CreationDate="2010-12-28T00:41:52.507" UserId="919" />
  <row Id="8773" PostId="5770" Text="&#xA;Thanks very much, Suncoolsu, for the explanation as well as the great resources!&#xA;&#xA;Yes, you are right. My split-plot treatment, or subject, should be the 4 individuals within each combination of the two fixed factors, they were sampled 4 times (with one day interval), forming a longitudinal sequences. (But I am totally confused by the terminology of repeated measurement, as I did destructive sampling, but the samples should have temporary correlation in between). &#xA;" CreationDate="2010-12-28T01:09:53.450" UserId="2536" />
  <row Id="8774" PostId="5770" Text="The BLOCK in my question referred to three whole plots (I call them groups) with 5 days interval (destructive sampling). As a result, all the individuals were sampled in 12 samplings (｛3 groups｝*｛4 samplings in each group｝), the dates from the first sampling is 1-4, 6-9, 11-14 (I also tested the model where all the sampling were taken as one random factor, but it is still not correct, as I only have pseudo-replications, right?). " CreationDate="2010-12-28T01:10:35.557" UserId="2536" />
  <row Id="8775" PostId="5770" Text="&#xA;The reason for the complicated design is because I have a little bit long term variables (i.e., biomass) which takes effect on the group level, and short term variables (i.e., 14C activity) which could respond quickly within one day. Each group was fertilized, irrigated, and labeled together, and received different water, nutrient, but the same labels. &#xA;Now, I would like to arrangethe whole experiment like this&#xA;&#xA;model&lt;-aov(Var~Group+A*B+Error(Group/A/B/Block),data)&#xA;&#xA;But, I do not if it is correct.&#xA;" CreationDate="2010-12-28T01:11:01.843" UserId="2536" />
  <row Id="8776" PostId="5770" Text="I know you probably already feel crazy about the shit design, but I really appreciate if you could help me out of these mess." CreationDate="2010-12-28T01:11:30.343" UserId="2536" />
  <row Id="8777" PostId="5772" Text="@Basil I think most statistical packages will output sth like $\log\big(\Pr(Y=2)/\Pr(Y=1)\big)=b_{10}+\sum_ib_{1i}x_i$ and $\log\big(\Pr(Y=3)/\Pr(Y=1)\big)=b_{20}+\sum_ib_{2i}x_i$; and the interpretation of a regression coefficient is straightforward: for a one-unit increase in a (continuous) $x$, the log of the ratio of either of these probabilities will be $b$." CreationDate="2010-12-28T08:16:00.277" UserId="930" />
  <row Id="8778" PostId="5779" Text="mpiktas, I am missing something basic here.  When should one use the normal distribution approach I outlined vs. simply multiplying the p directly with no normal approximation?  " CreationDate="2010-12-28T08:48:26.680" UserId="1279" />
  <row Id="8779" PostId="5779" Text="However, the chance that the person who wins the second round is the same as the first is 1/8 (p = 0.125) assuming H0 for all players." CreationDate="2010-12-28T09:00:00.797" UserId="183" />
  <row Id="8780" PostId="5779" Text="@stats-teacher-stats-studentZZZ, normal distribution approach is an approximation. You use it when you do not know the exact distribution, or the exact distribution is too hard to calculate. The approximation error decreases as the sample length (in your case number of rounds) increases." CreationDate="2010-12-28T09:14:10.423" UserId="2116" />
  <row Id="8781" PostId="5779" Text="@stats-teacher-stats-studentZZZ: A rule of the thumb is, that if n*p*(1-p) &gt;= 9 one can approximate the binomial distribution with a normal distribution." CreationDate="2010-12-28T09:26:11.740" UserId="264" />
  <row Id="8782" PostId="5736" Text="@Noah, how about multivariate GARCH? Each individual series is GARCH with innovations from multivariate normal with non-diagonal covariance matrix. Then the returns will not have normal distribution, but they will be correlated." CreationDate="2010-12-28T09:39:49.830" UserId="2116" />
  <row Id="8783" PostId="5779" Score="1" Text="Simple question, but why should the null hypothesis be rejected? What are you comparing the result of 0.015625 to conclude that the nullH should be rejected?" CreationDate="2010-12-28T10:07:33.740" UserId="1497" />
  <row Id="8784" PostId="5779" Text="@FreshCode, I assume the usual: if the p-value is lower than 0.05, the null hypothesis is rejected." CreationDate="2010-12-28T10:10:11.633" UserId="2116" />
  <row Id="8785" PostId="5744" Text="Hi Ronaf.  This came up when I learned about Cramer's V statistic for contingency tables.  I was wondering what the CI for it are, and found a way to get them - but hoped to validate my results using bootstrapping.  Cheers, Tal" CreationDate="2010-12-28T10:35:47.380" UserId="253" />
  <row Id="8786" PostId="5782" Text="Delta method is used for asymptotic distributions. You cannot use when you have only one random variable." CreationDate="2010-12-28T14:35:31.440" UserId="2116" />
  <row Id="8787" PostId="5778" Text="@mpiktas (+1) Thanks for the link.It is also possible to consider a heterogeneous correlation matrix (see, e.g. `hetcor()` from the [polycor](http://cran.r-project.org/web/packages/polycor/index.html) package). Provided the VC matrix is SDP, it should do the work--mostly in the spirit of Factor Analysis. Nominal variables might be dummy coded." CreationDate="2010-12-28T15:24:20.967" UserId="930" />
  <row Id="8788" PostId="5779" Text="Ok, so with only 2 trials, it is smarter to just to the raw probability calculation.  Once you get to 9, it's a pain to multiply to the 9th power, so you THEN start using normal distribution estimations, b/c they are close enough.  I should do both calculations and compare, in fact.  Thanks for the insight.  BTW, my friend DID indeed win both rounds of Texas Hold'em.  Clearly, some skill!  " CreationDate="2010-12-28T15:48:53.747" UserId="1279" />
  <row Id="8789" PostId="5788" Text="@jebyrnes Are these the same statistical units that are measured at the two time points?" CreationDate="2010-12-28T17:55:05.517" UserId="930" />
  <row Id="8790" PostId="5789" Text="@whuber (+1) I won't post my answer since it's basically your 2nd § (less well explained) + some digressions about an additional test for equal intercept (if we cannot reject the null for the test of the equality of slopes, compute a common slope for both regression lines and ask whether the new lines are parallel or identical)--but, it's always under the assumption that the sampled units are independent at the two time points." CreationDate="2010-12-28T18:12:13.147" UserId="930" />
  <row Id="8791" PostId="5789" Score="2" Text="@chl I refrained from commenting on the option of computing a common slope because that implicitly assumes homoscedasticity in the combined year 1 + year 2 datasets.  Given that the slope appears to have changed substantially I would want to check for a change in residual variance, which seems easiest to do by conducting a separate regression for each year--which the OP already has done.  The advantage of modeling the combined datasets comes from the additional power it offers of evaluating serial correlation of the residuals." CreationDate="2010-12-28T18:27:33.117" UserId="919" />
  <row Id="8792" PostId="5789" Score="1" Text="@chl You bring up an interesting question concerning equal intercepts.  However, an intercept is an artifact of the origin of the x coordinate system.  We can handle that problem by comparing two models in which a pair of linear splines (plus one common intercept) is nested within two slopes+two intercepts.  The splines would of course have a &quot;knot&quot; at the transition from year 1 to year 2.  Assuming the splines aren't a significantly worse fit than the bigger model, we can then test for equality of their coefficients to see whether the slope changed." CreationDate="2010-12-28T18:33:46.343" UserId="919" />
  <row Id="8793" PostId="5793" Score="1" Text="@chl +1, pressed post the same second your answer appeared. I would add that for a full tree rsq.val[nrow(rsq.val),] is probably what OP wants." CreationDate="2010-12-28T20:34:49.110" UserId="2116" />
  <row Id="8794" PostId="5789" Text="@whuber Thanks for the clarification and added comment. I'm afraid to say that my initial ideas were not that advanced, but I definitively have to think about this last suggestion. " CreationDate="2010-12-28T20:35:42.197" UserId="930" />
  <row Id="8795" PostId="5793" Text="@mpiktas Oups... Needless to say, your comment is very appreciated." CreationDate="2010-12-28T20:39:18.217" UserId="930" />
  <row Id="8796" PostId="5793" Text="Thanks a bunch.  I am (obviously) new to R, and never once did I think to look at the function as you first mentioned.  I was trying something similar to your answer, but that is perfect! Thx" CreationDate="2010-12-28T20:50:15.650" UserId="569" />
  <row Id="8797" PostId="5789" Text="Awesome, thanks, all.  In summary, calculate t where $t=\frac{b1-b2}{se_{b1-b2}}$.  To estimate the denominator, you can calculate this as $\sqrt{se_{b1}^2+se_{b2}^2}$.  You can also use the sums of squares and a different formula if you like, but you'll get the same result.  Then just pull out the df, and voila, answer." CreationDate="2010-12-28T20:52:23.447" UserId="101" />
  <row Id="8798" PostId="2207" Text="They're both distributions and loss functions in different contexts." CreationDate="2010-12-28T21:36:38.513" UserId="1119" />
  <row Id="8799" PostId="5785" Text="Great point. I also fell into the trap assuming that 1. is the true setup :(" CreationDate="2010-12-28T21:49:13.193" UserId="264" />
  <row Id="8800" PostId="2207" Text="I pressed enter too quickly on the previous reply - exponential loss is widely associated with boosting (see Friedman Hastie and Tibshirani's Statistical View of Boosting), where it's a loss rather than a distribution, logistic regression coresponds to log loss, laplace is a distribution but corresponds to absolute value loss - so for the most part I was being extremely sloppy, thanks for pointing it out.&#xA;&#xA;But while L1 loss has a geometric solution, it isn't analytically closed form, so I would hardly call its solution easy." CreationDate="2010-12-28T21:49:28.133" UserId="1119" />
  <row Id="8801" PostId="5786" Text="(+1) A very interesting question ..." CreationDate="2010-12-28T21:51:34.753" UserId="264" />
  <row Id="8802" PostId="5780" Text="Thanks Tal, I'll check out that Barnard's test code and have a think about your Wilcoxon suggestion." CreationDate="2010-12-28T21:55:06.147" UserId="174" />
  <row Id="8803" PostId="5780" Text="My pleasure.  I'm not sure the Wilcoxon test would answer something you want to know - but if you are interested in over all dropout &quot;earlier&quot; in the stages of one group vs the other - that seems to me to be a good way to test in.  Cheers." CreationDate="2010-12-28T21:56:25.320" UserId="253" />
  <row Id="8804" PostId="5773" Text="Basil, I love it! +1 for a plain English explanation. " CreationDate="2010-12-28T22:35:02.523" UserId="1329" />
  <row Id="8805" PostId="5753" Text="whuber, thanks the link was really helpful." CreationDate="2010-12-28T22:36:58.947" UserId="1329" />
  <row Id="8806" PostId="5756" Text="+1: Nice geometric explanation." CreationDate="2010-12-28T22:52:28.677" UserId="919" />
  <row Id="8807" PostId="5798" Score="6" Text="You will need to provide more information. Are the two &quot;populations&quot; represented as factors in R? Is the species a particular level of a factor? Please explain the context as well." CreationDate="2010-12-29T01:28:45.480" UserId="159" />
  <row Id="8808" PostId="5765" Text="How is Mean Density of Numbers and Quartiles different? I mean, statistically, dont you think quartiles convey this information in a better way?" CreationDate="2010-12-29T04:15:21.960" UserId="2535" />
  <row Id="8809" PostId="5791" Text="What assumptions are you making about the distributions? Are they all, for example, lognormal?" CreationDate="2010-12-29T05:10:38.657" UserId="919" />
  <row Id="8810" PostId="5791" Text="@whuber each distribution has it's own assumptions, priors are either beta, gamma, logN, or weibull and only six are updated with data; in these cases the posterior distributions are simulated " CreationDate="2010-12-29T05:51:38.330" UserId="1381" />
  <row Id="8811" PostId="5795" Text="Hey Mariana, thanks for your answer, but we had a misunderstanding: I by &quot;kernel methods&quot; mean methods such as the Support vector machine using the &quot;kernel trick&quot;, not kernel smoothing methods. " CreationDate="2010-12-29T10:00:44.477" UserId="2549" />
  <row Id="8813" PostId="5782" Text="@mpiktas: Actually I dont know much about Delta method, I've just read something on wikipedia. This is quotation from wiki: &quot;The delta method uses second-order Taylor expansions to approximate the variance of a function of one or more random variables&quot;." CreationDate="2010-12-29T13:08:40.030" UserId="1643" />
  <row Id="8814" PostId="5790" Text="I dont need to know the exact value of the variance, approximation should works for me." CreationDate="2010-12-29T13:10:18.733" UserId="1643" />
  <row Id="8815" PostId="5782" Text="@Tomek Tarczynski, it seems wikipedia has exactly what you want: http://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables. I will reedit my answer, it seems that I underestimated Taylor expansion." CreationDate="2010-12-29T13:39:10.650" UserId="2116" />
  <row Id="8816" PostId="5790" Text="Indeed, the approximate formula for $\mathbb{E}[f(X)]$ in the OP is often used in risk analysis in economics, finance and insurance." CreationDate="2010-12-29T14:50:19.560" UserId="2036" />
  <row Id="8817" PostId="5790" Text="@Raskolnikov, yes but it contradicts my admitedly stale knowledge of Taylor expansion. Clearly the remainder term must be taken into account. If the random variable is bounded, then no problem, since polynomials approximate continuous functions on bounded interval uniformly. But we deal with unbounded random variables. Of course for random normal  we can say that it is effectively bounded, but still in general case, some nasty surprises can arise, or not. I will fix my answer when I'll have the clear answer." CreationDate="2010-12-29T15:11:47.007" UserId="2116" />
  <row Id="8818" PostId="5778" Text="Thanks a lot for suggestions, these are great guides. " CreationDate="2010-12-29T15:25:34.353" UserId="2540" />
  <row Id="8819" PostId="5784" Text="Thank you for very much for your help. much appreciated!!" CreationDate="2010-12-29T15:28:14.227" UserId="2528" />
  <row Id="8820" PostId="5772" Text="Thank you for very much for your help. much appreciated!!" CreationDate="2010-12-29T15:29:29.017" UserId="2528" />
  <row Id="8821" PostId="5763" Text="Thank you for very much for your help. much appreciated!!" CreationDate="2010-12-29T15:30:00.293" UserId="2528" />
  <row Id="8822" PostId="5790" Text="@mpiktas: I've done some simulations and accuracy of the approximation strongly depends on the distribution, for unimodal distribution it seems to work fine, but the error could be significant for uniform distribution. I was testing $f(x)=\sqrt{x}$ and in case of $\chi^{2}$ the relative error was about 3%, but in case of uniform distribution it was more than 10%. I'm talking about the variance, because mean is approximated far better. " CreationDate="2010-12-29T15:46:50.130" UserId="1643" />
  <row Id="8823" PostId="5809" Text="That's just plain wrong. One of the points of doing ANCOVA is to adjust for covariate effects that might be confounded with the predictor." CreationDate="2010-12-29T16:34:53.240" UserId="279" />
  <row Id="8825" PostId="5809" Score="2" Text="No. ANCOVA is meant as a way to enhance power when you have a covariate related to the predicted variable but unrelated to the predictor variables. It is lamentably common that folks fail to understand this and instead attempt to use ANCOVA to &quot;control for&quot;/&quot;regress out&quot; effects of the covariate when they are correlated with the predictors. See Miller 2001: http://www.ncbi.nlm.nih.gov/pubmed/11261398" CreationDate="2010-12-29T16:38:26.250" UserId="364" />
  <row Id="8826" PostId="5810" Score="1" Text="I thought that interaction between the covariates and the factors indicates violation of the homogeneity of regressions assumption (another assumption also mentioned in the first link I gave)" CreationDate="2010-12-29T16:53:05.757" UserId="339" />
  <row Id="8827" PostId="5810" Score="2" Text="This approach tests whether the effect of the predictor variables (on the predicted variable) is affected by the covariate (or vice versa). In other words, this tests assumption 2 on the wikiversity entry linked in the question. This routine cannot speak to assumption 3, the focus of the poster's question, which is about the relationship between the values of the covariate and the values predictor variables. My answer speaks to this assumption." CreationDate="2010-12-29T16:56:38.473" UserId="364" />
  <row Id="8828" PostId="5809" Score="1" Text="I stand corrected." CreationDate="2010-12-29T17:10:30.507" UserId="279" />
  <row Id="8829" PostId="5809" Score="1" Text="I should note that I myself was taught and employed (http://www.ncbi.nlm.nih.gov/pubmed/19154749) the inappropriate practice of using ANCOVA to &quot;control for&quot;  a covariate with a known relationship with other predictor variables. It's only been through later self-directed study that I discovered the Miller &amp; Chapman paper and the errors of my ways!" CreationDate="2010-12-29T17:24:40.367" UserId="364" />
  <row Id="8830" PostId="5796" Text="@Dikran (+1) Thanks for the link and advice on model selection issue." CreationDate="2010-12-29T18:17:14.680" UserId="930" />
  <row Id="8832" PostId="5812" Text="@mbq: Can you explain this idea of re-sampling CV? Are you proposing an algorithm where you create 10 groups (typical 10 fold CV) but within each fold you create 1 bootstrap sample and determine the expected error on those observations not in the training set of the bootstrap? So you still have 10 error measurements to average?" CreationDate="2010-12-29T18:42:13.923" UserId="2040" />
  <row Id="8833" PostId="5812" Text="@mbq: Also, Are you suggesting this (compared to say the regular 10 fold cross validation) because of the small sample size?" CreationDate="2010-12-29T18:46:43.170" UserId="2040" />
  <row Id="8834" PostId="5813" Text="@avj I don't think there's a single best answer, so this has probably to be made CW. Moreover, I think you need to clarify what you intend to do, and what's your background." CreationDate="2010-12-29T18:52:58.033" UserId="930" />
  <row Id="8835" PostId="5820" Score="2" Text="Another possible source of confusion is that the pdf of a _discrete_ random variable (also called pmf - probability mass function) indeed cannot exceed 1." CreationDate="2010-12-29T20:40:04.977" UserId="279" />
  <row Id="8836" PostId="5820" Text="@Aniko: This is indeed a source of confusion. I think I understand now." CreationDate="2010-12-29T20:48:17.387" UserId="977" />
  <row Id="8837" PostId="5812" Text="@user2040 I can't find a good reference, so I'll edit the answer to be more precise; due to the second thing, yes, I think OP needs a quite small N (to remove this possible homogeneity bias) for N-fold CV, and so may get too few repetitions for a meaningful result." CreationDate="2010-12-29T21:09:09.893" UserId="88" />
  <row Id="8838" PostId="5796" Text="I should add, don't use implementations based on the Laplace approximation - the posterior is highly skewed, and a symmetric approximation centered on the mode generally wont work very well. " CreationDate="2010-12-29T21:16:21.637" UserId="887" />
  <row Id="8839" PostId="5813" Text="Not sure this is answerable without a bit more info on the company's area(s) of activity and the job specification. 'Medical research' and 'programmer' are a bit vague." CreationDate="2010-12-29T21:18:22.730" UserId="449" />
  <row Id="8840" PostId="5814" Text="I'm having trouble following the question. It might help  a little if each opening parenthesis was matched by a corresponding closing parenthesis. Also are the vehicle flow rates for each lane averages? If not how do you allow for overtaking within a lane? Cycle speeds and times depend on the cyclist." CreationDate="2010-12-29T21:38:17.883" UserId="449" />
  <row Id="8841" PostId="5802" Text="Adam great answer.  That's exactly what I was asking.  However, you added a tid bit of information that I have had a hard time finding more information on.  The Mallow's CP stop criterion.  I've looked through the R source code, but it's too hard to figure how they are choosing to stop.  I roughly have an idea, but I need a paper or some discussion on how it works so I can finish implementing it.  Do you have any information about it?" CreationDate="2010-12-29T21:44:51.537" UserId="1929" />
  <row Id="8842" PostId="5722" Text="@vqv You're quite right. I was too hasty to attempt an answer before leaving for the christmas break. I hadn't come across that inequality relating the Pearson correlation of three variables before. Here's another ref for it complete with 3D visualisations: http://www.jstor.org/stable/2684832 . I still think it might have *some* relevance, so I won't delete my answer, though I can't see how to fix it either. " CreationDate="2010-12-29T21:50:31.677" UserId="449" />
  <row Id="8844" PostId="5821" Text="Have you used two different accounts for question and answer? they will probably be merged. You should edit your original question, instead of providing this 'answer'." CreationDate="2010-12-29T22:03:32.283" UserId="795" />
  <row Id="8845" PostId="5821" Text="@Noah Is this correct?  You used two different email addresses; which account is primary?" CreationDate="2010-12-29T22:19:44.133" UserId="5" />
  <row Id="8846" PostId="5823" Text="I think the robust analogue of CV would be the Winsorized standard deviation divided by the trimmed mean." CreationDate="2010-12-30T00:08:38.837" UserId="795" />
  <row Id="8847" PostId="5812" Text="@mbq: Thank you. From what I read of your edit, this is the usual bootstrap validation (e.g. Elements of Statistical Learning 2nd edition page 249), but with 10,15 or 30 iterations (compared to 100's or 1000's) typically used with large N data set." CreationDate="2010-12-30T00:35:54.757" UserId="2040" />
  <row Id="8848" PostId="5822" Text="@Mike Lawrence in your dataframe there should only be 100 trials in each cell that is a combination of the 2 tasks. Also as an aside, I'm not wed to nlme. I've just begun using (or trying to use) mixed-models. nlme() is the first function that I've been introduced to." CreationDate="2010-12-30T03:32:38.170" UserId="2322" />
  <row Id="8849" PostId="5821" Text="My mistake.  Didn't realize that I had created a second account.  How can I merge back to the original one?" CreationDate="2010-12-30T04:17:36.337" UserId="2566" />
  <row Id="8850" PostId="5802" Text="So, I checked the rpart reference manual, and apparently I was incorrect. While one of the criterion for stopping in rpart is called &quot;cp&quot; this is short for &quot;complexity parameter&quot; and is merely the minimum amount r^2 must increase by in order to pursue a particular split. I have corrected my post above to reflect this. The lack of statistical tests in splitting rules is one of the reasons why I use the party package over the rpart package. With the party package the default method implements a Bonferroni corrected p-value as a stopping criterion (default p=0.05). For details see the vignette." CreationDate="2010-12-30T06:14:04.810" UserId="2166" />
  <row Id="8851" PostId="5802" Text="I feel the fact that rpart does not use a stopping criterion is an asset, rather than a liability. A split may appear to be worthless, and yet open the way for subsequent splits further down the tree, which may be quite significant. A &quot;grow, then prune&quot; policy as implemented in rpart will avoid early stopping and seems to me a sensible approach." CreationDate="2010-12-30T09:00:44.500" UserId="892" />
  <row Id="8852" PostId="5790" Score="1" Text="@Tomek Tarczynski, the third derivative of $\sqrt{x}$ goes to zero quite quickly for large $x$, but is unbounded near zero. So if you picked uniform distribution with support close to zero, the remainder term can get large." CreationDate="2010-12-30T09:04:21.300" UserId="2116" />
  <row Id="8853" PostId="5790" Text="@mpiktas: Great answer, You put a lot of effort in it. I'm very grateful." CreationDate="2010-12-30T09:09:52.267" UserId="1643" />
  <row Id="8854" PostId="5790" Text="@Tomek Tarczynski, you're welcome. " CreationDate="2010-12-30T09:31:59.680" UserId="2116" />
  <row Id="8855" PostId="5825" Text="+1 for mentioning L-moments! Glad i'm not their only proselytizer around here. I'm not convinced that normalizing/standardizing by dividing by a measure of location is the best approach here though, which is why I suggested dividing by a measure of dispersion instead. See point number 2 involving temperature in the question." CreationDate="2010-12-30T09:49:15.027" UserId="449" />
  <row Id="8856" PostId="5812" Text="Thanks for the replies. The (small) dataset is not that homogeneous : the coefficient of variation reaches 26% for the modelled property they are modelling using 3 predictors. That is why I was looking more at the robustness of BRTs to the inclusion or exclusion of very few individuals (1 in case of LOOCV), and thus, more generally speaking the validity of LOOCV for BRTs." CreationDate="2010-12-30T10:15:03.470" UserId="2561" />
  <row Id="8857" PostId="5826" Text="What are the practical issues in using the gam fit?" CreationDate="2010-12-30T10:20:42.933" UserId="364" />
  <row Id="8858" PostId="5812" Text="Nevertheless, the resampling CV algorithm proposed for validation the whole BRT models is interesting and yes seems equivalent to bagging, already performed within the BRT fitting procedure." CreationDate="2010-12-30T10:23:25.380" UserId="2561" />
  <row Id="8859" PostId="5796" Text="Thank you Dikran! Could you explain to me the relation of KLR und Kernel smoothing? The KLR-model is built similar to the svm [loss + penalty]-formulation and solved via gradient descent. But the same time references (e.g. in &quot;Kernel Logistic Regression and the Import Vector Machine&quot;, Zhu and Hastie 2005) on KLR go to the smoothing-literature (e.g. &quot;Generalized Additive Models&quot;, Hastie and Tibshirani 1990)." CreationDate="2010-12-30T10:59:08.260" UserId="2549" />
  <row Id="8860" PostId="5826" Text="A bunch of things, but they mostly boil down to: whatever I fit has to be reproducible in SAS' `proc reg`." CreationDate="2010-12-30T11:25:05.680" UserId="1569" />
  <row Id="8861" PostId="5830" Text="Thanks Rob. I noticed that the `gam` in the mgcv package can also do penalised regression splines; would that be the same as in SemiPar?" CreationDate="2010-12-30T11:26:34.120" UserId="1569" />
  <row Id="8862" PostId="5821" Text="@Noah Your accounts are merged now, but you are still not registered -- so probably you'll loose your account again. You can register by visiting this link http://stats.stackexchange.com/users/login" CreationDate="2010-12-30T11:30:49.843" UserId="88" />
  <row Id="8863" PostId="5812" Text="@user2040 Well, the real criterion for T is large enough to stabilize result and small enough to finish analysis in a reasonable time." CreationDate="2010-12-30T11:36:52.413" UserId="88" />
  <row Id="8864" PostId="5812" Text="@backflip Again, you just need to test it making CV with larger tests, and I proposed resampling CV as the easiest solution to the problem with small overall number of objects. About bagging, yes it is the same. Indeed at first I thought BST is just a plain boosting, so I haven't mentioned bagging." CreationDate="2010-12-30T11:43:26.323" UserId="88" />
  <row Id="8865" PostId="5796" Text="I am not that familiar with the smoothing literature, but kernel models are closely related to spline smoothing.  I think the best place to look would be the publications by Grace Wahba (http://www.stat.wisc.edu/~wahba), whos work spans both smoothing and kernel methods." CreationDate="2010-12-30T11:43:41.410" UserId="887" />
  <row Id="8866" PostId="5796" Text="Thanks, i will have a closer look at wahba's publications. Can you recommend an implementation of KLR, at best in R?" CreationDate="2010-12-30T12:03:16.293" UserId="2549" />
  <row Id="8867" PostId="5809" Text="Do you mean that I'll have to perform a 3-way anova and look for significant effects?" CreationDate="2010-12-30T12:24:25.050" UserId="339" />
  <row Id="8868" PostId="5830" Text="It's similar, but a different implementation, so the results are not the same. I notice that Matt Wand (author of SemiPar) comments at http://www.uow.edu.au/~mwand/SemiPar.html that mgcv now does most of what was intended for SemiPar and that he advises people use mgcv instead." CreationDate="2010-12-30T12:44:25.727" UserId="159" />
  <row Id="8869" PostId="5814" Text="@onestop - In Traffic engineering, flow rates in this context could be considered as a snapshot. Let's say we're interested in finding the number of vehicles queued at an intersection every five minutes. You Look at the intersection, jot down the number of vehicles queued in the intersection, then you close your eyes for five minutes, open them up and count the vehicles again" CreationDate="2010-12-30T14:19:34.560" UserId="59" />
  <row Id="8870" PostId="5816" Text="...always remembering that you are reinventing the wheel with that: use it as a &quot;proof-of-concept&quot; that you can program, if they want to align short DNA fragments to a genome they would use BLAST." CreationDate="2010-12-30T14:56:33.600" UserId="582" />
  <row Id="8871" PostId="5827" Text="May I suggest that you talk to your friendly neighborhood statistician? This project seems to be too large and important to mess up, and it would be easy to do it incorrectly." CreationDate="2010-12-30T15:18:25.700" UserId="279" />
  <row Id="8872" PostId="5791" Text="@David Given these distributions come from different families, it seems that CVs will be difficult to interpret and compare among distributions.  Could you clarify then why you are tracking the change in CV and how you intend the CVs to be interpreted?" CreationDate="2010-12-30T15:23:44.140" UserId="919" />
  <row Id="8873" PostId="5820" Text="This question is a duplicate of http://stats.stackexchange.com/q/4220/919 ." CreationDate="2010-12-30T15:28:18.707" UserId="919" />
  <row Id="8874" PostId="5827" Score="1" Text="May I just emphasize what Aniko writes.  The biggest concern in my mind, after reading this question, is not about statistical procedure or software, but about what all this might *mean*.  Unless your data are an exhaustive survey (of a well-defined population) or obtained through a formal randomized procedure, considerable thought--of both a statistical and epidemiological nature--will be necessary to assure that you can learn *anything* from these data that is validly generalizable." CreationDate="2010-12-30T15:55:32.787" UserId="919" />
  <row Id="8875" PostId="5836" Text="So... why don't you write it as in the top graph?" CreationDate="2010-12-30T15:59:58.460" UserId="582" />
  <row Id="8876" PostId="5834" Text="You need to clarify what you mean, exactly, by &quot;DTMC.&quot;  For instance, let the states be $0, 1, \ldots, n, \ldots$.  Define a Markov chain with transition probabilities $p_{i,i+1} = 1$.  Because $f_{ii}=0$ for all $i$ it is transient.  Because every state is eventually reached from $0$ it is irreducible.  Nevertheless, *your intended conclusion is obviously not true* in this example." CreationDate="2010-12-30T16:03:33.963" UserId="919" />
  <row Id="8877" PostId="5791" Text="@whuber I would like to quantify the amount of information that we have about each parameter before and after adding data. I would like the figure to show something like CV as an index of how well a parameter is constrained, so that this can be compared to sensitivity of the model to the parameter and the contribution of the parameter to model output variance." CreationDate="2010-12-30T16:15:40.353" UserId="1381" />
  <row Id="8879" PostId="5327" Text="Since I'm trying to learn the assumptions of truncated regression, I've updated my question to try to focus the answers in that direction." CreationDate="2010-12-30T16:55:45.317" UserId="1583" />
  <row Id="8880" PostId="5835" Text="I'd agree with most of this except that instead of including age as a time-varying covariate you'd almost certainly be better using age as the time scale. See Kom, Graudbard &amp; Midthune 1997 http://aje.oxfordjournals.org/content/145/1/72." CreationDate="2010-12-30T16:57:08.577" UserId="449" />
  <row Id="8881" PostId="5834" Text="Perhaps the questioner forgot to add the condition that the state space is finite." CreationDate="2010-12-30T17:25:04.563" UserId="449" />
  <row Id="8882" PostId="5840" Text="Thanks, I'll take that into account for the future." CreationDate="2010-12-30T17:37:27.830" UserId="2443" />
  <row Id="8883" PostId="5838" Text="Thanks, this will work nicely I think." CreationDate="2010-12-30T17:38:21.513" UserId="2443" />
  <row Id="8884" PostId="5836" Text="@nico Lack of expertise on how to do it in R, basically." CreationDate="2010-12-30T17:39:11.713" UserId="2443" />
  <row Id="8885" PostId="5839" Text="Thanks, that's also a solution." CreationDate="2010-12-30T17:39:44.037" UserId="2443" />
  <row Id="8886" PostId="5836" Score="2" Text="@Jakob Borg: `paste` is your friend. Just construct a string like `legend.string &lt;- paste(&quot;In traffic [Max&quot;, max(values),&quot;]&quot;)` and then pass that as the `legend` parameter of `legend`" CreationDate="2010-12-30T17:53:48.597" UserId="582" />
  <row Id="8887" PostId="5834" Text="@onestop I initially suspected that, but then it occurred to me that in the finite case there must exist either an absorbing or a recurrent state; neither of those can be transient.  Thus a finite transient Markov chain does not exist." CreationDate="2010-12-30T18:07:28.647" UserId="919" />
  <row Id="8888" PostId="5839" Text="@dickoa (+1) Your solution is at least more elegant than mine because you don't duplicate the grouping variable! " CreationDate="2010-12-30T18:20:49.017" UserId="930" />
  <row Id="8892" PostId="5847" Score="1" Text="2500 - 2000 = $500.  Whether that's &quot;meaningful&quot; or not depends on how much you value this money ;-)." CreationDate="2010-12-30T20:09:48.667" UserId="919" />
  <row Id="8893" PostId="5847" Text="@whuber I can tell that one made more money, but I want to know if how to tell if that difference is likely due to chance, or if its due to the inherent superiority of the other page. " CreationDate="2010-12-30T20:12:08.597" UserId="142" />
  <row Id="8894" PostId="5847" Score="1" Text="@yc01 You can't answer that with the information given.  If your pages are offered randomly (using a random number generator) to all comers during the same period *and* you have the individual &quot;conversion&quot; values, then you can use simple, standard techniques for testing whether the difference can be attributed to non-chance mechanisms." CreationDate="2010-12-30T20:15:18.240" UserId="919" />
  <row Id="8895" PostId="5844" Text="what kind of connection you are looking for? KLR is for classification, smoothing splines are for non-linear approximation of conditional expectation. The way I see they are non-linear extensions of logistic and ordinary regressions respectively." CreationDate="2010-12-30T20:15:20.470" UserId="2116" />
  <row Id="8896" PostId="5848" Text="@whuber (+1) Ah... we cited the same book :)" CreationDate="2010-12-30T20:21:05.730" UserId="930" />
  <row Id="8897" PostId="5849" Text="@chi and @whuber: Thank you, I was wondering if The Good book was a *good* choice. For the problem types I laid out - basically ANCOVA with violations, am I on the right track with permutation or bootstrapping? " CreationDate="2010-12-30T20:22:29.047" UserId="2040" />
  <row Id="8898" PostId="5809" Text="@gd047: If your covariate is numeric, then yes, you can do an ANOVA predicting the covariate from the 3 predictor variables and their interactions. If your covariate is nominal, then you'll have to do a multinomial regression (again, predicting the covariate from the 3 predictor variables and their interactions). If you find suggestion that any of the predictor variables or their interactions predict the covariate, then you may need to reconsider using the covariate in the full model." CreationDate="2010-12-30T20:44:09.887" UserId="364" />
  <row Id="8900" PostId="5847" Text="I probably misunderstood the question, but where is the &gt;100% value coming from?" CreationDate="2010-12-30T21:09:36.650" UserId="582" />
  <row Id="8901" PostId="5844" Text="KLR seems to be coming out of nowhere: I found the name KLR to be mentioned first around 2000 (for example in &quot;kernel logistic regression and the import vector machine&quot;, Zhu and Hastie, 2001). But they say in there it is well known and reference smoothing spline literature. My Question is: When and where was KLR first introduced and why do they always reference smoothing-splines?" CreationDate="2010-12-30T21:15:13.230" UserId="2549" />
  <row Id="8902" PostId="5753" Text="As usual, a most thorough explanation you get a well deserved &quot;Best Answer&quot; check mark." CreationDate="2010-12-30T21:17:48.767" UserId="1329" />
  <row Id="8903" PostId="5753" Text="@Gaetan Lion You are very kind.  I have enjoyed reading *all* the answers to this question (and marked them all up)." CreationDate="2010-12-30T22:13:12.627" UserId="919" />
  <row Id="8904" PostId="5849" Text="@user2040 It's hard to go wrong with permutation tests.  Good has chapters specifically on multifactor designs, categorical data, and multivariate analysis (including MANCOVA).  Although I don't fully comprehend your specific problem, I'm sure you're find something useful there." CreationDate="2010-12-30T22:20:36.867" UserId="919" />
  <row Id="8905" PostId="5848" Text="@chl I don't think so: they're by the same author but have slightly different titles and different publishers.  Maybe we should each say a little more about them so we can determine which might be more appropriate for the OP.  I added a few details in a comment to your response." CreationDate="2010-12-30T22:23:49.373" UserId="919" />
  <row Id="8906" PostId="5848" Text="@whuber I deleted mine after seeing yours." CreationDate="2010-12-30T22:26:57.533" UserId="930" />
  <row Id="8907" PostId="5849" Text="@user2040 I will add some references but I found your two points difficult to understand as well. To my knowledge, there's no *exact* permutation test when the covariate is continuous." CreationDate="2010-12-30T22:29:37.763" UserId="930" />
  <row Id="8908" PostId="5849" Score="1" Text="@chl I think, if I correctly understand your comment about continuous covariates, that exactness depends on the role played by randomness in the data.  When randomization occurs *by design*, it doesn't seem to matter what kind of data you have.  The permutation test takes the data as given and simply lets us glimpse how the statistical results would turn out if our random number generators had (for example) resulted in different assignments of subjects to treatment and control groups." CreationDate="2010-12-30T22:36:17.477" UserId="919" />
  <row Id="8909" PostId="5848" Text="@chl Ah, I see.  So there's no redundancy." CreationDate="2010-12-30T22:37:05.633" UserId="919" />
  <row Id="8910" PostId="5834" Text="@whuber - thanks for clarifying that. I started suspecting as much as I pondered after submitting my previous comment so tried to add a question mark to its end but was outside the arbitrary 5-minute edit window." CreationDate="2010-12-30T22:49:11.437" UserId="449" />
  <row Id="8911" PostId="5847" Score="2" Text="@Nico You're right to wonder that.  The references to percentages and, indeed, to chi squared are red herrings.  In the new situation the observations are total dollar returns on page visits, not a percentage of anything (so of course a chi-squared calculation is wholly invalid).  In the old situation the observations are counts of &quot;conversions&quot; and visits for each page.  Chi squared enters as one (of several) possible approximations to a *test of proportions.*  (In both situations it's necessary that conversions be serially independent, which is a dubious but often necessary assumption.)" CreationDate="2010-12-30T23:53:24.303" UserId="919" />
  <row Id="8912" PostId="5849" Text="@chi and @whuber, Thanks again. I will see which of the Good books is the best (many puns intended). As far as my problem, basically it is a two sample experiment (treatment and control/no treatment) where there exist a pre-experiment baseline measure and a post treatment measure, the latter being the dependent variable (actually it is change in the measure from pre to post). So it would be a typical ANCOVA or ANOVA (depending on if the change is the dependent or post is, with the pre as a covariate) except that many of the post measurements are zero (the customer did not purchase anything)." CreationDate="2010-12-31T00:55:11.070" UserId="2040" />
  <row Id="8913" PostId="5780" Text="Wow, that Barnard's code is *slow*. The test doesn't seem to give substantively different results on my first trials, so I'll probably stick with the faster/more conservative Fisher. Haven't yet tried Wilcoxon -- overall dropout might be of interest, but only in a very peripheral way. Anyway, thanks for your help. Holm correction it is." CreationDate="2010-12-31T02:16:26.887" UserId="174" />
  <row Id="8914" PostId="5799" Text="(+1)  This is a great idea generally.  For large correlation matrices, however, there are so many statistics and so many of them can simultaneously be spuriously large that it usually pays to adjust.  Otherwise you wind up chasing a large number of misleadingly &quot;significant&quot; correlations that just disappear in the hold-out data.  (Run a simulation with, say, a few hundred draws from 50 uncorrelated standard normal variates.  It's an eye-opener.)" CreationDate="2010-12-31T05:14:57.070" UserId="919" />
  <row Id="8915" PostId="5775" Text="(+1) Good advice in light of the OP's comment to @Peter Flom's response." CreationDate="2010-12-31T05:16:26.850" UserId="919" />
  <row Id="8916" PostId="5837" Text="Thank you very much for the advise, I will think about this.. I agree that I need a statistician but: 1. I underestimated the problem, the previous smaller questions were easier and could be handled by ourselves; b. I'm in a progrma for a master in public health with special in epidemiology, the advanced statistics modules are only in 2011 and I wanted to have an idea now; c. I like to understand my statistical friend when we are discussing this. I'm a professor ath the University of Antwerp, we even have a epidemiology and biostatistical department ;-) (politics....)" CreationDate="2010-12-31T05:18:13.403" UserId="2571" />
  <row Id="8917" PostId="5835" Text="Thanks! Can you recommend any books on a. survival analysis of this kind; b. programming advise in any program (I'm not married to Stata, my university has a licence for all major programs; I tried R but the learning curve of both statistics and R was a bit to steep...)" CreationDate="2010-12-31T05:24:20.093" UserId="2571" />
  <row Id="8918" PostId="5853" Score="2" Text="If your set changes time from time it is impossible to fit it to the same bell-curve. Suppose you have the set $A=\{1,2,3,4\}$, so 4 should be 800, but if a new observation 5 arrives, then it should become 800.  " CreationDate="2010-12-31T07:43:23.000" UserId="2116" />
  <row Id="8919" PostId="5854" Text="@dev, what exactly are you displaying? It is not clear for me what kind of data your one data point represents. " CreationDate="2010-12-31T07:49:23.180" UserId="2116" />
  <row Id="8920" PostId="5854" Text="How about a kernel density plot? http://www.statmethods.net/graphs/density.html" CreationDate="2010-12-31T09:14:26.310" UserId="144" />
  <row Id="8922" PostId="5854" Text="@mpiktas: My data is Census data for villages. My website will allow the user to select an area on the map, and then will find all the villages in that area. The census data for a village consists of various values like: Male Population, Female population, Average household income etc for that village.  I hope to show the data distribution for a particular value (eg: Total Population) for all the villages falling in the user selected area." CreationDate="2010-12-31T09:25:56.697" UserId="2586" />
  <row Id="8923" PostId="5856" Text="@gd047 (+1) Forgot about that. It might be handy." CreationDate="2010-12-31T10:04:09.160" UserId="930" />
  <row Id="8925" PostId="5849" Text="@user2040 *Resampling Methods* is really a light version of the book that @whuber refered to, with computer illustrations (R, SAS, Stata, and even C++); so I'll recommend the latter for serious work. There are practical suggestions for dealing with classical exp. setups, unbalanced design, outliers, etc. (Chap. 6, 7 and 11). If you use a difference score, then it is basically a two-group comparison, but see [The Analysis of Pre-test/Post-test Experiments](http://j.mp/gbdPhn) by GE Dallal, and a permutation test should do the job. Bootstrap might be more interesting with unbalanced design." CreationDate="2010-12-31T10:56:16.317" UserId="930" />
  <row Id="8926" PostId="5861" Score="1" Text="I work in biological research. I know some colleagues (I mean, people with a PhD) who cannot really grasp boxplots. I would not use them to target a general audience." CreationDate="2010-12-31T11:17:57.843" UserId="582" />
  <row Id="8927" PostId="5863" Text="Note I wasn't suggesting he use violin plots for his applications, but a histogram with logarithmically spaced bins.  Violin plots was the answer to the question in the title (which was rather different to the question in the post itself)." CreationDate="2010-12-31T11:21:42.303" UserId="887" />
  <row Id="8928" PostId="5857" Score="3" Text="Quoting a friend of mine: if you want to &quot;hide&quot; something in a paper, put it in the text rather then in a figure. If you want to make sure nobody ever reads it put it in a table! ;) Just joking of course, but having a website with interactive maps for users to click etc. all of that to get a table... well that would be disappointing!" CreationDate="2010-12-31T11:28:12.830" UserId="582" />
  <row Id="8929" PostId="5853" Score="1" Text="Welcome to CrossValidated, NealWalters. You may find that a common response to asking a statistician &quot;How do i do this&quot; is &quot;Why do you want to do that?&quot;" CreationDate="2010-12-31T11:37:51.180" UserId="449" />
  <row Id="8930" PostId="5857" Text="@nico, yeah but sometimes tables are much more informative than graphs. I for example prefer table instead of a bad graph. In this case the table still can be represented by graph, and I suggested quantiles because they do not have problems with outliers.  " CreationDate="2010-12-31T11:46:09.140" UserId="2116" />
  <row Id="8931" PostId="5849" Text="@whuber I was referring to nuisance parameters like those encountered in fMRI studies (e.g., age) that might invalidate the exchangeability assumption, as discussed by [Tom Nichols (2008)](http://j.mp/dTr6O2), with reference to [Permutation Tests for Linear Models](http://j.mp/fxkhLe) by Anderson and Robinson, 2001, but this is probably off-topic there. Good makes it clear that &quot;we must assume that the residuals are exchangeable and both the concomitant variables and the regression coefficients are unaffected by the treatment&quot;. (p. 101)" CreationDate="2010-12-31T11:47:17.017" UserId="930" />
  <row Id="8932" PostId="5861" Text="@nico That's a fair point. But, this is not a reason not to use efficient graphical summary. A schematic illustration of what a boxplot actually does might help the reader." CreationDate="2010-12-31T11:53:31.160" UserId="930" />
  <row Id="8933" PostId="5861" Text="@chl: it really depends on what the target audience is and what the aim of the site is. Explaining boxplots would definitely help, but still, some people struggle a lot with the concept of distribution." CreationDate="2010-12-31T12:00:25.313" UserId="582" />
  <row Id="8934" PostId="5863" Score="1" Text="@nico You will probably like [Many Eyes](http://www-958.ibm.com/software/data/cognos/manyeyes/), [dataviz](http://www.improving-visualisation.org/visuals/), [datavisualization.ch](http://datavisualization.ch/), and [Ideas2evidence](http://www.ideas2evidence.com/showcase.html), to name a few." CreationDate="2010-12-31T12:12:31.200" UserId="930" />
  <row Id="8935" PostId="5861" Text="@nico Yes, I agree. Although boxplot is not mentioned in [A Tour through the Visualization Zoo](http://queue.acm.org/detail.cfm?id=1805128) -- but these are for large and complex data sets, I simply like it and I'm sorry to see that it is not much used in experimental sciences. Superimposing raw data is a way to help the reader visualizing the distribution." CreationDate="2010-12-31T12:17:41.333" UserId="930" />
  <row Id="8936" PostId="5861" Text="@chl: I know! I always try to &quot;convert&quot; my colleagues to boxplots, at least when it comes to writing papers, making presentations etc., but sometimes is though!" CreationDate="2010-12-31T12:25:38.093" UserId="582" />
  <row Id="8937" PostId="5863" Text="nice links @chl!" CreationDate="2010-12-31T12:30:10.603" UserId="582" />
  <row Id="8938" PostId="5862" Text="@onestop. Good point. Histograms ( or density plots with experiment with bandwidth) are a great solution here." CreationDate="2010-12-31T12:51:33.050" UserId="1307" />
  <row Id="8939" PostId="5864" Text="With the inclusion of all the non-donators, it seems that 80-90% of the data will be 0. Is the $t$-test still sufficiently robust if there's such a strong deviation from normality? It would also be interesting to know if the data is essentially categorical because people typically donate 5, 10, 15 dollars, ... That would mean a lot of ties, and Mann-Whitney wouldn't be appropriate." CreationDate="2010-12-31T12:52:11.800" UserId="1909" />
  <row Id="8940" PostId="5802" Text="@Tusell that's the whole idea behind CART's research is that choosing a stop criterion is suboptimal.  Grow a full tree then use pruning to find an optimal tree.  However, as @Adam has pointed out rpart doesn't exactly work like that.  Part of it I think is an optimization for large datasets.  If it can chop off a subtree early without exploring it can save way more CPU time than going through a full expansion and pruning session.  As you point out though that means some trees aren't fully explored and you have to tweak the cp param to make it work." CreationDate="2010-12-31T14:57:09.867" UserId="1929" />
  <row Id="8941" PostId="5853" Text="@onestop - see my link to previouspost.  In school, I remember of hearing of teachers who graded &quot;on the curve&quot;. So no matter what score you got on a test, only a certain percentage of people would get each grade.  I have no idea how credit scores work, but that's very similar to what I'm simulating.  I give people points for example when they pay off a loan.  But some people will pay off dozens of loans.  I want to condense my wild points into a set of normalized points. " CreationDate="2010-12-31T14:59:10.517" UserId="2585" />
  <row Id="8942" PostId="5802" Text="@Adam is R^2 calculated from the testing dataset or is it using the pruning set to calculate MSE?  " CreationDate="2010-12-31T15:04:15.197" UserId="1929" />
  <row Id="8944" PostId="5873" Text="If your goal is to provide evidence that the true proportion is greater than 95% then you probably want to switch your null and alternative hypothesis around.  Typically you put what you want to show in your alternative hypothesis." CreationDate="2010-12-31T16:10:21.523" UserId="1028" />
  <row Id="8945" PostId="5873" Text="@Dason, Good point. One of my profs said that you draw a stronger conclusion from rejecting the null hypothesis." CreationDate="2010-12-31T16:16:23.927" UserId="2591" />
  <row Id="8946" PostId="5870" Text="The quality of an approximation tends to depend on the values of the parameters.  Can you stipulate anything about the values of $n$ or $\beta$ that you are interested in?  For example, how small can $n$ be?  How close to $1/2$ can $\beta$ be?" CreationDate="2010-12-31T16:17:29.817" UserId="919" />
  <row Id="8947" PostId="5870" Text="@Whuber: It can be assumed that $\beta$ is far from $\frac{1}{2}$ lets say $\beta \approx 0.1$. Case when n is big is simple, so I'm more interested in case where n is rather small: $10 \le n \le 20$" CreationDate="2010-12-31T16:45:35.107" UserId="1643" />
  <row Id="8948" PostId="5872" Text="@Didier Piau: I'm looking for the distribution of $R_{n}$ for fixed n, but I would be also satisfied with first two moments. When n approaches infinity it is easy to calculate them." CreationDate="2010-12-31T16:48:03.467" UserId="1643" />
  <row Id="8949" PostId="5873" Score="2" Text="&quot;Sample 10% of the population&quot; has almost no basis in statistical theory or practical application.  (In fact, it is contrary to statistical theory, which shows that usually the absolute sample *size* matters, not its percentage of the population.)  More importantly, any such rule is assuming something about the trade-off between the costs of sampling and the value of improved decision making *to you, in your particular circumstance*.  That's *exactly* the sort of thing you don't want a stats cookbook or glib Web site to decide for you!" CreationDate="2010-12-31T16:48:55.250" UserId="919" />
  <row Id="8950" PostId="5873" Score="2" Text="A good guiding principle is **Wald's sequential testing theory**.  The Wikipedia article includes a (brief) example identical to your question (but testing for a 99% proportion rather than a 95% proportion): http://en.wikipedia.org/wiki/Sequential_probability_ratio_test" CreationDate="2010-12-31T16:54:31.203" UserId="919" />
  <row Id="8952" PostId="5871" Text="What is true, however, is that if there exists a state $i$ such that $f_{ii}=1$, or if there exist two different states $i$ and $j$ such that $f_{ij}=f_{ji}=1$, then the Markov chain is recurrent." CreationDate="2010-12-31T17:08:32.753" UserId="2592" />
  <row Id="8953" PostId="5872" Text="@Tomek: &quot;to calculate them&quot; in the sense of &quot;to compute some simple asymptotic equivalents of them&quot;, I presume." CreationDate="2010-12-31T17:13:39.393" UserId="2592" />
  <row Id="8954" PostId="5872" Text="@Didier Piau: Yes, of course I meant to calculate asymptotic equivalent." CreationDate="2010-12-31T17:22:06.170" UserId="1643" />
  <row Id="8955" PostId="5875" Text="@whuber: Calculating this by brute force is deffinitely a solution, but I wanted to avoid, but if there is no better idea I will use it. Thanks for the code, it will save me some of the work." CreationDate="2010-12-31T17:25:31.080" UserId="1643" />
  <row Id="8956" PostId="5872" Text="@Tomek: then the asymptotic equivalents you are interested in are $2\beta n$ for the expectation of $R_n$ and $2n(\frac14 -\beta^2)$ for its variance." CreationDate="2010-12-31T17:37:47.843" UserId="2592" />
  <row Id="8957" PostId="5875" Text="@Tomek For $n \lt 10$ we can't hope to find good approximations.  Given that the brute force computation is easy to code and executes fairly quickly, the main reason for looking for *any* approximations would be to improve our understanding of the behavior of the sample range.  If that's your situation, the sum in your question can be reduced to a case of the hypergeometric function.  That's a trivial result but it suggests analytical ways of studying the range and creating approximations." CreationDate="2010-12-31T17:52:03.073" UserId="919" />
  <row Id="8958" PostId="5872" Text="@Didier Comparisons with the examples I posted for $n=20$ show we'll need much larger $n$ before these approximations are reasonably close.  Note, too, that for extreme values of $\beta$ (including near $0$), the distribution tends to be quite skewed, implying that more than the expectation and variance are needed for decent characterization.  Until $n$ gets huge, then, @Tomek's suggestion to use Normal approximations looks more worthwhile." CreationDate="2010-12-31T17:56:28.640" UserId="919" />
  <row Id="8959" PostId="5872" Text="@whuber: thanks for your comment. Why do you write that the linear approximations of the expectation and variance are not &quot;reasonably close&quot; to the actual values? I see nothing about this in your post, so... did you compute these values and choose not to post them? About the last sentence of your comment: the equivalent of the variance I wrote in my last comment is precisely a consequence of the normal approximation you advocate." CreationDate="2010-12-31T18:18:20.023" UserId="2592" />
  <row Id="8960" PostId="5876" Text="@whuber +1, I've deleted my answer, since your was way better and made mine obsolete :)" CreationDate="2010-12-31T18:42:04.997" UserId="2116" />
  <row Id="8961" PostId="5868" Text="What exactly are you trying to estimate? If you want to estimate maximum of the data as each new observation arrives the most simple estimate is the sample maximum. Maybe you want some distribution properties, like mean, variance, quantiles?" CreationDate="2010-12-31T18:49:43.930" UserId="2116" />
  <row Id="8962" PostId="5872" Score="1" Text="@Didier Good questions. (1) $2\beta n$ in my examples equals about 0, 6.7, and 13.3.  The distributions have modes at 4, 8, and 15, respectively.  Especially in the case $\beta=0$ the approximation is pretty far off (more than 1 SD wrong).  (2) The normal approximation @Tomek proposes differs from yours: one approximates each of the $X_i$ with a Gaussian and from that does the integration to estimate the distribution of the range.  The result is decidedly non-normal yet, when $(1/2-\beta)n \ge 5$ or so, should work quite well with a continuity correction." CreationDate="2010-12-31T19:45:05.410" UserId="919" />
  <row Id="8963" PostId="5878" Text="Hmm... you think it will be enough? Well, I guess that would be easier for me. And, on second thought, after I gather my data, I will always be able to change the algorithm if I see that another one gives better results." CreationDate="2010-12-31T20:14:38.340" UserId="2590" />
  <row Id="8964" PostId="5878" Text="(Oh, btw, you have a typo - it's &quot;winsorized&quot;, not &quot;windsorized&quot;. Took me a moment to figure it out, since I hadn't heard about it before.)" CreationDate="2010-12-31T20:18:27.963" UserId="2590" />
  <row Id="8965" PostId="5878" Text="@Vilx, thanks, I fixed a typo and added wikipedia links. I posted answer via mobile, so that is why I missed typo and did not add links. Also in your place I would wait for other answers, since you cannot expect a lot of them in new year's eve :)" CreationDate="2010-12-31T20:40:57.217" UserId="2116" />
  <row Id="8966" PostId="5872" Text="@Didier I did a calculation to check @Tomek's normal approximation for $n=20, \beta=1/3$.  It qualitatively reproduces the pdf.  The errors in the pdf are usually less than 0.01 and the error in the cdf has a max of about 0.02 at $k=14$.  We can expect better performance for $\beta$ closer to $0$ and worse performance for smaller $n$.  The numerical integration, which takes some effort, will become computationally more efficient than exact calculation somewhere around $n=50$ I guess." CreationDate="2010-12-31T20:43:24.003" UserId="919" />
  <row Id="8967" PostId="5878" Text="I know, but recently I learned that the &quot;accepted&quot; mark can be removed and re-assigned at any time, so I better accept the answer now, because later I'll just forget to do it. If there is a better answer, I'll accept that. :)" CreationDate="2010-12-31T20:47:29.807" UserId="2590" />
  <row Id="8968" PostId="5878" Text="@Vilx, cheers:)" CreationDate="2010-12-31T21:26:23.327" UserId="2116" />
  <row Id="8969" PostId="5835" Text="Applied Survival Analysis by Hosmer and Lemeshow is good; if you use SAS, then Survival Analysis Using SAS by Allison is very good and clear." CreationDate="2010-12-31T22:08:25.927" UserId="686" />
  <row Id="8970" PostId="5878" Text="Cheers to you too! :)" CreationDate="2010-12-31T22:13:24.523" UserId="2590" />
  <row Id="8971" PostId="5879" Text="Sounds complicated and unstable. On one hand I understand that statistics and predictions are inherently unstable and imprecise, on the other hand the programmer in me wants an absolute and 100% correct answer. XD I think now that it might indeed better to start out simple (simple winsorized mean sounds good), and then, when I have gathered a significant amount of raw data, I can start thinking about upgrading the algorithm." CreationDate="2010-12-31T22:18:54.023" UserId="2590" />
  <row Id="8972" PostId="5879" Text="P.S. Happy new year! :) (It's 2011 here already)" CreationDate="2010-12-31T22:19:16.033" UserId="2590" />
  <row Id="8973" PostId="5879" Text="@Vilx Definitely more complicated than simple averages! -- but you stated you are &quot;going for maximum accuracy,&quot; which I took as a strong hint that complications wouldn't be an obstacle for you.  As far as &quot;unstable&quot; goes, it's hard to tell what you mean here.  What you might do is monitor the total completion time for the first few users just to see how much variation there is.  This will indicate how much (or little) would be gained by accounting for their individual differences." CreationDate="2010-12-31T22:22:00.267" UserId="919" />
  <row Id="8974" PostId="2763" Text="@shabbychef would you be willing to share your ascii_hist function by any chance?" CreationDate="2011-01-01T00:02:14.660" UserId="2594" />
  <row Id="8975" PostId="5857" Text="That is what I am currently doing(Showing the deciles on a graph), but after showing it to some of our target audience, we received feedback, that the graphs were not easy to understand." CreationDate="2011-01-01T05:04:10.337" UserId="2586" />
  <row Id="8976" PostId="5863" Text="Thank you for the Links!" CreationDate="2011-01-01T05:10:02.183" UserId="2586" />
  <row Id="8977" PostId="5879" Text="Well, no, complications aren't an obstacle, as long as someone helps me write out the relevant formulas. XD And sorry about the &quot;unstable&quot;. :( I know that the estimates are going to be just that - estimates, and that they will only approximately represent the users remaining time. But since you're even guessing at the method to calculate them (logarithms? square roots?) then that gives the impression of _very_ approximate results. But then - you know your craft much better than I do and it's not really for me to say. Sorry! :(" CreationDate="2011-01-01T08:36:06.650" UserId="2590" />
  <row Id="8978" PostId="5872" Text="@whuber: Three comments. First, about &quot;Tomek's normal approximation&quot;: to what post/comment are you referring? Once again, the asymptotic variance $2n(\frac14-\beta^2)$ follows from a **normal** approximation. Second, when $\beta&gt;0$, did you compare $2\beta n$ and the **expectations** (not the **modes**) of your simulations? Third, when $\beta=0$, $R_n$ is of order $\sqrt{n}$ and $2R_n/\sqrt{n}$ converges in distribution to a non degenenerate positive random variable $\max(N_1,N_2,N_3)-\min(N_1,N_2,N_3)$ with $(N_i)$ i.i.d. standard normal." CreationDate="2011-01-01T11:55:46.493" UserId="2592" />
  <row Id="8980" PostId="5869" Text="Thanks Dikran -- however book publishers have also studied extreme values; would anyone know of an online introduction ?" CreationDate="2011-01-01T16:00:05.403" UserId="557" />
  <row Id="8981" PostId="5872" Text="@Didier (1) The original question.  (2) Look at the plot in my response.  The expectation for $\beta=0$ is larger than the mode; the expectations for the other values are close to the modes.  (3) Fine.  But &quot;convergence&quot; tells us what happens for arbitrarily large $n$.  Without additional information it tells us *nothing* about particular values of $n$.  When we turn to the actual distributions, it's clear the asymptotic estimates are not very good even for small $\beta$ and $n=20$ and it's also apparent the distributions are not approximately normal for $\beta$ near $0$ or near $1/2$." CreationDate="2011-01-01T17:13:57.870" UserId="919" />
  <row Id="8982" PostId="5879" Text="@Vilx The business about logarithms or roots has to be a guess *for me* because I don't have the data.  For *you* it won't be a guess because there are standard and rigorous ways to ascertain appropriate re-expressions of data; they are part of exploratory data analysis." CreationDate="2011-01-01T17:16:30.210" UserId="919" />
  <row Id="8983" PostId="5881" Text="+1 Nice answer!" CreationDate="2011-01-01T17:18:50.037" UserId="919" />
  <row Id="8984" PostId="5869" Score="2" Text="http://books.google.com/books?id=kXCg8B5xSUwC&amp;lpg=PP1&amp;dq=Statistics+of+Extremes+gumbel&amp;pg=PP1&amp;hl=en#v=onepage&amp;q&amp;f=false" CreationDate="2011-01-01T17:21:27.897" UserId="919" />
  <row Id="8985" PostId="5870" Text="@Tomek Just for clarification (because it came up as a question in some comments), I take it that your allusion to the normal approximation should be interpreted to mean we can approximate each $X_i$ with a normal variable and perform the integrations to estimate the distribution of the range.  (Note that this is *not* the same as a &quot;sum&quot; of normal distributions, nor is it easy to calculate.)  Is this what you were thinking of, or have I misinterpreted your idea?" CreationDate="2011-01-01T17:26:51.590" UserId="919" />
  <row Id="8986" PostId="5879" Text="Ahh, I see. Well then, it's even more paramount that I get the data first. I'll start with that, and then, when I have the data, I'll return here. :) Thank you for the explanation!" CreationDate="2011-01-01T18:07:30.787" UserId="2590" />
  <row Id="8987" PostId="5870" Text="@whuber: To be honest I was considering both versions, but the one that I wrote about was different from what you proposed. $X_{1}$ and $X_{3}$ can be approximated using normal distribution and then we just approximate the range by the $X_{1}-X_{3}$ which have normal distribution. It works quite well for big n. Your method gives better results, but it is much harder to calculate." CreationDate="2011-01-01T21:52:06.110" UserId="1643" />
  <row Id="8988" PostId="5870" Text="@Tomek Thank you for the clarification.  Yes, when $n$ is large, the problem is an easy exercise, because you can be practically certain the range equals $X_1 - X_3$.  Thus the challenge is to compute the distribution when there's an appreciable chance that $X_2$ will exceed $X_1$ or be less than $X_3$.  This happens when $2\beta^2n/(1-2\beta^2)$ is less than 6 or so." CreationDate="2011-01-01T22:06:39.237" UserId="919" />
  <row Id="8989" PostId="5883" Text="Actually this is pretty close to the kind of stuff I'm looking for." CreationDate="2011-01-02T00:12:52.640" UserId="2293" />
  <row Id="8992" PostId="2763" Text="@btown the margin is too small to contain it!" CreationDate="2011-01-02T05:34:12.510" UserId="795" />
  <row Id="8994" PostId="5885" Text="@moderators: Community Wiki?" CreationDate="2011-01-02T10:05:43.967" UserId="449" />
  <row Id="8995" PostId="5885" Text="@onestop Sure, done." CreationDate="2011-01-02T11:23:39.303" UserId="88" />
  <row Id="8997" PostId="5892" Score="2" Text="@Ben It would be better to post a new question with reference to this one. Your data and hypothesis are likely to be somewhat different, and different clues could be provided to you." CreationDate="2011-01-02T12:02:51.683" UserId="930" />
  <row Id="8999" PostId="5887" Score="1" Text="How did you divide the data? Randomly, by hand, or some other method? Though, in truth, the part about &quot;successfully created a model&quot; is a MUCH larger part of the issue. Before doing expensive things, you should see if you're using the appropriate type of model, if you've overfit your training data, and if you have the appropriate data for what you are trying to predict." CreationDate="2011-01-02T15:06:44.310" UserId="1764" />
  <row Id="9000" PostId="5879" Text="&quot;on the other hand the programmer in me wants an absolute and 100% correct answer&quot;. I'd suggest changing your user base to automatons of some sort." CreationDate="2011-01-02T15:22:57.477" UserId="1764" />
  <row Id="9001" PostId="5874" Text="How much freedom do you have to change question ordering? For example, a first page that has three or four questions that divide users into several classes (business, private person one job over last X years, private person Y jobs over last X years, private person who does consulting on the side, etc) that might have more homogenous question paths and times. Also, can questions be skipped or must they be answered in order?" CreationDate="2011-01-02T15:56:05.983" UserId="1764" />
  <row Id="9002" PostId="5874" Text="Also, I'd suggest that you display a range, like &quot;Remaining time: 20-45 minutes&quot; that narrows as time goes by, not just a point estimate." CreationDate="2011-01-02T15:59:22.563" UserId="1764" />
  <row Id="9004" PostId="5871" Text="understood, but it is true that for a transient DTMC Lt k-&gt;infinity pij(k) =0 for all i,j . How to prove that? We know that E(Mj|X0=i)=sigma k=1 to infinity pij(k). Here Mj= number of visits to state j. We can show that E(Mj|X0=i)=fij/1-fij . We need to show that for transient DTMC E(Mj|X0=i)=finite then it is obvious that pij(k) when k-&gt;infinity is zero. But we can't say that fij&lt;1 for all i,j. Then how to prove that this expectation will be finite. Hope I am making my point clear." CreationDate="2011-01-02T17:54:44.460" UserId="2516" />
  <row Id="9005" PostId="5893" Score="2" Text="presumably you could get something about geographic location as well, if you could get the appropriate matching data? You could also use information about popularity of first names over time (google &quot;baby name wizard&quot;) to make inferences about age ..." CreationDate="2011-01-02T18:25:22.133" UserId="2126" />
  <row Id="9006" PostId="5871" Text="The point here is that, if the Markov chain is transient, then $f_{jj}&lt;1$ for every state $j$. To visit $j$ at least $n+1$ times starting from $i$, one first has to reach $j$, which happens with probability $f_{ij}$, and then to come back to $j$ at least $n$ times, which happens with probability $(f_{jj})^{n}$. Thus $E(M_j|X_0=i)$ is the sum over $n\ge0$ of $f_{ij}(f_{jj})^n$, which is finite if and only if $f_{jj}&lt;1$, and then $E(M_j|X_0=i)=f_{ij}/(1-f_{jj})$. (Be careful: your comment is awfully written and you write &quot;equals zero&quot; instead of &quot;converges to zero&quot;.)" CreationDate="2011-01-02T19:22:26.973" UserId="2592" />
  <row Id="9007" PostId="5896" Score="2" Text="@Didier It would be interesting to expand that response a little bit, especially to show how you get this result." CreationDate="2011-01-02T19:55:59.290" UserId="930" />
  <row Id="9008" PostId="5897" Text="Thanks. That's what I started with. But was using an old edition of Walpole which did not have much information on it." CreationDate="2011-01-02T19:58:52.403" UserId="2610" />
  <row Id="9009" PostId="5874" Text="@Wayne - yes, I too had a range in mind, though didn't voice it. I think that's a rather minor implementation detail. But anyway, nice to know that others think the same way. :) About the ordering - for the most part I can order them any way I like. It makes more sense to group similar questions together though. And, of course, if a question depends on a previous question, then that order is fixed. But big class-defining questions will definitely go up front." CreationDate="2011-01-02T19:59:06.223" UserId="2590" />
  <row Id="9010" PostId="5874" Text="Questions cannot be skipped as per se, but some can be left empty. Like, if you didn't have any income abroad, then you just leave that table empty. However in most cases I think that will be preceded by a yes/no question in the form &quot;did you have any income abroad&quot;? Or something like that. If you answer &quot;no&quot;, then the question with the table will be skipped. It's the &quot;not applicable&quot; point in the original question." CreationDate="2011-01-02T20:00:47.737" UserId="2590" />
  <row Id="9011" PostId="5879" Text="@Wayne, ehh, wishful thinking...." CreationDate="2011-01-02T20:02:02.337" UserId="2590" />
  <row Id="9012" PostId="5899" Text="Why does candidate 1 have only 3 votes? Does that mean he also has 2 &quot;no hire&quot;? In that case one may choose number 2!!!" CreationDate="2011-01-02T21:23:31.923" UserId="582" />
  <row Id="9013" PostId="5901" Text="Why limit tries to 27?" CreationDate="2011-01-02T22:34:17.020" UserId="449" />
  <row Id="9014" PostId="5899" Score="1" Text="Or were two members of the committee out to lunch while candidate 1 was being interviewed? In that case one might treat their votes as 'missing data'." CreationDate="2011-01-02T22:40:19.187" UserId="449" />
  <row Id="9015" PostId="5901" Text="I was thinking that there had to be some kind of limit and that after checking 27 times that you would stop. But thats a presumption that I probably should not have made.. What would you suggest?" CreationDate="2011-01-02T23:32:29.293" UserId="2612" />
  <row Id="9016" PostId="5739" Text="Indeed it is a very good idea!" CreationDate="2011-01-03T01:16:41.213" UserId="1709" />
  <row Id="9017" PostId="5899" Text="These kinds of examples are also better understood if you list all of the ballots cast.In the example you gave, the ballots are based on a point system, so a board member could theoretically give +2 for several candidates, while another board member might only give one candidate a +2." CreationDate="2011-01-03T02:10:43.323" UserId="660" />
  <row Id="9018" PostId="2563" Text="This isn't just a data analysis question, because many users become aware of the voting system and vote strategically. In other words, the kind of measurement is changing the behavior. Economics (political economy and game theory in particular) has a lot of interesting things to say about choosing ranking algorithms." CreationDate="2011-01-03T02:16:17.643" UserId="660" />
  <row Id="9019" PostId="5862" Text="You are completely right, that the Histogram is the most understood way to show a distribution. I will try to make histograms with both the axes in log scale." CreationDate="2011-01-03T03:47:14.080" UserId="2586" />
  <row Id="9020" PostId="5871" Text="yes, I understood my mistake of finding the expectation after posting this. Thanks for pointing it also. " CreationDate="2011-01-03T04:26:12.783" UserId="2516" />
  <row Id="9021" PostId="5887" Text="BTW, I forgot to turn on the cynicism mode before the &quot;successfully created a model&quot; " CreationDate="2011-01-03T07:28:19.093" UserId="2604" />
  <row Id="9022" PostId="5862" Score="2" Text="I'm only suggesting using a log scale for the x-axis. I don't think a log scale for the frequency axis would be a good idea, as then the shaded area of each bar of the histogram wouldn't be proportional to the number of observations." CreationDate="2011-01-03T09:06:18.880" UserId="449" />
  <row Id="9023" PostId="5900" Text="Looks set to kick Gilks, Richardson &amp; Spiegelhalter (1996) into the long grass when that comes it in May." CreationDate="2011-01-03T09:35:05.333" UserId="449" />
  <row Id="9025" PostId="5893" Text="The guys at [Cross Validated](http://stats.stackexchange.com/) might be able to better serve this question." CreationDate="2011-01-03T01:14:40.623" />
  <row Id="9026" PostId="5910" Text="What about people who have the same name as you... there are a number of &quot;Dean Harding&quot;s out there, one of them was even a professional footballer! The &quot;DeanHarding&quot; on twitter is not me, there's hundreds of &quot;Dean Harding&quot;s on Facebook, etc etc..." CreationDate="2011-01-03T02:06:36.493" />
  <row Id="9027" PostId="5910" Text="That depends on chance, of course. Usually you can find out which one is it by profession, location, etc. though I saw cases where there were 3 persons with the same full name, in the same profession and living roughly in the same area. Then of course it becomes harder :)" CreationDate="2011-01-03T04:02:50.243" />
  <row Id="9028" PostId="5761" Text="@James, could you point to some (real or synthetic) data for your problem ?" CreationDate="2011-01-03T11:19:51.490" UserId="557" />
  <row Id="9029" PostId="5905" Text="Good info Dave. Why do you have strong reservations about my proposed &quot;preference ranking&quot; based voting system? Note that the CEO bit was just an example... my domain is actually different." CreationDate="2011-01-03T11:19:56.217" UserId="2605" />
  <row Id="9030" PostId="5904" Text="I added some code using JAGS where I seem to be getting a different answer. Where is my mistake? Is this happening because of the priors?" CreationDate="2011-01-03T12:20:31.243" UserId="2617" />
  <row Id="9031" PostId="5896" Score="2" Text="@chl: Yessir! The OP asks for $E(N)$ where $N\ge1$ is the date of the first success in a series of independent trials, each with probability $p=1/27$ of success. The event $[N\ge n+1]$ corresponds to the $n$ first trials being unsuccessful, hence (by independence) $P(N\ge n+1)=(1-p)^n$. Now, for every nonnegative integer valued $N$, $E(N)$ is the sum of $P(N\ge n+1)$ from $n=0$ to $+\infty$. For every complex number $a$ such that $|a|&lt;1$, the sum of $a^n$ from $n=0$ to $+\infty$ is $1/(1-a)$. For $a=1-p$, one gets $E(N)=1/p=27$." CreationDate="2011-01-03T13:30:12.853" UserId="2592" />
  <row Id="9034" PostId="5896" Text="@Didier (+1) Thank you." CreationDate="2011-01-03T14:22:38.980" UserId="930" />
  <row Id="9036" PostId="5917" Text="rajah9's suggestion to have the stakeholders divvy up $100 is a variation of a range voting system (with a rating ballot)." CreationDate="2011-01-03T15:11:50.763" UserId="660" />
  <row Id="9037" PostId="5905" Score="1" Text="@varuman - I tend to prefer voting systems when you have relatively large constituencies that are unlikely, unwilling, or unable to engage in dialog and deliberation as part of the selection process. On the other hand, small groups that already have open lines of communication would benefit strongly by attempting to select the CEO by a deliberative and consensus-oriented approach. If the group does end up needing to vote numerically, I would be agnostic about which voting system to use until I know more details about the problem space." CreationDate="2011-01-03T15:16:08.700" UserId="660" />
  <row Id="9038" PostId="5919" Text="Sorry - I should have told what @CP suggested in his comments. Theoretically, b|y under a flat prior and classical CI are the same, but you can't do that practically in JAGS unless you use a very very diffuse prior like CP suggested and use a lot of MCMC iterations." CreationDate="2011-01-03T16:17:23.443" UserId="1307" />
  <row Id="9039" PostId="5924" Text="Thanks very much for your answer. I will pass it on to my colleague. " CreationDate="2011-01-03T17:06:42.220" UserId="1157" />
  <row Id="9040" PostId="5926" Score="1" Text="Who says they shouldn't be &quot;too strongly correlated&quot;, i.e. what's the source of that quote?" CreationDate="2011-01-03T17:38:14.300" UserId="449" />
  <row Id="9041" PostId="5915" Text="+1 For noticing the strange treatment of critical values in the OP's code!" CreationDate="2011-01-03T17:59:28.553" UserId="919" />
  <row Id="9042" PostId="5918" Text="@user2040 (+1) Those questions are very good ones, indeed! A somewhat related question can be found here: [Feature selection for “final” model when performing cross-validation in machine learning](http://stats.stackexchange.com/q/2306/930)." CreationDate="2011-01-03T19:02:13.523" UserId="930" />
  <row Id="9043" PostId="5928" Text="I'm still working on it.  My initial model was similar to the one @Srikant Vadali proposed, but with an equality constraint on total population imposed.  I implemented it as a Poisson GLM with *linear* link.  It worked rather poorly on realistic test data.  My suspicion is that the population density in an area may depend as much, or even more, on the characteristics of the *surrounding* area as it does on the characteristics of that area itself.  I'm still open to creative suggestions or other references!" CreationDate="2011-01-03T19:08:13.580" UserId="919" />
  <row Id="9044" PostId="5927" Score="2" Text="The determinant is not invariant with respect to reparameterization of the problem.  Thus, you cannot expect there to be any universal measure of flatness: you need to translate the &quot;flatness&quot; into something *meaningful for the problem.*  Looking at simultaneous confidence regions (which are typically estimated with the Hessian) might be a good place to start." CreationDate="2011-01-03T19:13:31.297" UserId="919" />
  <row Id="9045" PostId="5823" Text="@onestop you make a good point about reporting the change in means and the change in variability separately, and your proposed metrics are nice and intuitive. I am going to have to play around with these, and research some of the leads given by @shabbychef before deciding which path to choose, thanks for your answer" CreationDate="2011-01-03T19:20:26.693" UserId="1381" />
  <row Id="9046" PostId="5825" Text="@shabbychef thanks for these leads, the windsorization approach seems to be the most straightforward, but the L-moments approach also looks promising based on the Vigilone 2010 article, so I will check them out. Can you recommend a good textbook reference?" CreationDate="2011-01-03T19:22:02.763" UserId="1381" />
  <row Id="9047" PostId="5825" Text="@David, my favorite text on 'robust statistics' is Wilcox' 'Introduction to Robust Estimation and Hypothesis Testing.' (2nd edition; 3rd ed to appear in July) http://amzn.to/hcpZm2 This text describes trimmed means, the McKean-Schrader estimate, etc, but does not put them together in terms of coefficients of variation. The author has an R package, see http://www-rcf.usc.edu/~rwilcox/  I do not have a good reference on L-CV other than Viglione, but I believe the L-CV is just the Gini coefficient, up to a rescaling. @onestop would know better than I about that." CreationDate="2011-01-03T19:41:02.243" UserId="795" />
  <row Id="9048" PostId="5825" Text="@David, Wilcox has a more 'accessible' text, 'Applying Contemporary Statistical Techniques', but I prefer the 'Introduction' text. YMMV." CreationDate="2011-01-03T19:42:46.080" UserId="795" />
  <row Id="9049" PostId="5825" Text="@shabbychef thanks for these, off to the library" CreationDate="2011-01-03T19:47:30.223" UserId="1381" />
  <row Id="9050" PostId="5927" Text="whuber's caveat notwithstanding, I do think you're on the right track: The expectation of the Hessian (i.e., the inverse Fisher-Information) is the asymptotic covariance of the (asymptotically normal) MLE, which is the main reason you can use it to construct the confidence regions whuber mentions. Its determinant is a suitable (albeit nor invariant, see above) measure of the &quot;flatness&quot; of the likelihood around the MLE. " CreationDate="2011-01-03T20:18:15.233" UserId="1979" />
  <row Id="9051" PostId="5919" Text="@Ringold I have merged your accounts so you can edit your questions and add comments. Yet, please register your account by clicking here: http://stats.stackexchange.com/users/login ; you can use your Gmail OpenID to do it in a few seconds and you'd not loose your account here any more." CreationDate="2011-01-03T20:18:16.697" UserId="88" />
  <row Id="9052" PostId="5919" Text="Thanks, I have registered. And many thanks to those who answered this question. I will try the gamma prior." CreationDate="2011-01-03T20:37:14.450" UserId="2617" />
  <row Id="9054" PostId="5928" Text="Also this one: P. A Zandbergen and D. A Ignizio, “Comparison of Dasymetric Mapping Techniques for Small-Area Population Estimates,” Cartography and Geographic Information Science 37, no. 3 (2010): 199–214. http://www.ingentaconnect.com/content/acsm/cagis/2010/00000037/00000003/art00004&#xA;&#xA;Which seems to call out for blending." CreationDate="2011-01-03T21:08:31.680" UserId="82" />
  <row Id="9055" PostId="5928" Text="This paper might be useful: Hwahwan Kim and Xiaobai Yao, “Pycnophylactic interpolation revisited: integration with the dasymetric-mapping method,” International Journal of Remote Sensing 31, no. 21 (2010): 5657. http://www.informaworld.com/10.1080/01431161.2010.496805&#xA;&#xA;" CreationDate="2011-01-03T21:10:51.773" UserId="82" />
  <row Id="9056" PostId="5927" Text="@whuber, @fabians," CreationDate="2011-01-03T22:00:46.083" UserId="1004" />
  <row Id="9057" PostId="5927" Text="@whuber, @fabians,  What about using the largest eigenvalue of the hessian of -2 x loglikelihood, ( the hessian should be ideally postive definite at the mle), if largest eigenvalue is &quot;small&quot; in some sense, then the likelihood is flat. Also, is there a way to reduce to simultaneuous regions into one number?" CreationDate="2011-01-03T22:06:19.777" UserId="1004" />
  <row Id="9058" PostId="5893" Score="1" Text="I have merged the transferred question with the duplicate." CreationDate="2011-01-03T22:20:21.507" UserId="88" />
  <row Id="9059" PostId="5927" Text="@acarchau That idea does not overcome the arbitrariness-of-scale objection.  For example, suppose you're estimating location and scale for a distributional family and you find that both eigenvalues are 1000.  What does this mean, pray tell?  Why, we don't even know what the units of measurement are!  Unless I give you more information, these statistics are *absolutely meaningless.*" CreationDate="2011-01-03T22:23:43.643" UserId="919" />
  <row Id="9061" PostId="5936" Text="@user2040. I don't understand how would you do the &quot;type 3&quot; test? Is it something SAS related? (sorry my SAS knowledge is very limited). I would have done a logistic regression as you suggested but with 2 dummy variables. Also, given that I understand correctly, if you do logistic regression, testing if some or all the coefficients are 0 is done by deviance (or likelihood ratio) and it IS asymptotically Chi-Sq (not necessarily with df = 1)" CreationDate="2011-01-03T22:52:01.170" UserId="1307" />
  <row Id="9063" PostId="5937" Text="@tal can you define 'tiny'? Does 'manual' calculation include use of a calculator?" CreationDate="2011-01-03T23:05:15.167" UserId="1381" />
  <row Id="9064" PostId="5944" Text="I still need to think about the contrast. I will correct it whenever I get time. Sorry about that" CreationDate="2011-01-03T23:30:56.427" UserId="1307" />
  <row Id="9065" PostId="5942" Text="Is it just me or is their closed form incorrect?  Seems like their *n* changes meaning to 365-*n* in the final step." CreationDate="2011-01-03T23:35:12.740" UserId="2622" />
  <row Id="9066" PostId="5923" Text="Thanks, this worked." CreationDate="2011-01-03T23:37:26.047" UserId="2617" />
  <row Id="9067" PostId="5936" Text="@suncoolsu: Yes, practically speaking you should get the same conclusion. I should not have said &quot;equivalent&quot; (I work with big data so they end up the same). I added some code in the answer to help clarify." CreationDate="2011-01-03T23:42:11.503" UserId="2040" />
  <row Id="9068" PostId="5802" Text="If you use &quot;rsq.rpart(fit)&quot;, where fit is your rpart model, you can see that cp is calculated using the decrease in relative error. " CreationDate="2011-01-04T08:47:36.180" UserId="2166" />
  <row Id="9069" PostId="5937" Text="Hi David - calculator is fine - but the idea is to not have too many numbers (Since they all need to be written on the board) " CreationDate="2011-01-04T09:24:35.620" UserId="253" />
  <row Id="9070" PostId="5940" Text="Hi David - the site you linked to is really great - thank you." CreationDate="2011-01-04T09:37:36.967" UserId="253" />
  <row Id="9071" PostId="5938" Text="Good idea wolfgang, I didn't think of it." CreationDate="2011-01-04T09:38:00.350" UserId="253" />
  <row Id="9072" PostId="5943" Text="Very nice - thank you." CreationDate="2011-01-04T09:38:42.380" UserId="253" />
  <row Id="9073" PostId="5949" Text="Good example Andy - I've learned something new today :)" CreationDate="2011-01-04T09:39:00.427" UserId="253" />
  <row Id="9074" PostId="5956" Text="@Gavin (+1) Far more complete response than mine; I just like it." CreationDate="2011-01-04T10:25:53.780" UserId="930" />
  <row Id="9075" PostId="5956" Text="@Gavin The &quot;Via the segmented package&quot; section didn't work for my data. I got a &quot;No breakpoint estimated&quot; after running the `segmented` command. " CreationDate="2011-01-04T10:35:10.893" UserId="339" />
  <row Id="9076" PostId="5954" Text="+1 Great question..." CreationDate="2011-01-04T10:42:41.697" UserId="88" />
  <row Id="9077" PostId="5955" Text="+1 ... and a great answer!" CreationDate="2011-01-04T10:43:56.233" UserId="88" />
  <row Id="9078" PostId="5946" Score="1" Text="I might just be showing my ignorance here, but isn't the assumption behind ANOVA that the residuals are normal? In that case it doesn't really matter if the variable itself is non-normal, as long as the residuals fit the pattern." CreationDate="2011-01-04T10:56:08.293" UserId="656" />
  <row Id="9079" PostId="1185" Text="That is very true, and also in psychology, there's an annoying confusion between non-parametric and non-normal, which seems to hinder understanding." CreationDate="2011-01-04T10:58:48.107" UserId="656" />
  <row Id="9080" PostId="5956" Text="@gd047: Apologies, there was an error in the code I showed. You need to supply argument `seq.Z` with a one sided formula of the variable(s) that have a segmented relationship with the response. I've edited my answer to include `seq.Z = ~ sqft` and added a note about having `segmented` choose values of `psi` for you." CreationDate="2011-01-04T11:52:34.943" UserId="1390" />
  <row Id="9081" PostId="5956" Text="@gd047 I would like to remove my answer as this one addresses your original question in a more better way. Would mind accepting this one instead of mine?" CreationDate="2011-01-04T12:00:51.510" UserId="930" />
  <row Id="9082" PostId="5956" Text="@chl Of course, even though I still get an error : Error in if (model) objF$model &lt;- mf : &#xA;  argument is not interpretable as logical&#xA;In addition: Warning message:&#xA;In if (model) objF$model &lt;- mf :&#xA;  the condition has length &gt; 1 and only the first element will be used" CreationDate="2011-01-04T12:16:19.787" UserId="339" />
  <row Id="9083" PostId="5769" Text="PyIMSL is not open source, is it?" CreationDate="2011-01-04T12:19:47.663" UserId="2454" />
  <row Id="9084" PostId="5956" Text="@gd047: Sorry, first day at work for 2 weeks and the brain isn't working. I've edited the code and it now works for me on R 2.12-patched with the latest version of segmented (0.2-7.3). This did work fine for me at home so I think I introduced errors when copying code into the answer. Check if it works now and if not, check your R and packages are up to date; Vito released a new version of the package in early December." CreationDate="2011-01-04T12:29:58.170" UserId="1390" />
  <row Id="9085" PostId="5956" Text="@gd047 The last command from @Gavin works provided we pass `model=TRUE` when calling `segmented(...)` (don't know why, even after looking at the source code of `segmented.lm`, line 104?!)." CreationDate="2011-01-04T12:30:33.610" UserId="930" />
  <row Id="9086" PostId="5956" Text="@chl: strange, both work for me without `model = TRUE`, and that is the default for this argument in `segmented`, so I wouldn't expect this to cause a problem." CreationDate="2011-01-04T12:33:34.423" UserId="1390" />
  <row Id="9087" PostId="5956" Text="@Gavin I'm on a Mac (os x 10.6) with R 2.12 and `segmented` 0.2-7.3. And yes, `model=T` is the default (didn't notice that BTW). And now, it works! Ok, found it: there was an extra `~ sqft` in your original formula." CreationDate="2011-01-04T12:37:23.337" UserId="930" />
  <row Id="9088" PostId="5956" Text="@chl: Yes, that's right. In my first edit to fix code, I didn't see `seg.Z` explicitly, so added it not realising I had included it as an unnamed argument already. I fixed that problem with my second edit. Not sure why the first version (before all my editing) failed as it worked for me at home with up to date packages and R. My first edit was in error. Apologies for the confusion!" CreationDate="2011-01-04T12:47:07.547" UserId="1390" />
  <row Id="9089" PostId="5956" Text="@Gavin No probs. Congrats for such a nice job and thx for the follow-up!" CreationDate="2011-01-04T12:52:44.567" UserId="930" />
  <row Id="9090" PostId="5921" Text="Thanks for all your wonderful advice; it’s much appreciated! My colleague [posted some more info here](http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5961#5961); does this affect anything?" CreationDate="2011-01-04T13:20:03.547" UserId="2616" />
  <row Id="9091" PostId="5915" Text="Thanks for all your wonderful advice; it’s much appreciated! My colleague [posted some more info here](http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5961#5961); does this affect anything?" CreationDate="2011-01-04T13:20:38.007" UserId="2616" />
  <row Id="9092" PostId="5916" Text="Thanks for all your wonderful advice; it’s much appreciated! My colleague [posted some more info here](http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5961#5961); does this affect anything?" CreationDate="2011-01-04T13:22:53.933" UserId="2616" />
  <row Id="9093" PostId="5918" Text="@chi Thank you, I had seen that post. Do you think I am on the right track with my thought process at least? It seems that an independent test set allows us to be more liberal in our use of CV for feature selection and model tuning / selection. Otherwise, nested loops appear required to train, tune and estimate error generalization all using the same training data." CreationDate="2011-01-04T13:25:25.510" UserId="2040" />
  <row Id="9094" PostId="5915" Text="@Mathias Bynens, if you are really going the way of rounding degrees of freedom, the only fix is to drop the division by 2." CreationDate="2011-01-04T13:25:33.103" UserId="2116" />
  <row Id="9095" PostId="5960" Score="2" Text="Intriguing question. I've know nothing about this so won't attempt an answer, but googling bimodality+test gives quite a few promising leads. &#xA;http://www.google.co.uk/search?q=bimodality+test" CreationDate="2011-01-04T13:29:09.263" UserId="449" />
  <row Id="9096" PostId="5942" Text="I don't think so. Just write out the definition of the binomial coefficient, and you'll see that the equality works. Note that ${n\choose k} = {n\choose {n-k}}$" CreationDate="2011-01-04T13:54:29.007" UserId="279" />
  <row Id="9098" PostId="5916" Text="A sample size of 5 is a bit low. Can you increase the minimum a bit, say to 10 or 20? It's not just a question of rounding the d.f. The test is more sensitive to non-normality with small samples, and strictly speaking, your rule of sampling until the margin of error is below some threshold should affect the critical values of the test. " CreationDate="2011-01-04T14:33:00.493" UserId="449" />
  <row Id="9099" PostId="5961" Text="Please delete the t-table: it is pointless.  It would suffice to say your are using the upper 97.5 percentile for your critical region.  Of more import is a description of how the testing works and some examples of how the timings are distributed.  These variances differ so greatly that it makes no sense to test for equality of means." CreationDate="2011-01-04T16:06:17.813" UserId="919" />
  <row Id="9100" PostId="5948" Text="How does it help to suggest that this is an &quot;undergrad&quot; question?  Is that relevant to the answer?  Does it suggest ways to search for an answer on the Web?  I hope this remark wasn't intended to belittle the questioner." CreationDate="2011-01-04T16:29:06.123" UserId="919" />
  <row Id="9102" PostId="5916" Text="@onestop: Could you please explain a bit further? Especially the part about affecting the critical values of the test. A larger sample changes the degrees of freedom, since `df = sample size - 1`, but beyond that I’m afraid I don’t really understand what you mean. Thanks in advance!" CreationDate="2011-01-04T17:01:38.713" UserId="2616" />
  <row Id="9103" PostId="5961" Text="@whuber, The t-table is used to calc the margin of error of each benchmark result (each benchmark has a sample of runs).&#xA;&#xA;Because of such difference in variances would a simple comparison of mean+-margin of error suffice?&#xA;&#xA;You can see a sample test http://jsperf.com/benchmark-js-test-page.&#xA;&#xA;Testing is done by determining the min time to run a test (in most cases the smallest unit is 1ms so the time needed for a percentage uncertainty of 1% is 50ms ((1 / 2) / 0.01), so a test is executed repeatedly until the time passed is &gt;= 50ms, the time is clocked, and the executions per second are computed" CreationDate="2011-01-04T17:11:06.040" UserId="2634" />
  <row Id="9104" PostId="5961" Text="@John-David You might must as well show us a multiplication table to illustrate the multiplying you do :-).  The sample test page is well done.  You could enhance it slightly by indicating how many iterations each result is based on.  You might also screen out obvious outliers.  What I'm still unsure of is what you are comparing to what!  If you're comparing my ops/sec to someone else's for the same benchmark, then you are probably best off using *logarithms* of the timing for your comparison.  That should equalize the variances, eliminating that complication." CreationDate="2011-01-04T17:25:16.357" UserId="919" />
  <row Id="9105" PostId="5967" Score="1" Text="Why does heteroscedasticity in the *independent* variables bother you?  Are you using a mixed model in which they are treated as random variables?" CreationDate="2011-01-04T17:28:16.057" UserId="919" />
  <row Id="9106" PostId="5961" Text="I believe they are comparing different approaches, not different users (though it is not clear if users are aggregated). That could justify different variances, as one algorithm might be amortized or somesuch. However, since runtimes may not be normal, I would recommend a brief look at some data to see if, for example, an Erlang distribution offset from below and censored from above would be a better fit than the Normal. Also, the test stopping condition isn't independent of the data, which should ideally be addressed." CreationDate="2011-01-04T17:47:56.697" UserId="2456" />
  <row Id="9107" PostId="5916" Text="The property that distinguishes the t-distribution from the normal is that it takes into account the sample size. See for example http://www.math.unb.ca/~knight/utility/t-table.htm - note how the critical values are substantially larger at small degrees of freedom." CreationDate="2011-01-04T17:55:42.493" UserId="2456" />
  <row Id="9108" PostId="5961" Text="@whuber I posted the table to show we are not using a formula to resolve the critical value but rather a simple table and to show you/others the values in the table in case you/others took issue with them or caught something." CreationDate="2011-01-04T17:57:37.937" UserId="2634" />
  <row Id="9109" PostId="5961" Text="@whuber, &#xA;&#xA;&quot;You might also screen out obvious outliers&quot; &gt;&#xA;So one pass compute standard deviation (SD) and things, remove outliers, then a second pass recompute SD and margin of error, and things?" CreationDate="2011-01-04T18:00:19.450" UserId="2634" />
  <row Id="9110" PostId="5961" Text="@whuber, &quot;What I'm still unsure of is what you are comparing to what!&quot; &gt;&#xA;In the example *(I linked to earlier)* when you click &quot;Run Tests&quot;, and it finishes, it will rank each test (&quot;Sort ascending&quot;, &quot;Sort descending&quot;, &quot;Don’t sort at all&quot;, and so on) based on their own mean ops/sec *(so it's comparing different tests to each other)*. The Browserscope results show the results of other browsers users ran the benchmarks in." CreationDate="2011-01-04T18:01:32.070" UserId="2634" />
  <row Id="9111" PostId="5967" Text="whuber, I am not sure in what direction our model will ultimately go.  I don't think we will use our independent variables as random variables.  And, if we do I'd be concerned that their respective standard error around the regression coefficient is unstable because of heteroskedasticity.  Otherwise, I may be completely wrong to be bothered by heteroscedasticity in the independent variable.  But, I don't know if I should or should not be bothered.  That's the essence of my question. " CreationDate="2011-01-04T18:12:13.110" UserId="1329" />
  <row Id="9112" PostId="5974" Text="I'm measuring the time taken to perform a specific operation, and then running an application that performs that operation repeatedly.  The output is just a dump of the latencies for each operation." CreationDate="2011-01-04T18:29:45.347" UserId="2638" />
  <row Id="9113" PostId="5961" Text="@John-David Thank you for explaining.  I was confused because the heading for the Browserscope says &quot;Results&quot; and it is the only tabular summary.  I now understand you are (a) expressing standard errors as percentages and (b) expressing mean speeds as percentages of the max.  Both indicate you are already thinking in terms of *relative* (*i.e.*, proportional) comparisons, which strongly suggests basing the analysis on logs.  (A difference in logarithms translates to a proportional difference in the original units.)" CreationDate="2011-01-04T18:32:32.773" UserId="919" />
  <row Id="9114" PostId="5961" Text="@John-David As far as the t-table goes, it's just laying out the results of 100 calculations, exactly like a multiplication table would lay out 100 multiplications.  It doesn't matter *how* you get the t-values; what matters is *what they represent.*  You are using them to construct two-sided 95% confidence intervals around the mean; that's all we (or anyone else) need to know." CreationDate="2011-01-04T18:34:17.450" UserId="919" />
  <row Id="9115" PostId="5974" Score="1" Text="so if you're getting different regression outputs for each test right? i may be misunderstanding, but can you're question also be stated as &quot;does this model A offer more explanatory power than this model B&quot;? " CreationDate="2011-01-04T18:37:41.833" UserId="2644" />
  <row Id="9116" PostId="5961" Text="@whuber, Thanks for your advice. Would you point me to some resources, or specific formulas for comparing logarithms, I can use/learn about?" CreationDate="2011-01-04T18:37:54.657" UserId="2634" />
  <row Id="9117" PostId="5963" Text="How will you identify the centers of the two normal distributions?" CreationDate="2011-01-04T18:41:59.587" UserId="2535" />
  <row Id="9118" PostId="5973" Text="@user2643: Do you have any references to share on how you created the nested CV? Was it along the same lines as the pdf I linked to in my question? Also.....is this data marketing data by chance?" CreationDate="2011-01-04T18:50:18.943" UserId="2040" />
  <row Id="9119" PostId="5916" Text="Oh, that's not actually a relevant. The question was about your sampling rule affecting the test. The issue there is that the t-statistics you are comparing to are built with the assumption that consecutive measurements are independent, but yours are not. Since you stop after a margin is sufficiently low, you are stopping all cases where the margin dips below the threshold but would soon come back up, and therefore you slightly overstate significances." CreationDate="2011-01-04T19:05:20.607" UserId="2456" />
  <row Id="9120" PostId="5973" Text="@user2643 The problem with that approach (which is correct) is that it only yields a single criterion for accuracy (classification) or precision (regression); you won't be able to say &quot;those are the features that are the most interesting ones&quot; since they vary from one fold to the other, as you said. I've been working with genetic data (600k variables) where we used 10-fold CV with embedded feature selection, under a permutation scheme (k=1000, to be comfortable at a 5% level) to assess reliability of the findings. This way, we are able to say: &quot;our model generalizes well or not&quot;, nothing more." CreationDate="2011-01-04T19:09:30.337" UserId="930" />
  <row Id="9121" PostId="5980" Text="May I ask you to clarify your answer.  I can't tell if you think the independent level variables are a problem and should be transformed or not." CreationDate="2011-01-04T19:13:13.093" UserId="1329" />
  <row Id="9122" PostId="5977" Text="Canyou clarify answer.  You start by suggesting that heteroskedasticity is a problem.  And, you move on by suggesting that multicollinearity is more of a problem and that heteroskedasticity is Ok.  " CreationDate="2011-01-04T19:15:42.783" UserId="1329" />
  <row Id="9123" PostId="5968" Text="@Berkan +1 but (1) You can format your code using indentation (with 4 spaces or through the online editor); (2) Some comments around your code might be appreciated because users that are not familiar with Matlab syntax will have hard time to understand what the above code chunk actually does. As far as I understand, predicted class memberships on the test sample are stored at each iteration, and then averaged to yield the classification CR." CreationDate="2011-01-04T19:16:37.083" UserId="930" />
  <row Id="9124" PostId="5972" Score="1" Text="You would probably want to treat k/n&gt;0.25 cases as $\alpha=0.5$ and not 0, but that does not change the bias issue." CreationDate="2011-01-04T19:17:24.453" UserId="279" />
  <row Id="9125" PostId="5962" Score="1" Text="Re disappointment: You wouldn't expect each problem to have the same parameters, so why *would* you expect the problems to share good values for the hyperparameters (log gamma and C)? " CreationDate="2011-01-04T19:19:26.017" UserId="1739" />
  <row Id="9126" PostId="5972" Text="@Aniko: Of course You are right, I've already changed it." CreationDate="2011-01-04T19:21:16.220" UserId="1643" />
  <row Id="9127" PostId="5945" Text="I'm not really up on SVM theory, but the 'flatness' in the kernel-machines discussion you link to seems to amount to: 'has small *second* derivative' (think of the typical motivation for spline smoothing models)." CreationDate="2011-01-04T19:32:00.497" UserId="1739" />
  <row Id="9128" PostId="5873" Text="@whuber, thank you for directing me toward Wald's SPRT. What's the correct way to compute the likelihood ratio for this experiment?And does http://www.quantitativeskills.com/sisa/statistics/sprt.htm count as a glib web site, or does their calculator take everything into account?" CreationDate="2011-01-04T19:34:23.270" UserId="2591" />
  <row Id="9129" PostId="5977" Text="well, without seeing the data I cant tell either way, which is a problem. I was simply stating the actual effects of multicoll and heterosked, because it seems as if in your answer you are concerned with heterosked and the standard errors around the regression plane, but as I understand it, multicollinearity is more of a problem when concerning the standard errors and when estimating coefficients, not heterosked. I'm just unsure of which one actually might be the problem." CreationDate="2011-01-04T19:41:54.587" UserId="2644" />
  <row Id="9130" PostId="5950" Text="so this is basically regression with a 'tube' loss function (0 penalty for points +/- epsilon of the prediction) rather than the quadratic loss function from OLS?" CreationDate="2011-01-04T19:43:51.930" UserId="1739" />
  <row Id="9131" PostId="5982" Text="Could you add a bit more context please? e.g. what R function and package are you using?" CreationDate="2011-01-04T19:50:44.087" UserId="449" />
  <row Id="9132" PostId="5962" Text="@Conjugate Prior: The training sets are subsets of the same experiment, and the negative training sets are sampled from the same population, so I had hoped that the same RBF kernel width ɣ would be effective.  Because the positive sets are being discriminated from the same background (negative) population, I had hoped that the ideal penalty C would be similar as well.  If this is not the case, it makes SVM really hard to apply.  Gentle boosting, for instance, seems much easier to tune." CreationDate="2011-01-04T19:51:39.807" UserId="220" />
  <row Id="9133" PostId="5950" Text="@Conjugate Prior: yes, usually kernel regression minimizes an 'epsilon-insenstive loss' function, which you can think of as $f(x) = (|x| - \epsilon)^+$ see e.g. http://kernelsvm.tripod.com/ or any of the papers by Smola _et al_." CreationDate="2011-01-04T19:51:47.690" UserId="795" />
  <row Id="9134" PostId="5962" Text="Aha.  But it seems to me that although it is the same experiment in the physical sense, you are nevertheless attacking separate and different problems in the statistical sense.  Particularly if the negative cases are resampled for each treatment." CreationDate="2011-01-04T19:59:07.473" UserId="1739" />
  <row Id="9135" PostId="5950" Text="@shabbychef  Thanks.  I always wondered what was going on there." CreationDate="2011-01-04T20:00:03.967" UserId="1739" />
  <row Id="9136" PostId="5950" Text="@Conjugate Prior: I don't think this is actually the desired loss function, but the math ends up working out well, so they ran with it. At least that's my suspicion." CreationDate="2011-01-04T20:02:19.327" UserId="795" />
  <row Id="9137" PostId="5982" Text="A link explaining why random-walk needs a burn-in also would help." CreationDate="2011-01-04T20:12:08.037" UserId="2116" />
  <row Id="9138" PostId="5980" Text="@Dmitrij Celov, I've upvoted your answer, since I was intending to write something similar, but please fix the language" CreationDate="2011-01-04T20:18:44.600" UserId="2116" />
  <row Id="9139" PostId="5948" Text="@whuber, no the remark was made with a smile and happy memory of Prof. Bentley's combinatorics class. No disparagement was intended." CreationDate="2011-01-04T20:21:04.110" UserId="2591" />
  <row Id="9140" PostId="5948" Text="@rajah9 I am very glad to read your clarification.  It's easy to misunderstand parenthetical comments when they are only written and not spoken." CreationDate="2011-01-04T20:24:56.530" UserId="919" />
  <row Id="9141" PostId="5977" Text="I think heteroskedasticity is a problem regarding the standard error and the related confidence intervals around a specific regression coefficient.  " CreationDate="2011-01-04T20:33:34.450" UserId="1329" />
  <row Id="9142" PostId="5873" Score="1" Text="@rajah9 That site looks balanced and reasonable.  I did not read it in detail, but I looked for unsupported assertions (like &quot;always do this&quot; or &quot;10 is a minimum&quot;, etc.) and didn't find any.  I did find a straightforward applet and lots of references, both of which are good signs of quality.  As always, don't rely on *anybody's* software until you have checked it out with data where you know the answer." CreationDate="2011-01-04T20:33:46.427" UserId="919" />
  <row Id="9143" PostId="5961" Text="@John-David I'm sure the Web is full of such resources.  I posted some materials relevant to statistics, for post-graduate professionals who may have forgotten all about logarithms, at http://www.quantdec.com/envstats/notes/class_03/gm_etc.htm ." CreationDate="2011-01-04T20:36:29.170" UserId="919" />
  <row Id="9144" PostId="5980" Text="Even though you try to apply stepwise selection it is not a good practice to pool into one set of the variables those that are in levels (trendy one) and flows (% changes, changes, etc.), especially when you try to model variable that is a % change. My guess is if you do include the lagged level of mortgage loans you could be surprised even more. In this case think about error-correction models if they can help you. Pardons for bad English.   " CreationDate="2011-01-04T20:37:40.580" UserId="2645" />
  <row Id="9145" PostId="5967" Text="whuber, giving this more thought here is why an independent level variable bothers me...  It is because of its temporal dimension.  Let's say you consider what happens to the dependent variable when the Dow is at 5000.  Well, a couple of decades ago this may be an economically bullish indicator several thousands points above the relevant level at the time.  But, today it has a completely different meaning.  It would entail another wrenching crash with dire economic implication.  Thus, I don't see how such level variables could really work. " CreationDate="2011-01-04T20:38:42.967" UserId="1329" />
  <row Id="9146" PostId="5982" Text="Well we will use diffinv() directly, but the burn-in default could be found in arima.sim() function. " CreationDate="2011-01-04T20:39:01.733" UserId="2645" />
  <row Id="9147" PostId="5969" Text="It seems that you're on the right track but you might want to look at the interaction between cognitive ability and your other predictors." CreationDate="2011-01-04T20:42:10.907" UserId="601" />
  <row Id="9148" PostId="5969" Text="@John that's exactly what I want to do. Are you suggesting that I'm not able to do that with mixed-models?" CreationDate="2011-01-04T21:01:46.387" UserId="2322" />
  <row Id="9149" PostId="5973" Text="If features vary from fold to fold it means that there isn't enough information to confidently identify the useful features, so I'd view that as an advantage of cross-validation (as just looking at the results from a single model is likely to have over-fit the feature selection criterion and hence be misleading).  For problems with many features and few observations, ridge regression often gives better performance, so unless identifying features is a key goal, it is often better not to do any feature selection." CreationDate="2011-01-04T21:08:48.567" UserId="887" />
  <row Id="9151" PostId="5989" Text="Hah!  I think my proposal is worse than yours :-)." CreationDate="2011-01-04T21:18:28.043" UserId="919" />
  <row Id="9152" PostId="5986" Text="I've got two questions (at least for now):$&#xA;&#xA;$&#xA;1) What is the value of estimator g when n=16 and k=8? I'm asking because it seems to be complex.&#xA;2) Could You elaborate $h_{\lambda}(k,n)=\lambda(n)k/n$? I'm not sure what $\lambda(n)$ is." CreationDate="2011-01-04T21:20:53.630" UserId="1643" />
  <row Id="9153" PostId="5989" Text="The no-free lunch theorem suggest that there will be problems where it works well and other algorithms dont! ;-P" CreationDate="2011-01-04T21:21:59.787" UserId="887" />
  <row Id="9154" PostId="5990" Text="+1 If you wanted a straw man, that would indeed be a good one!" CreationDate="2011-01-04T21:25:13.497" UserId="887" />
  <row Id="9155" PostId="5986" Text="@Tomek Good catch: I simply left out the most crucial part of the definition of $\delta$!  It's in now.  It is the obvious one: when the argument of the square root is negative, just set it to zero.  In other words, estimate $\alpha = 1/2$ whenever $n/4 \le k$.  (There was no such error in the plots, however: this was only a typo.)  $\lambda$ is simply a number but it can change from one $n$ to another.  For a fixed $n$ that makes $h$ a linear function of $k$." CreationDate="2011-01-04T21:47:29.780" UserId="919" />
  <row Id="9156" PostId="5942" Text="Yes, &quot;more manageable&quot; for the question of doubles, assuming only a d6 and not a d365.  For triples, etc., you need more powerful methods." CreationDate="2011-01-04T22:04:56.787" UserId="919" />
  <row Id="9157" PostId="5947" Text="I just want to comment that anyone familiar with the Binomial theorem and multinomial coefficients can easily use this generating function to write down a closed-form formula for any specified pattern (a partition of the number of dice).  It is a product of two multinomial coefficients.  The point of this response is that you can go pretty far without any such knowledge, solely by exploiting generating functions." CreationDate="2011-01-04T22:08:29.437" UserId="919" />
  <row Id="9158" PostId="5989" Text="@Dikran Thank you.  Do you have a favorite reference or references to these NFL theorems?" CreationDate="2011-01-04T22:09:39.887" UserId="919" />
  <row Id="9159" PostId="5990" Text="@Dikran You have read my answer as it was intended: tongue in cheek.  It's amusing to try to think of anything that could be worse and somewhat amusing to think of more sophisticated attempts that, in the end, haven't performed any better than a coin flip!" CreationDate="2011-01-04T22:11:05.517" UserId="919" />
  <row Id="9160" PostId="5989" Text="The original papers were by David Wolpert, there are some references and info at http://www.no-free-lunch.org/ .  &quot;no free lunch for the sandwich&quot; would be a favourite reference, but only because of the title ;o)" CreationDate="2011-01-04T22:51:10.740" UserId="887" />
  <row Id="9161" PostId="5986" Text="@Whuber: Great, now I know how to make the calculations that You have made, just to clarify $h_{\lambda}=\lambda(n)k/n$ should be changed to $h_{\lambda}=min(0.5,\lambda(n)k/n)$. Thank You very much for the answer, I'm still considering whether something better might be done." CreationDate="2011-01-04T23:16:15.717" UserId="1643" />
  <row Id="9162" PostId="5980" Text="Dmitrij, intuitively I really agree with you.  And, that is why I am asking the question.  But, what is the mathematical argument supporting our position?  I came up with a reasonably good logic argument as shown in my comments above answering whuber.  But, logic is not math...  " CreationDate="2011-01-04T23:16:41.550" UserId="1329" />
  <row Id="9163" PostId="5962" Score="1" Text="BTW, grid search is rather inefficient, the Nelder-Mead simplex optimisation algorithm is very effective, as are gradient descent optimzation methods.  Grid search is simple, but a bit &quot;brute force&quot;." CreationDate="2011-01-05T00:59:36.573" UserId="887" />
  <row Id="9164" PostId="5990" Score="1" Text="I am curious if one can do even _worse_ than a coin flip in the case of unknown proportions of classes, but it seems probably not." CreationDate="2011-01-05T01:07:32.037" UserId="795" />
  <row Id="9165" PostId="5986" Text="@Tomek Yes, good observation.  In fact, the $h$ I ended up using is $\min(1/2, \lambda k / n, \lambda (n-k) /n)$.  Although this is not precisely linear in $k$, it is linear in a &quot;constrained&quot; sense.  Linearity doesn't really matter; what matters is to find an effective estimator of *any* form." CreationDate="2011-01-05T01:42:04.977" UserId="919" />
  <row Id="9166" PostId="5994" Text="A fascinating account, but completely wrong (where it makes sense at all)." CreationDate="2011-01-05T03:23:58.383" UserId="919" />
  <row Id="9167" PostId="4899" Text="Did you mean $n=1$ instead of the (undefined) $n=0$?" CreationDate="2011-01-05T03:25:55.793" UserId="919" />
  <row Id="9168" PostId="5969" Text="not at all... I'm suggesting that what you want to find is an interaction between cognitive ability and your other predictors.  All you need to do is add them to the model." CreationDate="2011-01-05T03:39:37.527" UserId="601" />
  <row Id="9169" PostId="4899" Text="Yes indeed, thx" CreationDate="2011-01-05T03:40:52.250" UserId="1709" />
  <row Id="9170" PostId="5950" Text="@shabbychef: I'm still lost. Consider the one-dimensional case: $y = \theta x$. All minimizing $\theta$ does is give you a more *horizontal* line. It seems to have nothing to do with the second derivative, which I think you're referring to (&quot;smoothness&quot;). And if my sample points are (0,0) and (1,1e9), why would I prefer a flatter line? I.e., say my $\epsilon$ tolerance is 1 - why would I prefer the flatter line from (0,0) to (1,1e9-1) ($\theta=1e9-1$) instead of the line through (1,1e9) ($\theta=1e9$) or the line through (1,1e9+1) ($\theta=1e9+1$)?" CreationDate="2011-01-05T05:43:10.450" UserId="1720" />
  <row Id="9171" PostId="5950" Text="@Yang: I do not see it as a &quot;smoothness&quot; issue either. The page at http://kernelsvm.tripod.com also refers to flatness as  &quot;model complexity,&quot; which is more in line with my view. For example, if most of the elements of the vector $\theta$ were zero, you would consider this a less complex model. Flatness is a more continuous version of this. The main problem with your counterexample is that the $\epsilon$ tolerance is far too small for the units in which you measure the response $y$. " CreationDate="2011-01-05T06:29:28.210" UserId="795" />
  <row Id="9172" PostId="5996" Text="Well, I do know that the process is not stationary. Burn-in period's purpose is to be more or less confident that the starting history values are already present in the stochastic trend, in the sense that we may assume that we have started from the long long past. It is not clear why I have to go from zero if I don't want to. And else it would be interesting to hear some suggestions about the same questions when the process is random walk with drift component." CreationDate="2011-01-05T07:12:00.373" UserId="2645" />
  <row Id="9173" PostId="5980" Text="Gaetan, search for the information in Brooks &quot;Introductory Econometrics&quot; or whenever about the &quot;spurious regression&quot;, &quot;error correction models&quot;. Working with time series non-experimental objects is a bit (but significantly!) different from what you see in general statistics. So you may be concerned about at least several things: are your regression results significant purely at random (an easy check is to perform rolling regression and see the evolution of parameter estimates), are the level parameters more or less the same (then linear restriction? but opposite in sign!). Be cont." CreationDate="2011-01-05T07:25:59.357" UserId="2645" />
  <row Id="9175" PostId="5973" Text="@Dikran I should have give more precision about my study: Yes, the objectives was feature selection (in an $n\ll p$ context, as commonly found in [genome-wide association studies](http://en.wikipedia.org/wiki/Genome-wide_association_study)). When the outcome is univariate, I still prefer the elasticnet criterion, but anyway it's not the purpose of the question. I like your response (+1)." CreationDate="2011-01-05T07:54:24.203" UserId="930" />
  <row Id="9176" PostId="5963" Text="@venkasub, the center of normal distribution is its mean. If you know that bimodality comes from two normal distributions you can fit mixture model to estimate the means, which will be two modes." CreationDate="2011-01-05T09:00:10.597" UserId="2116" />
  <row Id="9177" PostId="5990" Text="I suspect it is possible to make a classifier that is worse than coin flipping by using an overly complex model (such as an ANN with a large hidden layer) and too little data.  While it will out-perform coin flipping on the training set, it may well be worse on the test set.  AFAIK, there is no guarantee that an overfit model won't be arbitrarily bad on the test set." CreationDate="2011-01-05T09:07:42.863" UserId="887" />
  <row Id="9181" PostId="5969" Score="1" Text="I think it would be great if you could provide a few example of the exact nature of the independent variables (i.e., how many factors and how many levels). I think you can get pretty far with traditional GLM depending on the exact nature of your designs." CreationDate="2011-01-05T10:16:57.187" UserId="442" />
  <row Id="9182" PostId="5991" Text="+1 for the appropriate citation" CreationDate="2011-01-05T11:17:42.250" UserId="2116" />
  <row Id="9183" PostId="5990" Text="I would say flipping a coin is the best you can do when the information you have is not related with the classification you want to make. Hence there are worst things to do ? " CreationDate="2011-01-05T13:18:03.583" UserId="223" />
  <row Id="9184" PostId="6007" Score="1" Text="for this purpose you can see the stringr package of H. Wickam" CreationDate="2011-01-05T13:56:03.123" UserId="1154" />
  <row Id="9186" PostId="5990" Score="1" Text="In addition to Dikran, the worst classifer I can imagine would be a classifier trained on a small amount of data which performs sometimes better and sometimes worse than random so that the negation is not possible, i.e. the is quality systematically unpredictable (but this seems ridiculous, so I'll stick with whuber). " CreationDate="2011-01-05T14:06:01.827" UserId="264" />
  <row Id="9188" PostId="6009" Text="@mpitkas (+1) Nice one!" CreationDate="2011-01-05T14:13:48.870" UserId="930" />
  <row Id="9189" PostId="6009" Text="@mpitkas Ah, and yours is faster (by 0.004, on my fake data set). This the way to go..." CreationDate="2011-01-05T14:18:52.020" UserId="930" />
  <row Id="9190" PostId="6009" Text="@mpiktas And sorry to have mispelled your name (two times) :(" CreationDate="2011-01-05T14:23:14.263" UserId="930" />
  <row Id="9191" PostId="5990" Text="@steffen I think you're right: complete unpredictability can be worse than random.  There is a difference!  But although there is no probability model available, we can analyze this situation with game theory.  It's you *vs* the classifier.  Assume the classifier is out to get you: it will do its worst.  What is your optimal strategy?  The answer, in the absence of any other information, is to flip the coin: that guarantees you won't lose more than half the time in the long run. What ruins this analysis is the eternal hope that *our* classifier is actually good, so we rarely use this strategy." CreationDate="2011-01-05T14:23:57.053" UserId="919" />
  <row Id="9192" PostId="6009" Text="@chl, thanks to Hadley Wickham :)" CreationDate="2011-01-05T14:32:07.460" UserId="2116" />
  <row Id="9193" PostId="6009" Text="@chl, no worries, my surname is constantly misspelled by my native language speakers, so my expectations for not misspelling the nickname are really low :)" CreationDate="2011-01-05T14:35:39.157" UserId="2116" />
  <row Id="9194" PostId="5817" Text="Given the silence here thusfar, you should try sending this query to the R-SIG-Mixed-Models mailing list." CreationDate="2011-01-05T14:44:09.813" UserId="364" />
  <row Id="9195" PostId="5817" Score="1" Text="Also, during the Bayesian Data Analysis session at the recent annual meeting of the Psychonomics Society, it was mentioned that Bayesian approaches can capture scenarios like this where you have two tasks but want to analyze the data simultaneously, letting the data from each task influence inference about the other. I'm working my way through the happy puppy Bayes book (http://www.indiana.edu/~kruschke/DoingBayesianDataAnalysis/), but still not to the point where I can describe how to achieve this analysis." CreationDate="2011-01-05T14:48:45.440" UserId="364" />
  <row Id="9196" PostId="5969" Text="@Henrik Does traditional GLM do well with repeated measures (i.e. can it remove the subject variance)and continuous IVs? If so, can you provide me with examples or references?" CreationDate="2011-01-05T15:19:14.377" UserId="2322" />
  <row Id="9197" PostId="6011" Text="Thanks - I know I will need the log-likelihood to get the DIC, but I still don't know how to actually calculate P(D|theta)" CreationDate="2011-01-05T15:39:35.677" UserId="2654" />
  <row Id="9199" PostId="5714" Text="I develop milk. If either of you run into any problems, please let me know by email (lpc at cmu dot edu). Bug reports generally get fixed in under 24 hours." CreationDate="2011-01-05T16:25:30.213" UserId="2067" />
  <row Id="9200" PostId="6002" Text="mpiktas, thanks this is an excellent answer.  " CreationDate="2011-01-05T16:49:14.087" UserId="1329" />
  <row Id="9201" PostId="5987" Text="Mathematically speaking, in binary classification, knowing the worst is equivalent to knowing the best. If you know the best you reverse the rule and it becomes the worst. Why do you want the worst rule ? what do you mean by worst ? " CreationDate="2011-01-05T17:07:26.863" UserId="223" />
  <row Id="9202" PostId="6011" Text="Thanks onestop - I suspected this was model dependent. So if I have a model that is say, a Poisson regression with a bunch of covariates, then for each Y_rep data point, I calculate the probability that it came from a Poisson(lambda) distribution? - where lambda is a fitted parameter value." CreationDate="2011-01-05T17:35:15.517" UserId="2654" />
  <row Id="9203" PostId="6005" Text="Just a programming note: it is useless (and will slow down your script) to create a function that just calls another function. There is no real advantage in calling `find_domain` if the only thing it does it's calling `grep`. You are just adding the overhead of another function call." CreationDate="2011-01-05T17:47:59.513" UserId="582" />
  <row Id="9204" PostId="6007" Text="Thanks for pointing out the rstring library!! Awesome approaches! Thanks!" CreationDate="2011-01-05T17:53:09.863" UserId="201" />
  <row Id="9205" PostId="6005" Text="you are right! this was a dirty approach and just sounded sort of wrong design though it worked. Am glad that i asked!" CreationDate="2011-01-05T17:55:52.870" UserId="201" />
  <row Id="9206" PostId="6011" Text="Yes, that's it." CreationDate="2011-01-05T18:02:56.263" UserId="449" />
  <row Id="9207" PostId="5916" Text="@sesqu One of the reasons for stopping it after it reaches 1% is so tests don’t run forever or all run for 8 seconds. Would it be better to just stop after some time (like 8 seconds) with no percent threshold or lower the threshold to below 1%? We also use 1% as a cutoff for calibrations that are used to counter the cost of some testing overhead, and taking it down to 1% can take some time and 750+ sample sizes. Is there a way to factor in the cutoff to counter any negative affects on significance?" CreationDate="2011-01-05T18:04:54.577" UserId="2616" />
  <row Id="9208" PostId="5996" Score="2" Text="If I understand correctly, you want to simulate a stochastic trend. Why not initialize $Y_0$ to a random value, say a sample from $N(0,\sigma^2)$ with $\sigma^2$ large?" CreationDate="2011-01-05T18:22:51.777" UserId="1670" />
  <row Id="9209" PostId="5996" Text="One more comment.  The problem with burn-in is that, in addition to being non-stationary, the process is not ergodic.  So what you are asking for is not really possible.  You will always be able to infer (roughly) what the initial value of the process is.&#xA;" CreationDate="2011-01-05T18:39:40.617" UserId="1670" />
  <row Id="9211" PostId="592" Text="That's all? I'm surprised, cos it is a really good book.&#xA;&#xA;The same author wrote an excellent introduction to Time Series Analysis as well." CreationDate="2011-01-05T19:41:23.207" UserId="57" />
  <row Id="9212" PostId="5916" Text="Yes. The easy way would be to stop at 8 seconds every time. To counter the effects, you would divide each p by 0.95^i-1, depending on what you're really doing." CreationDate="2011-01-05T20:01:31.050" UserId="2456" />
  <row Id="9213" PostId="6000" Text="Depending on how this suggestion is interpreted, it may amount to either Fisher's Exact Test or Barnard's Test.  See http://en.wikipedia.org/wiki/Fisher%27s_exact_test ." CreationDate="2011-01-05T20:06:21.827" UserId="919" />
  <row Id="9214" PostId="6015" Score="1" Text="+1 for addressing the apparent (rather than the stated) question, which is much more to the point." CreationDate="2011-01-05T20:07:45.010" UserId="919" />
  <row Id="9215" PostId="5817" Text="@Mike Lawrence Thanks for the tip. You'll probably see this on the R-Sig list soon. Although I'd lose power doing it, I can address the problem you mention below about the cell with 0 trials by running 2 separate models. I missed the Bayes session at pnomics, but find the idea of using a Bayesian approach to dual-tasks intriguing. I don't know much about Bayesian analysis, but it has me thinking about how the Bayes approach may be different from fitting a covariance matrix in mixed-models." CreationDate="2011-01-05T22:28:17.900" UserId="2322" />
  <row Id="9216" PostId="5603" Text="I think my use of the word &quot;state&quot; probably does imply that -- although I did want to think about a &quot;continuous&quot; situation, where the &quot;birth rate&quot; of the process declines gradually (perhaps precipitousness) as the population grows. " CreationDate="2011-01-06T00:29:29.797" UserId="446" />
  <row Id="9217" PostId="6018" Score="2" Text="This question needs a lot more information to be answerable, as well as considerable clarification of what is meant by &quot;independently exist,&quot; &quot;overall average,&quot; and &quot;buying state allocations.&quot;  Do you have any references?" CreationDate="2011-01-06T03:31:33.233" UserId="919" />
  <row Id="9218" PostId="5769" Text="Correct. The PyIMSL algorithms are Python wrappers to the IMSL C Numerical Library (~500 math and stat functions), which is not open source. PyIMSL is free to use for non-commercial use though." CreationDate="2011-01-06T04:20:35.717" UserId="1080" />
  <row Id="9219" PostId="6020" Score="1" Text="the correlation is calculated for each pair of variables separately. So conceptually there is no difference, you just need to repeat the analysis; in R you could use lm(y~x)" CreationDate="2011-01-06T06:41:54.410" UserId="1381" />
  <row Id="9220" PostId="6025" Score="1" Text="Thanks for the advice.  Re: lognormal - I could make the ratios of percentiles to median work out by subtracting 7077 from everything, then adding it back in at the end.  How bad an idea would that be?" CreationDate="2011-01-06T09:04:44.860" UserId="2665" />
  <row Id="9221" PostId="6020" Text="Thanks, I will look into using 'lm'.  Do you have any further information of how I can analyze such a dataset?  Is it called correlation or regression?" CreationDate="2011-01-06T09:26:53.213" UserId="2664" />
  <row Id="9222" PostId="6014" Text="Thank you for the great answer! I do have a couple of questions: is it correct to assume that &quot;chance of chi-square variate exceeding X&quot; is essentially the chance that a random selection between 1-2.3E-111 would produce the number? or? And, is there any software that I can use to quickly calculate these metrics, or is doing it by hand the best option? Thank you!" CreationDate="2011-01-06T10:56:36.447" UserId="2648" />
  <row Id="9223" PostId="6025" Score="1" Text="Good point, that would give a 'shifted log-normal distribution'. The log-normal and the log-logistic are pretty similar in shape apart from the heavier tails of the latter, so you could try both and compare results." CreationDate="2011-01-06T11:40:34.420" UserId="449" />
  <row Id="9224" PostId="6030" Text="@mbq (+1) I forgot the `Sys.sleep()` function. I think the $x$- and $y$-axis range should bet set up in advance, no?" CreationDate="2011-01-06T12:26:01.880" UserId="930" />
  <row Id="9225" PostId="6024" Text="Seems to me there might be some better way of formulating or parameterising the problem, but it's hard to tell without knowing a bit more.. could you give us some idea what the $x_i$s are and how you know the underlying model is of that form? " CreationDate="2011-01-06T12:33:08.347" UserId="449" />
  <row Id="9226" PostId="6030" Text="@chl, yes the $x$ and $y$ axis ranges should be set up in advance.  " CreationDate="2011-01-06T13:30:45.077" UserId="2116" />
  <row Id="9227" PostId="6030" Score="1" Text="@mpiktas Or just init the plot with a single call to `plot()`, possibly with `0,0,type=&quot;n&quot;` if there is nothing to plot yet... Indeed this is much easier than ding `plot.new()` and adding all the stuff like axes or labels by hand." CreationDate="2011-01-06T13:49:09.310" UserId="88" />
  <row Id="9228" PostId="6033" Score="4" Text="This is similar to http://stats.stackexchange.com/questions/5700/finding-the-change-point-in-data-from-a-piecewise-linear-function " CreationDate="2011-01-06T14:09:13.907" UserId="2116" />
  <row Id="9229" PostId="5996" Text="So if I understand well building up some common stochastic trend (to have cointegrating relationship) do not affect the Monte Carlo simulation results whatever the starting value will be? On the other hand it is the ergodicity property that insures the &quot;memory&quot; length, that after some period of development you can not tell what the starting value was.Again in Gaussian random walk case how could you infer (roughly)the initial value if you drop sufficiently large burn-in part, if you don't or do know the length of burn-in (say T) the confidence interval of +/- 2$\sqrt{T}$ is too rough isn't it? " CreationDate="2011-01-06T14:15:01.373" UserId="2645" />
  <row Id="9230" PostId="6019" Text="@Josh Thanks for a pointer to some sample code, I'll check it out!" CreationDate="2011-01-06T15:10:51.720" UserId="2659" />
  <row Id="9231" PostId="5996" Text="Suppose $Y_t = \alpha Y_{t-1} + \epsilon_t$ where $\epsilon_t$ are i.i.d. $N(0,1)$ and $|\alpha| \leq 1$.  The conditional distribution of $Y_t$ given $Y_0$ is $N(\alpha^t Y_0, \sigma_{\alpha,t}^2)$, where $\sigma_{\alpha,t}^2 \leq t$ depends only on $\alpha$ and $t$.  If $\alpha = 1$, note that $Y_t$ given $Y_0$ remains centered at $Y_0$ even as $t \to \infty$.  In that sense, you can always infer $Y_0$.  On the other hand, if $|\alpha|&lt; 1$ then the conditional mean of $Y_t$ given $Y_0$ tends to 0 as $t\to\infty$ so that asymptotically $Y_t$ is conditionally independent of $Y_0$. " CreationDate="2011-01-06T15:42:26.200" UserId="1670" />
  <row Id="9232" PostId="6014" Text="@Sam The chi-square statistic is a measure of association of the two variables (&quot;attribute&quot; and renewing).  When there is no association, the chi-square statistic should act like the 500 renewers occurred at random in the population of buyers, regardless of the values of &quot;attribute.&quot;  In this model, the number 3E-111 estimates the chance that chi-square could be as large as it turned out to be.  You can do all the calculations in Excel, although stats software like R (free), Stata, SAS, JMP, Mystat (free), etc., is easier and more reliable." CreationDate="2011-01-06T15:48:31.877" UserId="919" />
  <row Id="9233" PostId="5996" Text="If $|\alpha| = 1$, then $\sigma_{\alpha,t}^2 = t$. If $|\alpha| &lt; 1$, then  $\sigma_{\alpha,t}^2 = (1-\alpha^t) / (1-\alpha)$.  So although you can infer $Y_0$ from $Y_t$ when $\alpha = 1$ the standard error will be of the order $\sqrt{t}$.  I think to better answer your question you need to be less specific and explain what you will use $Y_t$ for." CreationDate="2011-01-06T15:49:48.410" UserId="1670" />
  <row Id="9234" PostId="6035" Text="I have merged your accounts; but please, register it here http://stats.stackexchange.com/users/login so you'd not loose it again. And please use comments instead of adding answers." CreationDate="2011-01-06T15:52:02.077" UserId="88" />
  <row Id="9235" PostId="6033" Text="Yes, and the answers will be the same, too. " CreationDate="2011-01-06T15:54:15.817" UserId="919" />
  <row Id="9236" PostId="6032" Score="1" Text="Any three-parameter distribution is guaranteed to fit three quantiles *perfectly.*  Thus it makes sense to use this approach to fit only one or two parameters.  It also does not make any sense to compare a one-parameter fit to a two-parameter fit (with a different family) based on likelihood alone." CreationDate="2011-01-06T16:04:42.640" UserId="919" />
  <row Id="9237" PostId="6027" Score="1" Text="Polynomials and splines are unlikely to be valid CDFs." CreationDate="2011-01-06T16:05:25.043" UserId="919" />
  <row Id="9238" PostId="6025" Text="@onestop Compare how?  The shifted lognormal is guaranteed to fit the quantiles perfectly.  Almost any three-parameter family will fit perfectly.  How do you compare two perfect fits?" CreationDate="2011-01-06T16:31:41.153" UserId="919" />
  <row Id="9239" PostId="5980" Text="Dmitrij, thanks for the additional references. " CreationDate="2011-01-06T16:53:53.173" UserId="1329" />
  <row Id="9240" PostId="6025" Text="@whuber I meant compare the resulting predictions for the percentiles corresponding to other values" CreationDate="2011-01-06T17:13:01.523" UserId="449" />
  <row Id="9241" PostId="6025" Text="@onestop I'm missing something: what other values?  The OP states that *only* three percentiles are available, nothing else." CreationDate="2011-01-06T17:38:31.793" UserId="919" />
  <row Id="9242" PostId="6038" Text="@whuber, +1, great answer." CreationDate="2011-01-06T17:39:54.367" UserId="2116" />
  <row Id="9243" PostId="6027" Text="Good observation. In this case, the usual quadratic polynomial fails to work, but there are infinitely many quadratic splines to choose from (think Bézier) that should not have the same problem (though some might still require domain cropping). Similarly, it should be possible to find a suitable monotonic cubic spline. I am aware of spline algorithms that guarantee monotonicity, but am unable to find one just now, so I have to leave the matter at &quot;pick something you like that works as cdf&quot;." CreationDate="2011-01-06T17:46:40.633" UserId="2456" />
  <row Id="9244" PostId="6039" Score="2" Text="Since Principal Components Analysis and sampling have almost nothing to do with each other--one analyzes data and the other determines what data to collect--I am wondering whether you mean something else by &quot;PCA&quot;. &quot;Point of Closest Approach&quot;? &quot;Prompt Corrective Action&quot;? &quot;Physical Configuration Audit&quot;? &quot;Permanent Corrective Action&quot;? &quot;Power Control Algorithm&quot;? &quot;Process Control &amp; Automation&quot;? &quot;Parallel Cellular Automata&quot;? &quot;Product Complaint Analysis&quot;? &quot;Probable Course of Action&quot;? &quot;Poodle Club of America&quot;?" CreationDate="2011-01-06T17:47:26.590" UserId="919" />
  <row Id="9245" PostId="5996" Text="Thanks for the explanation, and even though it is correct I'm still a bit confused. My point was since you do drop the burn-in period, you can't track back to the original point, but do continue from some point $Y_T$, pretending that there was some history to get you to the point. Therefore you can't say for sure what was the original $Y_0$ value you started from, and even more due to self-similarity it will be $Y_T$ the new point to build the mean value (again for a particular realization! not on average). Building the question I was just curious what usually people do for RW in Simulations." CreationDate="2011-01-06T18:24:04.470" UserId="2645" />
  <row Id="9247" PostId="5996" Text="I was not originally planning to give more details on the experiment we do perform, but it seems that we need more details on the DGP in simulation experiments. Suppose we try to simulate to factors $F_{i,t} = \alpha_i F_{i,t-1} + \beta_i G_{t} + \nu_{i,t}$, $i = 1,2,\dots,k$, where $\nu_{i,t}$ (and any other residuals) are i.i.d $N(0,1)$, where $G_t$ is a common stochastic trend. Then we build many &quot;factor-based&quot; series $x_{j,t} = \sum_i \lambda_{i,j}F_{i,t} + \varepsilon_{j,t}$, $j = 1,\dots,n$, where $\lambda_{i,j}$ are randomly drawn from $N(0,\sigma_j^2)$. The same factors are used for" CreationDate="2011-01-06T18:42:36.060" UserId="2645" />
  <row Id="9248" PostId="5996" Text="the dependent variable $y_{t} = \sum_i \gamma_{i}F_{i,t} + \varepsilon_{t}$. The point is how to bring common stochastic trend into the model, and when we can say that the trend is nice. May be in this case it indeed doesn't matter if we start from &quot;looking as in real data&quot; trend or actually take any part of it... in any case we will standardize the auxiliary processes latter on, but would like to see at least by simulations if the problem we are working on (actually it is temporal disaggregation) could be done for non-stationary but co-integrated data. Thanks again for details you provided :)" CreationDate="2011-01-06T18:48:58.830" UserId="2645" />
  <row Id="9249" PostId="6028" Text="Thanks a lot for your reply! I'm having a hard time de-ciphering your text-based formulas. Can you humanize them? ;) Great article you're referencing. I'll read it thouroughly and get back... " CreationDate="2011-01-06T19:05:54.843" UserId="2652" />
  <row Id="9250" PostId="6043" Text="Looking at this document, http://support.sas.com/publishing/pubcat/chaps/59498.pdf, their also appears to be differences in definitions between SAS and SPSS that could cause confusion. Formats in SPSS appear to be referred to as informats in SAS. Value labels in SPSS appear to be referred to as formats in SAS. " CreationDate="2011-01-06T19:10:07.580" UserId="1036" />
  <row Id="9251" PostId="6038" Text="@whuber +1 **You need more data** and **what you intend to use the results for** deserve extra emphasis." CreationDate="2011-01-06T19:20:11.310" UserId="1670" />
  <row Id="9252" PostId="6028" Text="Ahhh... Internet Explorer failed to render the formulas. Firefox fixed this. :)" CreationDate="2011-01-06T19:21:15.630" UserId="2652" />
  <row Id="9253" PostId="6038" Text="@whuber It sounds like there's lots of wisdom in your answer.  I'll have to consult more with the people who posed me the problem about just what they want.  Thank you for the links and the advice." CreationDate="2011-01-06T19:29:03.600" UserId="2665" />
  <row Id="9254" PostId="6038" Text="@vqv Thanks.  I made the changes you suggested." CreationDate="2011-01-06T19:33:07.250" UserId="919" />
  <row Id="9255" PostId="6038" Text="@Mark Best of luck!" CreationDate="2011-01-06T19:33:47.233" UserId="919" />
  <row Id="9256" PostId="6044" Score="1" Text="To keep the responses relevant to the question you have in mind, could you say a little more about what these data are, what you mean by &quot;difference in percentage,&quot; and why that's important to you?" CreationDate="2011-01-06T19:40:15.887" UserId="919" />
  <row Id="9257" PostId="6040" Text="+1 For pointing out the collinearities." CreationDate="2011-01-06T19:42:14.213" UserId="919" />
  <row Id="9258" PostId="6037" Text="I'm actually unclear as to how a likelihood ratio (LR) does not achieve everything that an effect size achieves, while also employing an easily interpretable scale (the data contains X times more evidence for Y than for Z). An effect size is usually just some form of ratio of explained to unexplained variability, and (in the nested case) the LR is the ratio of unexplained variability between a model that has an effect and one that doesn't. Shouldn't there at least be a strong correlation between effect size and LR, and if so, what is lost by moving to the likelihood ratio scale?" CreationDate="2011-01-06T19:54:51.660" UserId="364" />
  <row Id="9259" PostId="6037" Text="Mike -  You've got me interested, but do your points extend to effect sizes as simple as mean differences between groups?  These can be easily interpreted by a lay person and can also be assigned confidence intervals.  " CreationDate="2011-01-06T20:22:53.617" UserId="2669" />
  <row Id="9260" PostId="6044" Text="Alright. I will edit the post." CreationDate="2011-01-06T20:46:12.147" UserId="2676" />
  <row Id="9261" PostId="6032" Text="@whuber, re: &quot;Any three-parameter distribution is guaranteed to fit three quantiles perfectly&quot;. I hadn't realized that, so good to know! re: &quot;It also does not make any sense to compare a one-parameter fit to a two-parameter fit (with a different family) based on likelihood alone.&quot; Ah yes, indeed; I failed to mention that one would have to apply some complexity correction (AIC, BIC, ...) if comparing fits to distribution flavors with different numbers of parameters. Thanks for pointing that out." CreationDate="2011-01-06T21:10:59.380" UserId="364" />
  <row Id="9263" PostId="6044" Score="3" Text="It's still not clear what you mean. Do you mean the number of points which are different? Do you mean the percent difference in the averages? Do you mean the mean percent difference for each of the pairs of points? There's no obvious way to interpret what you're asking for. Percentage difference is not something people normally think about for sets of points. Percent difference can fully specify how two individual points are different, but you need a lot more information to fully specify how different two sets of points are." CreationDate="2011-01-06T21:24:02.667" UserId="1146" />
  <row Id="9264" PostId="6043" Text="Thank you for the suggestion Andy.  I forwarded it to my friend - and I hope she will find it useful." CreationDate="2011-01-06T21:53:52.777" UserId="253" />
  <row Id="9265" PostId="5951" Text="I'd even say [box-counting dimension](http://en.wikipedia.org/wiki/Box-counting_dimension) for a more practical statistic." CreationDate="2011-01-06T22:01:53.663" UserId="2036" />
  <row Id="9266" PostId="6045" Text="+1 Nice find!  The paper also has a brief discussion of the PCA approach (as well as some other methods)." CreationDate="2011-01-06T23:08:31.400" UserId="919" />
  <row Id="9267" PostId="6032" Text="@Mike I exaggerated a little bit, because I was thinking of two of the parameters being scale and location and the third comprising a wide range of shapes.  Even so, most three-parameter families have sufficient flexibility to fit three percentiles provided they are all distinct." CreationDate="2011-01-06T23:11:12.367" UserId="919" />
  <row Id="9268" PostId="6037" Text="Ah, so by effect size, you mean absolute effect size, a value that is meaningless unto itself, but that can be made meaningful by transformation into relative effect size (by dividing by some measure of variability, as I mentioned), or by computing a confidence interval for the absolute effect size. My argument above applies to the merits of LRs vs relative effect sizes. There may utility to computing effect CIs in cases where the actual value of the effect is of interest (eg. prediction), but I still stand by the LR as a more intuitive scale for talking about evidence for/against effects." CreationDate="2011-01-06T23:11:56.483" UserId="364" />
  <row Id="9269" PostId="6037" Text="I guess the use of LRs vs CIs will likely vary according to the context, which may be usefully summarized as follows: More exploratory stages of science, where theories are roughly characterized by the existence/absence of phenomena, may prefer LRs to quantify evidence. On the other hand, CIs may be preferred in more advanced stages of science, where theories are sufficiently refined to permit nuanced prediction including ranges of expected effects or, conversely, when different ranges of effect magnitudes support different theories. Finally, predictions generated from any model need CIs." CreationDate="2011-01-06T23:18:49.550" UserId="364" />
  <row Id="9270" PostId="6031" Text="Thank you very much, that was a great pointer in the right direction!" CreationDate="2011-01-07T00:14:10.003" UserId="2664" />
  <row Id="9271" PostId="6044" Text="@John: Exactly, percentage difference works fine for a pair of points, but when someone says that certain result (as a function or as a set of numbers) is inaccurate by 10% or 15% with respect to another result, how can you arrive to such conclusion?" CreationDate="2011-01-07T00:52:54.747" UserId="2676" />
  <row Id="9272" PostId="6044" Text="By the way, I don't mean any particular question as &quot;How to calculate percent difference&quot;. I want to know what is the correct way to reach the kind of conclusion I mentioned before." CreationDate="2011-01-07T01:00:21.773" UserId="2676" />
  <row Id="9273" PostId="6050" Text="@nerdbound I think it is hard to answer this question in a general context. If possible, can you pls provide some insights about what is your data? what do you want to infer? possible assumptions, etc. Providing these details might help us in answering your questions better." CreationDate="2011-01-07T01:02:48.120" UserId="1307" />
  <row Id="9274" PostId="6049" Text="@carlosdc: Thanks for your answer. Unfortunately, those questions are not what I want to know. I updated the question, though." CreationDate="2011-01-07T01:02:52.487" UserId="2676" />
  <row Id="9275" PostId="6052" Score="1" Text="Thanks for the response. I guess my concern is that, in your example, if one country of birth is a fantastic predictor of a disease, and all the other countries of origin are poor predictors, then the backward elimination model will tell me to remove country of birth, even though country of birth is hardly useless as a covariate. In other words, it might make sense to have a single variable for 'are you Lithuanian?' even though country of birth as a whole does not predict well. Are there sensible ways to handle that sort of problem?" CreationDate="2011-01-07T01:53:30.930" UserId="2308" />
  <row Id="9276" PostId="6050" Text="@suncoolsu I'm not really at liberty to discuss any of that. If it's possible to answer the question with a decision tree ('if you're assuming X, then do Y'), that could be really helpful." CreationDate="2011-01-07T02:11:31.843" UserId="2308" />
  <row Id="9277" PostId="6050" Score="1" Text="@nerdbound For clarity, are you interested in interpreting the model as causal or are you simply interested in prediction?" CreationDate="2011-01-07T02:18:23.813" UserId="1036" />
  <row Id="9278" PostId="6052" Score="2" Text="(+1) If one country is a great predictor, then the whole group of countries will be significant and it will be retained.  @Matt Parker is correct: keep groups of dummy variables together and base the entering and leaving criteria on the joint significance level for the entire group, not on the significance levels for the individual dummies." CreationDate="2011-01-07T02:18:47.520" UserId="919" />
  <row Id="9279" PostId="6052" Score="1" Text="@whuber OK one more potentially dumb comment from me and then I'll be at peace when I understand why I'm wrong :-P It seems to me that if I'm looking at AIC and I exclude, say, 200 country dummies as a group, the AIC will improve because the new model will have 200 fewer variables, and it will get worse only to the extent that those dummies were good predictors. If only one of the variables was a good predictor, it seems to me that on net the AIC will make the model with 200 fewer variables look better, but then I'll be missing out on that one variable..." CreationDate="2011-01-07T03:03:50.557" UserId="2308" />
  <row Id="9280" PostId="6030" Text="Indeed this seems the simplest... but too bad that the axes are not updated. As I want to update online a time series, and I know at which speed the x axis moves, I guess that I can replot everything every n updates." CreationDate="2011-01-07T03:06:12.740" UserId="1709" />
  <row Id="9281" PostId="6052" Score="2" Text="@nerdbound That's not dumb at all.  However, if only one dummy in 200 works well, you really have a different categorical variable: it's a new binary (or reclassified) variable.  @Matt Parker addresses this issue in his second paragraph.  This looks like a matter for exploratory analysis (which is what stepwise regression is anyway).  It's perfectly fine to torture your data in this way, but after all the smoke clears, you need to test your model on independent data." CreationDate="2011-01-07T03:09:20.440" UserId="919" />
  <row Id="9282" PostId="6044" Text="Let me give you an example, say you have 1 population of 50 individuals and 2 bathroom scales. You want to know if both bathroom scales work equally well? Or if one scale with give readings higher or lower than the other?" CreationDate="2011-01-07T04:11:41.240" UserId="1540" />
  <row Id="9283" PostId="6044" Text="@carlosdc: No, sorry. Did you read the update?. I put an example there of what I want to know. " CreationDate="2011-01-07T05:16:40.097" UserId="2676" />
  <row Id="9284" PostId="5381" Text="+1 Good reference; thank you." CreationDate="2011-01-07T05:17:11.733" UserId="919" />
  <row Id="9285" PostId="5358" Text="@onestop Thanks, I did mean the sum.  I haven't done the calculations, but it's clear many of the problems are due to having some largish p's in with a mix of smaller ones.  But since the OP hasn't responded to a request for clearer characterization of the distribution of p's, we should try to keep the discussion fairly general and at least characterize when various suggested approaches work and when others should be considered.  My second response to this thread covers so many of the possibilities with perfect accuracy that I would be reluctant to use an approximation now." CreationDate="2011-01-07T05:35:51.733" UserId="919" />
  <row Id="9286" PostId="5969" Score="2" Text="@Matt If you have two levels on a within subject variable you can use the difference as dv (no need to go for repeated measures models). The same logic applies if you have two two-level within subject variables. The interaction is the difference of the differences (avoiding repeated measure models). However, if one of your within-variables has more than two levels, this approach does not work anymore, but you have to go multilevel. I recommend you read the special issue of the Journal of Memory and Language referred to in chl's answer: J. Mem. Language, 2008 59(4): Emerging Data Analysis" CreationDate="2011-01-07T09:41:57.480" UserId="442" />
  <row Id="9287" PostId="6060" Text="@gd047 (+1) Nice answer. The [arules vignette](http://cran.r-project.org/web/packages/arules/vignettes/arules.pdf) also have some good references about *lift*." CreationDate="2011-01-07T09:45:59.490" UserId="930" />
  <row Id="9288" PostId="6042" Text="Invite your friend here!" CreationDate="2011-01-07T11:02:38.473" UserId="88" />
  <row Id="9289" PostId="6042" Text="I totally want to, but she's too busy writing academic articles.  In general, I wonder how many non-R statisticians we've got on the site." CreationDate="2011-01-07T11:48:47.160" UserId="253" />
  <row Id="9291" PostId="6044" Text="@Robert Smith, Yeah, sorry what you said doesn't make much sense. I'm trying to put it in statistical terms." CreationDate="2011-01-07T12:43:35.713" UserId="1540" />
  <row Id="9292" PostId="6022" Text="i've added the r tag so the R code is highlighted in my comment" CreationDate="2011-01-07T13:50:04.063" UserId="2116" />
  <row Id="9293" PostId="6065" Text="Great!  Thanks for all the effort that went into this, mpiktas.  I'm not familiar with R, but your code is explained well enough that I can still easily tell what you're doing." CreationDate="2011-01-07T13:57:07.503" UserId="2665" />
  <row Id="9294" PostId="6048" Score="2" Text="While I don't doubt the veracity of your claims, it may be more useful to the OP and to the community in general if you describe how the lme4 package handles missing values." CreationDate="2011-01-07T14:11:13.067" UserId="1036" />
  <row Id="9295" PostId="6046" Text="How many participants (i.e., animals) do you have? If the ratio of animals to missing data is too big, this is not that good. In all cases (e.g., after using mixed effect modeling as proposed by Mike Lawrence) you should compare the results of the data with listwise deleted cases (i.e., all animals with missing data omitted) with the modeled results..." CreationDate="2011-01-07T14:19:52.913" UserId="442" />
  <row Id="9296" PostId="6047" Score="1" Text="I think it depends on the context.  Note that $$Q = \frac{\Pr(A|B)}{\Pr(A)} = \frac{\Pr(B|A)}{\Pr(B)}$$ so that $\Pr(A|B) = Q \Pr(A)$ and $\Pr(B|A) = Q \Pr(B)$.  This form has more of a Bayesian inference flavor." CreationDate="2011-01-07T14:23:19.830" UserId="1670" />
  <row Id="9297" PostId="6060" Text="Thanks, that's probably where I've seen it before.  I think I've seen lift with a slightly different definition in machine learning context before though...I hate that sometimes there is a lack of consensus over a definition while other times there are many terms for the same concept." CreationDate="2011-01-07T14:59:04.023" UserId="2485" />
  <row Id="9298" PostId="6048" Text="Thanks for answers. Sorry, but I'm not an expert neither a statistician,  I don't handle R. I've just two statistical softwares, SPSS 16 or STATISTICA 7.0 from StatSoft and I'm quite lost! I've tried mixed models with SPSS with fixed effect for reproductive status and dose as the repeated effect. however I'm quite confuse, I'dont know how interprete the results." CreationDate="2011-01-07T16:04:05.243" UserId="2677" />
  <row Id="9299" PostId="6055" Text="I took the problem to not be actually writing a usable file, but to transporting the value labels to SAS. I don't have a copy of SAS handy to see if the export to SAS7dat files normally saves this or not." CreationDate="2011-01-07T17:00:38.500" UserId="1036" />
  <row Id="9300" PostId="6066" Score="1" Text="This is a problem with stepwise regression and categorical variables.  See this (very) recent question for more information:&#xA;&#xA;[How should I handle categorical variables with multiple levels when doing backward elimination?](http://stats.stackexchange.com/questions/6050/how-should-i-handle-categorical-variables-with-multiple-levels-when-doing-backwar)" CreationDate="2011-01-07T17:01:57.017" UserId="71" />
  <row Id="9301" PostId="6062" Text="Thanks!.That sounds like what I was looking for." CreationDate="2011-01-07T17:22:31.070" UserId="2676" />
  <row Id="9302" PostId="6044" Text="@carlosdc: See the answer posted by Dennis." CreationDate="2011-01-07T17:23:35.203" UserId="2676" />
  <row Id="9303" PostId="6049" Text="+1 For a valiant attempt at deciphering the question and providing good advice." CreationDate="2011-01-07T19:29:48.183" UserId="919" />
  <row Id="9304" PostId="6071" Text="Are $e_i$ and $\delta_i$ (assumed to be) independent? Also, what are the distributional assumptions? In particular, are $e_i$, $\delta_i$, and $g_i$ assumed to be normally distributed? Finally, did you really mean $\beta \mu_i$ and not $\beta + \mu_i$?" CreationDate="2011-01-07T19:31:33.107" UserId="1934" />
  <row Id="9305" PostId="4247" Text="This question originated as http://math.stackexchange.com/q/9057/1489 .  The respondents there were just as tentative as we are because they weren't sure what is being asked, either." CreationDate="2011-01-07T19:39:18.830" UserId="919" />
  <row Id="9306" PostId="6042" Score="1" Text="I think you should change the title to reflect the concern with keeping the formats and labels. Actually saving the file in the other softwares format is trivial, keeping the labels and value labels is the hard part." CreationDate="2011-01-07T20:20:27.340" UserId="1036" />
  <row Id="9307" PostId="6067" Score="2" Text="It is hard to imagine how this could be. Perhaps you are cutting the predicted probability at 0.5? If so, try varying the cutoff." CreationDate="2011-01-07T20:30:02.617" UserId="279" />
  <row Id="9308" PostId="6071" Text="They could be correlated, but literature indicates that this is a minor concern. All errors are assumed normal, and the variance is **known** for $e_i$ and $\delta_i$" CreationDate="2011-01-07T20:33:37.757" UserId="1364" />
  <row Id="9309" PostId="6071" Text="yes $\beta \mu_i$, indicating the relationship between $X_i$ and $Y_i$." CreationDate="2011-01-07T20:34:41.533" UserId="1364" />
  <row Id="9310" PostId="862" Text="As a note, by default (so, if you don't specify `breaks`) R " CreationDate="2011-01-07T21:36:26.267" UserId="582" />
  <row Id="9311" PostId="6066" Text="I don't understand. If one categorical variable group (or partition, or what I have called &quot;comparison&quot;) is not significant and all the groups have to be removed from the model, why in my case I find that one group is retained for nominal regression and the other two ones are discarded?" CreationDate="2011-01-07T23:13:25.313" UserId="1219" />
  <row Id="9312" PostId="6074" Text="I doubt it is a Naive Bayes, rather some decision tree extended each time it fails to recognize someone." CreationDate="2011-01-07T23:39:24.007" UserId="88" />
  <row Id="9313" PostId="6075" Text="Wow... very nice follow-up." CreationDate="2011-01-07T23:52:00.940" UserId="71" />
  <row Id="9314" PostId="564" Text="Does anyone know how to estimate a difference in difference regression in gretl? Do I have to work with OLS or panel data?" CreationDate="2011-01-07T23:54:44.187" UserId="2687" />
  <row Id="9315" PostId="564" Text="@Pyca It sounds like an inappropriate use of comments there. You should post a new question, with reference to this one." CreationDate="2011-01-07T23:54:44.267" UserId="930" />
  <row Id="9316" PostId="6074" Text="Wouldn't such decision tree be too unwieldy to update? Plus, I see no easy way to account for accidentally-wrong/honest-mistake answers and still get it right in the end with decision tree" CreationDate="2011-01-07T23:55:07.883" UserId="2696" />
  <row Id="9317" PostId="6074" Score="3" Text="Looks like a reincarnation of the twenty year old 20-questions guesser, now at http://www.20q.net/. Here's a popular explanation how it works http://www.mentalfloss.com/blogs/archives/13725" CreationDate="2011-01-08T01:01:21.187" UserId="511" />
  <row Id="9318" PostId="5347" Text="Sum of IID bernoulli variables is known as the Binomial random variable, so what you want is binomial tail probability. Quick google search seems to give some R recipes for getting it exactly" CreationDate="2011-01-08T01:05:09.560" UserId="511" />
  <row Id="9319" PostId="6078" Text="@alan Sorry to ask, but do you want to test trails and patients are independent? If no, can you please specify what does the Chi-sq test intend to do?" CreationDate="2011-01-08T03:49:57.583" UserId="1307" />
  <row Id="9320" PostId="6071" Text="The 'simex' package is popular for measurement error models...." CreationDate="2011-01-08T04:08:14.980" UserId="2310" />
  <row Id="9321" PostId="6079" Text="Could you explain where the 82/15/3 values come from?  You appear to be using numbers that you haven't yet shared with us, such as &quot;the monthly CDDs&quot; (whatever those may be) and the &quot;total predictions for the year&quot; (whatever those may be)." CreationDate="2011-01-08T04:25:48.910" UserId="919" />
  <row Id="9322" PostId="6078" Text="Yes, that's what I am looking for. I have edited the question to reflect that. Please let me know if anything is unclear." CreationDate="2011-01-08T05:24:30.163" UserId="52" />
  <row Id="9323" PostId="6075" Text="I am somewhat surprised it reduced to 201 words. Although for first word played, our house rules accept 'I' and 'A' as words, which would probably further reduce the number of minimal words.  I was hoping to see someone bust out the inclusion-exclusion analysis, which should be pretty hairy..." CreationDate="2011-01-08T06:05:39.157" UserId="795" />
  <row Id="9324" PostId="5993" Text="Whatever approach I end up using, Monte Carlo will certainly be used to test the implementation." CreationDate="2011-01-08T06:10:29.017" UserId="795" />
  <row Id="9325" PostId="6074" Score="2" Text="Excuse me, but I think that using &quot;artificial intelligence&quot; and &quot;neural networks&quot; without any context hardy counts as explanation. And I cannot see how one could use neural net for this kind of thing - what would be the output function, for example?" CreationDate="2011-01-08T07:03:31.570" UserId="2696" />
  <row Id="9326" PostId="6062" Text="You're welcome Robert; could you post what you ended up using (in a week or so) ?" CreationDate="2011-01-08T11:36:57.057" UserId="557" />
  <row Id="9327" PostId="6082" Score="3" Text="Good answer. Regarding the last part (with respect to running the three models): The model with the intercept alone won't give you an $R^2$ value. Also, the $R^2$ values from the two models with just one predictor are probably not going to add up to the $R^2 = .85$ value for the full model with both predictors. That will only happen when the two predictors are uncorrelated. In other words, the two predictors will probably account for overlapping parts of the variance in the dependent variable." CreationDate="2011-01-08T13:03:19.417" UserId="1934" />
  <row Id="9329" PostId="6082" Text="Thanks! I've filled in more information, above, if that makes any difference to your answer." CreationDate="2011-01-08T14:20:21.247" UserId="1764" />
  <row Id="9330" PostId="6079" Text="Hopefully it's more clear now, thanks!" CreationDate="2011-01-08T14:21:09.390" UserId="1764" />
  <row Id="9331" PostId="6086" Score="1" Text="If you don't know the operational class frequencies, they can be estimated by EM without knowing the labels of the test/operational samples.  The details are in Saerens et al. &quot;Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure&quot;, Neural Computation, vol. 14, no. 1, pp. 21-41, 2002 ( http://dx.doi.org/10.1162/089976602753284446 ).  I've used this a couple of times and was impressed at how well it worked.  Note however that the theoretical correction is not normally optimal, and setting it via e.g. cross-validation is often better." CreationDate="2011-01-08T17:17:37.383" UserId="887" />
  <row Id="9333" PostId="6075" Text="@shabbychef There are no 1-letter words in the lexicon.  Most minimal words are 2- and 3-letter words.  Here is the full distribution of minimal word lengths: 2: 73, 3:86, 4:31, 5:9, 6:2.  The 6-letter words are: GLYCYL and SYZYGY." CreationDate="2011-01-08T17:33:36.907" UserId="1670" />
  <row Id="9334" PostId="6045" Text="Thanks very much, I think that this is the closest to what my colleague was looking for." CreationDate="2011-01-08T19:17:28.807" UserId="1157" />
  <row Id="9335" PostId="5951" Text="Thanks for your answer. I will pass it on!" CreationDate="2011-01-08T19:18:09.290" UserId="1157" />
  <row Id="9336" PostId="6083" Text="Thank you Dwin.  I wonder which of the different residuals should be explored (besides the Within one).  Cheers, Tal" CreationDate="2011-01-08T19:37:57.800" UserId="253" />
  <row Id="9337" PostId="6083" Text="npk.aovE is a list of three elements. The first two are parameter estimates and normality is not assumed for them, so it wouldn't seem appropriate to test anything except $Within. `names(npk.aovE)` returns `&#xA;[1] &quot;(Intercept)&quot; &quot;block&quot;       &quot;Within&quot;`   " CreationDate="2011-01-08T19:57:36.283" UserId="2129" />
  <row Id="9338" PostId="6075" Text="@shabbychef I updated my answer to include a sketch of an exact inclusion-exclusion approach.  It is worse than hairy." CreationDate="2011-01-08T20:18:57.030" UserId="1670" />
  <row Id="9339" PostId="6087" Text="Thanks gd047 - the question is what do we do when we have a more complex scenario of aov(yield ~  N*P*K + Error(block/(N+K)), npk)  would the test you propose do the work? " CreationDate="2011-01-08T20:46:08.047" UserId="253" />
  <row Id="9340" PostId="5996" Text="@Dmitrij I don’t think I can help you further.  But generally, when performing simulations you want the model to be close to reality as possible.  I sometimes fit my model to some typical data and then simulate from the fitted model as if it were true.  When simulating from a stationary random walk, people often want to sample from its stationary distribution, but its stationary distribution is hard to sample from directly so that initializing $X_0$ to the stationary distribution is not possible.  In that case burn-in makes sense so that $X_t$ for some large $t$ is approx. stationary." CreationDate="2011-01-08T21:07:20.940" UserId="1670" />
  <row Id="9341" PostId="6087" Text="Would you be kind enough to explain the difference between the scenarios? Unfortunately I am not familiar enough with the use of the Error term in the model (by the way, can you suggest a good book on that?). What I just proposed is the SPSS way of checking the normality assumption, as I have learned it. See this as an example  http://goo.gl/p45Bx" CreationDate="2011-01-08T21:38:08.413" UserId="339" />
  <row Id="9342" PostId="862" Text="@nico. The default in R is breaks=&quot;Sturges&quot; which does not always give good results." CreationDate="2011-01-08T21:48:45.580" UserId="159" />
  <row Id="9343" PostId="6087" Text="Hi gd047. Thank you for the link.  The things I know about these models are all linked to from here: http://www.r-statistics.com/2010/04/repeated-measures-anova-with-r-tutorials/   If you'll get to know of other resources - I'd love to know about them.  Cheers, Tal" CreationDate="2011-01-08T22:16:07.067" UserId="253" />
  <row Id="9344" PostId="862" Text="@Rob Hyndman: for whatever reason my comment was truncated... I meant to write &quot;by default (so, if you don't specify breaks) R uses the Sturges algorithm&quot;... odd!" CreationDate="2011-01-09T00:15:54.283" UserId="582" />
  <row Id="9345" PostId="6093" Text="possibly related http://stats.stackexchange.com/questions/421/what-book-would-you-recommend-for-non-statisticians" CreationDate="2011-01-09T00:52:24.687" UserId="22" />
  <row Id="9346" PostId="6092" Score="1" Text="stackoverflow.com is probably a better place to ask this question, or r-devel list. I suspect that this is possible, but that is not the matter of changing some options. This means that probably you will need to write your own custom install.packages function. I may be wrong of course." CreationDate="2011-01-09T05:36:45.397" UserId="2116" />
  <row Id="9347" PostId="6090" Text="OK, I have two predictors (hdd and cdd), and the squared semi-partial correlations for cdd ~ hdd is 0.755, and for hdd ~ cdd is 0.055, for a total of 0.810 versus the R-squared of 0.845." CreationDate="2011-01-09T06:25:34.327" UserId="1764" />
  <row Id="9348" PostId="6094" Text="Thanks, simple but works!" CreationDate="2011-01-09T07:15:31.060" UserId="2664" />
  <row Id="9349" PostId="6092" Text="I think it is equally off-topic here and on SO, so I'll leave it here. But I doubt this is possible to do easily -- it would need switching R to cross-compilation toolchain." CreationDate="2011-01-09T12:44:53.210" UserId="88" />
  <row Id="9351" PostId="5268" Text="I'm not clear on whether you are after unsupervised clustering (where all variables are treated equally) or in supervised classification (where you have a Y variable and a set of X variables). K-NN is a supervised method, while k-means clustering and PCA are unsupervised.&#xD;&#xA;&#xD;&#xA;Can you explain the purpose of clustering the users?" CreationDate="2011-01-09T12:49:34.520" UserId="1945" />
  <row Id="9352" PostId="5268" Text="Both of you are right, I may have jumped the gun on clustering.  The goal is to find &quot;archetypes&quot; for users in the database, that is, some number of user groups much less than the total number of users, where each group can be described using the fewest number of features while minimizing the average distance between the description and each user.  This is definitely unsupervised." CreationDate="2011-01-09T12:49:34.597" UserId="2305" />
  <row Id="9353" PostId="5876" Text="Where are you getting the 5 in the 1-5?  4 is the average. Should it be 1-4?  I'm testing today with Python, thanks again. " CreationDate="2011-01-09T15:56:10.060" UserId="2585" />
  <row Id="9354" PostId="5876" Text="@NealWalters Sorry, that was a slip of the finger.  I have fixed it now so it reads (1-4)/2*100+500." CreationDate="2011-01-09T16:55:27.400" UserId="919" />
  <row Id="9355" PostId="6097" Score="1" Text="Have you considered vector-valued autoregressive processes as a model?  http://en.wikipedia.org/wiki/Autoregressive_model ." CreationDate="2011-01-09T17:51:59.960" UserId="919" />
  <row Id="9356" PostId="6097" Text="Your second equation, $O^1_j = f(O^1_{j-l}, O^2_{j-l}, O^3_{j-l}) + \epsilon$, is confusing.  Is this the same $f$ as previously?  (I hope not, because that wouldn't make much sense.)  What exactly do you mean by setting $l$ to 1, 2, 3?  That seems to give three separate equations for $O_j^1$." CreationDate="2011-01-09T17:53:48.807" UserId="919" />
  <row Id="9357" PostId="6100" Text="So something like `var calibrated = a.mean - cal.mean;` and then take their standard errors `var error = Math.sqrt( Math.pow(a.SEM, 2) + Math.pow(cal.SEM, 2));`? How does that get factored in for reporting margin of error?" CreationDate="2011-01-09T17:55:05.247" UserId="2634" />
  <row Id="9358" PostId="6097" Text="@whuber: No, the f is different (but note that it is unknown). You're right, probably the equation is confusing... I meant that essentially you have 9 parameters, $O^1$, $O^2$ and $O^3$ for the past 3 observations. I will have a look to the autoregressive models, I am not completely familiar with them but I guess it's a good pointer as a start." CreationDate="2011-01-09T18:01:40.330" UserId="582" />
  <row Id="9359" PostId="6100" Text="@John Sure; for the second thing, I'm afraid I don't understand... how to call it in a report?" CreationDate="2011-01-09T18:53:42.537" UserId="88" />
  <row Id="9360" PostId="6100" Text="@mbq - So how would I report the `error`, if it's not margin of error *(MoE)*, or standard error of the mean *(SEM)*? Would I do something like `moe = error * getCriticalValue(a.size - 1);` similar to if `error` was the SEM?" CreationDate="2011-01-09T19:36:07.133" UserId="2634" />
  <row Id="9361" PostId="6102" Text="Might help if you told us what software package you're using, and what command(s) you typed." CreationDate="2011-01-09T19:55:56.937" UserId="449" />
  <row Id="9362" PostId="6102" Text="@onestop, the article in the link mentions stata, so probably stata is used. Does not matter though, since any statistical package would encounter problems in this particular case." CreationDate="2011-01-09T20:17:26.303" UserId="2116" />
  <row Id="9363" PostId="6104" Text="If you have shown independence and zero means, you only need to check whether the standard errors are ones. In your case if we put $\sigma^2$ inside the parenthesis, your statement will follow if you prove that $EY_i^2/\sigma^2=1$ " CreationDate="2011-01-09T20:53:47.177" UserId="2116" />
  <row Id="9364" PostId="6106" Text="Thank you mpiktas, I will have a look at those functions and see if they can help me with this problem." CreationDate="2011-01-09T21:09:02.207" UserId="582" />
  <row Id="9365" PostId="6090" Text="@caracal (+1) A nice step-by-step illustration." CreationDate="2011-01-09T21:17:57.083" UserId="930" />
  <row Id="9366" PostId="6101" Score="1" Text="Quick look at http://en.wikipedia.org/wiki/Queueing_model says that the case N servers one queue is also well analyzed." CreationDate="2011-01-09T21:19:56.920" UserId="2116" />
  <row Id="9367" PostId="6107" Text="This method is equivalent, and would probably be the way to go, had I not already calculated the means and variances of the Yi's. Either way, it gets stuck at the same point, I believe." CreationDate="2011-01-09T21:22:23.557" UserId="1118" />
  <row Id="9368" PostId="6048" Text="@Rom, if you do not know how to interpret results, try formulating it as a new question." CreationDate="2011-01-09T21:34:10.980" UserId="2116" />
  <row Id="9369" PostId="6094" Text="@Fojtasek, if you use sapply, you won't need to use unlist" CreationDate="2011-01-09T21:36:23.770" UserId="2116" />
  <row Id="9370" PostId="6107" Text="Not really. You have (or at least I now have)  proved that ((Y1^2 + Y2^2 + Y3^2) == (X1^2 + X2^2 + X3^2) and surely that is sufficient to establish the chi-sq-ness when you have Xi ~ N(0, sigma^2)" CreationDate="2011-01-09T21:43:42.297" UserId="2129" />
  <row Id="9371" PostId="6090" Text="One more question: this illustrates how each predictor contributes to the R-squared &quot;variance accounted for&quot;. Is it legitimate to use the coefficients &amp; intercept as I mentioned (in the edited version of) the original posting, to talk about each component's contribution to a yearly total? If so, how do you see the R-squared playing into this?" CreationDate="2011-01-09T23:17:08.603" UserId="1764" />
  <row Id="9372" PostId="6110" Text="Well, I'm no expert in physics. I googled few &quot;Fermi-Dirac statistics&quot; articles, but I can't see any biparametric distribution. Also please note that I am not interested with higher variance but in variance as a parameter." CreationDate="2011-01-09T23:23:47.650" UserId="217" />
  <row Id="9373" PostId="6107" Text="That's where the problem lies. I'm looking for how to do that. I'm not sure how to go from having a sum of three squared normal pdfs to a chi-square pdf with df=3." CreationDate="2011-01-10T00:32:55.853" UserId="1118" />
  <row Id="9374" PostId="6107" Score="1" Text="I'm confused. Isn't that precisely the definition of a chi-sq variate with 3 df? If you are worried about the sigma being different than unity, it's just a constant and so easily factors out. http://en.wikipedia.org/wiki/Chi-square_distribution" CreationDate="2011-01-10T01:09:50.357" UserId="2129" />
  <row Id="9375" PostId="6104" Text="@Christopher (I TeXified your question to make it readable.)  You should double-check the distributional form for $Y_i^2$, because it is incorrect: it is not Gaussian, but exponential.  (That is, the argument of exp should be linear in $x$, not $x^2$.)  There also needs to be a factor of $x^{-1/2}$.  Once you fix that, all you need to show is that a sum of Gamma distributions is also Gamma." CreationDate="2011-01-10T02:05:15.223" UserId="919" />
  <row Id="9376" PostId="6097" Text="@nico You may find it helpful, both conceptually and notationally, to think of $O$ as a sequence of **vectors** $(O_j^1, O_j^2, O_j^3)$ indexed by &quot;time&quot; $j$.  You can then write $O_j = f(O_{j-1}, O_{j-2}, O_{j-3}) + \epsilon$ (and $\epsilon$ is a random vector with zero expectation).  It would help immensely if you can assume $f$ is linear, at least to a good approximation: that's the AR model.  (If this is a correct representation of the situation, then your last formula needs additional fixing...)" CreationDate="2011-01-10T02:24:17.157" UserId="919" />
  <row Id="9377" PostId="6112" Score="3" Text="It's simpler to note that the probability of 4 or more agreements equals the probability of 3 or fewer and, since that exhausts all possibilities, both numbers must equal **1/2** exactly." CreationDate="2011-01-10T02:30:06.123" UserId="919" />
  <row Id="9378" PostId="6114" Text="The &quot;closed formula&quot;, expressed as a polynomial in the original $p_i$, has 2187 terms--more than the number of values you would have to enumerate!" CreationDate="2011-01-10T02:37:02.077" UserId="919" />
  <row Id="9379" PostId="6117" Text="(+1) good and a very fundamental point." CreationDate="2011-01-10T02:47:43.493" UserId="1307" />
  <row Id="9380" PostId="6117" Text="I thought, may be @Lew is trying to estimate the parameter and the corresponding CI." CreationDate="2011-01-10T02:49:00.327" UserId="1307" />
  <row Id="9381" PostId="6108" Score="3" Text="First hit when Googling &quot;negative binomial confidence interval&quot;: http://www.bepress.com/ucbbiostat/paper242/ (abstract only).  The seventh hit gets you a full paper along with its references to the literature: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0000180 ." CreationDate="2011-01-10T02:51:30.860" UserId="919" />
  <row Id="9382" PostId="6104" Text="@Whuber: Thanks for TeXing me out! $Y_i$ is distributed normal, being a linear transformation of two random normal variables. I didn't specify the distribution of $Y_i$, but I'd figure it to be $\chi^2(1)$. The sqrt(x) also confused me. I was not sure where that should come from. No one in the normal pdf could I see a sqrt(x) coming from. I was hoping someone would have pointers." CreationDate="2011-01-10T04:13:50.747" UserId="1118" />
  <row Id="9383" PostId="6104" Text="@Christopher. If Yi are the sum of normal variates with mean zero then they cannot be χ2(1)." CreationDate="2011-01-10T05:33:36.017" UserId="2129" />
  <row Id="9384" PostId="6118" Text="I would suggest you check out these links to other questions on the site, http://stats.stackexchange.com/q/421/1036 , http://stats.stackexchange.com/q/170/1036 as well as searching the other questions on this site (looking at the questions tagged book would be a good start,  http://stats.stackexchange.com/questions/tagged/books ). Your question will likely be closed as recent others have, http://stats.stackexchange.com/q/6093/1036 . Your particular field makes little to no difference in what intro content to statistics people would suggest." CreationDate="2011-01-10T06:10:08.847" UserId="1036" />
  <row Id="9386" PostId="6111" Text="added r tag, so the R code in the comment is highlighted" CreationDate="2011-01-10T07:17:19.807" UserId="2116" />
  <row Id="9387" PostId="6104" Text="Sorry if I was unclear, I meant each $Y_i$ was distributed $\chi^2(1)$." CreationDate="2011-01-10T07:33:10.153" UserId="1118" />
  <row Id="9388" PostId="6121" Text="Thanks, mpiktas. I figured there must've been a mistake somewhere in my pdf." CreationDate="2011-01-10T08:31:01.340" UserId="1118" />
  <row Id="9389" PostId="6111" Text="@mpiktas This question is quite independent of R, IMHO. Moreover, code highlighting is handled through [Google Prettify](http://code.google.com/p/google-code-prettify/) on SE sites, so there's no need to add extra tag to enable syntax highlight." CreationDate="2011-01-10T09:10:59.933" UserId="930" />
  <row Id="9390" PostId="6111" Text="@chl, I asked the question on meta concerning adding r tags, since I felt too, that adding r tag just for syntax highlighting is not appropriate. Google prettify did not work for me. " CreationDate="2011-01-10T09:22:26.953" UserId="2116" />
  <row Id="9391" PostId="6090" Text="@Wayne You need to calculate the semi-partial correlation between your DV `elec` and each of your predictors. Taking the coefficients as a measure of &quot;influence&quot; doesn't seem like a good idea to me: a) the coefficients are scale dependend b) for a coefficient $b_{i}$, it is $\hat{Y}$ that changes by $b_{i}$ units when predictor $i$ changes 1 unit, not $Y$ itself. Before delving further into regression theory, I'd take the increase in $R^{2}$ as a measure for the improvement in prediction wenn adding a predictor." CreationDate="2011-01-10T09:25:06.707" UserId="1909" />
  <row Id="9393" PostId="6084" Text="I'm not really interested in the effect of the other coravirates, I just put them in the model to control for the effect they have. If I understand you correctly, my statistic &quot;indicates a genuine effect even though it is only borderline significant&quot;?" CreationDate="2011-01-10T10:41:55.630" UserId="2652" />
  <row Id="9394" PostId="6085" Text="Can I acheive this interval-censored model by calculating a new variable as the LN(age)? I remember reading about that somewhere..." CreationDate="2011-01-10T10:46:22.073" UserId="2652" />
  <row Id="9395" PostId="6108" Text="I downvoted the question because it's eminently google-able." CreationDate="2011-01-10T13:20:59.427" UserId="1979" />
  <row Id="9396" PostId="6048" Score="1" Text="...if by &quot;handle&quot; you mean ignore/remove all observations with missing values for any of the covariates or responses. " CreationDate="2011-01-10T13:22:41.157" UserId="1979" />
  <row Id="9397" PostId="6090" Text="@Carcal: I think your point B is pretty important: the distinction between $\hat{Y}$ versus $Y$. I'm wanting to compare $\sum_t{b_i} p_{it}$ to $\sum_{t=1}^n{\sum_i{b_i p_{it}}} + n a$ where the intercept is $a$ and predictor $i$ at time $t$ is $p_{it}$. Not comparing coefficients, but rather how much that predictor contributed to the sum. Does that make sense?" CreationDate="2011-01-10T13:29:43.207" UserId="1764" />
  <row Id="9398" PostId="6128" Text="Thanks for your great answer! 2 more questions: Does your second example mean, that coin in fact does provide all possible permutation and is an exact test? Is there any benefit of not providing an exact test in my case?&#xA;" CreationDate="2011-01-10T13:45:37.237" UserId="442" />
  <row Id="9399" PostId="6104" Text="@Christopher I suspect that in computing the pdf of $Y_i^2$ you made a substitution but did not back-substitute again.  Letting $y=x^2$, you can rewrite $C\exp(-2x^2/(2\sigma^2))dx$ = $C\exp(-y/\sigma^2)dy/(2\sqrt{y}),$ which is a (scaled) gamma(1/2) distribution." CreationDate="2011-01-10T13:46:10.027" UserId="919" />
  <row Id="9400" PostId="6129" Text="The scores are not proportions but estimates by participants on a 0 to 100 scale (the presented data are means of estimates on several items with that scale)." CreationDate="2011-01-10T13:57:04.573" UserId="442" />
  <row Id="9401" PostId="6128" Score="4" Text="(+1) It is no surprise that the (unpaired) t-test yields essentially the same p-value, 0.000349.  Despite what the reviewer said, the t-test *is* applicable to these data.  The reason is that the *sampling distributions* of the means are approximately normal, even though the distributions of the data are not.  Moreover, as you can see from the results, the t-test actually is more conservative than the permutation test.  (This means that a significant result with the t-test indicates the permutation test is likely to be significant, too.)" CreationDate="2011-01-10T14:09:22.393" UserId="919" />
  <row Id="9402" PostId="6127" Text="+1: This is a really well formulated question." CreationDate="2011-01-10T14:11:32.160" UserId="919" />
  <row Id="9403" PostId="6129" Text="Then non-parametrics would seem the traditional way to go. That said, I've wondered if such scale data might usefully be inferred to derive from a binomial process and thereby analyzed as such. That is, you say that each score is the mean of several items, and let's say each item is a 10 point scale, in which case I'd represent a response of, say, &quot;8&quot; as a series of trials, 8 of which have the value 1 and two of which have the value 0, all labelled with the same label in an &quot;item&quot; variable. With this expanded/binomial-ized data, you could then compute the binomial mixed effects model." CreationDate="2011-01-10T14:24:13.833" UserId="364" />
  <row Id="9404" PostId="6129" Text="Following from my previous comment, I should note that in the expanded/binomial-ized data, you could model the &quot;item&quot; variable as either a fixed or random effect. I think I'd lean toward modelling it as a fixed effect because presumably you might be interested in not only accounting for but also assessing the item differences and any possible interaction between item and other predictor variables." CreationDate="2011-01-10T14:26:21.960" UserId="364" />
  <row Id="9405" PostId="6128" Score="1" Text="@Henrik For certain situations (chosen test and numerical complexity), `coin` can indeed calculate the exact permutation distribution (without actually going through all permutations, there are more elegant algorithms than that). Given the choice, the exact distribution seems preferable, but the difference to a Monte-Carlo approximation with a high number of replicates should be small." CreationDate="2011-01-10T14:26:48.900" UserId="1909" />
  <row Id="9406" PostId="6128" Text="@Caracal Thanks for the clarification. One question remains: The data I presented is paired. Hence, I need the equivalent to the paired t-test. Is `oneway_test` the accurate function? And if so, which one is the correct for non-paired data?" CreationDate="2011-01-10T14:34:16.303" UserId="442" />
  <row Id="9407" PostId="6128" Text="@Henrik An exact permutation test needs 19! = 121 645 100 408 832 000 permutations.  Assuming a really fast 10^9 applications per second, that would require 3.87 years of computation.  (A clever algorithm would exploit all the repetitions among the x's to reduce that to one second, but I think you would have to hand-code that.)" CreationDate="2011-01-10T14:34:31.077" UserId="919" />
  <row Id="9408" PostId="6128" Text="@Henrik You're in trouble for paired data with a nonparametric test: the p-value is around 40%.  The problem is that there are some (inconsequential?) increases from x to y, a number of tied values, and some really big decreases from x to y.  By ignoring the magnitude of the decreases you lose too much information." CreationDate="2011-01-10T14:36:26.800" UserId="919" />
  <row Id="9409" PostId="6131" Text="Attack of the R code syntax highlighting! Is it possible to not have the code highlighted?" CreationDate="2011-01-10T14:50:12.483" UserId="1036" />
  <row Id="9410" PostId="6128" Text="@whuber: when using `wilcox.test` from stats with paired = T, I get a p-value of .02, which is reasonable. So I cannot replicate the .4 you mention. However, I am still unable to get a paired test to work with coin. (When using `wilcoxsign_test` I get an error: Not an unreplicated complete block design)" CreationDate="2011-01-10T14:52:20.547" UserId="442" />
  <row Id="9411" PostId="6048" Text="@AndyW &amp; fabians: I've updated the post to reflect my understanding of how mixed effects models handle missing data, please feel free to let me know if my understanding is erroneous in any way!" CreationDate="2011-01-10T14:57:27.813" UserId="364" />
  <row Id="9412" PostId="6130" Text="I wouldn't suggest resorting to python to accomplish this task unless you need it to be standardized to run on multiple datasets with different variable names. The UCLA website is full of useful examples though of interest to anyone who uses SPSS." CreationDate="2011-01-10T14:58:54.863" UserId="1036" />
  <row Id="9413" PostId="6085" Text="Interesting suggestion, to which I have not had prior exposure. I am reasonably sure that would not achieve the precise goal I had in mind. It might achieve similar objectives by dampening the effect of age, though. The effect of age on the hazard scale would now only be linear rather than exponential. Whether that would correspond to the &quot;true&quot; age effect on the response under study would still be open to question." CreationDate="2011-01-10T15:21:08.637" UserId="2129" />
  <row Id="9414" PostId="6084" Text="Hardly. I would not accept that as a fair paraphrase of what I said. I say one would first need to do a LRT examination before doing further discussion of whether the borderline Wald test even met conventional standards for significance. And  I think the work &quot;genuine&quot; is a far reach in the absence of better context." CreationDate="2011-01-10T15:25:37.367" UserId="2129" />
  <row Id="9415" PostId="6131" Text="(+1) For SPSS Code - Go SPSS!!" CreationDate="2011-01-10T15:31:58.593" UserId="1307" />
  <row Id="9416" PostId="6131" Text="@Andy What's wrong with highlighting SPSS code?  Code highlighting isn't restricted to R.  If you don't want it highlighted, don't indent it..." CreationDate="2011-01-10T15:38:56.377" UserId="5" />
  <row Id="9417" PostId="6131" Text="@Andy The highlighting engine used on SE guesses language on its own; it is even not capable of highlighting R (it is  interpreted as C); this seems to be interpreted as Pascal because of &quot;end&quot;." CreationDate="2011-01-10T16:10:32.273" UserId="88" />
  <row Id="9418" PostId="6114" Text="Yes and No answers have a probability of 0.5. This is correct: 0.2734375 0.1640625 0.0546875 0.0078125. There are 16 possible combination of Yes answers for two users: {4,4} {4,5} {4,6} {4,7} {5,4} {5,5} {5,6} {5,7} {6,4} {6,5} {6,6} {6,7} {7,4} {7,5} {7,6} {7,7}. And we can estimate the probability of each of these 16 pairs occuring. However once a pair occurs - the probability of a match is different. In other words for a {4,4} the probability is 1/35 as there are 35 combinations of 4 out of 7. For {4,5} the probability of a match increases. How do we take those in account." CreationDate="2011-01-10T16:20:43.080" UserId="2715" />
  <row Id="9419" PostId="6131" Score="1" Text="@Shane, it is not the blocked code highlighting (the grey background) that is the problem, it is the fact that the wrong keywords within the syntax are highlighted. For instance my loop variable in the simulated data is highlighted with a lighter grey shade, suggesting the code after the pound sign is a comment. I think it is much easier to read as a blocked code segment, but the highlighting can only be confusing. Is there anyway to keep the blocked code but not have highlighting?" CreationDate="2011-01-10T16:22:26.883" UserId="1036" />
  <row Id="9420" PostId="6128" Text="@Henrik Wilcox.test assumes the distribution of x-y is *symmetric* about its mean.  It is also unclear whether wilcox.test handles ties appropriately.  Thus I think your p-value of 0.02 is invalid." CreationDate="2011-01-10T16:25:35.913" UserId="919" />
  <row Id="9421" PostId="6128" Text="@whuber, re: exhaustive permutations -&gt; The exhaustive variant of the permutation test is sometimes termed a &quot;randomization test&quot; (eg, by Mewhort et al, 2009, http://www.springerlink.com/content/pg17u7141845w082/) and advances by Gill (2007, discussed in the Mewhort paper) dramatically reduce the computational requirements." CreationDate="2011-01-10T17:04:50.217" UserId="364" />
  <row Id="9422" PostId="6134" Text="I'm curious if you'd consider the following an appropriate quasi-visualized approach: bootstrap estimates for the moments of the two groups (means, variances, and higher moments if you so desire) then plot these estimates and their confidence intervals, looking for the degree of overlap between groups on each moment. This lets you talk about potential differences across the variety of distribution characteristics. If the data are paired, compute the difference scores and bootstrap the moments of this single distribution. Thoughts?" CreationDate="2011-01-10T17:21:47.220" UserId="364" />
  <row Id="9423" PostId="6128" Text="@Mike Thank you.  I don't have access to the paper, so I can only guess that it might be doing something similar to what I had in mind when I suggested 4 years of calculation could be reduced to one second :-)." CreationDate="2011-01-10T17:54:49.470" UserId="919" />
  <row Id="9424" PostId="6132" Score="1" Text="Exactly right. Aggregating prematurely is likely to obscure important relationships. If meat prices are going up in concert, is it because energy or transportation prices went up in prior intervals? I see no mention of methods for examining auto-correlation or cross-correlation which would clearly be needed for a proper treatment." CreationDate="2011-01-10T17:56:47.283" UserId="2129" />
  <row Id="9425" PostId="6134" Text="(+1) Good analysis. You're perfectly right that the results are obvious and one doesn't need to press the point with a p-value.  You may be a little extreme in your statement of (3), because the t-test does not require normally distributed data.  If you are concerned, there exist adjustments for skewness (e.g., Chen's variant): you could see whether the p-value for the adjusted test changes the answer.  If not, you're likely ok.  In this particular situation, with these (highly skewed) data, the t-test works fine." CreationDate="2011-01-10T17:58:13.803" UserId="919" />
  <row Id="9426" PostId="6135" Text="+1 Adjoining the lexicon and then re-minimizing it is a clever idea.  The algebra is beyond me, but it feels like you are calculating a multinomial probability, rather than hypergeometric.  So the probability is for sampling _with_ replacement.  I think that explains why your answer of 1.08% is so much larger than my estimate of 0.4%.  Is there a way to modify your approach to handle sampling without replacement?" CreationDate="2011-01-10T18:08:04.573" UserId="1670" />
  <row Id="9427" PostId="6131" Text="@Andy (+1) Good that you provide a pure SPSS solution." CreationDate="2011-01-10T18:21:50.923" UserId="930" />
  <row Id="9428" PostId="6134" Text="(+1) Nice catch! And very good comments." CreationDate="2011-01-10T18:26:32.507" UserId="930" />
  <row Id="9429" PostId="6114" Text="I'm not sure I fully understand your question, but in the `for()` loop, each number of pairwise agreements is considered separately (4, 5, 6, 7 in your case): for each of these numbers, all possible answer patterns leading to that agreement are enumerated (`poss`), and their respective probabilities calculated. One way to check that the result is correct, you can set `k` (all numbers of agreement) to `0:7`. The sum for all probabilities should then be 1." CreationDate="2011-01-10T19:11:03.180" UserId="1909" />
  <row Id="9430" PostId="6128" Text="@whuber: The preprint of the Mewhort paper can be accessed via his website: http://psyc.queensu.ca/~hiplab/LAB_PUBS/BSC903_Mewhort_Kelly_Johns_BRM.pdf" CreationDate="2011-01-10T19:47:49.060" UserId="364" />
  <row Id="9431" PostId="6135" Score="2" Text="@vqv Yes.  Now that we have a list of the half million or so racks with no words, it's straightforward (by changing the last two lines of code) to compute the chance of each rack (without replacement) and obtain the hypergeometric result.  The exact answer equals 349870667877/80678106432000 = **0.43366%**.  With N=100K trials your SE is 0.021%, so your answer should have come between 0.38% and 0.49% (two-sided 99% CI).  I'm so glad our answers agree!" CreationDate="2011-01-10T19:48:48.140" UserId="919" />
  <row Id="9432" PostId="6139" Score="1" Text="[Cross Validated](http://stats.stackexchange.com/)" CreationDate="2011-01-10T11:17:44.437" UserId="2036" />
  <row Id="9433" PostId="6135" Text="@whuber Could you run the calculation using the Words With Friends (WWF) tile distribution?  My estimate of 0.4% is based on the WWF lexicon and WWF tile distribution.  I think you are using the Scrabble tile distribution with the WWF lexicon." CreationDate="2011-01-10T20:34:57.437" UserId="1670" />
  <row Id="9434" PostId="6114" Text="@Rado It looks like you are conditioning on the numbers of yes answers; *e.g.*, {6,4} means respondent 1 had 6 yeses and respondent 4 had 4 yeses.  This is a valid approach but is more complex than necessary.  @caracal is modeling it question by question: on question $i$, the chance of agreement is like flipping a coin with probability $p_i^2 + (1-p_i)^2$.  By looking at all possible outcomes of these 7 coin flips--there are only 128 of them--we can compute the chance of any pattern of agreement." CreationDate="2011-01-10T20:37:55.523" UserId="919" />
  <row Id="9435" PostId="5853" Text="@NealWalters: reading your last comment, I wonder what approach is best: give, e.g. 1 point for each loan paid off, other points for other things, then try to tame the enormous score at the end, or perhaps do something more meaningful with each component that makes up the final score? For example, if you give points for loans paid off you could calculate your Loans Paid Off (LPO) points as log (1 + NumberOfLoansPaidOff). So someone with zero loans paid off gets an LPO of 0, someone with 3 loans paid off gets an LPO of 1.1, while someone with 100 loans paid off gets an LPO of 4.6. Cap LPO at 5." CreationDate="2011-01-10T20:38:38.050" UserId="1764" />
  <row Id="9436" PostId="6114" Text="I might be missing somenthig - ( res ) is probability of selecting 4, 5,6 or 7 &quot;yes&quot;, e.g. for 4 it is 35/128. sum (res) - are you suggesting that this is the probability of the the two picking the same answers? To me sum (res) is the probability of anyone selecting 4 or more Yes out of all possible combinations." CreationDate="2011-01-10T20:40:38.370" UserId="2715" />
  <row Id="9437" PostId="6135" Text="Oops.  The exact answer actually is 349870675899 (I was 8022 off due to an error in my dictionary.)  This makes no practical difference, fortunately." CreationDate="2011-01-10T20:43:07.797" UserId="919" />
  <row Id="9438" PostId="6135" Text="@vqv I'm not familiar with the various tile distributions.  I copied mine directly from your code (and I used your dictionary) :-).  If you mean the distribution at http://osxreality.com/2010/01/01/beginners-guide-to-words-with-friends-2/ , then I obtain **1.15444%** (with replacement), **0.43366%** (without replacement).  The second number actually does differ from the Scrabble frequencies at the 8th significant figure." CreationDate="2011-01-10T20:51:44.200" UserId="919" />
  <row Id="9439" PostId="6139" Text="You might be interested in the recent thread at http://stats.stackexchange.com/q/5750/919 ." CreationDate="2011-01-10T20:55:32.173" UserId="919" />
  <row Id="9440" PostId="6114" Text="@Rado No, `res` contains the probabilities for exactly 4, 5, 6, or 7 pairwise agreements. I.e., the probability that for exactly 4 (5, 6, 7) questions, either both participants answer &quot;yes&quot; or both participants answer &quot;no&quot;. `sum(res)` therefore is the probability of 4 or more pairwise agreements." CreationDate="2011-01-10T20:57:18.700" UserId="1909" />
  <row Id="9441" PostId="6135" Text="@whuber I have both in my code.  The Scrabble distribution was commented, but looking above it seems like you used the Scrabble distribution.  btw, great follow up.  We now have a complete solution for the specific problem in the title." CreationDate="2011-01-10T20:59:10.453" UserId="1670" />
  <row Id="9442" PostId="6114" Text="@Rado It sounds like you're both right, at least when all $p_i$ are 1/2.  In both cases the outcomes can be modeled by seven flips of a fair coin.  In your case you interpret the sides of the coin as &quot;yes&quot; and &quot;no&quot; on one response; in @caracal's case the sides of the coin mean &quot;agree&quot; and &quot;disagree&quot; for a given pair of responses." CreationDate="2011-01-10T21:11:46.343" UserId="919" />
  <row Id="9443" PostId="6134" Text="We seem to be accepting the notion that the underlying distribution is &quot;similar&quot; to the random instantiation. So couldn't one pose the question: are these both from beta(0.25, 0.25) and then the test would be whether they have the same (non-)centrality parameter. And wouldn't that justify using either a permutation test or Wilcoxon?" CreationDate="2011-01-10T21:13:05.857" UserId="2129" />
  <row Id="9444" PostId="6135" Text="@vqv I see it now.  I just happened to grab the Scrabble frequencies.  BTW, simulation (at least with N &lt; 1000000 or so) is faster.  My aim was to find a solution that required little coding and still was computationally efficient.  For the greatest computational efficiency, I think both methods can be beaten by a suitable Boyer-Moore type of algorithm.  That is, one needs to sequentially construct a DAG whose nodes represent subsets containing no words and whose leaves are words.  (This is basically what the polynomial algebra is doing automatically.)" CreationDate="2011-01-10T21:20:33.543" UserId="919" />
  <row Id="9445" PostId="6140" Text="and a good short discussion and with two excellent references on [Paul Hippel's pages](http://www.sociology.ohio-state.edu/people/ptv/faq/weights.htm) " CreationDate="2011-01-10T21:31:40.460" UserId="1739" />
  <row Id="9446" PostId="6135" Text="@whuber @vqv The algebra is also beyond me (although clearly exposed), but I'd like to say that I still admire your tenacious efforts (and the follow-up you gave)." CreationDate="2011-01-10T22:01:27.513" UserId="930" />
  <row Id="9447" PostId="5928" Text="You know, dasymetric mapping ultimately as an ecological inference problem. The recent work of K. Imai might be helpful: http://pan.oxfordjournals.org/content/16/1/41.abstract" CreationDate="2011-01-10T22:04:16.043" UserId="82" />
  <row Id="9448" PostId="6114" Text="thanks that clarifies it. I will test the R code shortly." CreationDate="2011-01-10T22:10:05.067" UserId="2715" />
  <row Id="9449" PostId="6145" Text="Thank you for the answear.&#xA;But is the first model pooled? I have added the LAND term wich is the entity dummy variable (as it's a facor variable.&#xA;And as I can se the parameter estimates (EURO) are identical?" CreationDate="2011-01-10T22:27:09.540" UserId="2724" />
  <row Id="9450" PostId="6114" Text=" [C(7,4) + C(7,5) + C(7,6) + C(7,7)]*(1/2)^7 = (35+21+7+1)*1/128 = 64/128 = 1/2." CreationDate="2011-01-10T22:30:24.480" UserId="2715" />
  <row Id="9451" PostId="6145" Text="@Skolnick, I've missed that at first, so I changed my answer accordingly" CreationDate="2011-01-10T22:31:03.123" UserId="2116" />
  <row Id="9452" PostId="6135" Text="@chl &quot;algebra&quot; = &quot;magic&quot; ;-)." CreationDate="2011-01-10T23:05:25.670" UserId="919" />
  <row Id="9453" PostId="6147" Score="2" Text="http://en.wikipedia.org/wiki/Multivariate_normal_distribution .  But probably your prof mis-spoke (or you misheard): you need to understand how to work with the Normal distribution, not the multivariate normal.  To this end I added the &quot;homework&quot; tag because that will give you links to worked examples and clear explanations, such as http://stats.stackexchange.com/questions/5504/normal-distribution-probability ." CreationDate="2011-01-10T23:23:45.293" UserId="919" />
  <row Id="9454" PostId="6119" Score="1" Text="You simply must read Mosteller &amp; Wallace, *Applied Bayesian and Classical Inference, The Case of the Federalist Papers.*  It is a classic, it's readable, and if by page 12 you aren't thoroughly convinced it has plenty to show you, just return it to the library or bookseller." CreationDate="2011-01-10T23:42:25.577" UserId="919" />
  <row Id="9455" PostId="6151" Score="1" Text="@noli please clarify your question - it is not clear that you need a 'treatment' to answer your question. can you provide a list or some examples of the questions in the survey? Do you plan to have 100 individuals, 50 sibling pairs, or something else?" CreationDate="2011-01-11T01:21:45.663" UserId="1381" />
  <row Id="9456" PostId="6153" Text="@DWin I am not sure if it is clear, but the histograms are of the 30 values of variance, and the barplots are the raw values of the variance, i.e. `var &lt;- c(0,0,1,3,10,100,150), hist(var), barplot(var)`, so I interpret this as a few parameters explain most of the variance, not that most of the explained variance is in the tail. Does that make more sense? Sorry if it was unclear." CreationDate="2011-01-11T04:07:48.073" UserId="1381" />
  <row Id="9457" PostId="6154" Score="1" Text="I am not an expert on SAS or SEM, but if you get t-values, can't you calculate p-values yourself?" CreationDate="2011-01-11T04:24:26.237" UserId="2116" />
  <row Id="9458" PostId="6147" Text="Most likely I misheard :) Thank you." CreationDate="2011-01-11T04:29:12.113" UserId="2729" />
  <row Id="9459" PostId="6155" Text="I have tried to use http://cran.r-project.org/web/packages/graph/vignettes/graph.pdf but it seams that the package &quot;Rgraphviz&quot; is not available anymore. Anyone have another solution?" CreationDate="2011-01-11T05:03:10.347" UserId="1709" />
  <row Id="9460" PostId="6155" Text="Rgraphviz is on bioconductor: http://www.bioconductor.org/packages/release/bioc/html/Rgraphviz.html" CreationDate="2011-01-11T05:07:07.760" UserId="159" />
  <row Id="9461" PostId="6151" Text="Jealousy____ Competition for parents' attention____ Difference in terms of gender____     These are sample choices in the questionnaire. They have to rank them from 1 to 15. 1 being the top and 15 as the least., I have 50 pairs of siblings as my respondents, How can I detect if there is really a rivalry between siblings, what will be my statistical treatment on these?, Thanks David for your response, I'll be waiting for your next post" CreationDate="2011-01-11T05:20:58.280" UserId="2731" />
  <row Id="9462" PostId="6067" Text="The area under the ROC-curve is .585, a rather poor result. This implies that there really isn't a cutoff value where the specificity/sensitivity trade-off is worth it. Fiddling with the cutoff won't improve classifications much, as it would just decrease the specificity by roughly as much as it increases the sensitivity. " CreationDate="2011-01-11T07:52:43.050" UserId="2690" />
  <row Id="9463" PostId="6086" Text="Yeah, I should have mentioned that the results from the ROC-curve weren't convincing also. In this case I think there isn't a threshold that gives satisfactory results." CreationDate="2011-01-11T07:58:11.313" UserId="2690" />
  <row Id="9464" PostId="6126" Text="Our current policy is to not weight, since I, like you, decided that the weights were rather arbitrary and were probably no better than the unweighted results.  When we sample, we target a specific distribution using each segment's proportional size, but this is also using a relatively arbitrary target distribution, so we may need to change this policy as well." CreationDate="2011-01-11T09:38:03.150" UserId="1195" />
  <row Id="9465" PostId="6140" Text="thanks very much, that's the answer I was looking for.  It was more of a curiosity thing and having the answer on hand to be able to explain to someone if they asked me about it, but I agree with you and @Mohit" CreationDate="2011-01-11T09:39:27.500" UserId="1195" />
  <row Id="9467" PostId="6128" Text="@caracal. Thanks for the update. But what is the rational for not using the exact statistics but an approximation? (The results are markedly different...)" CreationDate="2011-01-11T09:50:47.767" UserId="442" />
  <row Id="9468" PostId="6134" Text="+1 Thanks for the detailed answer. One point: The data is paired. I.e., Each data pair is responses from a single participant on different trial types. We constructed the trial in such a way to expect differences in that way. Furthermore, I have to present the results (i.e., there exists differences) as briefly as possible. So, I have to see how I can use your analysis." CreationDate="2011-01-11T10:22:31.297" UserId="442" />
  <row Id="9470" PostId="6156" Text="(+1) It seems my response came after yours. As your response highlights somewhat a different perspective (cross-platform, algorithms), I feel like our responses are not so redundant, but I can remove mine without any prob." CreationDate="2011-01-11T10:23:29.347" UserId="930" />
  <row Id="9471" PostId="6160" Text="@Thomas (+1) Never used this package, but your overview suggest I shoud try it. Seems good at first sight." CreationDate="2011-01-11T10:26:01.560" UserId="930" />
  <row Id="9472" PostId="6156" Text="@chl: don't worry I think we can leave both with no problem. :)" CreationDate="2011-01-11T11:06:59.777" UserId="582" />
  <row Id="9473" PostId="6138" Text="That's a good start, but I think there is more to the problem. For example: how do they get the answers to the questions? Presumably they use the answers given by previous players (a reinforcement learning problem), in which case we face the tradeoff of choosing a question that (a) solves the problem for the current player, and (b) provides information for future players." CreationDate="2011-01-11T11:10:53.033" UserId="495" />
  <row Id="9474" PostId="6132" Score="1" Text="Yes, collinearity is a major problem, but the relationships change over time, since one factor may affect one group of commodity prices at one point, then affect a different group when the factor changes in a different way." CreationDate="2011-01-11T13:09:26.660" UserId="1195" />
  <row Id="9475" PostId="6163" Text="@johanvdw. Are you trying to calculate the variance of y_pred in Deming Regression. I am sorry, its not clear to me." CreationDate="2011-01-11T14:34:32.687" UserId="1307" />
  <row Id="9476" PostId="6163" Text="Sorry for being unclear. I would like to know the prediction error when I use a new x (not one used to fit the model) trying to predict y (y_pred). The standard error of x is known." CreationDate="2011-01-11T14:44:22.687" UserId="2732" />
  <row Id="9477" PostId="6154" Text="Yes which is what I do, but in every other SAS procedure the p-value is given along with the test statistic." CreationDate="2011-01-11T14:46:10.473" UserId="2310" />
  <row Id="9478" PostId="6162" Text="Thanks.  I should have made it clear that I know how to calculate a p-value given a test statistic but I find it strange that they don't output the p-value for you as in almost all the other SAS procedures." CreationDate="2011-01-11T14:47:12.937" UserId="2310" />
  <row Id="9479" PostId="6110" Text="Comparing Gamma random variables is exactly the kind of idea I needed. How can I download the article?" CreationDate="2011-01-11T14:51:50.003" UserId="217" />
  <row Id="9480" PostId="6166" Score="1" Text="@mpiktas. Good answer. However, I think, if @johanvdw wants to predict for new x, the prediction error will be larger than you estimate. If I understood him correctly, var(pred_y) = var(\hat y) + var(error)." CreationDate="2011-01-11T15:30:39.607" UserId="1307" />
  <row Id="9481" PostId="6128" Text="@caracal I would be grateful if you could give some hints. as the difference is huge: p = .01 (approx) and p = .15 (exact)" CreationDate="2011-01-11T15:31:41.460" UserId="442" />
  <row Id="9482" PostId="6166" Text="@mpiktas - thanks, that was fast!&#xA;@suncoolsu I think the error in y should not be added. The regression coefficients already include that uncertainty. You wouldn't add it in a normal linear regression either." CreationDate="2011-01-11T15:45:59.693" UserId="2732" />
  <row Id="9483" PostId="5994" Text="No, I am correct.  If I'm wrong show me.  Try it and see the difference in the distribution around the mean and the different scale in the bell curve.  Assumptions about math/finance formulas are value based not empirical.  There is no other explanation.  The need to forecast takes precendents over complexity every time." CreationDate="2011-01-11T17:04:35.797" UserId="2647" />
  <row Id="9484" PostId="6149" Text="I think they discuss the (central) hypergeometric distribution, if I am not mistaken. Do correct me if I am wrong." CreationDate="2011-01-11T17:19:19.993" UserId="2728" />
  <row Id="9485" PostId="6166" Score="1" Text="@johanvdw. I think we do need to add the error in normal linear regression for prediction of a new Y (for a new X). Please see : equation 2.38 in section 2.5 of Applied Linear Statistical Models - Kutner, Nachtsheim, Neter, and Li (2005). McGrawHill" CreationDate="2011-01-11T18:12:56.457" UserId="1307" />
  <row Id="9487" PostId="6114" Text="@Rado Your formula answers two questions: (1) the chance of four or more &quot;agrees&quot; in seven draws and--reversing the sides of the &quot;coin&quot;--(2) the chance of four or more &quot;disagrees&quot;.  These events are complementary: they don't overlap and they include all possibilities.  Thus the chance of each of these events must equal 1/2.  No calculation needed!  The point of @caracal's response is to show how to compute the chances when the frequencies of &quot;yes&quot; responses to one or more questions depart markedly from 1/2.  This might be a more meaningful (and accurate) approach." CreationDate="2011-01-11T18:58:33.843" UserId="919" />
  <row Id="9488" PostId="6164" Text="@Asif, could you be more specific? What kind of model did you fit exactly? Could you post the outputs illustrating the problem?" CreationDate="2011-01-11T19:01:31.453" UserId="2116" />
  <row Id="9489" PostId="6149" Text="@highBandWidth you are correct, I have changed my answer. Perhaps it would be better to rephrase the question 'Is there a conjugate prior...'" CreationDate="2011-01-11T19:20:34.707" UserId="1381" />
  <row Id="9490" PostId="6128" Score="1" Text="@Henrik The `coin` author wrote me that `oneway_test()` cannot calculate the exact distribution for the dependent case, you have to go with the MC approximation (only `wilcoxsign_test()` is suitable for the exact test). I didn't know this and would prefer an error in that case, but MC should be accurate enough with a high number of replicates." CreationDate="2011-01-11T19:30:15.453" UserId="1909" />
  <row Id="9491" PostId="6151" Text="@noli it is still not clear what your questions are, and I think that you are getting your predictors and responses confused. For example, 'difference in terms of gender' would most commonly be used as a predictor, since 'sex' can be assumed to be binary (male or female) whereas 'gender' can have more nuanced meaning; to evaluate &quot;how different is your gender&quot; as a response on a scale of 1 to 15 would require a clear definition of gender." CreationDate="2011-01-11T19:33:11.693" UserId="1381" />
  <row Id="9492" PostId="6138" Text="At the first glance, ability to draw analogy between 20 questions and Huffman coding hinges on the ability to ask &quot;range questions&quot;. That is, instead of &quot;Have you character ever been to space?&quot; we are asking &quot;blanket&quot; questions like &quot;Has he ever been to space, or is a male, or is bald, or was in a movie, or ... (100500 other options)?&quot;&#xA;&#xA;Am I right? If so, then I should probably edit my question to make it clear that I am interested in &quot;ask one by one&quot; variety" CreationDate="2011-01-11T19:41:37.317" UserId="2696" />
  <row Id="9493" PostId="6138" Text="Plus, most of the articles that use Huffman codes as model for 20 questions, imply that questioner is free to make up his own questions, which in essence boil down to &quot;Does bit $i$ in codeword for the object is $0$&quot;? However, in my case (or, rather, akinator.com's case) the set of questions is predefined, and it (obviously) has nothing to with Huffman codewords.&#xA;&#xA;Right now I can't see how to make transition from my question to Huffman codes. Perhaps you could elaborate?" CreationDate="2011-01-11T19:52:44.150" UserId="2696" />
  <row Id="9494" PostId="6166" Text="@suncoolsu and @johanvdw,  I've updated the answer to reflect your discussion. @suncoolsu is right, we need to add the error, but only if we want to forecast true $y$, if we want the forecast of $y^*$ the addition is not necessary. @johanvdw in normal linear regression error addition is necessary, since normal linear regression does not postulate the latent variable $y^{*}$ " CreationDate="2011-01-11T20:14:45.600" UserId="2116" />
  <row Id="9498" PostId="6172" Text="+1 for pglm, did not know about it. " CreationDate="2011-01-11T20:36:14.897" UserId="2116" />
  <row Id="9499" PostId="6173" Score="3" Text="@Matt (+1) A possible exception is when we have a set of indicators (e.g., items in a questionnaire) with a fixed number of categories (e.g., Likert-type items), as discussed in [Visualizing Likert Item Response Data](http://stats.stackexchange.com/questions/3921/visualizing-likert-item-response-data), or a single crossing factor (e.g., gender), in which case an horizontal (we are better at visually discriminating along the horizontal than the vertical axis) stacked barchart might be interesting, IMO." CreationDate="2011-01-11T20:39:46.760" UserId="930" />
  <row Id="9500" PostId="6172" Text="Thanks, this does indeed look very useful. Do you know how one would go about predicting future values from the pgml() function?" CreationDate="2011-01-11T20:44:05.303" UserId="2704" />
  <row Id="9501" PostId="6110" Text="Added link. Wasn't able to find source of it that was free but perhaps you can get it through your university library." CreationDate="2011-01-11T20:45:39.443" UserId="2129" />
  <row Id="9502" PostId="6173" Score="1" Text="Excellent point - that mostly negates the concern about varying group sizes, assuming non-response is about even across the questions.  I still have difficulty comparing the &quot;slightly concerned&quot; proportions across groups, but the horizontal layout does seem easier.  That post is a great resource - thanks for linking it here." CreationDate="2011-01-11T20:50:15.710" UserId="71" />
  <row Id="9503" PostId="6167" Text="How many classes are there?  If there are substantially fewer than $10^4$, then why not estimate all the probabilities directly from the data as $\hat{P}(A&gt;B)$ = # cases where A is preferred to B / # cases where A is compared to B?  (This is ML and it's as simple as they get.)" CreationDate="2011-01-11T20:56:23.833" UserId="919" />
  <row Id="9504" PostId="6174" Text="Thanks David...I will start with that.  Another small question I had....is there some guidence on how I can interperate the notation you have used above?" CreationDate="2011-01-11T20:56:46.753" UserId="2734" />
  <row Id="9505" PostId="6172" Text="@Thomas Jensen, forecasting the future values is the weak point of plm package. There are no readily available functions even to get fitted values. So I do not expect that pglm will have them. If you chose to estimate model via pglm, you will have to code the prediction yourself. " CreationDate="2011-01-11T20:59:10.867" UserId="2116" />
  <row Id="9507" PostId="6174" Text="@mike $P(Y_{t}=y_{t})$ is the probability that you sell $y$ units in a time $t$; $P(Y_{t}=[0,1,2,3]$ is the probability that you sell 0, 1, 2, or 3 units, $\lambda$ is the poisson parameter, the average rate of sales such that the average units sold in time $t$ is $Y_t=\lambda t$." CreationDate="2011-01-11T21:06:54.340" UserId="1381" />
  <row Id="9508" PostId="6176" Score="1" Text="math.stackexchange.com might be more appropriate place to ask this, and it may already contain the answer. " CreationDate="2011-01-11T21:06:55.433" UserId="2116" />
  <row Id="9509" PostId="6135" Text="@whuber +1, what can I say, awesome follow-up. " CreationDate="2011-01-11T21:17:45.157" UserId="2116" />
  <row Id="9510" PostId="6175" Text="@mpiktas, Thanks a lot for this! I am still a bit new to all of this, how would one write a function for forecasting? Is it a simple matter of fixing the independent variables, and then feed the predicted values back into the lag term in the model to get new predicted values?" CreationDate="2011-01-11T21:20:08.783" UserId="2704" />
  <row Id="9511" PostId="6174" Text="Thanks..is there a general document somewhere on the notation you are using?  Is it some statistical notation?  For example, in &quot;\frac{e^{-\lambda t}(\lambda t)^{y_{t}}}{y_{t}!}$$&quot; I am not clear on the parameter precedence and do I interperate the ! as factorial, or does it have a special meaning there (and what does &quot;frac&quot; mean?).  Thanks for the help." CreationDate="2011-01-11T21:27:08.853" UserId="2734" />
  <row Id="9512" PostId="6175" Score="1" Text="@Thomas Jensen, something like this. When you have data up until period $t$, you can calculate the forecast of $y_{t+1}$. You use this forecast when forecasting $y_{t+2}$ and so on. To implement this some kind of loop construct will be necessary." CreationDate="2011-01-11T21:28:02.933" UserId="2116" />
  <row Id="9513" PostId="6176" Score="2" Text="@mpiktas Good suggestion, but bear in mind that the stated interest is *technique* rather than *theory*.  Recommendations at math.SE will likely favor the latter.  Moreover, you don't need to know measure theory (beyond absolute basics) to learn about NP Bayes methods, so the main focus here should be on intros to probability that focus on statistical applications." CreationDate="2011-01-11T21:28:44.447" UserId="919" />
  <row Id="9514" PostId="6175" Text="@mpiktas, Great I really appreciate the help!" CreationDate="2011-01-11T21:35:33.307" UserId="2704" />
  <row Id="9515" PostId="6174" Text="Ok...ignore that last message (and my first one!) (although I will leave them there in case anyone else has the same question).  My browser was not resolving the notation into its MathJax image." CreationDate="2011-01-11T21:50:55.207" UserId="2734" />
  <row Id="9516" PostId="6173" Score="1" Text="+1 For the distinction between bar charts and histograms.  (The difference becomes very clear when you consider how best to plot histograms with varying intervals, as illustrated at http://www.stata.com/support/faqs/graphics/histvary.html.)" CreationDate="2011-01-11T21:52:29.137" UserId="919" />
  <row Id="9517" PostId="6174" Text="@Mike thats okay. The programming language used by MathJax is $\LaTeX$" CreationDate="2011-01-12T00:20:26.517" UserId="1381" />
  <row Id="9518" PostId="6177" Text="@whuber thank you for the explanation and proper terminology / reference. The audience is readers of a scientific journal and the topic is variance decomposition; understanding the concept of a log transform is a pre-requisite but I still wasn't sure if this presentation required further justification - roots are a good alternative. Thanks." CreationDate="2011-01-12T00:30:34.880" UserId="1381" />
  <row Id="9519" PostId="6173" Text="I updated to reflect I want bar plots not histograms. " CreationDate="2011-01-12T00:58:36.020" UserId="559" />
  <row Id="9520" PostId="6182" Text="The traditional way you could do this is through hypothesis testing. This will allow you to test whether the plausibility that the lifetime is equal to some value, say 10,000 times.  http://en.wikipedia.org/wiki/Statistical_hypothesis_testing" CreationDate="2011-01-12T01:39:06.967" UserId="1118" />
  <row Id="9521" PostId="6166" Text="(+1) @mpiktas. Thanks for the answer." CreationDate="2011-01-12T01:46:29.483" UserId="1307" />
  <row Id="9522" PostId="6169" Text="@Elpezmuerto. Good question. I have been always confused about their uses. Thanks for asking." CreationDate="2011-01-12T02:11:16.280" UserId="1307" />
  <row Id="9523" PostId="6182" Score="2" Text="@tcrosley You have to make strong assumptions about the lifetimes.  Without them, you would need to test approximately 3/(0.1%) = 3000 devices.  For example, suppose 1% of the devices have a relatively rare defect that causes them to fail after 100 reprogrammings.  Your chance of encountering one of them during your testing of only four devices is approximately 4%: pretty low.  Thus, *based on the data alone,* you can have very little confidence that such defects don't exist in the population of microcontrollers." CreationDate="2011-01-12T02:35:24.480" UserId="919" />
  <row Id="9524" PostId="6181" Text="Isn't http://en.wikipedia.org/wiki/Multiple_correlation good enough?" CreationDate="2011-01-12T02:39:35.053" UserId="919" />
  <row Id="9525" PostId="6184" Score="1" Text="(+1) Nice answer.  This will become a reference for similar questions in the future, I think.  However, cross-posting is unusual; in fact, I believe it is frowned on, because it screws up many aspects of the feedback/referencing/threading/commenting system.  Please consider removing one of the copies and replacing it by a link in a comment." CreationDate="2011-01-12T02:49:48.307" UserId="919" />
  <row Id="9526" PostId="6184" Text="@whuber thanks for the feedback. I have removed the other copy." CreationDate="2011-01-12T02:53:43.990" UserId="1381" />
  <row Id="9528" PostId="6182" Text="@whuber - is my answer consistent with your point about assumptions? " CreationDate="2011-01-12T05:20:53.527" UserId="1381" />
  <row Id="9529" PostId="6188" Score="1" Text="another useful function is `any`. For instance if you want to check that at least one element of a vector is `= 10` you could write (`any(v==10)`)." CreationDate="2011-01-12T07:38:28.980" UserId="582" />
  <row Id="9530" PostId="6185" Text="+1 for the R code illustrating fast solution of the problem" CreationDate="2011-01-12T07:40:39.780" UserId="2116" />
  <row Id="9531" PostId="6188" Text="@nico yeah, but **10 %in% v** has 9 characters and variant with **any** 10 :). Although for numeric vectors it is better to use equality, since R is smart and if your data is integers mixed with real numbers it will correctly recognize that you have 10 in your data set." CreationDate="2011-01-12T07:42:35.283" UserId="2116" />
  <row Id="9532" PostId="6189" Text="what is the mean or median?" CreationDate="2011-01-12T08:04:06.537" UserId="2116" />
  <row Id="9533" PostId="6180" Text="Those statements sound fine to me." CreationDate="2011-01-12T08:25:19.317" UserId="449" />
  <row Id="9534" PostId="6191" Text="+1 for the link to the blog, first citation is a gem." CreationDate="2011-01-12T08:45:16.250" UserId="2116" />
  <row Id="9535" PostId="6188" Text="what about the case for regular expression? assuming that you do not want to work with an exact match? Do they have something close to ~= as in other languages? what is most close  to that expression in R" CreationDate="2011-01-12T08:56:35.180" UserId="201" />
  <row Id="9536" PostId="6128" Text="@caracal Thank you so much!" CreationDate="2011-01-12T09:28:59.017" UserId="442" />
  <row Id="9537" PostId="6192" Score="4" Text="@onestop This prompts me to think whether IQR should be an interval or a scalar. From Wikipedia, it is defined as a scalar and is consistent with what I have learnt." CreationDate="2011-01-12T09:37:29.040" UserId="2742" />
  <row Id="9538" PostId="6190" Text="@mbq . Thanks you for the mathematical explanations." CreationDate="2011-01-12T09:41:13.560" UserId="2742" />
  <row Id="9539" PostId="6192" Score="2" Text="I've always known that the IQR is the **difference** between the 3rd and the 1st quartile, hence a scalar." CreationDate="2011-01-12T10:50:40.063" UserId="582" />
  <row Id="9540" PostId="6161" Text="Thanks for the help! I will try to implement the code in Mathematica Environment." CreationDate="2011-01-12T10:58:04.053" UserId="1637" />
  <row Id="9541" PostId="6105" Text="@mpiktas thanks a lot for your answers!!!!" CreationDate="2011-01-12T12:24:13.280" UserId="1251" />
  <row Id="9542" PostId="6188" Text="@Biorelated See `grep` and `agrep`, and related documentation on POSIX 1003.2 or Perl-compatible regex." CreationDate="2011-01-12T12:26:40.907" UserId="930" />
  <row Id="9543" PostId="6188" Text="@Biorelated, I've update the answer with the example of grep." CreationDate="2011-01-12T12:37:09.860" UserId="2116" />
  <row Id="9544" PostId="6105" Score="1" Text="@Ama, your are welcome. At this site and similar stackexchange sites, there is a standard way of thanking for the answers. You simply accept the answer. Just press the check box outline to the left of the answer. I am bringing this up only because I noticed that you haven't accepted any of the answers you asked, so maybe this something you do not know. " CreationDate="2011-01-12T12:44:13.063" UserId="2116" />
  <row Id="9545" PostId="6192" Score="1" Text="Though IQR might be defined as a scalar, if it is reported as an interval it carries much more information. For example [-1,1] and [499,501] will have the same range, but implications are different." CreationDate="2011-01-12T13:23:36.243" UserId="2116" />
  <row Id="9546" PostId="6188" Text="Thanks a tonne on this!!!! the explanation and the references! awesome." CreationDate="2011-01-12T13:32:58.177" UserId="201" />
  <row Id="9547" PostId="6182" Score="1" Text="@David Yes it is.  Good work.  I was thinking along the lines of borrowing strength from other datasets or knowledge about failure modes and distributions derived from similar devices.  Such information could justify a Bayesian estimate, too." CreationDate="2011-01-12T15:19:49.223" UserId="919" />
  <row Id="9548" PostId="6185" Text="@David if $n=4$ is too small to justify most models, why is it enough to justify *any* model?" CreationDate="2011-01-12T15:21:01.967" UserId="919" />
  <row Id="9549" PostId="6191" Text="Despite the quip, it helps to know enough measure theory that you won't be afraid to read articles in JASA (or wherever) that could be useful or instructive.  If you're going to work in stochastic processes and mess about with Ito integrals and the like, and if you care to understand the tools you'll be using, then you actually do need a serious dose of measure theory." CreationDate="2011-01-12T15:37:09.770" UserId="919" />
  <row Id="9550" PostId="6193" Text="+1 for offering a classic and for the remark on brevity!" CreationDate="2011-01-12T15:37:52.327" UserId="919" />
  <row Id="9551" PostId="6185" Text="@whuber my point was that, given $n=4$, it might be most appropriate to fit a single parameter model. I can see your point though. Perhaps it would help to incorporate his prior knowledge into a gamma prior with $P(Y&lt;100,000)=0.5$ and $P(Y&lt;10,000)\simeq 0)$?" CreationDate="2011-01-12T15:57:46.283" UserId="1381" />
  <row Id="9552" PostId="6196" Text="Exactly how would you apply a multinomial logit model, whose dependent variable is categorical, to prices, which are not categorical?" CreationDate="2011-01-12T16:00:12.603" UserId="919" />
  <row Id="9553" PostId="6185" Text="@David I don't know enough about this particular situation to comment on any specific proposal.  The point I am making is that there is no magic in statistics that will generate &quot;99.9% certainty&quot; from a sample of four.  A single-parameter model might fit beautifully but anybody would be foolish to rely on a 99.9% CI or percentile estimated with it *unless* they had substantial, *independent* evidence that the model is appropriate." CreationDate="2011-01-12T16:04:21.947" UserId="919" />
  <row Id="9555" PostId="5973" Text="@user2040: Sorry for the late reply. I created my own software implementation of the nested CV approach. Because my research is related to bioinformatics, I'm planning to submit a description of the software soon to a bioinformatics journal. But it can be used in any research domain. If you're interested in trying it out, please let me know. goldfish1434 at yahoo dot com" CreationDate="2011-01-12T16:29:17.633" UserId="2643" />
  <row Id="9557" PostId="6185" Text="@whuber I also don't know much about failure analysis and never think about such small probabilities, but I know that tail probabilities are much more sensitive and difficult to estimate - is that the reason for your concern?" CreationDate="2011-01-12T16:31:54.347" UserId="1381" />
  <row Id="9558" PostId="6185" Text="@David I didn't plead ignorance concerning the *subject matter* :-)  I just don't know what additional information the OP has, so I have no idea whether your gamma prior is reasonable.  (Choice of prior is a big deal with this small sample.)  The concern is only indirectly about high uncertainties attached to tail probabilities.  More fundamentally it's about the model itself: if you use a bad model for the failure distribution, you will likely get bad (or awful) estimates.  How, then, can you tell whether the model is appropriate?  Not by testing just four units!" CreationDate="2011-01-12T16:36:30.997" UserId="919" />
  <row Id="9559" PostId="6182" Score="1" Text="It may be there is a significant variation in chips from different wafers (batches), but low variation within a batch.  This means that if the four chips are all from the same batch, it may give a misleading view of the lifetime of a chip selected at random from the whole production run.  It may be the low lifetime quoted is because there is occasionally a bad batch and it is easier to quote a low life expectancy than to detect the bad batches.  If your chips are all from the same batch, it may not be possible to estimate the true population lifetime." CreationDate="2011-01-12T18:28:10.677" UserId="887" />
  <row Id="9560" PostId="6185" Text="@David, thank you for your answer.  We do know that the failure will be due to the Flash memory &quot;wearing out&quot;.  Any devices failing less than the minimum can be ignored and treated as burn-in failures.  I had one of the four fail with less than 100 events, but it was a &quot;soft&quot; failure and I restarted the test.  I am only interested in hard failures where retries are not possible.  Right now the counters on the four devices are approximately 721,000, 166,000, 166,000 and 172,000 so none have failed yet. (continued)" CreationDate="2011-01-12T18:32:05.620" UserId="2698" />
  <row Id="9561" PostId="6185" Text="(continued from above)  I picked the number of 99.9% somewhat arbitrarily -- our current overall return rate for the product in the field after several months is between one and two percent, and I was just wanting to make sure this new process didn't contribute significantly to this return rate.  If I wanted to use your formula in R for other percentages (say 99.5, 99 and 98) I would just subsitute 0.005, 0.01 and 0.02 in the qexp line?" CreationDate="2011-01-12T18:37:18.007" UserId="2698" />
  <row Id="9562" PostId="6185" Text="@David, I have installed R version 2.12.1.  Since you are using the MASS library, would &quot;Modern Applied Statistics with S&quot; be worthwhile for me to get (I notice there is an edition available for my Kindle DX), or would it be over my head since I have never taken a stats course?  Or do you recommend other book(s)?  (I do have an MS in computer science.)  Thanks." CreationDate="2011-01-12T19:35:01.067" UserId="2698" />
  <row Id="9563" PostId="6185" Text="@tcrosley yes, that is correct; MASS is one of the best books for learning R - I don't think that it would be over your head. I used MASS for the `fitdistr`, which provides a maximum likelihood estimate (MLE) of a distribution's parameters given data. So, if you are interested in the theory, use `?fitdistr` and read up on MLE. But you might want to consider comparing the fits of different distributions, perhaps contacting the manufacturer for additional data that could be used in a hierarchical model." CreationDate="2011-01-12T19:48:29.313" UserId="1381" />
  <row Id="9564" PostId="6198" Text="@whuber: Sold! That sounds exactly what I'm looking for! As I'm all new to this I'm having trouble comparing suggestions, so I'm wondering how would least squares regression compare to multiple-regression and &quot;hedonic pricing&quot;. These are suggestions I got in the mathematics site where I initially posted. What am I fixing when using least squares regression for instance? Basically, is there something I need to be aware of when using this approach?" CreationDate="2011-01-12T19:49:04.243" UserId="2746" />
  <row Id="9565" PostId="6196" Text="@Dmitrij Celov: Thanks for your suggestion. I'll try to answer your questions. 1) No price is available, this is the unknown which I'd like to answer by looking at similar cars. 2) I don't know which variable is weighing the most - this I was hoping to get. 3) I would like to based on a list of cars with features and prices be able to price any car with any features." CreationDate="2011-01-12T19:54:14.860" UserId="2746" />
  <row Id="9566" PostId="6196" Text="@whuber: The&quot;trick&quot;with categorical attributes is to introduce the dummy variables that correspond to $K_j - 1$ level of the $j$-th attribute. The &quot;not categorical&quot; prices are introduced as they are, so it is an ordinary independent variable in (multinomial) logit model. Whuber, now I doubt that we need multinomial here, probably it is simple binary dependent variable with $1$ if chosen and else $0$. You do compare with rival collection of attributes, so some kind of differences between estimates like $P(y_i = 1| y_j = 0) = \frac{1}{1 + e^{-\beta^\prime (X_i-X_j)}}$ comparing $y_i$ and $y_j$. " CreationDate="2011-01-12T19:54:58.090" UserId="2645" />
  <row Id="9567" PostId="6198" Text="@whuber: also thanks for this suggestion. It seems very good. I'll have to read up more to get an idea how I can get started to see how to use it." CreationDate="2011-01-12T19:55:02.537" UserId="2746" />
  <row Id="9568" PostId="6196" Text="@murrekatt: 1) So you just look for the most &quot;valuable&quot; attributes? 2) Logit estimated parameters are nicely interpreted like odds and odds ratios, but multinomial logit has a weak feature known as **independence from irrelevant alternatives** 3) Can you be sure that the listed prices are relevant, i.e. that the cars were actually purchased? @whuber: simple regression works here fine, if the dependent is price, but again what price? published where? or is it the actuall transaction?" CreationDate="2011-01-12T20:05:28.357" UserId="2645" />
  <row Id="9569" PostId="6201" Text="@mpiktas: thanks. I understand what you mean. This was something I was thinking about, but didn't know exactly how to ask or add to the question. How does one deal with what you explain? Is this a problem which is separate and as you write to be taken into account when interpreting results, or is this integrated in some other approaches and not part of least squares regression? Not sure how to formulate myself, but what I mean is that are there approaches which take this into account and others which don't? Which means that for the &quot;don't&quot; we must interpret results?" CreationDate="2011-01-12T20:06:45.360" UserId="2746" />
  <row Id="9570" PostId="6082" Text="Hi Wolfgang - thanks!  Actually, I think the intercept gives you an R^2 of 0, doesn't it?  But you are right about the overlapping variance, that's why you need do each alone; because each would add a different amount to a null (intercept only) model, and to a model with the other variable in there (unless they are exactly orthogonal, which seems very unlikely)" CreationDate="2011-01-12T20:10:00.153" UserId="686" />
  <row Id="9571" PostId="6196" Text="@Dmitrij Celov: 1) I don't know how to answer this. Maybe I am, but then it's a side-product when pricing a given car. Maybe after analyzing the car list with prices some features are not really important in pricing the car (e.g. color maybe). 3) Yes, they are and yes those prices are real deals. Still back in time, so if e.g. a list if 1 year old, all cars on the list are now one year older too, which will decrease the price. Not sure how to take this into account." CreationDate="2011-01-12T20:11:16.697" UserId="2746" />
  <row Id="9572" PostId="6201" Score="3" Text="@murekatt, if you do not have additional data on demand, but you need the model for price, you deal with this by taking extra care. This means less attention to statistical significance of coefficients, but more attention to forecasting performance. Essentialy this means treating regression as black-box and use the model forecasting performance as measure of model validity. This means using cross-validation, data division to train and test samples, etc." CreationDate="2011-01-12T20:19:42.500" UserId="2116" />
  <row Id="9573" PostId="6202" Score="1" Text="$R^2$ can also be negative. This happens when other methods are used to calculate the coefficients in linear regression model specification, and the usual $R^2$ formula is used. I think $R^2$ can also get negative if regression is without the intercept." CreationDate="2011-01-12T20:23:52.543" UserId="2116" />
  <row Id="9575" PostId="6196" Text="@Dmitrij Celov: I'll try to answer your added questions. By the way, I didn't -1 your answer. I'll start with 2) The prices in the list are actual deals and no price if no deal for a car. All prices are deals. If we have to know who set it, let's say seller&amp;buyer agreed. 1) Not sure here how to answer other than let's imagine we want as a seller to generate prices across all cars somehow. Or let's imagine as a buyer you'd like to generate prices across all available cars." CreationDate="2011-01-12T20:34:54.950" UserId="2746" />
  <row Id="9576" PostId="6196" Text="@murrekatt: than I give up, simple regression actually all that you need here, but if you observe the &quot;real deals&quot; and some deals that were not purchased, that logit analysis is still possible. To your reply, it is the buyers position, or the demand side that limits you here, so if you would like to discriminate the prices some how you need to know the features consumers value most, may be you still observe some suggestions that have not passed to real deals? By the way how to model &quot;hedonic&quot; prices is a very interesting question :)" CreationDate="2011-01-12T20:35:04.280" UserId="2645" />
  <row Id="9577" PostId="6201" Text="@mpiktas: what do you mean with &quot;additional data&quot;? Could you please give an example of this in the car context?" CreationDate="2011-01-12T20:37:28.957" UserId="2746" />
  <row Id="9578" PostId="6201" Score="1" Text="@murrekatt, look at the end of updated Dmitrij's answer. The demand data is important, so if you have how much cars were sold with given price this would help tremendously. Furthemore if you have data of how price changes for the given car with fixed attributes this also should be reflected in your model" CreationDate="2011-01-12T20:47:20.603" UserId="2116" />
  <row Id="9579" PostId="6196" Score="1" Text="@Dimitrij Price is not an independent variable: it's the *dependent* variable: &quot;I would like to understand how to model prices for any car based on this base information.&quot;  I fear that with this misapprehension you may be taking @murrekatt very far afield." CreationDate="2011-01-12T21:01:19.203" UserId="919" />
  <row Id="9580" PostId="6196" Text="Applicability of the logit model aside, I recommend looking into [Biogeme](http://biogeme.epfl.ch/) for a dedicated software package that allows extreme flexibility in estimating a variety of GEV models...and it's free." CreationDate="2011-01-12T22:57:01.023" UserId="696" />
  <row Id="9581" PostId="6144" Text="Thanks, It works." CreationDate="2011-01-13T00:34:52.623" UserId="2725" />
  <row Id="9582" PostId="6207" Text="I hope I am not old fashioned if I use lattice :-)" CreationDate="2011-01-13T02:47:52.010" UserId="1307" />
  <row Id="9583" PostId="6207" Score="2" Text="I also hope that if this is a HW problem, you will not simply copy paste." CreationDate="2011-01-13T02:54:59.210" UserId="1307" />
  <row Id="9584" PostId="6161" Text="I could run the code in Mathematica Environment, but since I'm not familiar with this topic, can you tell me how can I plot the impulse response of this filter? I just could plot the amplitude response." CreationDate="2011-01-13T03:06:47.253" UserId="1637" />
  <row Id="9585" PostId="6207" Text="Thanks. This is not a HW question and the answer is helpful for me to understand my model.   " CreationDate="2011-01-13T04:25:39.350" UserId="2755" />
  <row Id="9586" PostId="6203" Text="@F. Tusell: that was a good description. I already puzzled this together from other posts, but this summarized things well for a beginner like me." CreationDate="2011-01-13T07:00:20.240" UserId="2746" />
  <row Id="9587" PostId="6201" Text="@mpiktas: I see. Price change of a car I suppose can be gathered by looking at historical data somehow. I have been thinking of this too, but not knowing how to utilize it. Maybe this is something that can be added later on?" CreationDate="2011-01-13T07:03:28.927" UserId="2746" />
  <row Id="9588" PostId="6201" Score="1" Text="@murekatt, in principle yes. I think you need to start small and add additional features later. The initial results will tell you what direction to take further." CreationDate="2011-01-13T07:52:51.377" UserId="2116" />
  <row Id="9589" PostId="6204" Text="@Jana could you please provide code example illustrating your problem? In my experience the defaults of ggplot2 are really good, so there should be a good reason to change them. " CreationDate="2011-01-13T07:58:33.130" UserId="2116" />
  <row Id="9590" PostId="6201" Text="@mpiktas: very good! thanks for all your help!" CreationDate="2011-01-13T08:05:26.273" UserId="2746" />
  <row Id="9591" PostId="6207" Text="@suncoolsu, oh yes you are :)" CreationDate="2011-01-13T08:09:30.527" UserId="2116" />
  <row Id="9592" PostId="6196" Text="@whuber @murrekatt I was too quick to post the answer, after some sort of discussion I completely remade the suggested solution. Note the threats with least square regression I mentioned. Thanks for a good lesson on how to write the answers here, I'm still a newbie :) And, all attributes are interdependent, including the price, in our discussion I simply wanted to show that the fact of willingness to pay for the attributes comparing to not bought could not be ignored here." CreationDate="2011-01-13T09:36:14.553" UserId="2645" />
  <row Id="9593" PostId="6208" Text="Shortly - yes.  Simply to allow consistency with other packages (even if the result was &quot;wrong&quot; in some way - in which case a warning massage would be fine).  And thank you for your amazing package!" CreationDate="2011-01-13T11:08:45.243" UserId="253" />
  <row Id="9594" PostId="6212" Score="1" Text="To add one further requirement: It would be totally awesome if the function would also be able to run all kinds of contrasts from this ANOVA, for example, using the `multcomp` package (which, as far as I remeber, requires `aov` objects). Otherwise, I agree totally and normally use `ez` for my daily duty, because it is so easy..." CreationDate="2011-01-13T12:06:38.623" UserId="442" />
  <row Id="9598" PostId="6216" Text="This is pretty clear, but the question was suggesting a restriction between b0 and b1. Should I also create a new variable Z = 2X + 1 and fit a model without intercept?" CreationDate="2011-01-13T13:04:54.677" UserId="339" />
  <row Id="9599" PostId="6216" Text="@Dmitrij, I think usualy **I** is used instead of eval in formulas,  i.e. Y~I(1+2*X1)+X2+X3-1" CreationDate="2011-01-13T13:21:04.807" UserId="2116" />
  <row Id="9600" PostId="6216" Text="@gd047: I have updated with a code pieces, yes it is as you say. @mpiktas: will change this, yes it is shorter ;)" CreationDate="2011-01-13T13:21:29.613" UserId="2645" />
  <row Id="9602" PostId="6212" Score="1" Text="@caracal: adding a &quot;method&quot; argument (with possible values as &quot;oneway&quot;,&quot;aov&quot;, and &quot;Anova&quot;, and with a &quot;...&quot; argument to pass var.eq=FALSE to oneway and type=2 to Anova) is an intriguing idea, and it certainly seems that it would be easy enough to incorporate. Thanks for the suggestion." CreationDate="2011-01-13T13:32:16.473" UserId="364" />
  <row Id="9604" PostId="6212" Text="@Henrik: I have to be honest that my understanding of contrasts is very limited, and I never use them in my research, so I haven't prioritized incorporation of contrasts into ez." CreationDate="2011-01-13T13:36:57.420" UserId="364" />
  <row Id="9605" PostId="6211" Text="Ah, I hadn't realized that the type II/III distinction arises whenever predictors are correlated, of which an unbalanced design is only one example. This certainly further undermines my first &quot;just collect more data&quot; argument." CreationDate="2011-01-13T13:46:25.043" UserId="364" />
  <row Id="9606" PostId="6211" Score="1" Text="I'm rather convinced by your argument that R will gain more SPSS converts if they are provided with easy means of achieving familiar analyses, even if these analyses require more nuanced consideration than is typically applied. Then, once we have them hooked on R, we can attempt to convince them that the familiar approach isn't necessarily always the appropriate approach. I think I'll still put the default as type=2 with a big warning in the documentation that this default may yield results that differ from SPSS and links do documentation on the type II/III distinction." CreationDate="2011-01-13T13:47:30.627" UserId="364" />
  <row Id="9607" PostId="6212" Text="@Mike If you can add the method argument and so your function would be able to return an `aov` element that would be great. So far I am usually to lazy to make my own `aov` element and use you `ezANOVA` instead with t.tests..." CreationDate="2011-01-13T13:49:57.893" UserId="442" />
  <row Id="9609" PostId="1527" Text="I agree with Robin, Anyhow, even if it is not usual to say that in a given set of observations the probability of success is 0.3, it is common to use the word proportion as a synonym of probability: search google for: binomial and &quot;proportion p of success&quot;" CreationDate="2011-01-13T13:56:55.287" UserId="1219" />
  <row Id="9610" PostId="6212" Text="@Mike I hadn't dared suggesting something like that since it's your work you're putting in. Thanks for that! The `ez` package has some great potential: in psychology, it's already recommended quite often (cf. German book &quot;R für Einsteiger&quot; by Luhmann). Making it even more flexible would certainly be appreciated." CreationDate="2011-01-13T14:01:05.763" UserId="1909" />
  <row Id="9612" PostId="6211" Text="Other variations on correlated predictors are usually with continuous predictors, but yes, that's the general problem. // That sounds great to me! Sometimes these shifts are slow; the use of sphericity corrections (rather than ignoring the problem) is one that I think progress has been made on in neuroscience. There were times when any mention of it (particularly if you reported the corrected, non-integer df) made referees think you were barking." CreationDate="2011-01-13T14:42:12.157" UserId="2761" />
  <row Id="9613" PostId="5791" Score="1" Text="If you want to measure the constrain-ness of a parameter, what about to use st.dev(posterior)? If you want to compare constrain-ness of different parameters, what about to use st.dev.(posterior)/st.dev(prior)?   " CreationDate="2011-01-13T14:50:11.210" UserId="1219" />
  <row Id="9614" PostId="6198" Score="2" Text="I want to acknowledge and express my agreement with the caveats posted by @mpiktas and @dimitrij celov.  Analyses of prices can be--and in many cases *should* be--as complex as the economic systems of which they are a part.  However, due to the intended application (a hobby) and the plainly signaled limitations in the OP's capabilities for statistical modeling, we should place great value on simplicity, ease of use, and interpretability.  Obviously someone not yet conversant with least squares is not going to jump right in and start creating full-blown econometric models." CreationDate="2011-01-13T15:57:02.667" UserId="919" />
  <row Id="9615" PostId="6221" Score="1" Text="Correct except for step 3: there are **two** parameters fitted, so you divide by $n-2 = 1$. (Check it out: do the math!)" CreationDate="2011-01-13T16:02:34.410" UserId="919" />
  <row Id="9616" PostId="6220" Text="Not correct: you need to use $n-2$, not $n$." CreationDate="2011-01-13T16:03:38.700" UserId="919" />
  <row Id="9617" PostId="6213" Score="1" Text="Examples of better plots?" CreationDate="2011-01-13T16:06:31.470" UserId="559" />
  <row Id="9618" PostId="6215" Text="Could you explain the distinction you are making between &quot;observed&quot; and &quot;declared&quot; level?  (I also suspect a good journalist would be bothered by the implication that you are &quot;discarding&quot; values and would follow up with some difficult questions :-).)" CreationDate="2011-01-13T16:08:00.910" UserId="919" />
  <row Id="9620" PostId="6221" Text="Thanks - I've correct it. I did do the math (and even used R to avoid mistakes). Unfortunately, that still didn't stop me typing n-1 :(" CreationDate="2011-01-13T16:43:55.217" UserId="8" />
  <row Id="9621" PostId="6220" Text="good point, fixed." CreationDate="2011-01-13T16:54:13.177" UserId="449" />
  <row Id="9622" PostId="6225" Score="3" Text="It depends on what you mean by &quot;prove.&quot;  As stated, this is a philosophical question, not a statistical one, and has no definitive answer (although, at least since David Hume's time, most people would answer &quot;no&quot;)." CreationDate="2011-01-13T16:54:16.800" UserId="919" />
  <row Id="9623" PostId="6224" Text="Please delete your previous version of this question." CreationDate="2011-01-13T16:56:04.830" UserId="919" />
  <row Id="9624" PostId="6226" Score="3" Text="NA is not the same thing as NaN. See http://stats.stackexchange.com/questions/5686/what-is-the-difference-between-nan-and-na." CreationDate="2011-01-13T16:56:51.443" UserId="449" />
  <row Id="9625" PostId="6221" Text="I sympathize: I make typos all the time even when the math is correct." CreationDate="2011-01-13T16:57:35.877" UserId="919" />
  <row Id="9626" PostId="6219" Text="Surprisingly, Wikipedia appears to have no articles related to this subject that explicitly and clearly show the correct formula in this least squares context!  (The formula is buried in articles on analysis of variance and least squares.)" CreationDate="2011-01-13T17:04:57.100" UserId="919" />
  <row Id="9627" PostId="6226" Text="Why are you doing &quot;data preparation&quot; in openoffice? Do you really have to? I try and load things into R as soon as possible and make all changes within R itself." CreationDate="2011-01-13T17:44:26.507" UserId="8" />
  <row Id="9628" PostId="6202" Text="@mpiktas: Is $R^2$ negative because of a poor approximation? You are estimating $R^2$ with error." CreationDate="2011-01-13T17:51:06.907" UserId="8" />
  <row Id="9630" PostId="6224" Text="I do not find the option to remove the previous (incomplete) post, could you tell me how?" CreationDate="2011-01-13T18:08:43.407" UserId="2486" />
  <row Id="9631" PostId="6226" Score="4" Text="If you save a file as csv and import it to R, blank cells will be represented as NA." CreationDate="2011-01-13T18:20:55.773" UserId="2116" />
  <row Id="9632" PostId="6226" Text="@csgillespie I find it convenient to edit some data in a spreadsheet UI as opposed to a text editor UI." CreationDate="2011-01-13T18:24:17.120" UserId="660" />
  <row Id="9633" PostId="6226" Text="@onestop Thanks for pointing out that NA is different from NaN. I edited the question accordingly. " CreationDate="2011-01-13T18:24:46.527" UserId="660" />
  <row Id="9634" PostId="6226" Text="@mpiktas Thanks, I think your response is useful as an &quot;answer&quot; (not just a comment) -- want to submit it?" CreationDate="2011-01-13T18:26:34.077" UserId="660" />
  <row Id="9635" PostId="6229" Text="@robin +1, your answer made mine redundant" CreationDate="2011-01-13T19:02:24.613" UserId="2116" />
  <row Id="9636" PostId="6219" Score="2" Text="I think you are all confused, rms stands for your favorite GNU advocate, [Richard Matthew Stallman](http://en.wikipedia.org/wiki/Richard_Stallman)" CreationDate="2011-01-13T19:14:49.907" UserId="696" />
  <row Id="9637" PostId="6224" Text="On the left below the text of the question you should find a list of options: &quot;link edit delete flag&quot;." CreationDate="2011-01-13T19:27:55.293" UserId="919" />
  <row Id="9638" PostId="6224" Text="@whuber I deleted the other question." CreationDate="2011-01-13T19:29:00.327" UserId="5" />
  <row Id="9639" PostId="6219" Score="1" Text="@Chase Good point; we shouldn't rely on acronyms: see http://stats.stackexchange.com/q/6039/919 .  However, &quot;Royal Meteorological Society&quot; gets more Google hits than &quot;Richard M Stallman&quot; :-)." CreationDate="2011-01-13T19:34:53.650" UserId="919" />
  <row Id="9640" PostId="6114" Text="Can @caracal's code be generalized to inlude multiple choice questions. i can vary but it will be nice to give a line like # p &lt;- c(0.4, 0.4, 0.4, 0.4, 0.1, 0.1, 0.1) but instead of the probability the user enters the number of choices, assuming equal likelihood." CreationDate="2011-01-13T19:38:14.240" UserId="2715" />
  <row Id="9641" PostId="6226" Text="@David, ok, I've posted it as an answer and included additional comments. Hope it helps." CreationDate="2011-01-13T19:39:32.537" UserId="2116" />
  <row Id="9642" PostId="6075" Text="@vqv: great work! I love that this question, which could be posed as one sentence (to those with sufficient background), has brought out monte carlo, inclusion-exclusion, DAGs, searching trees, polynomial algebra, and that your simulations are confirmed by the theoretical of @whuber. cheers!" CreationDate="2011-01-13T19:49:03.020" UserId="795" />
  <row Id="9643" PostId="6218" Text="Could you please state your initial problem at what you are trying to achieve. Also please explain what do you mean by graphing interactions, graphing the different regression lines, plotting the coefficient values?" CreationDate="2011-01-13T19:58:25.877" UserId="2116" />
  <row Id="9644" PostId="6208" Text="When I wrote &quot;packages&quot; I meant other statistical packages (like SPSS and SAS) - I'm glad others have driven the point further home then I did :) " CreationDate="2011-01-13T20:00:22.863" UserId="253" />
  <row Id="9645" PostId="6114" Text="@Rado If a question has $k$ choices with probabilities $p_{j}$, then the probability for pairwise agreement on a single such question is $\sum_{j=1}^{k}{p_{j}^{2}}$. You should only need to change the `getP()` function to reflect this, specifically the lines starting with `pp &lt;- ...` and `qq &lt;- ...`." CreationDate="2011-01-13T20:50:30.947" UserId="1909" />
  <row Id="9646" PostId="6232" Text="I'm having trouble wrapping my head around your design... At what level was the randomization done? At the standtype*treatment level? How many standtype*treatment level 'units' are there? 4×3×6=72? And then 72×6=432 plots?" CreationDate="2011-01-13T20:56:22.890" UserId="449" />
  <row Id="9647" PostId="6224" Text="thank Wuber and Shane" CreationDate="2011-01-13T21:13:17.080" UserId="2486" />
  <row Id="9648" PostId="6131" Text="@Andy Interesting suggestion: some mechanism for specifying that you want to shade the code block without highlighting?  I'd say it's unlikely since SO uses an open source code highlighting utility (don't remember the name).  Maybe open a meta ticket to suggest it?" CreationDate="2011-01-13T21:25:02.550" UserId="5" />
  <row Id="9649" PostId="6229" Score="1" Text="+1 Nice answer.  A simple rendering of the math is that the null and its alternatives are assumed to yield disjoint sets of outcomes; *e.g.,* either there is a zebra in this room or there isn't.  Of course &quot;prove&quot; here implicitly includes &quot;conditional on the model,&quot; which itself is never established with the same rigor as, say, a mathematical theorem; it implicitly includes &quot;conditional on the accuracy of the observations;&quot; and it implicitly includes that the hypotheses can be unambiguously interpreted.  (For criticism of the latter, see George Lakoff's *Women, Fire, and Dangerous Things.*)" CreationDate="2011-01-13T21:29:40.460" UserId="919" />
  <row Id="9650" PostId="6173" Text="Grouped bar chart is always preferable to stacked bar chart, because the latter makes the comparison of different series more difficult than the former. A brief discussion about different versions of stacked bar and grouped bar plot (marimekko chart, weighted grouped bar plot, etcetera) with some links can be found here: http://www.antoniorinaldi.it/data-visualization-for-two-ways-tables-1" CreationDate="2011-01-13T22:02:21.267" UserId="1219" />
  <row Id="9651" PostId="5986" Text="I've asked one of my professors and he said that he would use Bayes estimator: $p = \frac{k+1}{n+2}\cdot \frac{\beta_{k+2,n-k+1}(0.25)}{\beta_{k+1,n-k+1}(0.25)}$ , where $\beta$ denotes CDF of beta distribution and $p=\alpha-\alpha^2$. p is less than 0.25 so it can be easily transformed to $\alpha$. I havent checked those calculations yet, but several simulations, that I've made, have shown that for small n this is far from being great." CreationDate="2011-01-13T22:04:16.627" UserId="1643" />
  <row Id="9652" PostId="6192" Text="(IMHO) IQR is a scalar: it is the lenght of an interval. Obviously, since it is always possible to pass from the second (lying in a subset of R^2) to the first (lying in R+) but not viceversa, the latter is more informative than the former. " CreationDate="2011-01-13T22:05:50.330" UserId="1219" />
  <row Id="9653" PostId="5986" Text="@Tomek I thought of those Bayes estimators (with conjugate priors) when writing the original answer, and worked out their characteristics, but found them wanting.  The approach is a good one for estimating $\alpha$ but doesn't overcome the difficulties of estimating $\alpha - \alpha^2$.  (One problem is that the prior your professor is implicitly using just doesn't narrow the likely range of $\alpha$ sufficiently.)  But I still think, as I indicated, that this is a fruitful line of investigation." CreationDate="2011-01-13T22:07:02.280" UserId="919" />
  <row Id="9654" PostId="6235" Text="(+1) Nice answer! - I especially like the graphics. I was wondering if there is a way to do this in R?" CreationDate="2011-01-13T22:37:20.210" UserId="1307" />
  <row Id="9655" PostId="6232" Text="In addition to onestops's remark - I would also be interested to know - what was the aim of the experiment, what was the response variables, and any other specifics that are important in the data analysis. I am guess this to be a split-plot-type design, but we can help you better if we get more details." CreationDate="2011-01-13T22:42:52.437" UserId="1307" />
  <row Id="9656" PostId="4850" Text="Err, maybe I'm missing something but in some important cases, e.g. all of survey research, probabilities are not at all hypothetical, they're just population proportions.  In the question 'how many Ukrainians think X' the population is pretty clear - all Ukrainians - and the proportion that thinks X from a simple random sample estimates the proportion of the population that thinks X, which is exactly the probability of interest.  For frequentists, this is the easy case (and I, as a non-frequentist would concur with their analysis)." CreationDate="2011-01-13T22:46:41.977" UserId="1739" />
  <row Id="9657" PostId="4850" Text="@Conjugate In some cases a probability may *equal* a proportion but it *is not* a proportion.  What relates a proportion to a probability is the specific procedure of sampling uniformly at random with replacement from a well-defined population (which are rare, by the way: 20 Ukrainians have been born since you wrote your comment!).  This clearly is a special case of other sampling methods, including without replacement, with stratification, etc.  In those other cases the proportions no longer even equal the probabilities.  Doesn't this suffice to show the two concepts are distinct?" CreationDate="2011-01-13T23:05:18.250" UserId="919" />
  <row Id="9658" PostId="6238" Score="1" Text="The issue here lies in the distinction between *actual* coverage and *nominal* coverage.  You seem to assume the two are equal, but for most discrete distributions they are not.  Worse, the actual coverage depends on the proportion one is trying to estimate!  Before considering the question further, you might want to research the two methods @Michael Lew mentions." CreationDate="2011-01-13T23:08:06.610" UserId="919" />
  <row Id="9659" PostId="6237" Text="+1 A valiant, tactful, and informative attempt to respond to an unanswerable question!" CreationDate="2011-01-13T23:09:44.783" UserId="919" />
  <row Id="9660" PostId="4850" Text="On the 'special case of other sampling methods', I agree. The whole trick of survey research is to make one's stratified, clustered, etc. sample proportions match (in expectation at least) the population proportion one is interested in.  But I'm not sure that's a criticism, rather a statement of the problem." CreationDate="2011-01-13T23:56:24.247" UserId="1739" />
  <row Id="9661" PostId="4850" Text="Another tack: whenever a frequentist states a coverage probability she immediately asserts the existence of a population. In some situations the population is indeed infinite, e.g. infinite hypothetical replications of an experiment that takes place once. But in others, e.g. the survey context, the population is not infinite at all.  In my example it's the population of the Ukraine (at t). That's definitely finite and describable using proportions. I'm not disagreeing with your analysis - just pointing out that there is a simple situation where proportions and probabilities coincide." CreationDate="2011-01-14T00:03:48.340" UserId="1739" />
  <row Id="9662" PostId="6241" Text="Great! Short and sweet. Thanks." CreationDate="2011-01-14T00:17:11.880" UserId="660" />
  <row Id="9663" PostId="4850" Text="@Conjugate I am making a conceptual distinction here, so &quot;coincidence&quot; (mathematical equality), although noteworthy, is beside the point. In many cases, a &quot;population&quot; is a convenient fiction: one can find many references to *processes* as &quot;populations,&quot; for instance. In ideal cases we can indeed construct a rigorous data frame. But that solves only one of the difficulties.  Whatever we're interested in has to be measured and measurements are subject to error. (Even gender!) The ontological status of a &quot;population&quot; therefore is questionable: like probability, it too is a modeled construct." CreationDate="2011-01-14T00:44:12.473" UserId="919" />
  <row Id="9664" PostId="6240" Text="+1.  This is a nice example of the importance of being clear about one's standard of &quot;proof.&quot;  In many applications the one you invoke here--the &quot;act as if&quot; standard, if I may call it that--is so weak that nobody would accept it as &quot;proof.&quot;  I do not deny its utility, though, and advocate this kind of approach to support rational decision making.  (But maybe Bayesian methods are better... :-)" CreationDate="2011-01-14T00:47:33.357" UserId="919" />
  <row Id="9665" PostId="6220" Text="Can you confirm this thinking:  with n = 3, if order is 2, then k is 3.  So then n - k is zero, by which we don't want to divide. Is my understanding correct that if n = order + 1, there is always an exact fit, so calculating RMSE isn't even necessary?" CreationDate="2011-01-14T02:20:25.313" UserId="2767" />
  <row Id="9666" PostId="6235" Score="1" Text="I'm not aware of any way to do it; my webvis package provides a wrapper for Protovis, but it would be a lot of work to get it to make this graphic.  Incidentally, this paper introduces the &quot;arc diagram&quot; which is related: http://ieg.ifs.tuwien.ac.at/~aigner/teaching/ws06/infovis_ue/papers/arcdiagram_01173155.pdf" CreationDate="2011-01-14T04:10:18.577" UserId="5" />
  <row Id="9667" PostId="6112" Text="You're right. I just wanted to give an answer based on undergraduate probability and give where it comes from. Having the formula also gives you insight as to how the value changes for different i's and p's." CreationDate="2011-01-14T05:28:26.220" UserId="1540" />
  <row Id="9668" PostId="6241" Score="1" Text="Also note that window(qs,2010, c(2010, 4)) &lt;- 3 will change the qs accordingly." CreationDate="2011-01-14T06:34:27.537" UserId="2116" />
  <row Id="9669" PostId="6242" Score="1" Text="Please ask a new question" CreationDate="2011-01-14T07:18:10.627" UserId="2116" />
  <row Id="9670" PostId="6100" Text="@mbq I ended up computing the difference between means as the calibrated mean -- http://stattrek.com/AP-Statistics-4/Difference-Means.aspx" CreationDate="2011-01-14T07:46:47.717" UserId="2634" />
  <row Id="9671" PostId="6100" Text="@John-David MOE=SeM=deviation of mean=deviation of sample/$\sqrt{\text{size of sample}}$ -- that's why I used the word &quot;error&quot;. So, my result is identical to the magic formula on that site -- they just get deviation of sample as input, which is quite confusing. " CreationDate="2011-01-14T08:31:40.160" UserId="88" />
  <row Id="9672" PostId="4850" Text="@whuber Ahh, it's population as an idea you're not happy with. Fair enough.  (on the other hand it seems hard to talk about 'measurement subject to error' without that concept :-) " CreationDate="2011-01-14T08:35:08.237" UserId="1739" />
  <row Id="9673" PostId="5950" Text="@shabbychef: Why is a flatter (&quot;less vertical&quot; in 2D space) hypothesis the simpler one? Why is the $\theta$ with smaller components any simpler than one with larger components? For my example, what about training on (0,0) and (1,9) instead, then? I don't see why $\theta=8$ is any simpler than $\theta=10$, and similarly for more dimensions...." CreationDate="2011-01-14T08:39:12.630" UserId="1720" />
  <row Id="9674" PostId="6138" Text="@vqv: Re: &quot;I don’t think it is really a classification problem. 20 questions is often characterized as a compression problem.&quot; Aren't statistical inference and information compression directly related / the same problem?" CreationDate="2011-01-14T08:42:26.043" UserId="1720" />
  <row Id="9675" PostId="6240" Text="@dmk38 (+1) Nice answer. I added a link to an online version of Streiner's article; I hope you don't mind (feel free to remove)." CreationDate="2011-01-14T09:18:59.577" UserId="930" />
  <row Id="9676" PostId="6220" Text="Yes. When n=k, the model becomes an exact fit to the data, i.e. the fitted values are the same as the observed values, so you'd get 0/0 in the above formula, so the result is undefined. But logically, as the 'errors' are all zero, you'd expect the root mean square error to be zero too, and that's what the stats packages i've tried report." CreationDate="2011-01-14T10:18:59.637" UserId="449" />
  <row Id="9677" PostId="6243" Text="It may help if you explain a bit more about *how* you use the difference between means to calibrate the mean." CreationDate="2011-01-14T10:29:02.463" UserId="449" />
  <row Id="9678" PostId="6234" Text="Related, but for small number of sets (for reference): http://stats.stackexchange.com/questions/4211/how-to-visualize-3d-contingency-matrix" CreationDate="2011-01-14T12:51:06.377" UserId="88" />
  <row Id="9679" PostId="4850" Text="@Conjugate Measurement refers to individuals, not a population.  But that's taking us off on a tangent (for which I apologize): the question concerns distinguishing probabilities from proportions.  I hope we have succeeded in illuminating some of the differences." CreationDate="2011-01-14T13:04:28.020" UserId="919" />
  <row Id="9680" PostId="6235" Score="1" Text="@suncoolsu , the R package diagram may be able to do the same &quot;arc diagram&quot; Shane pointed to. It looks like it would be hard work though to get the &quot;plot web&quot; to look like the visual above though. http://cran.r-project.org/web/packages/diagram/vignettes/diagram.pdf . " CreationDate="2011-01-14T13:13:32.503" UserId="1036" />
  <row Id="9681" PostId="6221" Text="Great answer, csgillespie. Thanks for taking the time to edit my question and provide your answer. The reason I didn't accept it was that in your step 3, I couldn't figure out where the constant &quot;2&quot; came from!  I liked your presentation and clarity over the other answers, except for that.  I did give it +1 though. Thanks for the help. " CreationDate="2011-01-14T13:31:18.227" UserId="2767" />
  <row Id="9682" PostId="6220" Text="THANKS, onestop.  I've accepted your answer.  I appreciate it.  The frustrating thing about asking a statistics question on SO is there's no way I'll ever be able to &quot;pay it forward&quot; by helping someone else out.  " CreationDate="2011-01-14T13:34:28.783" UserId="2767" />
  <row Id="9683" PostId="6247" Text="Can you construct a simple, reproducible example of your problem? Just now I can't reproduce your difficultly since I don't have `mx`." CreationDate="2011-01-14T14:00:12.403" UserId="8" />
  <row Id="9684" PostId="6235" Text="@Shane and Andy. Thank you for your answers. @Shane, I have seen your webvis package. But I still need to explore it further. I do like protovis graphs a lot. They have a great website." CreationDate="2011-01-14T14:15:05.277" UserId="1307" />
  <row Id="9685" PostId="6222" Text="Thanks, Nico, for taking the time to answer.  You assumed that a beginner like me knows what &quot;R&quot; is. I didn't ... until I just Googled it ... so your R code didn't help." CreationDate="2011-01-14T14:28:02.460" UserId="2767" />
  <row Id="9686" PostId="6235" Text="@suncoolsu Regretfully, the package is far from &quot;complete&quot; and I have very little time to spend on it.  Maybe one of these days.  I'd love to expose things like the above graphic so that they would be trivial to create." CreationDate="2011-01-14T14:34:17.863" UserId="5" />
  <row Id="9687" PostId="4850" Text="I meant that it's measurement *error* (or any other notion of statistical error) that requires the concept.  But you're right, we've wandered a bit. Hope I'm not the only one who's been illuminated in this little exchange." CreationDate="2011-01-14T14:37:52.353" UserId="1739" />
  <row Id="9688" PostId="6243" Score="1" Text="@onestop See http://stats.stackexchange.com/questions/6096/correct-way-to-calibrate-means" CreationDate="2011-01-14T15:07:52.343" UserId="88" />
  <row Id="9689" PostId="6243" Text="Thanks, mbq. Seems to me the overhead would simply cancel out when taking the difference between two overhead-corrected means. Or am i missing something?" CreationDate="2011-01-14T15:18:22.680" UserId="449" />
  <row Id="9690" PostId="6114" Text="Thanks. That worked." CreationDate="2011-01-14T15:18:45.563" UserId="2715" />
  <row Id="9691" PostId="6253" Score="1" Text="I interpreted &quot;10% bonus&quot; as an additional 10% towards your return mark. For example, 30% goes to 40% with this bonus. However, it could be that your return mark is increased by 10% (30% goes to 33%). Would you clarify?" CreationDate="2011-01-14T15:25:52.750" UserId="8" />
  <row Id="9693" PostId="6222" Score="1" Text="Oh, OK! Well... if you did not find it, R is an open-source statistics environment and programming language. You can download it at http://www.r-project.org" CreationDate="2011-01-14T15:37:40.033" UserId="582" />
  <row Id="9694" PostId="6243" Text="@onestop - I correct the variance, moe, sd, and others but I am not sure if sample size also needs correcting and if so, the formula to do it." CreationDate="2011-01-14T15:43:46.447" UserId="2634" />
  <row Id="9696" PostId="6253" Score="1" Text="30% goes to 40%...." CreationDate="2011-01-14T16:58:12.037" UserId="2599" />
  <row Id="9697" PostId="6257" Text="+! Yup; @Max What do you thing this &quot;obvious&quot; clustering would be? " CreationDate="2011-01-14T17:33:42.630" UserId="88" />
  <row Id="9698" PostId="5950" Text="@Yang: thinking about this in the 1d case may not be instructive. I've amended my answer, see above." CreationDate="2011-01-14T17:49:17.863" UserId="795" />
  <row Id="9699" PostId="6220" Text="@Robert Frank: by asking this question you are effectively helping out anyone else who will in the future have the same doubt as you. :)" CreationDate="2011-01-14T18:11:25.637" UserId="582" />
  <row Id="9700" PostId="6257" Text="@mbq: Actually I don't know what would be a good clustering for this. By &quot;obvious&quot; I ment that (N-1, 1) is definitely not a good clustering for this. A better clustering would be only one cluster, so no clustering at all. Or maybe some clustering with the number of clusters more than 2. " CreationDate="2011-01-14T18:19:32.460" UserId="255" />
  <row Id="9701" PostId="6258" Text="thanks, I'll think about your answer suncoolsu" CreationDate="2011-01-14T18:20:13.137" UserId="2779" />
  <row Id="9702" PostId="6138" Text="@Yang Are you referring to Jorma Rissannen’s argument? Statistical inference and information compression both make use of probabilistic models to describe uncertainty, however their   perspectives and those if the researchers in those areas are generally very different.  What I mean to say above is that 20 questions can be more naturally formulated as a compression (specifically, source coding) problem rather than a classification problem. … continued below" CreationDate="2011-01-14T19:48:13.047" UserId="1670" />
  <row Id="9703" PostId="6138" Text="@Yang Here’s one aspect.  In a classification problem, we typically assume that there are a small number of classes and, certainly, that the number of classes is much smaller than the number of units.  In 20 questions, every unit is its own “class.”  The goal is to ask (on average) a minimal number of questions to uniquely identify each unit." CreationDate="2011-01-14T19:48:42.423" UserId="1670" />
  <row Id="9704" PostId="6138" Text="@ADEpt I don’t know what you mean by range questions.  The specifics behind the model that Akinator.com are… well… specific to them.  " CreationDate="2011-01-14T19:49:41.003" UserId="1670" />
  <row Id="9706" PostId="6243" Text="@mbq - Thanks for helping me clear things up. Still unsure what the composite `n` value would be. The tutorials I have seen don't mention a way to compute it." CreationDate="2011-01-14T20:23:06.307" UserId="2634" />
  <row Id="9707" PostId="6243" Text="@mbq I found a site that shows formulas for computing the new `n` value -- http://www.rad.jhmi.edu/jeng/javarad/samplesize/" CreationDate="2011-01-14T21:18:54.690" UserId="2634" />
  <row Id="9708" PostId="6240" Text="couple more things: (1) Treating failure to reject the null as evidence in *support* of null is a shockingly common error &amp; usual occasion for Streiner's point. This mistake essentially turns the strong aversion to type 1 error in &quot;p &lt; 0.05&quot; norm into license to make type 2. S says, &quot;wait--you need power...&quot; (2) Whuber cites Hume's famous argument. H's pt is actually just as subversive of empirical proofs rejecting the null as of proofs *of* the null. H says induction can't support causal inference. Ok; but there's no alternative for empirical study! Go Pearl (&amp; Bayes), not Hume, on causality!" CreationDate="2011-01-14T23:49:35.133" UserId="2347" />
  <row Id="9709" PostId="6265" Text="This does not directly pertain to the violation of independence, but have you considered multiple imputation to deal with your missing values?" CreationDate="2011-01-15T02:57:15.923" UserId="1118" />
  <row Id="9710" PostId="6243" Text="@mbq Seems that site wasn't much help :(" CreationDate="2011-01-15T05:18:28.867" UserId="2634" />
  <row Id="9711" PostId="6265" Score="2" Text="Why is there missing data? Is the data missing completely at random, or can the reason for a given observation being missing be explained by the non-missing data or the demographic data? The best method for handling missing data depends critically on the reason for missingness." CreationDate="2011-01-15T08:42:02.533" UserId="449" />
  <row Id="9712" PostId="6243" Text="@John This site you linked does not contain any equation valid in your case..." CreationDate="2011-01-15T09:43:40.113" UserId="88" />
  <row Id="9713" PostId="6270" Score="1" Text="@Dikran +1 for highlighting the distinction between model-based clustering *vs.* purely distance-based unsupervised clustering." CreationDate="2011-01-15T10:58:19.733" UserId="930" />
  <row Id="9714" PostId="6248" Text="It's usually considered good practice to save the old parameters before modifying them, so that new settings don't propagate throughout successive calls to `plot.*` functions. An example of use would be: `opar &lt;- par(mar=c(5,1,4,2)); ... plotting goes here ...; par(opar)`." CreationDate="2011-01-15T11:04:48.077" UserId="930" />
  <row Id="9715" PostId="6267" Text="I've tried using intra in inter cluster distance, but couldn't think of something usefull for a cluster with one point. Also I don't have a center point. I only have distances between points." CreationDate="2011-01-15T12:52:48.927" UserId="255" />
  <row Id="9716" PostId="6248" Text="@chl, OP uses postscript, so the plotting must end with dev.off(). This resets all the par settings. The same holds for usual plots. If dev.off is called between plot's, settings are reseted." CreationDate="2011-01-15T13:20:03.463" UserId="2116" />
  <row Id="9718" PostId="6233" Text="Thank you so much for the advice about R import of csv data with blank cells. I didn't know that, and have been spending a lot of time putting NA's into my spreadsheets. " CreationDate="2011-01-15T14:06:55.793" UserId="656" />
  <row Id="9719" PostId="6276" Text="Thank you for the links, the one to this PDF looks especially useful http://cran.r-project.org/doc/contrib/Farnsworth-EconometricsInR.pdf Not sure how i missed that." CreationDate="2011-01-15T14:32:35.890" UserId="656" />
  <row Id="9720" PostId="6276" Text="I don't think that the LearnR blog has any time series references..." CreationDate="2011-01-15T14:34:39.270" UserId="5" />
  <row Id="9721" PostId="6268" Score="1" Text="At the same time you are showing how good linear regression can be, please also show your audience how it fails in situations where the relationships are not well described by straight lines: `require(mlbench) ; cor( mlbench.smiley()$x );  plot(mlbench.smiley()$x)`" CreationDate="2011-01-15T15:12:05.037" UserId="2129" />
  <row Id="9722" PostId="6278" Text="Thank you, especially for the Zoonekynd reference, i found that before but couldnt get back to it. " CreationDate="2011-01-15T15:41:21.723" UserId="656" />
  <row Id="9723" PostId="6268" Text="Will do Dwin... :-)" CreationDate="2011-01-15T16:08:43.410" UserId="253" />
  <row Id="9724" PostId="6255" Score="1" Text="With a uniform prior distribution wouldn't guessing 30 be equivalent to guessing 35, since both will have 65 actual test grades that will result in a passing grade?" CreationDate="2011-01-15T16:12:56.930" UserId="1036" />
  <row Id="9725" PostId="6266" Text="+1, I think this is a better strategy if your goal is to maximize the grade and you have prior distribution of test grades. It seems a bit overkill if your only goal is pass the exam though. " CreationDate="2011-01-15T16:14:50.313" UserId="1036" />
  <row Id="9726" PostId="6276" Text="I searched it for time-series, and got some stuff about ggplot2. Useful, but not really with i was looking for." CreationDate="2011-01-15T16:16:00.023" UserId="656" />
  <row Id="9727" PostId="6233" Text="@richiemorrisroe, you're welcome. I usually find if something would be convenient, R implements it this way :)" CreationDate="2011-01-15T16:19:28.550" UserId="2116" />
  <row Id="9729" PostId="6281" Text="hm, if you know how to deal with numbers, then get them by multiplying true mean by 0.5%." CreationDate="2011-01-15T18:23:10.083" UserId="2116" />
  <row Id="9730" PostId="6279" Score="1" Text="@lcl23, why exactly are you testing for non-linearity? And what are you trying to model? What is IV?" CreationDate="2011-01-15T18:28:34.993" UserId="2116" />
  <row Id="9731" PostId="6248" Text="@mpiktas That's right. Sorry about that (forgot about the PS issue)." CreationDate="2011-01-15T18:59:15.803" UserId="930" />
  <row Id="9732" PostId="6248" Text="@chl, nothing to be sorry about, au contraire, your comment was useful, since this is good to know if you indent to use par more often." CreationDate="2011-01-15T19:13:35.233" UserId="2116" />
  <row Id="9733" PostId="6284" Text="That continuous result should be squared. $\sqrt{n}=\frac{Z_{c}\sigma}{E}$, and thus $n=(\frac{Z_{c}\sigma}{E})^2$" CreationDate="2011-01-15T20:07:03.340" UserId="1118" />
  <row Id="9734" PostId="6274" Text="@onestop - I replied to you and @mbq in an answer to my own question." CreationDate="2011-01-15T20:33:38.270" UserId="2634" />
  <row Id="9735" PostId="6273" Text="@mbq - I replied to you and @onestop in an answer to my own question." CreationDate="2011-01-15T20:34:11.813" UserId="2634" />
  <row Id="9736" PostId="6278" Text="@Shane (+1) Very useful references, thanks." CreationDate="2011-01-15T20:40:39.393" UserId="930" />
  <row Id="9737" PostId="6284" Text="@Christopher Aden: that is true, thank you for pointing out the error! I corrected in the answer." CreationDate="2011-01-15T21:22:26.733" UserId="2714" />
  <row Id="9738" PostId="6255" Text="@Andy-w Good point, I should have explained my reasoning. I've now updated my answer." CreationDate="2011-01-15T21:43:01.577" UserId="8" />
  <row Id="9740" PostId="6265" Text="I shine light on the skin and measure the properties of the light returning. The property I am analyzing is independent of the the mean intensity of the light unless it is below a certain intensity. Thus the device rejects any data with a mean intensity below the cutoff. Mean intensity is determined by the exposure time, set manually by a tech. Sometimes he chooses poorly and the mean intensity is too low. For the most part the missing values are randomly distributed throughout except two body sites need longer exposure times and he is more likely to underestimate the needed exposure time." CreationDate="2011-01-16T01:41:25.477" UserId="2788" />
  <row Id="9741" PostId="6279" Text="IV stands for independent variables. I thought the first step is testing assumption before we start the regression analysis? " CreationDate="2011-01-16T01:47:42.863" UserId="2793" />
  <row Id="9742" PostId="6288" Text="crosstab result is ok, with no zero values. " CreationDate="2011-01-16T01:52:57.037" UserId="2793" />
  <row Id="9743" PostId="6288" Text="crosstab result is ok, with no zero values. The problematic continuous variable gives me this result: B=9.62, SE=7.0, Wald=1.89, Sig=0.16, Exp(B)=15203.83. So, it is not sig., thus no issue of multicollinearity right? But the Exp(B) is so large which I think something wrong here. Another interaction (X*LnX) is sig. at 0.05, thus multicollinearity exists. From what I read from the book, the only solution is to transform this continuous variable to categorical?" CreationDate="2011-01-16T02:00:00.087" UserId="2793" />
  <row Id="9746" PostId="5770" Text="@Marco. Can you please make edits to your original question? This will make it easier to answer similar questions in the future. It will also help me comprehend your design and answer it correctly. Thanks in advance." CreationDate="2011-01-16T04:34:47.567" UserId="1307" />
  <row Id="9747" PostId="6280" Text="I’m writing a paper for a broad audience an wish to contrast those statements with the equivalent statement for the interval containing 95% of the population proportions that are likely to yield the observed number of successes. That interval is easily accessible from the cumulative beta function, Beta(x+1,n-x+1), and has exact coverage given the observed successes. It is fully frequentist, being a fiducial interval and is at the same time a Bayesian interval with a uniform prior. (Ross 2003, &lt;www.computersinbiologyandmedicine.com/article/S0010-4825(03)00019-2/abstract&gt;" CreationDate="2011-01-16T05:48:55.947" UserId="1679" />
  <row Id="9748" PostId="6279" Text="@lcl23, could you be more precise, your questions are for the results of non-linearity test, or for logistical regression? Also it would help if we knew more about your model, what is dependent variable, etc. " CreationDate="2011-01-16T07:32:41.687" UserId="2116" />
  <row Id="9749" PostId="6288" Text="@lcl23, what is $B$? If it is a coefficient from logistic regression, why are you interested in $\exp(B)$?" CreationDate="2011-01-16T07:33:56.007" UserId="2116" />
  <row Id="9750" PostId="6279" Text="it is the result of nonlinearity test using Ln transformation. My research question is to know the existence(yes/no) of management committee in public company. Predictors involved comprised of categorical &amp; continuous variable, such as board size, auditor, etc...." CreationDate="2011-01-16T10:28:56.517" UserId="2793" />
  <row Id="9751" PostId="6288" Text="yes, B is the coefficient. Exp(B) is the odd ratio. Does this extremely large odd ratio indicate something wrong? or I can just ignore it?" CreationDate="2011-01-16T10:55:47.673" UserId="2793" />
  <row Id="9752" PostId="6280" Text="Then I can see why you don't mention the Jeffreys interval:  Although I can't access the article itself, from the abstract it seems that Ross 2003 offers a Jeffreys' interval with a Beta(1,1) prior, rather than the same interval with a Beta(.5,.5) prior that Brown et al. examined.  It would seem odd if the coverage properties were wildly different for these two, but without the article it's hard to say." CreationDate="2011-01-16T11:55:50.440" UserId="1739" />
  <row Id="9753" PostId="6096" Text="About the flag -- this is no need to remove this post, it may be still useful for others." CreationDate="2011-01-16T12:18:59.683" UserId="88" />
  <row Id="9754" PostId="6294" Score="1" Text="The estimation is the same for ordinal &quot;X&quot; categorical variables as simple categorical ones - the interpretation is different though.  The fact that they are ordinal categories (a,b,c,d,e) simply tell you that a&lt;b&lt;c&lt;d&lt;e and you would need some structure saying in some way &quot;how much&quot; bigger each category is compared to the ones below it.  If you want to save a few degrees of freedom, then you would need to specify some additional structure saying how the ordering happens using 3 parameters or less." CreationDate="2011-01-16T12:34:26.443" UserId="2392" />
  <row Id="9755" PostId="6289" Text="Nice answer and great research. Appears to me from the 1st pdf that although they're defining this as the 'epidemic threshold', they're not declaring an epidemic each time it's exceeded for a given week - they're just declaring it a period of 'excess mortality'. They then focus on periods when it's exceeded for several weeks running (e.g. 10 or more)." CreationDate="2011-01-16T13:11:37.150" UserId="449" />
  <row Id="9757" PostId="6279" Text="I am perplexed about your recode of zero cells. Have you tried a sensitivity analysis? What happen if you use for example 0.01 or 0.00000001? Isn't to possible to make a conditional analysis after discarding observations with zeroes?" CreationDate="2011-01-16T13:37:21.180" UserId="1219" />
  <row Id="9758" PostId="6291" Text="&quot;Another thing to ask is what is the range of X values in your data?&quot; This variable ranges from 2 to 6. " CreationDate="2011-01-16T13:40:26.240" UserId="2793" />
  <row Id="9760" PostId="6192" Text="I expected this to be controversial and realise that it's not the *usual* definition, but i think it's a more useful one. In my experience of the medical literature, it's far more common to report both quartiles as the IQR rather than the difference between them, and I'm glad that's the case." CreationDate="2011-01-16T13:54:24.460" UserId="449" />
  <row Id="9761" PostId="6295" Score="1" Text="I would slightly disagree with onestop's advice, prefering to leave the ordinal variable as a categorical variable, and then check how the co-efficients vary as you move up the ordinal categories for some structure.  To fit the data as if it was continuous means you have to chose numerical values for the categories, and the results could easily be sensitive to this choice - example if I have 1,2,3,4,5 it gives the same ordering as -20,0,50,1000,1000000 - and there is no obvious way to chose between them prior to observing the data." CreationDate="2011-01-16T13:54:52.607" UserId="2392" />
  <row Id="9762" PostId="6295" Score="1" Text="The problem with leaving it as categorical is that you'll get a test with 4df that will have less power. In epidemiology it's common to perform a 'test for trend' by treating an ordinal covariate as continuous, but then to report the estimates and CIs for the model with it categorical." CreationDate="2011-01-16T14:07:01.137" UserId="449" />
  <row Id="9763" PostId="6298" Score="1" Text="I suspect they're hoping to keep that commercially confidential!" CreationDate="2011-01-16T14:11:03.977" UserId="449" />
  <row Id="9764" PostId="6289" Text="I don't think it takes 10 weeks of exceeding the mean+1.645 sd threshold to make an epidemic call ... more like 2 or 3." CreationDate="2011-01-16T14:46:42.613" UserId="2129" />
  <row Id="9765" PostId="6288" Text="I suppose it's possible that it means something but more likely it means something went wrong. Multi-colinearity will not necessarily show up in the cross-tabulations. You need to run some diagnostics. There should be documentation in your software about how to get a condition number for the XX' matrix or perhaps it will tell you how to get a report on variance inflation factors." CreationDate="2011-01-16T14:51:02.980" UserId="2129" />
  <row Id="9766" PostId="6192" Text="@onestop: obviously it is more informative to report the two quartiles, just I would not call that IQR! Just nomenclature nitpicking... :D" CreationDate="2011-01-16T14:57:17.147" UserId="582" />
  <row Id="9767" PostId="6288" Text="Addendum: If you are using the SPSS GUI, then check the &quot;Collinearity diagnostics&quot; at the time of model specification." CreationDate="2011-01-16T15:12:48.327" UserId="2129" />
  <row Id="9768" PostId="6225" Text="This is somewhat of an ill-posed question.  We need to know the conditions under which this &quot;proof&quot; is to occur." CreationDate="2011-01-16T15:39:26.503" UserId="2392" />
  <row Id="9769" PostId="6225" Text="Perhaps a better posed question is &quot;Under what conditions/assumptions is it possible to prove the null hypothesis?&quot;" CreationDate="2011-01-16T15:40:35.193" UserId="2392" />
  <row Id="9770" PostId="5564" Text="Don't forget the differences in how BUGS and JAGS deal with truncation, censoring and prior ordering of those distributions (section 8 of the manual).  In particular, JAGS has a `dinterval` distribution, where BUGS works with I()." CreationDate="2011-01-16T17:01:34.470" UserId="1739" />
  <row Id="9771" PostId="6289" Text="Wow, great research/answer.  Maybe it's the word &quot;Epidemic&quot; that threw me off.  To me, the term &quot;Epidemic Threshold&quot; is when certain conditions become unstable and the infection progresses at an increasing rate (due to the mechanics of the infection).  Other factors, such as a colder or wetter season could drive the count above the 95% level without it going unstable.  So, before today I would have expected the Epidemic Threshold to rise and fall due to &quot;other factors&quot;.  But, maybe simpler is better.   I'll play with the flubase package." CreationDate="2011-01-16T18:19:39.090" UserId="2775" />
  <row Id="9772" PostId="6215" Text="@whuber: I add an example to my answer. About the difficult questions, I suppose that they arise even if you answer refers to the normal distribution (as other answers to this question do), or some other concept. Could be simpler to explain a 50%-trimmed mean? If it is so, then say that the IQR is the length of the range of its observations" CreationDate="2011-01-16T19:00:51.643" UserId="1219" />
  <row Id="9773" PostId="6302" Text="@Elpezmuerto. Why do you need to report CDF (empirical or estimated) when you are happy with the PDF?" CreationDate="2011-01-16T21:15:40.223" UserId="1307" />
  <row Id="9774" PostId="6302" Text="@Suncoolsu, statement of work requires both" CreationDate="2011-01-16T21:26:02.503" UserId="559" />
  <row Id="9775" PostId="6302" Text="Also I think it is a good question regarding if people should use empirical or fitted cdfs. " CreationDate="2011-01-16T21:26:42.283" UserId="559" />
  <row Id="9776" PostId="6280" Text="The coverage properties with a uniform prior and the Jeffrey's prior are indeed very similar when expressed over all possible population parameter values. However, the uniform is superior when the coverage is expressed as conditional on the observed proportion. In that case it is exact. Not 'exact' in the Clopper-Pearson sense, by exactly exact. The Jeffries prior is not quite as good. It is important to note that intervals from the method do not have to be interpreted in the Bayesian manner, as it is entirely derivable as a fiducial interval." CreationDate="2011-01-16T22:05:18.500" UserId="1679" />
  <row Id="9777" PostId="6292" Text="Thanks for the thoughtful input. In terms of the words &quot;mean intensity,&quot; you are correct that I mean the mean of the light across the area of a spot (not of multiple subjects). In terms of the censoring of the data, I think I didn't explain it clearly.  My variable of interest isn't actually the intensity.  It is a different variable called contrast.  But when the intensity is too low, I am unable to reliably measure the contrast( I know this from previous experiments on objects with known properties).  Thus the values of the contrast, when the intensity is too low, are censored." CreationDate="2011-01-16T23:09:12.063" UserId="2788" />
  <row Id="9778" PostId="6295" Text="It happens that the ordinal variable is a series of equidistant samples on a continuous dimension, so it is ok to treat it as a continuous variable. I didn't include that information because I didn't want to bias responses to thinking only of continuous variables, but this should work. Thanks!" CreationDate="2011-01-16T23:44:55.020" UserId="2800" />
  <row Id="9779" PostId="6303" Text="Can I use a Q-Q plot instead? I am currently using Matlab and they have a built-in function for Q-Q plots (qqplot) but not R-R plots. " CreationDate="2011-01-17T03:09:29.573" UserId="559" />
  <row Id="9780" PostId="6303" Score="2" Text="@Elpezmuerto. Yes pp plot and qqplot try to do the same job. I think QQ plot is as good as PP plot (with subtle differences). But as the great Cleveland, W.S. says, QQ plot is one of the best ways to compare distributions." CreationDate="2011-01-17T05:00:50.733" UserId="1307" />
  <row Id="9782" PostId="6302" Text="I would go for consistency, if the pdf is fitted, then cdf should be fitted, if pdf is empirical, so is the cdf." CreationDate="2011-01-17T07:09:01.953" UserId="2116" />
  <row Id="9783" PostId="6304" Text="@suncoolsu : it does say 'nearly'..." CreationDate="2011-01-17T07:16:05.660" UserId="795" />
  <row Id="9784" PostId="6305" Score="1" Text="Hotelling's T^2:  (f − d + 1)/fd T^2 ∼ F (d , f + 1 − d ) " CreationDate="2011-01-17T08:04:40.140" UserId="2129" />
  <row Id="9785" PostId="6305" Text="@DWin, not so sure that Hotelling's $T^2$ is really applicable here. At least from formulas from wikipedia page it is not immediately clear that $T$ in OP question can be represented as $T^2$. Can you please elaborate on this more?" CreationDate="2011-01-17T08:25:08.437" UserId="2116" />
  <row Id="9786" PostId="6305" Text="will search for a convolution of $F(1,n)+F(1,n)$, afraid of some hypergeometric things, but have to be known somewhere." CreationDate="2011-01-17T09:41:14.213" UserId="2645" />
  <row Id="9787" PostId="6307" Score="1" Text="+1 for the link, did not know that characteristic function of F distribution was so complicated." CreationDate="2011-01-17T11:03:40.177" UserId="2116" />
  <row Id="9788" PostId="6124" Text="Thanks everyone! I realized I should write a script, I did it in Matlab." CreationDate="2011-01-17T11:16:36.447" UserId="2719" />
  <row Id="9789" PostId="6303" Text="Yes, a Q-Q plot would be fine too. Late last night I had some half-formed idea that a P-P plot would be a bit better in this case, but I've changed my mind in the cold light of morning so edited my answer to reflect this." CreationDate="2011-01-17T11:18:13.927" UserId="449" />
  <row Id="9790" PostId="6298" Text="This may be true, yet the video (from summer 2010) suggest that they had been still experimenting by that time; so I posted this Q hoping that some leaks appeared since then. " CreationDate="2011-01-17T11:29:43.167" UserId="88" />
  <row Id="9791" PostId="6308" Score="2" Text="For the fair and unfair coins this is a helpful read:&#xA;http://www.stat.columbia.edu/~gelman/research/published/diceRev2.pdf" CreationDate="2011-01-17T12:01:32.953" UserId="2116" />
  <row Id="9792" PostId="6293" Text="@probabilityislogic: you can use LaTeX styled math symbols (http://en.wikibooks.org/wiki/LaTeX/Mathematics) in your text between dollar signs `$` inline, or in new line aligned to the center between double dollar signs `$$`. Details: http://meta.math.stackexchange.com/questions/107/faq-for-math-stackexchange/117#117" CreationDate="2011-01-17T13:09:02.530" UserId="2714" />
  <row Id="9793" PostId="6138" Text="@vqv Let me try and put it the other way: all the articles on &quot;how the huffman codes and 20 questions are essentially the same&quot; are focused on finding out the best set of questions for the given set of objects. But the problem at hand deals with a predefined set of questions and predefined set of answers. I don't see how to bridge the gap between that and Huffman codes. Do you?" CreationDate="2011-01-17T13:51:18.257" UserId="2696" />
  <row Id="9794" PostId="6310" Text="&quot;Now maximise this quantity with respect to p&quot;: i think you mean minimise." CreationDate="2011-01-17T13:52:07.243" UserId="495" />
  <row Id="9795" PostId="6301" Text="@Gael (+1) Nice answer!" CreationDate="2011-01-17T14:13:56.710" UserId="930" />
  <row Id="9796" PostId="6105" Text="thanks again @mpkitas, you were right." CreationDate="2011-01-17T14:15:43.803" UserId="1251" />
  <row Id="9797" PostId="6310" Text="@Simon, yes, thanks. " CreationDate="2011-01-17T14:31:49.500" UserId="2116" />
  <row Id="9798" PostId="6311" Text="Thanks. Ok, I read in very beginning of the paper that they used Chi-square test for all categorical variables. The classification table written doesn't refer to a variable in particular, it is the output of a classification task. It isn't very clear! Now I suppose they did a classical test on proportion.. maybe Chi-square.." CreationDate="2011-01-17T14:53:35.530" UserId="2719" />
  <row Id="9799" PostId="6304" Text="@shabbychef. my apologies. didn't see that." CreationDate="2011-01-17T15:18:19.507" UserId="1307" />
  <row Id="9800" PostId="6306" Text="@Gil this is an interesting subject. I am curious why you (McKay et al 1999)  would choose 'War and Peace' as a control rather than, for example, random strings of letters (perhaps weighted by their observed frequencies). In other words, is it sufficient for the text to be sufficiently long, or does it have to be sufficiently long and comprehensible (or sufficiently long and of some literary value)? " CreationDate="2011-01-17T15:33:42.560" UserId="1381" />
  <row Id="9801" PostId="6306" Score="2" Text="David, the Choice of &quot;War and Peace&quot; as a control text (More precisely the beginning of the Hebrew translation of &quot;War and Peace&quot; of the same length as the Book of Genesis) was done by the original researchers. The story according to Aumann is this: When Bob Aumann who carefully followed the experiment told Kenneth Arrow about the marvelous findings in &quot;Genesis&quot;, Arrow asked what about &quot;War and Peace&quot;. Aumann then started reporting about the war and peace situation in Israel but it turned out that what Arrow asked about was if the same phenomenon cannot be found in &quot;War and Peace&quot;." CreationDate="2011-01-17T16:57:16.910" UserId="1148" />
  <row Id="9802" PostId="6302" Text="Consistency is nice, however if you using the empirical cdf, you can provide support (or argue against) your choice of distribution in the pdf " CreationDate="2011-01-17T17:14:32.247" UserId="559" />
  <row Id="9803" PostId="6318" Text="Could be a more focused question?  Perhaps one about likelihood oriented approaches to a specific inference problem?" CreationDate="2011-01-17T17:17:08.907" UserId="1739" />
  <row Id="9804" PostId="6318" Score="2" Text="But while we're here: On exposition: have you mixed up measures of effect size, usually identified with a parameter, for measures of comparative evidence for a complete model? LRs only look like candidates for the latter. Also, if you want Likelihood functions alone or in combination to tell you everything the data are trying to telling you about a model, then you're basically a Bayesian. Because that's the Likelihood Principle. (Come on in, the water's lovely :-)" CreationDate="2011-01-17T17:24:25.657" UserId="1739" />
  <row Id="9805" PostId="6318" Text="Your title and your concluding para seem to disagree on whether you're suggesting using confidence intervals or replacing them." CreationDate="2011-01-17T17:26:14.253" UserId="449" />
  <row Id="9806" PostId="6318" Text="@onestop: indeed, I just realized I forgot to change the title; I changed my mind regarding confidence intervals while writing the question. I have edited the title now. Apologies for the confusion." CreationDate="2011-01-17T17:43:24.123" UserId="364" />
  <row Id="9807" PostId="6305" Text="@mpiktas. I believe it reduces to your situation when  the variance matrix is diagonal. The off-diagonal elements from a sample should be near zero if the samples were from Normal, but might not be exactly zero if from t. Nonetheless, you asked for something approximate, so I think the answer is probably F under that proviso." CreationDate="2011-01-17T17:55:45.570" UserId="2129" />
  <row Id="9811" PostId="6320" Score="2" Text="@gappy, the answer will depend on what kind of distribution is estimated. Is your goal only to estimate the distribution, or you intend to estimate something else using this distribution? What about the structure of this $n$-dimensional vector? Is it also a sample, is it time-series, or something else? Does the distribution change with time?" CreationDate="2011-01-17T18:55:15.457" UserId="2116" />
  <row Id="9812" PostId="6320" Text="Just *how* large is 'very large'? Are you hoping to estimate a 100-dimensional multivariate distribution with no constraints on its form? You might indeed need a *very* large number of observations, maybe of the order of 10^100." CreationDate="2011-01-17T19:06:28.510" UserId="449" />
  <row Id="9814" PostId="6321" Score="1" Text="Use the Fisher Transformation or a t-test based on a scaled tangent tranformation: http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient#Determining_significance" CreationDate="2011-01-17T19:11:10.157" UserId="919" />
  <row Id="9815" PostId="6325" Text="@whuber: I was afraid of that, but I thought that summing would bring in the tails somewhat. " CreationDate="2011-01-17T20:53:01.823" UserId="795" />
  <row Id="9816" PostId="6305" Text="@DWin: it sure does look like a Hotelling with diagonal covariance matrix, but I am somewhat confused: from first principles, it does not seem like the sum of $F(1,n)$ RVs would be distributed like an $F$..." CreationDate="2011-01-17T20:55:53.430" UserId="795" />
  <row Id="9817" PostId="6325" Text="@whuber: I also thought to produce some sort of Monte Carlo experiments, trying to see for what $n$ and $k$ the approximation could be close enough to $\chi^2(k)$, probably $k(n)$ that we need here. But for small $k$ and especially $n$ it will be very heavy tailed indeed. May be you could add here these two histograms, just for lazy people like me?" CreationDate="2011-01-17T21:22:00.537" UserId="2645" />
  <row Id="9818" PostId="6322" Score="4" Text="For anyone else interested: this can be done in R with `cor.test()`" CreationDate="2011-01-17T21:58:50.760" UserId="900" />
  <row Id="9819" PostId="6325" Text="@Dmitrij The simulations are fast (it takes more time to draw the histograms), so I added 12 of them." CreationDate="2011-01-17T22:00:19.443" UserId="919" />
  <row Id="9820" PostId="6310" Text="@mpiktas (+1) Nice (updated) answer." CreationDate="2011-01-17T22:01:35.840" UserId="930" />
  <row Id="9821" PostId="6318" Text="@Conjugate Prior: Completely agree with your first two sentences. But you *can* accept the likelihood principle without being a Bayesian if you don't like the idea of priors and base inference on likelihoods alone - see books by Edwards http://books.google.com/books?id=2a_XZ-gvct4C and Royall http://books.google.com/books?id=oysWLTFaI_gC. Although someone (and i wish i remember who and where) once likened this to breaking eggs but not eating the omelette." CreationDate="2011-01-17T22:30:07.057" UserId="449" />
  <row Id="9822" PostId="6320" Text="As mentioned in the post: the sequence is iid, hence stationary. The goal is to estimate the PDF, not something else. I am open to learn about any know results for specific classes, e.g., PDFs of given smoothness, with finite support, with finite n-th moment, etc. I suspect that the literature is not very large." CreationDate="2011-01-17T23:26:13.440" UserId="30" />
  <row Id="9823" PostId="6267" Text="The higher the inter cluster distance the better, you can measure it by calculating the distances between the center of the clusters." CreationDate="2011-01-17T23:47:22.953" UserId="1808" />
  <row Id="9824" PostId="6320" Text="@gappy, it is still not clear for me, are the elements of the $n$-dimensional vector independent and distributed identicaly?" CreationDate="2011-01-18T07:49:44.717" UserId="2116" />
  <row Id="9825" PostId="6293" Text="Thanks you @daroczig for the math stuff, looks much more readable now!" CreationDate="2011-01-18T08:25:48.353" UserId="2392" />
  <row Id="9826" PostId="6318" Text="@onestop The quote is from Savage, I think, to the effect that &quot;to make the Bayesian omelet, you have to break the Bayesian eggs&quot;." CreationDate="2011-01-18T10:17:02.780" UserId="1739" />
  <row Id="9827" PostId="6084" Text="Thank you for taking your time to help me. And I'm sorry for the delayed response. I've separated the covariates in 2 blocks in the analysis. The first block contains the 5 covariates I want to control for, the second block contains only the 'risk' covariate. In Block 2 I get a 'Change from previous block'-statistic: Chi-square=6,298, df=1, sig=0,012. Doesn't this mean that the second changed the model significantly? Is this what you mean by Likelihood Ratio Test?" CreationDate="2011-01-18T10:35:30.290" UserId="2652" />
  <row Id="9828" PostId="6318" Text="@Conjugate Prior: You're right, the original metaphor is due to Savage in 1961. My memory had completely scrambled Morris's rejoinder to his 1983 paper on Empirical Bayes, at the end of which he lays out a 2×2 table for breaking eggs and enjoying omelettes: http://www.jstor.org/stable/2287105. It's Empirical Bayesians that break the egg without enjoying the omelette though. Likelihoodists aren't mentioned, but I guess they'd belong with frequentists as neither breaking eggs nor enjoying omelettes. Perhaps they're the vegans of statistics?? Time i got out of the kitchen.." CreationDate="2011-01-18T10:52:11.753" UserId="449" />
  <row Id="9829" PostId="6318" Text="I think the problem with p-values and confidence intervals is that they are too easy to interpret as something that they are not.  Likelihood ratios on the other hand have a name which corresponds precisely to what they are.  My issue with the usual null hypothesis testing is that often the alternatives are not well specified, and consideration is normally given only to &quot;type 1 errors&quot; or false positive.  A p-value also suffers from the &quot;double use&quot; of the data (discussion by James berger here http://www.isds.duke.edu/~berger/papers/interplay.html)." CreationDate="2011-01-18T11:17:03.337" UserId="2392" />
  <row Id="9830" PostId="6335" Text="@caracal I think the OP is after some kind of *treatment interference* (non-interference is defined as &quot;The effects of actions of each drug in the combination should be as good as or superior to its actions when used alone&quot;, e.g. FDA guidelines http://bit.ly/dIxYQH)." CreationDate="2011-01-18T11:38:25.217" UserId="930" />
  <row Id="9831" PostId="6310" Text="I think this example shows you exactly what a confidence interval is.  I find it easiest to interpret a CI as ONE observation from a Bernouli distributed random variable with a probability parameter equal to the level of confidence.  It only makes sense to me to use CIs if you are doing the experiment repetitively.  Another issue is that what is the alternative hypothesis? is it p=7/10, p&gt;0.5, p=1050/2000? p=527/1000?  Another issue is what do we mean by p=$\frac{1}{2}$? is it EXACTLY $\frac{1}{2}$ or is it $p \in \left(\frac{1}{2} \pm \epsilon \right)$ where $\epsilon$ is a small number." CreationDate="2011-01-18T11:42:52.410" UserId="2392" />
  <row Id="9832" PostId="6312" Text="How can you *estimate* the criterion? One physician's criteria for diagnosing e.g. chronic fatigue syndrome may well differ from another's, but I can't see it's the sort of thing you can *estimate*." CreationDate="2011-01-18T12:22:07.693" UserId="449" />
  <row Id="9833" PostId="6295" Text="@onestop - I would respond to the 4df having less power as a direct consequence of making the weaker assumption of ordinal categories.  Without any additional information on how far each category is from the other, one is effectively replacing degrees of freedom with assumptions (which are somewhat arbitrary without more info), and thus leaving the results open to model misspecification (i.e. choosing the wrong set of numbers to label the categories)." CreationDate="2011-01-18T12:57:57.197" UserId="2392" />
  <row Id="9834" PostId="6333" Text="Here's a question: what is the effect if you take logs and they're not called for? I've liked it when working with time series that require a log transform, because (as I understand it) the coefficients are ratios and at small values nearly percentages. (E.g. exp (0.05) = 1.051.)" CreationDate="2011-01-18T13:20:33.297" UserId="1764" />
  <row Id="9835" PostId="6312" Text="Nothing to do with the question, just some naïve questions on the text :) :  How would you define &quot;separation between the distributions&quot;  and what is the added value of the variance of the distribution ? what if the distributions are not gaussian ? have you heard about the total variation distance ?" CreationDate="2011-01-18T13:22:53.507" UserId="223" />
  <row Id="9836" PostId="6338" Text="Thanks for the answer and the links. I can't up-vote it yet :) I've tried the t-test on some of the data. The problem is that in some cases Wilcoxon and t-test give the same/similar results, but in other cases (as above), Wilcoxon gives a p-value ~ 0.05 and t-test gives a p-value ~ 0.5. Some of my data contain (a few) outliers." CreationDate="2011-01-18T13:51:44.373" UserId="2824" />
  <row Id="9837" PostId="6338" Text="@SabreWolfy But you can accept it by clicking on the check-mark icon." CreationDate="2011-01-18T13:58:54.257" UserId="88" />
  <row Id="9838" PostId="6338" Text="@Henrik I beg to differ: I do not &quot;always advocate the t-test.&quot;  In *one particular situation,* because the sample sizes were sufficiently large, the data did not appear to be extremely skewed, the data values were restricted to a finite set, and the t-test was likely more conservative than a permutation test (like the Wilcoxon), I argued that the t-test would perform well." CreationDate="2011-01-18T14:02:05.143" UserId="919" />
  <row Id="9839" PostId="6338" Score="1" Text="@Henrik Your response seems to assume, implicitly, that the Wilcoxon test is *not* a permutation test.  However, *both* Wilcoxon tests (the signed rank and rank sum tests) *are* permutation tests.  What other permutation tests are you specifically recommending here?" CreationDate="2011-01-18T14:03:52.440" UserId="919" />
  <row Id="9840" PostId="6338" Score="1" Text="@SabreWolfy Outliers are always critical to parametric tests (e.g., the t.test). One way of dealing with them, besides the permutation test I highly recommend, is Winsorizing (see e.g.,: http://www.unt.edu/rss/class/mike/5700/articles/robustAmerPsyc.pdf). However, with clinical data, outliers can be meaningful. This depends on what the outcome variable is (e.g., a single dead patient is pretty meaningful). No general solution to this problem." CreationDate="2011-01-18T14:07:34.040" UserId="442" />
  <row Id="9841" PostId="6338" Text="@whuber, you are absolutely correct in both cases. Sorry for misunderstanding you. No offense intended. Further, I refer to those class of permutation tests that do not take the rank but retain the original values and build an own test distribution (e.g., use `oneway_test`in `coin`). It is absolutely careless to oversee that Wilcoxon is a permutation test as well." CreationDate="2011-01-18T14:12:20.243" UserId="442" />
  <row Id="9842" PostId="6338" Text="@Henrik: So the outliers are the reason for the difference in results between the Wilcoxon and the T-test?" CreationDate="2011-01-18T14:13:08.193" UserId="2824" />
  <row Id="9843" PostId="6338" Text="@sabreWolfy Yes, see my updated answer." CreationDate="2011-01-18T14:20:38.887" UserId="442" />
  <row Id="9844" PostId="6338" Text="@Henrik I neglected to mention that your answer is spot-on and contains good advice.  Naturally I upvoted it right away ;-).  The added outlier example makes an excellent point." CreationDate="2011-01-18T14:26:53.823" UserId="919" />
  <row Id="9845" PostId="6338" Text="@Henrik: Thanks for the update re outliers; missed that in the help for t.test." CreationDate="2011-01-18T14:40:02.320" UserId="2824" />
  <row Id="9846" PostId="6339" Text="Thanks for pointing these out; I hadn't specifically considered these." CreationDate="2011-01-18T14:46:56.433" UserId="2824" />
  <row Id="9847" PostId="6336" Text="I used this Matlab function to generate the empirical cdf:&#xA;http://www.mathworks.com/matlabcentral/fileexchange/4426-cdfplot " CreationDate="2011-01-18T14:54:49.263" UserId="559" />
  <row Id="9848" PostId="6339" Text="@onestop (+1) I was about to post a response with [Simpson's paradox](http://en.wikipedia.org/wiki/Simpson's_paradox). " CreationDate="2011-01-18T15:02:23.487" UserId="930" />
  <row Id="9849" PostId="6339" Text="Just when I think that I've &quot;completed&quot; this analysis, I realize there is more to consider... :)" CreationDate="2011-01-18T15:06:24.577" UserId="2824" />
  <row Id="9850" PostId="6342" Text="This question appears to be a continuation of http://stats.stackexchange.com/q/6337/919 ." CreationDate="2011-01-18T15:08:22.710" UserId="919" />
  <row Id="9851" PostId="6340" Text="(+1) *Technical note*: gamma, beta, and hypergeometric functions are analytic almost everywhere in the complex plane.  They're actually fast and easy to calculate to high accuracy.  *All* functions &quot;require numeric evaluation&quot; at some point!" CreationDate="2011-01-18T15:10:25.793" UserId="919" />
  <row Id="9852" PostId="6342" Text="I thought it best not to mix the questions together as they cover different topics. Here I am asking how to bin the data; the previous question was asking about comparisons between subsets of the data." CreationDate="2011-01-18T15:11:23.260" UserId="2824" />
  <row Id="9853" PostId="6336" Text="Great advice, especially about overlaying the dots on the line plots. " CreationDate="2011-01-18T15:19:46.327" UserId="1080" />
  <row Id="9854" PostId="6344" Score="1" Text="@caracal (+1) The `cut2()` function from [Hmisc](http://cran.r-project.org/web/packages/Hmisc/index.html) is a very handy replacement to base `cut()`." CreationDate="2011-01-18T15:33:27.190" UserId="930" />
  <row Id="9855" PostId="6341" Text="What about using *plain english* (like the OP and everybody else on this website)?" CreationDate="2011-01-18T15:34:39.467" UserId="930" />
  <row Id="9856" PostId="6336" Text="Maybe its got something to do with the graphics on my PC or this web page, but the plots in the link do look like stair-case functions, but the plots in the question look smooth.  Maybe you have a lot of data and the &quot;steps&quot; are so close together you can't see them?" CreationDate="2011-01-18T15:35:20.633" UserId="2392" />
  <row Id="9857" PostId="6084" Score="1" Text="It's a bit hazy on this side of the crystal ball interface, since you are using some sort of software that has it's own particular terminology. It sounds as though there was a change in the 2*log-likelihood of 6.298 when risk was added. Given the limits of what you have offered,  it appears the evidence in favor of &quot;risk&quot; (whatever it might be) being associated with shorter survival time is considerably stronger than was indicated by the Wald test. " CreationDate="2011-01-18T15:42:31.000" UserId="2129" />
  <row Id="9858" PostId="6344" Text="@chl Thanks! Indeed, `cut2()` makes things easier, especially argument `g` and `minmax`." CreationDate="2011-01-18T15:48:12.147" UserId="1909" />
  <row Id="9859" PostId="6344" Text="Thanks for the reply and the example. I'll work through it carefully now. Seems I was on the wrong track trying to use 'hist(age)$...' to split the A variable." CreationDate="2011-01-18T16:29:31.547" UserId="2824" />
  <row Id="9860" PostId="6331" Text="That was my original logic; however, as caracal pointing out (and which now makes sense to me), it doesn't necessarily indicate that there is a difference between XB and AB (my study may not have been powered sufficiently to detect that difference)." CreationDate="2011-01-18T16:35:03.710" UserId="2816" />
  <row Id="9861" PostId="6335" Text="Thanks caracal. Some of the statistical lingo is a bit over my head but your answer has definitely helped me interpret my results. +1" CreationDate="2011-01-18T16:36:24.987" UserId="2816" />
  <row Id="9862" PostId="6331" Text="I actually didn't do any post hoc testing - I just used an anova to compare the four data sets. The only group that was significantly different than the others was XB. Our hypothesis expected the effect of B to be negated in the presence of A. +1" CreationDate="2011-01-18T16:40:49.633" UserId="2816" />
  <row Id="9863" PostId="6336" Text="@probabilityislogic, you would be correct, my posted figure is also from pgfplots to make it look pretty for the report" CreationDate="2011-01-18T16:56:47.037" UserId="559" />
  <row Id="9864" PostId="6325" Text="+1 for the figure. Illustrations are always nice to see." CreationDate="2011-01-18T17:16:58.370" UserId="2645" />
  <row Id="9865" PostId="6347" Score="3" Text="What exactly do you mean by denoise? Do you want to remove some data which satisfies certain criteria, or do you want to estimate the signal from your data?" CreationDate="2011-01-18T17:18:37.373" UserId="2116" />
  <row Id="9866" PostId="6348" Text="@Dmitrij, +1, nice write-up" CreationDate="2011-01-18T18:53:58.063" UserId="2116" />
  <row Id="9868" PostId="6351" Text="I think there is important points to add: in an ANOVA, the normality within each group (not overall) is equivalent to the normality of the residuals." CreationDate="2011-01-18T20:00:06.630" UserId="279" />
  <row Id="9869" PostId="6353" Text="I'm sure you *can*, but I'm afraid I've no idea *how*!" CreationDate="2011-01-18T20:41:33.160" UserId="449" />
  <row Id="9870" PostId="6320" Text="Surely you know SOMETHING about the nature of these 100+ variable observations. " CreationDate="2011-01-18T20:42:13.783" UserId="1795" />
  <row Id="9871" PostId="6352" Score="2" Text="The assumption is that these $SS$ are $\chi^2$-distributed *under the null hypothesis*, which is that the group means are all equal, i.e. $M_j=M$ for all $j$. When this is the case, $y_{ij}-M_j$ is normal implies $M_j-M$ is normal. So you only need to check the first, i.e. that the observation-level residuals are normal." CreationDate="2011-01-18T20:51:57.183" UserId="449" />
  <row Id="9872" PostId="6349" Text="By perfect you mean 0 training error? It doesn't seem that hard to get better than trivial classifier for any particular training dataset. Might be hard with decision stumps though" CreationDate="2011-01-18T20:54:44.860" UserId="511" />
  <row Id="9873" PostId="6350" Text="You can pretty much ignore anything else those sources that say if they claim the raw data needs to be normally distributed. And who said &quot;we&quot; were only checking the raw values with histograms, anyway. Are you in one of those Six Sigma classes???" CreationDate="2011-01-18T20:58:01.260" UserId="2129" />
  <row Id="9874" PostId="6350" Score="1" Text="@Andy W: I've just added a link to what appears to be the relevant section of the Wikipedia article on ANOVA." CreationDate="2011-01-18T20:58:07.550" UserId="449" />
  <row Id="9875" PostId="6350" Text="@DWin: http://blog.markanthonylawson.com/?p=296 (sorry, *completely* off-topic but couldn't resist)" CreationDate="2011-01-18T21:02:14.390" UserId="449" />
  <row Id="9876" PostId="6340" Text="@whuber - apologies on my abuse of the word *analytic* here.  I meant that these functions usually require a numerical evaluation of an integral or approximation to a sum, which is what it seems like @highbandwidth is trying to avoid." CreationDate="2011-01-18T21:20:20.203" UserId="2392" />
  <row Id="9877" PostId="6349" Text="I meant zero generalisation error. You cannot obtain it, even as the number of data points goes to infinity." CreationDate="2011-01-18T21:26:32.077" UserId="2067" />
  <row Id="9878" PostId="6340" Text="@probabilityislogic Then you should include exp, log, and the trig functions in your list!" CreationDate="2011-01-18T21:37:10.590" UserId="919" />
  <row Id="9879" PostId="6351" Text="@Aniko Could you please elaborate on what you mean by &quot;equivalent&quot; in your comment?  It is almost tautological that normality within a group is the same as normality of that group's residuals, but it is false that normality separately within each group implies (or is implied by) normality of the residuals." CreationDate="2011-01-18T21:39:33.613" UserId="919" />
  <row Id="9880" PostId="6351" Text="@whuber I really meant the tautological sense: if the groups are normal then the residuals are normal. The reverse is only true if homoscedascity is added (as in ANOVA). I don't mean to advocate for checking the groups instead of the residuals, but I think this is the underlying reason for the varying phrasing of the assumptions." CreationDate="2011-01-18T22:14:47.867" UserId="279" />
  <row Id="9881" PostId="732" Text="Especially if you are in the financial services sector." CreationDate="2011-01-18T22:21:46.553" UserId="2129" />
  <row Id="9882" PostId="6353" Text="I would suggest typing something like *regression* into the help documentation that comes with SPSS.  Should be bread and butter stuff for any stats package" CreationDate="2011-01-18T22:29:12.657" UserId="2392" />
  <row Id="9883" PostId="6351" Text="@Aniko Thank you for the clarification." CreationDate="2011-01-18T23:27:24.387" UserId="919" />
  <row Id="9884" PostId="6361" Text="That was a bit &quot;mathsy&quot; for me, and initially I didn't understand the formula, but I read it carefully about three times and it clicked! This is *exactly* what I was looking for, and your explanation was very clear, even for someone who isn't a mathematician or statistician at all. Thank you very much!" CreationDate="2011-01-19T00:11:22.943" UserId="2832" />
  <row Id="9885" PostId="6364" Text="I think maybe you wanted $S = PP^\top$, where $S$ is the covariance, and $P$ is the Cholesky factor." CreationDate="2011-01-19T01:39:06.543" UserId="795" />
  <row Id="9886" PostId="6364" Text="Actually I want $S$ (the square root, or in this case, the Cholesky factor) to be positive definite. I have clarified the question though; thanks!" CreationDate="2011-01-19T01:50:27.240" UserId="2833" />
  <row Id="9887" PostId="6365" Text="I am using `cholupdate` but my question is about making `R` (in this case) positive definite. I have a case where my `R` is non-pd, and `cholupdate(R,X,'-')` (a downdate) fails." CreationDate="2011-01-19T01:51:34.647" UserId="2833" />
  <row Id="9888" PostId="6366" Text="After a bit of maths, the class of distributions above all give $p(|\rho|&lt;1)=\frac{1}{2}$ so that can't be used as a criterion for making the solution unique." CreationDate="2011-01-19T03:03:21.730" UserId="2392" />
  <row Id="9889" PostId="6340" Text="tuchee @whuber, perhaps it is my &quot;comfortability&quot; with exp, log, and trig functions, compared to my somewhat &quot;phobia&quot; of gamma and beta functions (for non-integer arguments or their &quot;incomplete&quot; version), and the hypergeometric function, bessel functions.  My pure maths skills stop at about this kind of area." CreationDate="2011-01-19T03:33:07.570" UserId="2392" />
  <row Id="9890" PostId="1336" Text="Isn't this an &quot;economist&quot; joke? Saying that the economist Jargon of &quot;the long run&quot; we never actually get to &quot;the long run&quot;" CreationDate="2011-01-19T03:39:56.287" UserId="2392" />
  <row Id="9891" PostId="6350" Text="@onestop thank you. I only requested the link because I am lazy and did not want to look up ANOVA on wikipedia myself, not because it is essential for the question. " CreationDate="2011-01-19T03:44:39.213" UserId="1036" />
  <row Id="9892" PostId="968" Text="This is brilliant!" CreationDate="2011-01-19T03:51:22.827" UserId="2392" />
  <row Id="9893" PostId="6360" Text="+1 for pointing out (in the last paragraph) the assumption of homoscedasticity." CreationDate="2011-01-19T03:55:20.120" UserId="919" />
  <row Id="9895" PostId="6366" Text="Clearly $\Pr(|\rho| \lt 1) = 1/2$ by symmetry: no calculations are needed." CreationDate="2011-01-19T04:19:09.550" UserId="919" />
  <row Id="9896" PostId="6369" Text="I could also introduce a location parameter $\mu$ So that $Z_{t}$ effectively gets replaced with $\frac{X_{t}-\mu}{\sigma}$, but then the prior would just get multiplied by $\frac{1}{\sigma}$, so the example I have given doesn't make the task any easier" CreationDate="2011-01-19T05:16:56.250" UserId="2392" />
  <row Id="9897" PostId="6365" Score="1" Text="all online algorithms of this form (update &amp; downdate) suffer from precision issues like this. I had similar issues in 1d resulting in negative estimates of variance. My suggestion would be to keep a circular buffer of the last k vectors observed, and when `cholupdate` fails, recompute the covariance based on that circular buffer and eat the cost. If you have the memory, and can stand the occasional time hit when this happens, you will not find a better method in terms of accuracy and ease of implementation." CreationDate="2011-01-19T05:57:06.183" UserId="795" />
  <row Id="9898" PostId="2365" Text="This is a very poor example of the &quot;inferiority&quot; of Bayesian methods, of exactly the same type Jaynes speaks of in his 1976 paper.  You need to write down what the *numerical/mathematical equation* that the ML (or other frequentist method) does, *and the corresponding Bayesian method and its numerical answer!* You have written down the model, but no solution to the estimation of anything to do with it!  The rest of your answer would be greatly improved if you wrote down what the frequentist answer using ML actually is." CreationDate="2011-01-19T06:03:14.493" UserId="2392" />
  <row Id="9899" PostId="6369" Text="@probabilityislogic Yes, you're right that leaving out these parameters does not simplify the problem." CreationDate="2011-01-19T06:08:28.687" UserId="919" />
  <row Id="9900" PostId="2369" Text="Confidence intervals, in my opinion are *completely and utterly useless* UNLESS the experiment is to be repeated a moderate number of times (10 or more).  Because whether or not an $\alpha$ level CI contains the true parameter is basically a $Bernouli(\alpha)$ random variable which has been &quot;mixed up&quot; so that we don't know whether we have observed a &quot;success&quot; or a &quot;failure&quot;.  Also this problem it is impossible to give an &quot;exact&quot; CI, because $1^{12}$ times its 0.5 and 1 time its 0.1.  Show me 95% of this set? it doesn't exist!  Wouldn't you just give the set of two numbers {0.5,0.1}?" CreationDate="2011-01-19T06:18:14.933" UserId="2392" />
  <row Id="9902" PostId="6369" Text="I'm thinking from the lack of articles on it, perhaps no transformation group big enough exist? this group was the only one I could think of which left the problem unchanged to someone who was &quot;completely ignorant&quot;.  I may update my question to reflect this" CreationDate="2011-01-19T08:25:40.500" UserId="2392" />
  <row Id="9903" PostId="2365" Text="@probabilityislogic : I gave the references. This is a discussion site, not a scientific journal. Please read the comments and the reference I gave for more information. and before you call it a poor example." CreationDate="2011-01-19T09:10:01.387" UserId="1124" />
  <row Id="9904" PostId="6361" Score="1" Text="Very nice non-technical answer, and an approach I wouldn't have thought of myself. I'd only add that it's possible to add any number of fake 'observations' to each category instead of 1, including non-integer numbers. This gives you flexibility to decide how much you want to 'shrink' towards zero the scores of items with few votes. And if you happen to want an technical-sounding description of this method, you could say you're performing a Bayesian analysis of data from a multinomial distribution using a symmetric Dirichlet prior." CreationDate="2011-01-19T09:28:14.507" UserId="449" />
  <row Id="9905" PostId="6145" Text="I have tried to get reach of that book but the library has a copy that's not available for the moment. But I can perhaps ask which model that has the most accurate standard errors when HC-robust, LSDV or within?" CreationDate="2011-01-19T09:32:25.873" UserId="2724" />
  <row Id="9906" PostId="6352" Text="@onestop Edited to reflect your clarification, thanks!" CreationDate="2011-01-19T09:34:15.013" UserId="1909" />
  <row Id="9907" PostId="6375" Text="I guess @Shane will have something to say here while talking about webvis (parser for protovis) since protovis (dedicated to web visualisation) (http://vis.stanford.edu/protovis/) contains the possibility of interactive graphics... " CreationDate="2011-01-19T10:15:07.973" UserId="223" />
  <row Id="9908" PostId="6071" Score="1" Text="@Andrew Redd: have never searched for such models but Rseek also finds `gbev` package (or `MeasurementError.cor` for a simple model) as one that may be interesting. Do you have some known methods or do you know how it is implemented in other programs like S-Plus, Stata, SPSS or whenever?" CreationDate="2011-01-19T10:16:58.843" UserId="2645" />
  <row Id="9909" PostId="6349" Text="I'm not aware of any results that guarantee 0 generalization error given an unlimited supply of weak learners. However, there is a bound which requires a supply of weak learners in order to drive *training* error to zero, like eq.5 in Shapire's &quot;The Boosting Approach to Machine Learning&quot;" CreationDate="2011-01-19T10:29:10.283" UserId="511" />
  <row Id="9911" PostId="6363" Score="1" Text="@ rolando - good answer. That being said, missing pairwise approaches tend to confuse the comparison of effects, as they are based on different numbers of observations. Might be something to keep in mind." CreationDate="2011-01-19T11:32:46.570" UserId="656" />
  <row Id="9912" PostId="6341" Text="Fernando, thanks for your help. Regarding to correlation among fixed factor, not all value are low. There are four correlation of 0.7 (e.g., see the output above). This means that there is collinearity among fixed factors involved in each interraccion? Or are the whole interactions that are correlated?&#xA;Best regards&#xA;Silvina" CreationDate="2011-01-19T12:45:50.033" UserId="2486" />
  <row Id="9913" PostId="6361" Text="While they may seem like &quot;fake&quot; observations, they do have a well defined meaning when it is +1 (as opposed to +2 or higher, which really are &quot;fake&quot; numbers, or numbers from a previous data collection).  It basically describes a state of knowledge that it is *possible* for each category to be voted for, *prior* to observing any data.  This is precisely what the flat prior on the (N-1) simplex does." CreationDate="2011-01-19T13:25:23.337" UserId="2392" />
  <row Id="9914" PostId="6353" Text="I don't know what you mean by coding for one categorical variable. Can you give an example in syntax? Is your dependent variable continuous or categorical?" CreationDate="2011-01-19T13:35:16.863" UserId="1036" />
  <row Id="9915" PostId="6380" Text="there's an error in the above program: the line should be &quot;for(i in 1:plength)&quot;" CreationDate="2011-01-19T13:38:14.083" UserId="495" />
  <row Id="9916" PostId="6363" Text="I think your slightly confused, pair-wise missing only matters if your running entirely separate models (such as using a step-wise model selection procedure). If your entering all variables into the model it still drops missing values list-wise." CreationDate="2011-01-19T13:39:33.050" UserId="1036" />
  <row Id="9917" PostId="6380" Text="you are rigth, thx!" CreationDate="2011-01-19T13:40:05.257" UserId="2838" />
  <row Id="9918" PostId="6381" Text="please help me to modify my script..." CreationDate="2011-01-19T13:47:36.670" UserId="2838" />
  <row Id="9920" PostId="6383" Text="thank you very much. You are right, that with &quot;apply&quot; the script could be optimized. I just used my script as a minimal-example in order to get the message through... Thx a lot, your answer is exactly what I was looking for!!" CreationDate="2011-01-19T14:27:43.283" UserId="2838" />
  <row Id="9921" PostId="6381" Score="2" Text="Those are just hiding the loop from you. The real problem with @Produnis code is that forced copying is going on because the results vectors are being extended at each iteration of the loop." CreationDate="2011-01-19T14:43:04.383" UserId="1390" />
  <row Id="9922" PostId="6379" Text="Do you want to test the null hypothesis that all the players have the same strength, or check the fit of a model of player strength?" CreationDate="2011-01-19T14:58:18.867" UserId="449" />
  <row Id="9923" PostId="6385" Text="To be more specific. I have 8 players and only 18 games. So, there a lot of pair of players that did not play with each other and there a lot of pairs that played only one with each other. As a consequence, I cannot estimate the probability of win for a given pair of players. I also see, for example that there is a player that won 6 times in 6 games. But may be it is just a coincidence." CreationDate="2011-01-19T15:47:57.473" UserId="2407" />
  <row Id="9924" PostId="6384" Text="thanks Gavin, this is really helpfull!!!" CreationDate="2011-01-19T16:44:00.857" UserId="2838" />
  <row Id="9925" PostId="6381" Text="snowfall package can extend Gavin's solution like saying &quot;cake&quot;. Package has a plethora of apply function modified to do multicoring. For apply function, you would use sfApply(&lt;yourarguments as for apply&gt;). Snowfall is also well documented. I should point out that no extra software is needed for performing this on a multi-core processor. See http://stackoverflow.com/questions/4164960/which-list-element-is-being-processed-when-using-snowfallsflapply for a sfLapply example." CreationDate="2011-01-19T16:54:06.620" UserId="144" />
  <row Id="9926" PostId="6378" Text="Thank you very much!" CreationDate="2011-01-19T17:19:33.603" UserId="2816" />
  <row Id="9927" PostId="6349" Text="I did not come up with this argument. For example, &quot;Some Theoretical Aspects of Boosting in the Presence of Noisy Data&quot; mentions it (with the qualifier that the Bayes error is strictly positive; otherwise, it would be possible to get 0 generalisation error)." CreationDate="2011-01-19T17:28:29.960" UserId="2067" />
  <row Id="9928" PostId="6388" Text="@onestop (+1) It's funny, I just realized that this Stata command might be used for transmission/disequilibrium test in genetics :) I discussed R packages in an earlier response, http://stats.stackexchange.com/questions/5171/testing-paired-frequencies-for-independence/5258#5258." CreationDate="2011-01-19T18:21:32.800" UserId="930" />
  <row Id="9929" PostId="6373" Text="In your first main paragraph you seem to have confused $\alpha$ and $1-\alpha$.  Where does the value of 10^12+1 come in?  What do you mean by &quot;beheaded&quot;??  This text looks like it is need of proofreading and revision." CreationDate="2011-01-19T18:30:56.600" UserId="919" />
  <row Id="9930" PostId="6388" Text="Indeed, the TDT is one application discussed in the Stata help I linked above. It's also the context in which I first came across this test. Thanks for the link to that previous Q - looks like I was busy with other Qs when it was posted." CreationDate="2011-01-19T18:38:27.460" UserId="449" />
  <row Id="9931" PostId="6332" Text="I've edited my question to include a function I wrote to do that.  Does my new function make sense?" CreationDate="2011-01-19T18:41:26.557" UserId="2817" />
  <row Id="9932" PostId="6394" Text="What would be the structural difference between the Weibull and the Gamma distribution to tackle this modeling?" CreationDate="2011-01-19T18:55:05.737" UserId="2808" />
  <row Id="9933" PostId="6394" Text="Depends what you mean by 'structural'. The Weibull distribution is certainly more common in time-to-event analysis. It has a simpler cumulative distribution function, and it's the only distribution that has both the proportional hazards and accelerated failure time properties." CreationDate="2011-01-19T19:16:32.907" UserId="449" />
  <row Id="9934" PostId="6348" Text="@Dmitrij, thanks for the info.  With the GQ test, the lower the p value, the more likely the distribution is heteroskedastic?" CreationDate="2011-01-19T19:56:42.823" UserId="2817" />
  <row Id="9935" PostId="6394" Text="From watching some of the plots of weibull and gamma, it seems that weibull has more mass as it gets to the peak of the pdf whereas the gamma has more mass as it departs from the peak of the pdf.  I was wondering if this structure has any explanation." CreationDate="2011-01-19T20:03:02.367" UserId="2808" />
  <row Id="9937" PostId="6361" Text="One more observation, for future people who find this post: In implementing this in my model I took the final score and multiplied it by 20, which gives a range of -100 to 100 from worst to best possible score (though I suppose technically those are limits you can't ever quite reach, but you get the idea). This makes the output for users in my app very intuitive!" CreationDate="2011-01-19T20:14:47.820" UserId="2832" />
  <row Id="9938" PostId="6384" Text="FWIW, Matlab has the same issues regarding preallocation and expanding vectors, and is a classic code 'blooper'. In addition to your wager, it is probably faster to use the results of `rowSums` to compute the row means (unless I am missing something regarding _e.g._ Na or NaN). The code in your third approach sums each column *twice*." CreationDate="2011-01-19T20:16:01.170" UserId="795" />
  <row Id="9939" PostId="6367" Text="Thanks for your effort -- unfortunately, it did not work. (I was doing something very similar in my 3-line program: `[V,D] = eig(A); D(D &lt;= 1e-10) = 1e-6; Apd = V*A*V';`). This approach is similar to the one by Rebonato and Jackel, and it seems to fail for pathological cases like mine. " CreationDate="2011-01-19T20:35:10.280" UserId="2833" />
  <row Id="9940" PostId="6365" Text="Thanks, that's something to think about. Unfortunately, my covariance matrix goes through so many transformations that it is not clear at which point I have to a recomputation from the circular buffer. Nevertheless, if all else fails, I should be able to use the last known PD covariance matrix -- with the hope it doesn't produce a bias in my estimates." CreationDate="2011-01-19T20:38:38.653" UserId="2833" />
  <row Id="9941" PostId="6370" Text="Thanks, that's a technique I could potentially use. It seems like it's been implemented here: http://infohost.nmt.edu/~borchers/ldlt.html" CreationDate="2011-01-19T20:40:12.287" UserId="2833" />
  <row Id="9944" PostId="6400" Text="@Darren Could you explain what are $Y$, $X$, and $Z$?" CreationDate="2011-01-19T20:51:14.510" UserId="930" />
  <row Id="9945" PostId="6400" Text="Y is a vector of gene expression values. These are not gaussian, thus they are represented as a vector of ranks. X and Z are vectors of SNP data (genotypes at a particular location) for multiple samples. They are encoded as 0, 1, and 2 representing aa, ab and bb respectively." CreationDate="2011-01-19T20:54:53.790" UserId="2842" />
  <row Id="9946" PostId="6400" Text="Is that 'first order regression' in the sense of Karalič and Bratko (1997)?? http://dx.doi.org/10.1023/A:1007365207130 &#xA;If not, could you explain what you do mean by 'multiple first order regression analysis'?" CreationDate="2011-01-19T20:55:45.727" UserId="449" />
  <row Id="9947" PostId="6400" Text="No, it is first order in the sense that Y ~ X + Z + XZ + e, where X and Z are main effects, AB is the effect of the interaction and e is the error. Beta 1, 2 and 3 coefficients are omitted for clarity." CreationDate="2011-01-19T20:59:56.547" UserId="2842" />
  <row Id="9948" PostId="6367" Text="Thats too bad. I'd be interested in an example matrix you've found that causes this (and other methods you've tried) to fail if you have time to post one. &#xA;This is such an aggravating problem to keep running into, I hope you find a solution." CreationDate="2011-01-19T21:02:14.203" UserId="1913" />
  <row Id="9949" PostId="6394" Text="One difference is that the Wiebull has a subexponential tail (one type of heavy tail) when the shape parameter is less than 1, whereas the Gamma distribution has an exponential tail." CreationDate="2011-01-19T21:15:34.700" UserId="449" />
  <row Id="9951" PostId="6384" Text="@shabbychef you'll be surprised (see my edited answer). Yes the sums are notionally computed twice, but `rowSums` and `rowMeans` are highly optimised compiled code and what we gain in only computing the sums once, we loose again in doing the mean computation in interpreted code." CreationDate="2011-01-19T21:34:31.453" UserId="1390" />
  <row Id="9952" PostId="6395" Score="1" Text="Can you try to improve your answer? For start, elaborate a bit?" CreationDate="2011-01-19T21:49:33.150" UserId="88" />
  <row Id="9953" PostId="6400" Score="1" Text="1) &quot;The vectors of y-values do not all follow a normal distribution, therefore I need to implement a non-parametric regression&quot; is a non-sequitur. Linear regression is reasonably rubust to departures from normality with large samples, and as you're in genetics I suspect you have a *very* large sample. And non-parametric methods are not the only alternative; you could apply a transformation, or use generalized linear models.&#xA;2) You still haven't given us any idea information about $Z$, e.g. how many levels does it have?" CreationDate="2011-01-19T21:56:04.523" UserId="449" />
  <row Id="9954" PostId="4547" Text="drknexus You get the issue I was driving at,. The issue is that the national housing stats often report national medial house prices,, but I'm pretty sure they at best get the median values for a state so I believe that the &quot;national&quot; median house price statistic is pretty bogus. Particularly when one realizes that the mix of demand by state probably fluctuates quite a bit as well due to local economic conditions and local foreclosure trends (which vary widely). So, Is there a semi valid way to do this is is it as bogus as I suspect??" CreationDate="2011-01-19T22:33:05.793" UserId="1963" />
  <row Id="9955" PostId="6384" Text="@Gavin Simpson: not so fast: try instead `system.time({&#xA;	for (iii in c(1:1000)) {&#xA;	p1max3 &lt;- apply(p1, 1, max)&#xA;	p1mean3 &lt;- rowMeans(p1)&#xA;	p1sum3 &lt;- rowSums(p1)&#xA;	}&#xA;})` and similarly `system.time({&#xA;	for (iii in c(1:1000)) {&#xA;		p1max4 &lt;- apply(p1, 1, max)&#xA;    p1sum4 &lt;- rowSums(p1)&#xA;    p1mean4 &lt;- p1sum4 / ncol(p1)&#xA;	}&#xA;})`; the version which does not recompute the sum takes 1.368 seconds on my computer; the one which does takes 1.396. again, far from exhaustive, but more compelling..." CreationDate="2011-01-20T00:45:49.423" UserId="795" />
  <row Id="9956" PostId="4547" Text="&lt;shrug&gt; It is 'valid' to whatever extent it represents the information you are interested in/the extent to which it represents what it purports to represent.  If the people doing these analyses have the raw data that allows them to find the median by state, I see no compelling reason they would opt to report the median of state medians rather than just the overall median, because (as you suggest) the overall median has a clear interpretation whereas the median of medians does not." CreationDate="2011-01-20T00:57:53.813" UserId="196" />
  <row Id="9957" PostId="4547" Text="P.S. You can spot check their technique if you can find the median by state.  If their reported median national house selling price matches the median of reported state medians - then they are doing what you fear.  If not, then you can probably assume that they are using the simple median of all houses sold." CreationDate="2011-01-20T00:59:34.670" UserId="196" />
  <row Id="9958" PostId="4547" Text="If you had the medians by state (and not actual sale data) AND you had the number of sales per state, you could do a weighted median to get an approximate of the 'true' median e.g. (MedianState1 * NumberofSalesState1 + MedianState2*NumberofSalesState2 + ... MedianState50*NumberofSalesState50)/TotoalNumberOfSales." CreationDate="2011-01-20T01:02:04.760" UserId="196" />
  <row Id="9959" PostId="6380" Score="1" Text="doesn't this belong on StackOverflow?" CreationDate="2011-01-20T02:14:44.027" UserId="2544" />
  <row Id="9960" PostId="6373" Text="$10^12$ is for the trillion fair coins, and 1 is for the unfair coin.  And I haven't confused $\alpha$ and $1-\alpha$ the Clopper Pearson interval listed [here][1]" CreationDate="2011-01-20T02:43:56.857" UserId="2392" />
  <row Id="9961" PostId="6373" Text="{sorry typo]$10^12$ is for the trillion fair coins, and 1 is for the unfair coin, one over this is a rough approx. to the probability of having the &quot;bad&quot; coin.  Beheaded is the consequence of giving the wrong confidence interval.  And I haven't confused $\alpha$ and $1-\alpha$ the Clopper Pearson interval listed on the wiki page (search binomial proportion confidence interval).  What happens is one part of the C-P interval is a tautology $1 \geq \frac{\alpha}{2}$ when one 1 observation.  The side &quot;flips&quot; when X=1 to X=0, which is why there is $1-\theta$ and $\theta$." CreationDate="2011-01-20T02:52:36.367" UserId="2392" />
  <row Id="9962" PostId="6227" Score="1" Text="More generally the problem in statistics is not that you are unable to prove the null hypothesis, it is that you are not able to make any point estimates with certainty.  That is, just as you can't say &quot;there is no effect of the variable&quot; you are unable to say that &quot;the effect size of the variable is 1.95&quot;.  Statistics always have confidence intervals." CreationDate="2011-01-20T03:41:55.183" UserId="196" />
  <row Id="9963" PostId="2365" Text="@joris meys - I understand that you did give a reference, but your discussion does not talk about *how* the confidence interval solution is superior to the Bayesian credible interval.  This means that basically the confidence interval basically needs to be *uncalculable* using Bayesian methods.  By showing the Bayesian solution which gives the same interval, you can show what prior information was implicitly contained in the procedure to generate the confidence interval." CreationDate="2011-01-20T07:17:52.980" UserId="2392" />
  <row Id="9964" PostId="6348" Text="@Zach: exactly, take 5% for instance, of course if you are not planning to go for data mining. I personally start from the model assumptions." CreationDate="2011-01-20T07:52:47.553" UserId="2645" />
  <row Id="9965" PostId="6384" Text="@shabbychef we must have different ideas on what is or is not compelling ;-) In fact, your more rigorous simulations reinforce my main point, that as `rowMeans` and `rowSums` are implemented in efficient, optimised compiled code they are going to be difficult to beat." CreationDate="2011-01-20T08:14:17.530" UserId="1390" />
  <row Id="9966" PostId="5740" Text="Calling R from your preferred language seams the best solution in your case" CreationDate="2011-01-20T08:56:29.167" UserId="1709" />
  <row Id="9967" PostId="6402" Text="@probabilityislogic It's not clear to me whether your model treats all variables as categorical ones or not, and what is the advantage of modeling ranks using an adjacent or cumulative-logit model. Anyway, when dealing with SNP data, we often assume an additive model (aka, allelic dosage) and treat the frequency of minor allele {0,1,2} as a continuous predictor. " CreationDate="2011-01-20T09:13:45.087" UserId="930" />
  <row Id="9968" PostId="6404" Score="2" Text="Baseball is many US statisticians' favourite source of examples so a Google (/ Scholar) search will bring up several relevant articles, e.g. Morrison and Schmittlein (1981) http://www.jstor.org/stable/2630890 .&#xA;I'll leave it to someone more familiar with both baseball and R to answer your question." CreationDate="2011-01-20T09:16:32.733" UserId="449" />
  <row Id="9969" PostId="2365" Text="@probabilityislogic : the whole discussion revolves around Felstensteins claim that it is impossible to put a prior without making impossible assumptions about either time or mutation rate. Remember we're talking about phylogenetic trees. This concept makes for quite a different framework, as it's not a classical equational model in a space of real numbers. I'd suggest you read the chapter of his book to see his argument on how under certain condition the Bayesian approach can be proven to be wrong. I'd like to stress this is ONE example. It doesn't say anything about Bayesian in general." CreationDate="2011-01-20T09:33:33.593" UserId="1124" />
  <row Id="9970" PostId="2365" Text="@probabilityislogic : To show the difference in nature of the problem : you talk about confidence intervals. Now try to define a confidence interval around a phylogenetic tree..." CreationDate="2011-01-20T09:34:39.367" UserId="1124" />
  <row Id="9971" PostId="6390" Text="Thanks guys, sorry for not explaining myself more clearly.  I amended the question above...see above" CreationDate="2011-01-20T10:10:12.270" UserId="2405" />
  <row Id="9972" PostId="6390" Text="By the way, I would up-vote the comments but my rep is too low.  " CreationDate="2011-01-20T10:23:44.173" UserId="2405" />
  <row Id="9973" PostId="6390" Text="@Nick, some R example code for your solution would echo far. :)" CreationDate="2011-01-20T10:46:52.550" UserId="144" />
  <row Id="9974" PostId="6407" Text="Thank you very much for the answer. I wonder about the last one about adding no varying dummy variables. If I add only present members of the EURO-area those dummies will still vary since I use data from the years before they introduced the euro. But for countries that hasn't introduced the dummy will have a 0 for the the whole period.  " CreationDate="2011-01-20T11:12:00.063" UserId="2724" />
  <row Id="9975" PostId="6355" Text="@chubbard, see Beyer et al. under [when-is-nearest-neighbor-meaningful-today](http://stats.stackexchange.com/questions/6314/when-is-nearest-neighbor-meaningful-today): kNN is sometimes NOT meaningful.  What are your N, dim, k ?" CreationDate="2011-01-20T11:14:39.757" UserId="557" />
  <row Id="9976" PostId="6407" Text="@Skolnick: but if you don't your results could be biased, you would like to know the impact of being in the club as compared to the countries not in the club (hence you need more out of the club countries to be confident enough). If you have a particular question if the entrance into club has had an impact, than introduce one more dummy for the countries that were not originally in euro-zone. " CreationDate="2011-01-20T12:15:15.500" UserId="2645" />
  <row Id="9977" PostId="6404" Score="1" Text="I would also suggest you check out the work of J.C. Bradbury and his blog, Sabernomics, http://www.sabernomics.com/sabernomics/ . His book on measuring player worth will likely be insightful as to what characteristics are predictive of future productivity." CreationDate="2011-01-20T13:13:01.910" UserId="1036" />
  <row Id="9978" PostId="6402" Text="The questioner's sample data has as many values for Y as there are observations. If this is true of his real data, which I guess it is as they are ranks (ignoring any ties), maximum likelihood estimation isn't going to work, as the number of $\theta$s to estimate increases with the numbers of observations." CreationDate="2011-01-20T13:28:46.783" UserId="449" />
  <row Id="9979" PostId="6348" Text="@Dmitrij.  Thank you.  I just want to make sure I'm interpreting the output correctly." CreationDate="2011-01-20T14:30:17.880" UserId="2817" />
  <row Id="9980" PostId="6412" Score="4" Text="What is the purpose of your question? Why do you need such a list? Doesn't google search on statistical packages on linux answer your question?" CreationDate="2011-01-20T16:49:31.663" UserId="2116" />
  <row Id="9981" PostId="6410" Text="Can we assume that all machines are identical? Or you suspect that some of them misbehave and you want to detect them?" CreationDate="2011-01-20T17:15:24.900" UserId="2116" />
  <row Id="9982" PostId="6384" Text="@Gavin Simpson. Actually, the problem with my example is that most of the time is taken in the apply part to compute the maximum. I agree with you that a c-based vectorized function like `rowMean` is going to be hard to beat via a general-purpose R tool like `*apply`. However, you seem to suggest that it is faster to sum 10000 numbers *twice* via `rowMean` and `rowSum` rather than only once and use R's builtin division operator. I know R has some efficiency issues (_e.g._ the recent discovery of the curly braces vs. parenthesis issue), but that seems crazy." CreationDate="2011-01-20T17:53:47.843" UserId="795" />
  <row Id="9983" PostId="6384" Text="@Gavin Simpson: as a followup, try the following: &#xA;p1 &lt;- matrix(rnorm(10000), ncol=100)&#xA;&#xA;system.time({&#xA;	for (iii in c(1:20000)) {&#xA;    p1sum5 &lt;- rowSums(p1)&#xA;    p1mean5 &lt;- p1sum5 / ncol(p1)&#xA;	}&#xA;})&#xA;&#xA;system.time({&#xA;	for (iii in c(1:20000)) {&#xA;    p1sum6 &lt;- rowSums(p1)&#xA;    p1mean6 &lt;- rowMeans(p1)&#xA;	}&#xA;})" CreationDate="2011-01-20T17:54:04.507" UserId="795" />
  <row Id="9984" PostId="6390" Text="@Roman Luštrik, I added some sample code in Matlab for equi-depth histograms (I'm not a regular user of R), hope this will suffice! " CreationDate="2011-01-20T18:07:40.860" UserId="1913" />
  <row Id="9985" PostId="6418" Text="Maybe some remarks about the desired use? " CreationDate="2011-01-20T21:59:11.277" UserId="88" />
  <row Id="9986" PostId="6361" Text="@probabilityislogic: surely any strictly positive parameters for the Dirichlet prior describe that all the probabilities are strictly between 0 and 1? And this argument suggests setting them to 2/m, where m is the number of categories, rather than 1: http://en.wikipedia.org/wiki/Rule_of_succession#Generalization_to_any_number_of_possibilities" CreationDate="2011-01-20T22:32:25.490" UserId="449" />
  <row Id="9987" PostId="6418" Text="possible duplicate of [Weight a rating system to favor items rated highly by more people over items rated highly by fewer people?](http://stats.stackexchange.com/questions/6358/weight-a-rating-system-to-favor-items-rated-highly-by-more-people-over-items-rate)" CreationDate="2011-01-20T22:33:22.697" UserId="449" />
  <row Id="9989" PostId="6373" Text="Do you mean @Keith Winstein's answer?  BTW, $10^{12}$ is rendered by enclosing the exponent in curly braces, as in &quot;10^{12}&quot;." CreationDate="2011-01-20T23:00:31.950" UserId="919" />
  <row Id="9990" PostId="2365" Text="@Joris Meys - I do appreciate the reference to the book (but its seems as though without a link, I am to buy his book in order to read your reference), which is where all the arguments are.  The equation you presented for the model is simple enough (0&lt;P&lt;1, t&gt;0, u&gt;0 with a relation between each), in fact it could be expressed as the $P=Pr(Y&lt;u)$ where Y has an exponential distribution with rate parameter $t$ (or mean $\frac{1}{t}$).  The CI needs to be around one of the parameters defining the model you describe.  The interpretation will depend on the context, as you correctly point out." CreationDate="2011-01-20T23:17:00.363" UserId="2392" />
  <row Id="9993" PostId="6421" Score="3" Text="See http://stats.stackexchange.com/q/1883/159 for a similar question (which was closed as subjective &amp; argumentative)." CreationDate="2011-01-21T03:00:17.510" UserId="159" />
  <row Id="9994" PostId="6421" Text="I was about to bring up the same thread. Smells like a duplicate." CreationDate="2011-01-21T03:03:42.410" UserId="334" />
  <row Id="9995" PostId="2365" Text="Apologies (again), I wrote the fraction incorrectly (today is just not my day!).  So it should be that you can write $P=Pr(Y&lt;\frac{4u}{3})$ where $Y \sim Expo(t)$ so that $E(Y)=\frac{1}{t}$.  If we do not observe $u$ or $t$ then the model is not identifiable (i.e. there are an infinite amount of $u$ and $t$ values which give the same $P$)." CreationDate="2011-01-21T03:08:29.570" UserId="2392" />
  <row Id="9996" PostId="6412" Text="Purpose: to elicit actual experiences. Need: I think I may have large multivariate logistic regression in my near future. Google searches: no, it doesn't.  Have you done the searches?  Plenty of non-free, plenty of abandoned projects, etc." CreationDate="2011-01-21T04:25:19.583" UserId="2849" />
  <row Id="9997" PostId="6361" Text="@onestop - I would argue against the wiki page that you describe is only considering the case when 1 of 2 possible outcomes can happen.  So *if* we were indifferent between &quot;strongly like&quot; and everything else (the other three categories *combined*), then we are *not* indifferent between &quot;strongly like&quot; and &quot;like&quot;.  But the question as posed clearly give *four* categories, not two, and from the information in the question there is no *a priori* reason to favour any one rating.  I'll have another think about it, because the issue may be more subtle (e.g. why not use the reference prior?)." CreationDate="2011-01-21T04:44:56.133" UserId="2392" />
  <row Id="9998" PostId="6373" Text="@whuber, yes I do mean @keith winstein's answer (and have corrected my answer accordingly), and I must of forgot the braces in this case (the ^12 is correctly written in my answer).  But you can't edit comments - they stay forever!  And on the " CreationDate="2011-01-21T05:39:46.070" UserId="2392" />
  <row Id="9999" PostId="1491" Score="2" Text="gtk is fine on OSX." CreationDate="2011-01-21T05:44:07.067" UserId="46" />
  <row Id="10000" PostId="6361" Text="Okay, I've had a bit of a think and basically using the $Dir(k,\dots,k)$ prior to give posterior of $Dir(k+n_1,\dots,k+n_m)$.  The [wiki page][1] [1]:http://en.wikipedia.org/wiki/Dirichlet_distribution#Aggregation shows that collapsing categories, you just add the parameters. This means that collapsing to 2 categories gives you a Beta posterior. so the denominator will be the sum of all dirichlet parameters $n_1+\dots+n_m+mk = n+mk$ and the numerator will be the sum of all the categories labelled &quot;success&quot; which will be of the form $s+ck$ where c=number of categories for &quot;success&quot;." CreationDate="2011-01-21T06:34:55.743" UserId="2392" />
  <row Id="10001" PostId="6361" Text="... continuing, this means the probability of success is given by $\frac{s+ck}{n+mk}$.  So setting $k=\frac{2}{m}$ gives $\frac{s+\frac{2c}{m}}{n+2}$.  Now if m is &quot;evenly divided&quot; as the wiki page says, then 2c=m and we are left with the original rule of succession.  So what does this mean for the uniform prior (k=1)?  To me, it means that *mere knowledge of the existence of more than two possible categories (that we consider them possible is essential to this argument) represents important information about an equal aggregation of them*.  more later" CreationDate="2011-01-21T06:47:35.653" UserId="2392" />
  <row Id="10002" PostId="6421" Text="It's subjective, sure, but isn't it still okay for CW?" CreationDate="2011-01-21T06:53:21.470" UserId="1118" />
  <row Id="10003" PostId="6361" Text="...continuing again... So what state of knowledge does the constant $\frac{2}{m}$ represent?  Here's what I think.  It represents that we are only know that one category in the &quot;success&quot; labeled group, and one category in the &quot;failure&quot; group are *possible*, the remaining $m-2$ categories *we are not even sure that these categories are possible and will only consider them possible after observing them*.  This easily generalizes to a new constant $\frac{q}{m}$ where there are $q$ categories *possible* and $m-q$ categories where we are not sure of even the possibility that they exist." CreationDate="2011-01-21T06:55:11.717" UserId="2392" />
  <row Id="10004" PostId="6412" Text="@dfrankow, what is large? Do you have any special needs? The multivariate logistic regression has pretty standard algorithm for solving it (iterative reweighted least squares), so every software is simply reimplementing it. In this case you usually go with the most popular implementation. Or you can write the algorithm yourself, it is not so hard." CreationDate="2011-01-21T07:03:22.067" UserId="2116" />
  <row Id="10005" PostId="6361" Text="...continuing again (apologies for the lengthy &quot;comment&quot;)... Note that changing from $2$ to $q$ changes the probability to $\frac{s+\frac{q}{m}}{n+q}$.  One thing I have wondered with this argument, is do we need to specify which categories within &quot;success&quot; are they ones we are willing to assume are *possible a priori*?  Or can we just say &quot;one of them, but not sure which&quot; and then spread this out &quot;evenly&quot; over the aggregated categories?  If we can, then this gives an interpretation of the Jeffreys prior as assuming that we consider *only one outcome possible, but not sure which one* or $q=1$." CreationDate="2011-01-21T07:07:19.443" UserId="2392" />
  <row Id="10006" PostId="6361" Text="Apologies, I made a small error in the above comment, the probability should be $\frac{s+\frac{cq}{m}}{n+q}$ instead of $\frac{s+\frac{q}{m}}{n+q}$." CreationDate="2011-01-21T08:10:24.457" UserId="2392" />
  <row Id="10007" PostId="6390" Text="Effort is always a plus in my book." CreationDate="2011-01-21T08:39:09.223" UserId="144" />
  <row Id="10008" PostId="6416" Text="Just a slight tehcnical note: you say that &quot;taking averages won't help&quot;, but this is precisely what the Bartlett test does, to a degree.  It calculates logarithm of the ratio of the arithmetic average variance to the geometric average variance, with both averages *weighted* by the group sizes minus 1.  The factor multiplying this logarithm is fixed by the &quot;design of the experiment&quot; as it only depends on the number of samples and the number of groups.  So one is effectively rejecting the hypothesis of equal variance if the arithmetic average is &quot;too much&quot; bigger than the geometric average." CreationDate="2011-01-21T09:01:27.957" UserId="2392" />
  <row Id="10009" PostId="6410" Text="One important piece of information which is not included above is the number of bars you observed from each machine (i.e. the sample size for each group)" CreationDate="2011-01-21T09:09:05.763" UserId="2392" />
  <row Id="10010" PostId="6416" Score="1" Text="@probabilityislogic, yes but in Bartlett test case this average is used to determine whether the variances are identical or not. If they are not identical there is no point in averaging them, since this average does not estimate any parameter. " CreationDate="2011-01-21T09:10:00.430" UserId="2116" />
  <row Id="10011" PostId="6339" Text="You can *never* complete an analysis! you can only throw your hands in the air and say &quot;I'm sick of this problem, and I accept the errors that will come from not digging deeper!&quot; :)" CreationDate="2011-01-21T09:53:18.487" UserId="2392" />
  <row Id="10012" PostId="2365" Text="@probabilityislogic - I have Felsenstein's book, unfortunately his reasoning is faulty as he seems to think that all flat priors are uninformative and vice-versa and thus considers the fact that two flat priors on different parameterisations of the same thing give different conclusions is an indication there is a problem.  The premise is wrong, and the conclusion unsurprising to anyone familiar with the idea of transformation groups.  Essentially an uninformative prior on branch length should be insensitive to the choice of units, which would give a prior that was flat on a logarithmic scale." CreationDate="2011-01-21T10:11:16.297" UserId="887" />
  <row Id="10014" PostId="6406" Text="@mpiktas - I wouldn't say that @TMOD *needs* more data, just that the predictions are likely to be more accurate if @TMOD *had* more data.  There is enough information in the question to generate a prediction." CreationDate="2011-01-21T10:15:27.860" UserId="2392" />
  <row Id="10015" PostId="2365" Text="@Joris, can you give a specific page number?" CreationDate="2011-01-21T10:16:06.343" UserId="887" />
  <row Id="10016" PostId="2365" Text="comment removed - whatever..." CreationDate="2011-01-21T10:20:57.867" UserId="1124" />
  <row Id="10017" PostId="2365" Text="@Dikran : I'll look it up tonight. It's where he demonstrates the effect of the truncation on the t prior. Actually, it's almost a page big, you should have seen it when you read the chapter. It's pretty the center of his story..." CreationDate="2011-01-21T10:21:22.017" UserId="1124" />
  <row Id="10018" PostId="6404" Score="1" Text="The problem as stated is a bit like an *outlier* problem, but not in the normal way one thinks of outliers. To incorporate the *amazing* result (i.e. the outlier) you would need a &quot;sampling distribution&quot; with a heavy tail (Jose's result well over 3 standard deviations away from his average over the past data), so this may help fit you data better, and account for it in prediction." CreationDate="2011-01-21T10:21:47.903" UserId="2392" />
  <row Id="10019" PostId="6406" Text="@probabilityislogic, yes there is enough information to generate the prediction, but then the model will only have intercept. " CreationDate="2011-01-21T10:23:21.800" UserId="2116" />
  <row Id="10020" PostId="2365" Text="@probabilityislogic : the whole point Felstenstein makes is that t and u are linked. Meaning that a flat prior on t gives a greatly biased prior on u and vice versa. You'll have to use a prior that favorizes certain values for either of it in order to have a prior that actually makes **biologically** sense. So you need to know at least something about either the transformation rate or the mutation time to use eg mrBayes in phylogeny. " CreationDate="2011-01-21T10:29:15.623" UserId="1124" />
  <row Id="10021" PostId="2365" Text="@Joris, it is a while since I read the chapter in question, but IIRC Felseneteins problem was that a flat prior on branch length is biologically implausible.  I agree, but a flat prior on branch length is not necessarily an uninformative prior.  Felsensteing seems to think (incorrectly) that only flat priors are uninformative, and hence isn't aware of other choices that may be uninformative and biologically plausible.  I should point out though that if you have knowledge of what is and what isn't biologically plausible, then you are not entirely uninformed, and neither should be your prior!" CreationDate="2011-01-21T10:29:54.257" UserId="887" />
  <row Id="10022" PostId="2365" Text="@Joris &quot;the whole point Felstenstein makes is that t and u are linked. Meaning that a flat prior on t gives a greatly biased prior on u and vice versa.&quot;  It may be that this bias is what you get if you make a minimally informative prior that includes the prior knowledge that the units of measurement should have no effect on the conclusion (transformation groups)." CreationDate="2011-01-21T10:32:53.317" UserId="887" />
  <row Id="10023" PostId="2365" Text="@joris I can understand what you are trying to say, in setting a prior you are describing a *state of knowledge*, just as if you are setting a sampling distribution. Now in the uniform prior on $P$ you are describing a *state of knowledge* that it is possible for &quot;no change&quot; and &quot;one or more changes&quot; to occur on a given branch.  Probability theory tells you how to *coherently* transform this into *the same state of knowledge* about $t$, given your knowledge about the relationship between $P$ and $t$.  So a &quot;flat&quot; prior for $t$ necessarily is describing a *different state of knowledge*." CreationDate="2011-01-21T10:55:35.487" UserId="2392" />
  <row Id="10024" PostId="2365" Text="That the solutions are different is no more and no less surprising than if you used a different model between P and t." CreationDate="2011-01-21T10:56:16.577" UserId="2392" />
  <row Id="10026" PostId="4720" Text="Just a brief technical note, the &quot;arcsine&quot; transformation is one which has a faster convergence to normality than the logistic transformation.  Set $Y=\frac{2}{\pi}\arcsin{\sqrt{\frac{X}{N}}}$ (where $X$ is number of &quot;successes&quot; and $N$ the number of trials), and you can show with the so called &quot;delta method&quot; that the variance of $Y$ is approximately constant (and independent of $Y$, as it should be in the normal distribution)." CreationDate="2011-01-21T11:49:05.173" UserId="2392" />
  <row Id="10027" PostId="6431" Score="1" Text="Well said, especially about what question a CI actually answers.  In the Jaynes' article however, he does mention that CI's (and most frequentist procedures) are designed to work well &quot;In the long-run&quot; (e.g. how often do you see $n \rightarrow \infty$ or &quot;for large n the distribution is approximately...&quot; assumptions in frequentist methods?), but there are many such procedures that can do this.  I think this is where frequentist techniques (consistency,bias,convergence,etc.etc.) can be used to assess various Bayesian procedures which are difficult to decide between." CreationDate="2011-01-21T12:13:38.477" UserId="2392" />
  <row Id="10028" PostId="6431" Text="&quot;Jaynes was being a little naughty in his paper...&quot; I think the point that Jaynes was trying to make (or the point that I took from it) is that Confidence Intervals are used to answer question a) in a large number of cases (I would speculate that anyone who *only has frequentist training* will use CI's to answer question a) and they will think they are an appropriate frequentist answer)" CreationDate="2011-01-21T12:19:32.073" UserId="2392" />
  <row Id="10029" PostId="2356" Score="1" Text="I would add some clarity to the &quot;incorrect prior assumptions...&quot; by stating that the Bayesian answer and the frequentist answer must make use of *the same information*, otherwise you are just comparing answers to two different questions.  Great question though (+1 from me)" CreationDate="2011-01-21T12:26:40.900" UserId="2392" />
  <row Id="10030" PostId="6431" Text="yes, by &quot;a little naughty&quot; I just meant that Jaynes was making the point in a rather mischeiviously confrontational (but also entertaining) manner (or at least that is how I read it).  But if he hadn't then it probably wouldn't have had any impact." CreationDate="2011-01-21T12:28:05.417" UserId="887" />
  <row Id="10031" PostId="6421" Text="That was on a longer time scale. I don't think it's a duplicate. As for argumentative, it's up to the participants. I am not trying to award a trophy here, just to keep abreast of seminal papers I and others may have missed. Since there is no right answer, I am all for a CW. &#xA;&#xA;I find it interesting that so far all the answers are on bayesian innovations." CreationDate="2011-01-21T15:39:32.073" UserId="30" />
  <row Id="10033" PostId="6433" Score="1" Text="See this thread: http://stats.stackexchange.com/questions/5347/how-can-i-efficiently-model-the-sum-of-bernoulli-random-variables" CreationDate="2011-01-21T16:05:37.447" UserId="449" />
  <row Id="10034" PostId="2365" Text="I'm a bit curious, how does the ML solution work for $t$ if you just plug in $P$ into your likelihood.  The derivative will be (by chain rule) $\frac{dL}{dt}=\frac{dL}{dP}\frac{dP}{dt}=0$ but from the function for $P$ this means $\frac{dP}{dt}=\frac{4u}{3}e^{-\frac{4}{3}ut}$, so setting $u \rightarrow 0$ and $t\rightarrow\infty$ such that $P$ is unchanged (and equal to $P_{MLE}$) would solve the ML equation?  Or is there something about $u$ which is not stated in the information?" CreationDate="2011-01-21T16:08:43.437" UserId="2392" />
  <row Id="10035" PostId="6437" Text="Thanks, Chase, this is what I was looking for..." CreationDate="2011-01-21T16:15:24.507" UserId="2838" />
  <row Id="10037" PostId="6437" Score="1" Text="@Gavin - I guess I don't appreciate the difference in clarity or performance stand point. The R docs are pretty clear that the `...` are for additional parameters to `FUN` so point duly noted. However, it's just a small leap in complexity to needing that anonymous function capability and I'm guessing the OP didn't know about that &quot;trick&quot; previously. " CreationDate="2011-01-21T16:39:03.880" UserId="696" />
  <row Id="10038" PostId="6419" Text="Scipy has much more than BLAS. In terms of Fortran packs, it binds optimization packs such as minpack, fitpack, and other linear algebra packs. It also has signal or image processing tools, optimization. I personally find that SciPy is much more versatile than R. It addition, you seem to be comparing the complete set of R packages to the single Python package 'scipy'. To be fair, you should take in account the huge variety of Python packages, a lot of them useful for science. Now I will easily acknowledge that for the specific task of statistical data processing, R is much better equipped." CreationDate="2011-01-21T16:42:21.973" UserId="1265" />
  <row Id="10039" PostId="6402" Text="@chl, This model treats X and Z as categorical in the standard ANOVA-regression (factor contrasts) representation for categorical regressors.  That's why I have the $j$ and $k$ subscripts in the equations (if they were continuous only the $i$ subscript would be there).  And from what it sounds like your &quot;minor allele&quot; you have information about the scale of the categories, the &quot;ordinality&quot; of X and Z only affect the interpretation of coefficients.  But by treating it as a numerical value, you are effectively replacing &quot;degrees of freedom&quot; with &quot;model assumptions&quot;." CreationDate="2011-01-21T16:56:32.163" UserId="2392" />
  <row Id="10040" PostId="6437" Text="Indeed, hence a comment and not a down-vote. Nothing wrong with what you wrote at all. Wasn't meant as criticism." CreationDate="2011-01-21T17:06:04.553" UserId="1390" />
  <row Id="10041" PostId="6402" Text="@onestop - if this is true ($n=I$), then the model as I have described it is *unidentifiable* which means that the problem basically needs more structure in order to get a unique solution.  One way to add this &quot;structure&quot; is through the Bayesian approach, setting a slightly informative prior on the probabilities (such as the uniform prior)." CreationDate="2011-01-21T17:07:13.540" UserId="2392" />
  <row Id="10042" PostId="6437" Text="Actually, I sounded like and arse and that wasn't the intention. I'm going to delete that comment. The substantive bit (so the other comments mean something) was; you could do the same a bit simpler with `apply(data, 1, FUN, parameter = 1)`" CreationDate="2011-01-21T17:29:03.473" UserId="1390" />
  <row Id="10043" PostId="6433" Score="1" Text="The answer depends, in an essential manner, on precisely *how* your observations depart from independence and how they fail to be identically distributed.  In general the binomial distribution, with a single parameter, is too &quot;rigid&quot; to model your sum: you should be looking at families with more parameters to allow for under- or over-dispersion.  Perhaps you could tell us more about these data?" CreationDate="2011-01-21T17:31:25.950" UserId="919" />
  <row Id="10044" PostId="6437" Text="Hah, no worries and no offense taken. I think it's good to take a step back sometimes and make sure your fundamental operating assumptions about a mundane task are sound. I honestly had forgotten all about the `...` so it was good to reconsider that for a bit." CreationDate="2011-01-21T17:44:21.683" UserId="696" />
  <row Id="10045" PostId="6421" Text="I liked the 50-year question. I don't think it needed to be close.  I am just not so sure we needed a new question. &lt;Shrug&gt;" CreationDate="2011-01-21T18:15:11.387" UserId="334" />
  <row Id="10046" PostId="6428" Text="+1, nice answer, makes mine redundant" CreationDate="2011-01-21T18:37:07.033" UserId="2116" />
  <row Id="10047" PostId="6428" Text="@onestop, using function melt from package reshape, converting to long format is one line in R, melt(data,id=1:2)." CreationDate="2011-01-21T18:41:35.297" UserId="2116" />
  <row Id="10049" PostId="4665" Text="You are right I will do that next." CreationDate="2011-01-21T19:33:06.617" UserId="1808" />
  <row Id="10050" PostId="2365" Text="@Dikran : the graph about the truncation of T is shown on page 305 (fig 18.7)" CreationDate="2011-01-21T22:55:36.050" UserId="1124" />
  <row Id="10051" PostId="2365" Text="@probabilityislogic : we're talking about trees. The likelihood of the tree is the multiplication of all likelihoods at each site (node) of the tree, which is defined as the sum  over all possible nucleotides that may have existed at the interior nodes of the tree, of the probabilities of each scenario of events. And that probability is defined by a model which involves T (or u), the Jukes-Cantor model being the most easy one. As said, phylogeny does not fit into any classical framework. " CreationDate="2011-01-21T23:00:17.340" UserId="1124" />
  <row Id="10052" PostId="2365" Text="@probabilityislogic : There have been numerous frameworks built up by now around bayesian posterior probabilities as alternative for bootstrap support values, but most of the studies conclude - rightfully - that both cannot be compared. And for the estimates of the prior both birth-death processes (data-based) as theoretical distributions for branch lengths have been used extensively. Bayesian applications like mrBayes can reduce calculation time significantly, but discussion remains whether they perform better or worse, each side of the argument bringing &quot;proof&quot; for the claim." CreationDate="2011-01-21T23:04:51.357" UserId="1124" />
  <row Id="10053" PostId="2365" Text="@probabilityislogic : But again, most studies rigthfully conclude that they can't be compared. And I still follow Felsenstein that, in case no further knowledge is available, the risk on bias is far larger with a bayesian than with an ML estimate for a phylogenetic tree. If you dive into the literature on phylogeny ( and check the papers that are not online as well, science didn't start in 1998), you'll see that this controversy has been debated heavily for the past 50 years. You and @Dikran might disagree, but the comments here are far from the right place to discuss this properly. Cheers" CreationDate="2011-01-21T23:10:39.787" UserId="1124" />
  <row Id="10054" PostId="6439" Text="This is equivalent to my problem, but it is easier to pose the problem in terms of the left tail (smaller quantiles)." CreationDate="2011-01-21T23:11:26.753" UserId="795" />
  <row Id="10055" PostId="6371" Text="Since the Markov chain you consider has some absorbing states (and is irreducible, presumably), its stationary distribution is concentrated on these absorbing states. In other words, the stationary vector you computed should have exactly two non zero coordinates, one for each of the absorbing states. Is this really what you have in mind?" CreationDate="2011-01-22T00:13:15.910" UserId="2592" />
  <row Id="10056" PostId="6444" Text="Any particular reason you chose 1 and n as the two variables?  are they ordered in some way?" CreationDate="2011-01-22T00:25:46.063" UserId="2392" />
  <row Id="10057" PostId="6444" Text="What is the relationship between $X_{n}$ and $X_{1}$ without including the other variables?" CreationDate="2011-01-22T00:34:36.867" UserId="2392" />
  <row Id="10058" PostId="6406" Text="not necessarily, one could fit an AR(1) or AR(2) model to this data" CreationDate="2011-01-22T00:42:42.820" UserId="2392" />
  <row Id="10059" PostId="6406" Text="@probabilityislogic, ah yes, you are right." CreationDate="2011-01-22T03:53:56.550" UserId="2116" />
  <row Id="10060" PostId="6445" Text="@Zach. I think aeWiardll is implying that there is a correlation (or some similar structure) in the data. Therefore, the simple assumption of independence of `lm` is probably not what @eWizardll is looking for. I hope the answer to your question would make things clearer. " CreationDate="2011-01-22T04:50:58.457" UserId="1307" />
  <row Id="10061" PostId="6332" Text="If holdout=0, it is using insample fit which will favour the model with more parameters. But if holdout&gt;0, it makes sense, although you need quite a large holdout sample for the method to select the best model reliably. In general, I would choose the model to use based on other considerations rather than only consider the out-of-sample forecast performance on a short holdout sample from one series. For example, you could consider the out-of-sample performance across the whole ensemble of series (rather than one series at a time), and select the best model class that way." CreationDate="2011-01-22T07:56:32.553" UserId="159" />
  <row Id="10062" PostId="6449" Text="@mpitkas: Thank you very much for your helpful answer. Just a couple of typos in it for you to edit: The expectation operator is missing before the first parenthesis in the third line of the middle block of equations, and the coefficient $\beta_1$ is missing before the second term in the last line of that block and before the first term in the numerator on the RHS of the final equation." CreationDate="2011-01-22T10:23:33.217" UserId="2866" />
  <row Id="10063" PostId="6450" Text="(+1) I discovered this package only recently. It looks interesting." CreationDate="2011-01-22T11:17:55.440" UserId="930" />
  <row Id="10064" PostId="6440" Text="(+1) Good catch!" CreationDate="2011-01-22T11:19:43.017" UserId="930" />
  <row Id="10065" PostId="6430" Text="(+1) Thanks for sharing your experience. I often observed that a good compromise is found just by looking at the scree plot while considering Kayser's rule as a lower limit and simulated data as an upper limit. What I like in `psych` is that it displays simulated scree plots from both PCA and FA." CreationDate="2011-01-22T11:24:59.580" UserId="930" />
  <row Id="10066" PostId="6414" Text="(+1) I came across O'Connor's website some years ago and it has a lot of useful resources. Nice that you link it here." CreationDate="2011-01-22T11:25:43.980" UserId="930" />
  <row Id="10067" PostId="6402" Text="@probabilityislogic Thanks for clarifications. If you're interested in how to &quot;scale&quot; SNP data, there was a nice paper by Waaijenborg &amp; Zwinderman with a pretty nice use of optimal scaling, [Correlating multiple SNPs and multiple disease phenotypes: penalized non-linear canonical correlation analysis](http://j.mp/gIq9mW) (Bioinformatics 2009 25(21):2764). I'm still not clear myself with how we could convert a rank outcome to an ordinal one (hence echoing @onestop's comment), so I'm waiting for an helpful OP's feedback." CreationDate="2011-01-22T11:32:55.710" UserId="930" />
  <row Id="10068" PostId="6450" Score="1" Text="His paper about it got into *Biostatistics*: http://dx.doi.org/10.1093/biostatistics/kxp050" CreationDate="2011-01-22T11:45:43.160" UserId="449" />
  <row Id="10069" PostId="6449" Text="@John, thanks, I fixed them." CreationDate="2011-01-22T12:41:28.760" UserId="2116" />
  <row Id="10070" PostId="2365" Text="@Joris, Figure 18.7 on page 305 just shows that using an informative (not uninformative) prior, the maximum likelihood estimate lies outside the Bayesian credible interval.  There is nothing in the least surprising about that.  As has already been pointed out, a flat prior on branch length is unlikely to be uninformative (transformation groups), especially when needlessly truncated (it is possible to use improper priors)." CreationDate="2011-01-22T13:25:32.057" UserId="887" />
  <row Id="10071" PostId="6448" Text="The wiki article might help out:  http://en.wikipedia.org/wiki/Continuity_correction" CreationDate="2011-01-22T13:30:21.400" UserId="253" />
  <row Id="10072" PostId="6446" Score="3" Text="(+1) I've tried a few experiments in Stata on their standard auto.dta example dataset and they agree with this. The two $t$-statistic, and hence the two $p$-values, are always identical. So i think the questioner's intuition is right and he is indeed doing something wrong with the software." CreationDate="2011-01-22T15:49:29.000" UserId="449" />
  <row Id="10074" PostId="6450" Text="@onestop Thx for the reference." CreationDate="2011-01-22T16:21:09.910" UserId="930" />
  <row Id="10077" PostId="6402" Text="one way to dodge the &quot;ranks&quot; issue is to just use broader rankings, such as percentiles, deciles, quintiles, quartiles, etc.,etc..  However this introduces the &quot;messy&quot; problem of &quot;how much to aggregate?&quot; - could treat this as a decision problem and use decision theory (for a formal approach).  I would suggest going the &quot;formal&quot; way in this case because you run the risk of &quot;quasi-fudging&quot; the significance (i.e. choosing the aggregation in which the results are most significant).  Apologies for my possible ignorance, but what does &quot;SNP data&quot; stand for?" CreationDate="2011-01-22T17:00:15.230" UserId="2392" />
  <row Id="10078" PostId="6449" Score="1" Text="I'm not sure this is the answer to the question asked, because @John Bentin is not trying to fit the two least square equations simultaneously but *separately*, so there is no endogeneity in terms of the coefficients - only in terms of the model you have posed in your answer; *joint* estimation was not part of the question, only *conditional* estimation was." CreationDate="2011-01-22T17:30:54.317" UserId="2392" />
  <row Id="10079" PostId="6453" Score="1" Text="The statement &quot;hypothesis A is proven true...&quot; is not something statistics can be much help with *in absolute terms*.  Statistics is much more useful at *comparing* different hypothesis, it would be better put in terms of &quot;the data supports hypothesis B compared to hypothesis A&quot; and other comparative statements." CreationDate="2011-01-22T18:13:26.150" UserId="2392" />
  <row Id="10080" PostId="6459" Text="@bil_080, the OP does not mention R, and help page for Box.test in R mentions the correction and has an argument to allow for the correction, although you need to supply it manualy." CreationDate="2011-01-22T20:50:33.660" UserId="2116" />
  <row Id="10081" PostId="6449" Text="@probabilityislogic, if both equations are fitted and deemed adequate (with $\beta_1$ and $\alpha_n$ non-zero) you get a contradiction, so it does not matter that they are fitted separately.  " CreationDate="2011-01-22T20:59:45.937" UserId="2116" />
  <row Id="10085" PostId="6459" Text="@mpiktas, Oops, you're right. I assumed this was an R question.  As for the second part of your comment, there are several R packages that use Ljung-Box stats.   So, it's a good idea to make sure the user understands what the package's &quot;lag&quot; means." CreationDate="2011-01-22T22:29:06.497" UserId="2775" />
  <row Id="10086" PostId="6461" Score="2" Text="But what if the two effects have the opposite signs? Likelihood ratios may be an alternative to $p$-values, but I wouldn't recommend meta-analysing $p$-values either unless they're the only thing reported." CreationDate="2011-01-23T00:03:04.810" UserId="449" />
  <row Id="10087" PostId="6458" Text="Technical Note: &quot;...However, if you non-randomly select two papers from among many that have been published, it would introduce bias into your study...&quot; This *bias* you speak of is in the *generality* of the results to other studies, not in the combination of the two pieces of analysis.  The correct analysis will *never* be biased no matter how the data was obtained, as long as the conclusions are contained to populations *sufficiently similar* to that which was observed.  The potential bias comes in when the population you are inferring about is not *sufficiently similar* to the sample." CreationDate="2011-01-23T02:01:53.797" UserId="2392" />
  <row Id="10088" PostId="6461" Text="@onestop - I'd say the &quot;opposite sign&quot; effect is a direct consequence of using a 2-sided test.  In this test, the &quot;null hypothesis&quot; is no effect, whereas the alternative is for some effect, regardless of sign $effect = 0$ vs $effect \neq 0$.  If the likelihood is symmetric in the effect, then a change of sign (with no decrease in absolute value) is further evidence in favour of the alternative hypothesis." CreationDate="2011-01-23T02:11:18.980" UserId="2392" />
  <row Id="10089" PostId="6461" Text="@onestop meta-analysis uses the mean and standard deviation, and $P$ value itself is insufficient since sd can only be estimated from $P$-values if $n$ is also provided, the statistical model is clearly stated. A method called 'vote-counting' tallies $P$-values indicating positive, negative, and no effects, but I am not sure that this is useful. " CreationDate="2011-01-23T02:18:03.647" UserId="1381" />
  <row Id="10090" PostId="6461" Text="@onestop - just adding a further clarification to my comment above.  I think why it is *intuitively silly* for a change of sign to increase the evidence, is that the *hypothesis* we think of changes between experiments, from $effect=0$ vs $effect \neq 0$ to $effect=E_1$ vs $effect \neq E_1$ where $E_1$ is the estimate from the first study.  Shows that one needs to be *careful* when combining data, to make sure you know which hypothesis test you are doing." CreationDate="2011-01-23T02:22:20.137" UserId="2392" />
  <row Id="10092" PostId="6458" Text="@probabilityislogic Your point is on the same lines as mine: validity of the analysis is contingent on interpretation. Still, study selection is a special case on which many assumptions of meta-analysis are based. Thes assumptions are too many to review here, but the conclusions would be different if two studies are done on the same population by the same lab vs by two independent labs. The difference is not the population itself but in the investigator, and a common assumption tested by meta-analysis is the absence of an investigator effect." CreationDate="2011-01-23T02:36:17.517" UserId="1381" />
  <row Id="10093" PostId="6428" Text="An interesting extension/alternative to this is to fit a hierarchical model with a Possion sampling distribution with a *sampled* rate parameter (1 rate per year), but a *Cauchy* sampling distribution for the rate parameter (instead of normal or normal mixture).  The Cauchy distribution will allow for the *extreme* event to occur (by sampling a large rate parameter).  An intermediate case (between normal and Cauchy) is the t-distribution.  (Cauchy is easier to sample from as it can use the inverse CDF method)." CreationDate="2011-01-23T02:49:49.133" UserId="2392" />
  <row Id="10094" PostId="6444" Text="@probabilityislogic: I chose $1$ and $n$, rather than (say) $1$ and $2$, for convenience of notation; otherwise the naming of the variables has no significance beyond their enumeration. I didn't posit any relationship between $X_1$ and $X_n$ with the other variables excluded. However, testing this might be a good simple way to get insight into whether we are implementing the software correctly---thanks for the suggestion!" CreationDate="2011-01-23T08:44:06.183" UserId="2866" />
  <row Id="10095" PostId="6446" Text="@whuber and onestop: Thank you for your help, in particular for pointing out that the significance levels should be the same, from both a theoretical and an experimental perspective. Clearly we need to look at our implementation. " CreationDate="2011-01-23T08:54:47.197" UserId="2866" />
  <row Id="10096" PostId="6465" Text="+1 I also recommend Quantum GIS from the freely available, opensource softwares." CreationDate="2011-01-23T10:20:17.630" UserId="2714" />
  <row Id="10097" PostId="6463" Score="4" Text="You might be better off asking on http://gis.stackexchange.com ." CreationDate="2011-01-23T10:38:23.423" UserId="449" />
  <row Id="10098" PostId="6462" Score="1" Text="What does 'FCM' stand for?" CreationDate="2011-01-23T10:52:59.797" UserId="449" />
  <row Id="10099" PostId="6449" Score="1" Text="@mpiktas: You've assumed the questioner is trying to estimate an underlying *model*, when all he said is that he's fitting a couple of linear regressions. http://www.jstor.org/stable/2676681" CreationDate="2011-01-23T11:02:10.390" UserId="449" />
  <row Id="10100" PostId="6449" Text="@onestop, thanks for the article! You gave me some powerful ammunition for the discussions with my colleagues and students. Still in doing those 2 regressions I would be careful. " CreationDate="2011-01-23T13:42:18.257" UserId="2116" />
  <row Id="10101" PostId="6299" Text="Not so odd really. One reason for using a numerical method to find an answer to a problem that can be solved analytically is to ensure that one is using the software correctly." CreationDate="2011-01-23T14:49:23.903" UserId="2617" />
  <row Id="10102" PostId="6462" Text="Thanks for clarifying that mbq! Now I can be *sure* I'm not qualified to answer the question!" CreationDate="2011-01-23T14:57:50.693" UserId="449" />
  <row Id="10103" PostId="6467" Text="@Ringold (+1) Nice catch! It is limited to the US, isn't it?" CreationDate="2011-01-23T16:10:20.780" UserId="930" />
  <row Id="10104" PostId="6459" Text="Thanks--I am using R, but the question is a general one.  Just to be safe, I was doing the test with the LjungBox function in the portes package, as well as Box.test." CreationDate="2011-01-23T16:16:58.073" UserId="2875" />
  <row Id="10105" PostId="6298" Score="2" Text="There are &quot;several&quot; algorithms that the Prediction API can choose from when training/predicting your data.  The engine chooses the one that it decides is best.  &#xA;&#xA;Some users have requested a little more control over that selection, http://goo.gl/mod/5EoA, even if the algorithm is unknown.  &#xA;&#xA;Redditors have speculated on the guts here, http://www.reddit.com/r/MachineLearning/comments/evdxb/what_are_your_thoughts_on_google_prediction_api/, but the stat-speak is lost on me." CreationDate="2011-01-23T16:40:11.743" UserId="2884" />
  <row Id="10106" PostId="6298" Score="1" Text="@hyperslug Post it as an answer, it is quite useful so I'd like to accept it." CreationDate="2011-01-23T16:48:12.283" UserId="88" />
  <row Id="10107" PostId="6463" Score="2" Text="http://en.wikipedia.org/wiki/List_of_geographic_information_systems_software" CreationDate="2011-01-23T17:37:45.843" UserId="919" />
  <row Id="10108" PostId="6472" Score="2" Text="I don't understand how small sample size, _by itself_, necessarily limits statistical inferences.  Sometimes the effect size is so huge that you only need enough samples to estimate variability and to check any model assumptions." CreationDate="2011-01-23T17:55:08.140" UserId="919" />
  <row Id="10109" PostId="6404" Text="If you would consider a crude little shortcut in addition to all the more sophisticated commentary appearing here, there's Dixon's Test for Outliers which you can perform on a sample as small as 4.  See http://www.cee.vt.edu/ewr/environmental/teach/smprimer/outlier/outlier.html#detect " CreationDate="2011-01-23T18:38:08.773" UserId="2669" />
  <row Id="10110" PostId="6347" Text="Or, along those lines but slightly different, do you want to perform an especially selective PCA that only utilizes the most informative components?  For example many people default to extracting components with eigenvalue &gt;=1.0 when a stricter criterion might serve their needs better and reveal more replicable results." CreationDate="2011-01-23T18:47:57.913" UserId="2669" />
  <row Id="10111" PostId="6474" Text="I've run into the same problem as the OP, and also found that dlm and the Dynamic Linear Models with R book are very useful." CreationDate="2011-01-23T19:21:19.887" UserId="1764" />
  <row Id="10112" PostId="6455" Text="@user2875, I've deleted my answer. Fact is that for large $h$ the test is not reliable. So the answer really depends for which $h$, $p&lt;0.05$. Furthermore what is exact value of $p$? If we decrease the threshold to $0.01$, does the result of the test changes? Personally in case of conflicting hypotheses I look for other indicators whether model is good or not. How well model fits? How does the model compare to alternative models? Do the alternative model have the same problems? For what other violations the test rejects the null?" CreationDate="2011-01-23T19:52:36.607" UserId="2116" />
  <row Id="10113" PostId="6475" Score="1" Text="@Pegah I think you will get an answer soon, but maybe you could add some background information, i.e. what are you trying to show (or equivalently, what is/are your null hypothes{i|e}s), what is the sample size, etc. In the meantime, I think @caracal's brilliant response to this question [What is the NULL hypothesis for interaction in a two-way ANOVA?](http://stats.stackexchange.com/questions/5617/what-is-the-null-hypothesis-for-interaction-in-a-two-way-anova/5622#5622) might be helpful." CreationDate="2011-01-23T21:02:17.297" UserId="930" />
  <row Id="10114" PostId="6402" Text="@probabilityislogic SNP stands for [single nucleotide polyporphisms](http://j.mp/hdK2Vr) and are widely used as DNA markers to explain phenotypic variations in [GWAS](http://j.mp/eNHd79). As I said, we usually consider the frequency of the minor allele as a numerical variable, but other coding schemes are possible depending on the model we consider. I'd recommend [Genetic association studies](http://j.mp/hFFyWJ) by Cordell and Clayton (*Lancet* 2005; 366: 1121–31) for a thorough review on that subject. " CreationDate="2011-01-23T21:14:38.520" UserId="930" />
  <row Id="10115" PostId="6475" Text="thanks. I am reading it. Best. Pegah" CreationDate="2011-01-23T21:15:23.950" UserId="2885" />
  <row Id="10116" PostId="6402" Text="@probabilityislogic (Con't) btw, I appreciate your interest -- it's always good to know what kind of data we are dealing with... I spent a lot of time figuring out what SNPs really are!" CreationDate="2011-01-23T21:15:29.067" UserId="930" />
  <row Id="10117" PostId="6457" Text="@probabilityislogic (+1) Your 2nd paragraph is particularly well formulated (I initially interpreted the question as an experimental study with *a priori* hypotheses); I also like the 3rd paragraph, and I recommend this approach when finding interesting but unexpected interaction effects. Finally, I don't know about any application of cross-validation in ANOVA designs: do you have any references?" CreationDate="2011-01-23T21:26:07.807" UserId="930" />
  <row Id="10118" PostId="6455" Text="@mpiktas, The Ljung-Box test is based on a statistic whose distribution is asymptotically (as h becomes large) chi-squared.  As h gets large relative to n, though, the power of the test decreases to 0.  Hence the desire to choose h large enough that the distribution is close to chi-squared but small enough to have useful power.  (I do not know what the risk of a false negative is, when h is small.)" CreationDate="2011-01-24T00:05:21.110" UserId="2875" />
  <row Id="10119" PostId="6470" Text="Can I use the &quot;predict&quot; function on a new dataset, and retain the same error structure?  How does the gls command know what order the observations are in?" CreationDate="2011-01-24T00:23:45.313" UserId="2817" />
  <row Id="10120" PostId="6471" Text="What is the 0.5 for in &quot;corr=corAR1(0.5,form=~1)?&quot;" CreationDate="2011-01-24T00:29:06.647" UserId="2817" />
  <row Id="10121" PostId="6476" Text="What is the 0.5 for in &quot;corr=corAR1(0.5,form=~1)?&quot;" CreationDate="2011-01-24T00:30:10.610" UserId="2817" />
  <row Id="10122" PostId="6476" Text="It gives a starting value for the optimization. It makes almost no difference if it is omitted." CreationDate="2011-01-24T02:11:25.510" UserId="159" />
  <row Id="10123" PostId="6462" Text="Oh, sorry FCM means Fuzzy c-means clustering" CreationDate="2011-01-24T02:30:00.927" UserId="2325" />
  <row Id="10124" PostId="6371" Text="Yes. The issue is computing some confidence intervals for said entries." CreationDate="2011-01-24T03:08:29.537" UserId="1144" />
  <row Id="10125" PostId="6463" Text="Thank you for the answers. I need to go out ans look at these various options. " CreationDate="2011-01-24T03:17:32.490" UserId="2878" />
  <row Id="10126" PostId="767" Score="1" Text="Difference can be viewed purely from mathematical standpoint -- BIC was derived as an asymptotic expansion of log P(data) where true model parameters are sampled according to arbitrary nowhere vanishing prior, AIC was similarly derived with true parameters held fixed" CreationDate="2011-01-24T05:57:44.100" UserId="511" />
  <row Id="10129" PostId="6380" Text="This does belong on StackOverflow. There's no statistics question here at all. Only a general programming question. " CreationDate="2011-01-24T08:31:08.247" UserId="29" />
  <row Id="10130" PostId="6479" Text="I remember the interpretation of the p-value as (in this case): when assuming that the null-hypotheses (i.e., homogeneity of variances) is correct, then the probability of obtaining this or a more extreme result (i.e., contrary the null) is 57% or 95%. But whatsoever, the conclusion is the same and correct." CreationDate="2011-01-24T08:52:00.350" UserId="442" />
  <row Id="10131" PostId="6478" Text="how does this question differ from http://stats.stackexchange.com/questions/5443/estimate-the-meaningful-predictors-for-a-value-in-a-cart-model-rpart ?" CreationDate="2011-01-24T09:14:21.530" UserId="264" />
  <row Id="10132" PostId="6478" Text="That question refers to knowing which predictor was relevant for a particular categorical value of the dependent variable.  This question is more broad (variable importance/ranking without detecting to which nominal value it effects).  Since that question wasn't answered I thought it worth to phrase it in a more general way in the hopes that someone might be able to help ..." CreationDate="2011-01-24T09:19:21.467" UserId="253" />
  <row Id="10133" PostId="6484" Score="2" Text="@lcl23 Listwise deletion is usually for subjects with missing data on any one of the covariates. Could you indicate how you define an outlier in your particular case?" CreationDate="2011-01-24T09:40:40.020" UserId="930" />
  <row Id="10134" PostId="6481" Text="@lokheart Would you still be interested in knowing how ordinal logistic regression work? It may have serious implications on model interpretation when you ignore the natural ordering of a variable." CreationDate="2011-01-24T10:50:38.463" UserId="930" />
  <row Id="10135" PostId="6481" Score="1" Text="Just to be clear, it is the outcome variable that's ordinal? What is the alternative to ordinal logistic regression you're considering? Multinomial logistic regression, or grouping the categories into two groups and using standard logistic regression?" CreationDate="2011-01-24T11:13:59.033" UserId="449" />
  <row Id="10136" PostId="6484" Text="Google leads me to believe `casewise` may be some 'feature' of SPSS. Not a package I've ever used, unfortunately(?)" CreationDate="2011-01-24T11:23:24.277" UserId="449" />
  <row Id="10137" PostId="6457" Text="@chl - cross-validation is different to what I meant by &quot;lock up the data&quot;.  this part would not be tested,looked at,analysed,etc.,etc. *at all* until your analysis was finished.  Cross validation would be done with the remaining data.  A good book on this (and many other things) is called *The elements of Statistical Learning*, but I don't have the full reference right now, but I can get it in a few weeks (when I go back to work).  You could try &quot;googling it&quot;." CreationDate="2011-01-24T12:00:38.873" UserId="2392" />
  <row Id="10138" PostId="6457" Text="@chl - When you do cross validation, it is usually done with respect to prediction, and you need to specify a criterion for &quot;good&quot; and &quot;bad&quot; predictions, similar to a loss function in a decision theoretical framework.  You also need to specify the competing models (I don't think cross-validation can do this for you).  Cross validation then gives one way of deciding which model *out of the alternatives you gave it* was the best *according to the criterion of &quot;best&quot; that you gave it*.  In ANOVA one usually isn't interested in prediction, but inference, so it may not be so useful in this case." CreationDate="2011-01-24T12:09:28.300" UserId="2392" />
  <row Id="10139" PostId="6457" Text="...continuing... although, having said that, the model which gives the &quot;best&quot; predictions does seem like a good model to investigate further (as oppose to the one which gives the &quot;worst&quot; predictions)." CreationDate="2011-01-24T12:11:20.247" UserId="2392" />
  <row Id="10140" PostId="6455" Text="@user2875, this the third time you changed the question. First you ask about the strategy of picking $h$ with smallest value, then how to interpret the test if $p&lt;0.05$ for some values of $h$, and now what is the optimal $h$ to choose. All three questions have different answers and may even have different answers depending on the context of particular problem." CreationDate="2011-01-24T12:12:31.600" UserId="2116" />
  <row Id="10141" PostId="6457" Text="@probabilityislogic Ah, but I know what CV is (btw, the ESL book is available here, http://www-stat.stanford.edu/~tibs/ElemStatLearn/). I was just wondering whether you know references where CV is used in applied work. " CreationDate="2011-01-24T12:24:05.677" UserId="930" />
  <row Id="10142" PostId="6402" Text="@chl - thanks for the clarification.  I feel much better about my ignorance, seeing as I know virtually nothing about genetics, I can't even spell the actual word for DNA (its &quot;deoxy-narglyic-acid&quot; right? lol)." CreationDate="2011-01-24T12:34:49.033" UserId="2392" />
  <row Id="10143" PostId="6485" Text="Phenomenal answer chl, thank you!" CreationDate="2011-01-24T12:51:14.863" UserId="253" />
  <row Id="10144" PostId="6484" Text="Hi chl, casewise list is a table produced for analysis of residual in logistic regression, nothing related to missing data. i learnt from this link: http://www.uk.sagepub.com/burns/website%20material/Chapter%2024%20-%20Logistic%20regression.pdf. Page 17. " CreationDate="2011-01-24T14:22:50.427" UserId="2793" />
  <row Id="10145" PostId="6488" Text="There is no need to merge -- this is an example of &quot;good duplicate&quot;" CreationDate="2011-01-24T15:22:55.680" UserId="88" />
  <row Id="10146" PostId="6484" Score="1" Text="@lcl23 Ok, I just browsed the attached doc; p. 583, it is said: &quot;the casewise list produces a list of cases that didn’t fit the model well.&quot; (where outlying criterion is considered by default to be individuals outside a 2 SD band, p. 577). So why did you run so many instances of your logistic regression? Did you first look at the 44 individuals that were flagged when fitting the initial model? " CreationDate="2011-01-24T15:27:07.040" UserId="930" />
  <row Id="10147" PostId="6490" Score="1" Text="Well said, another thing is that a p-value tells you nothing about *what to do* if the null hypothesis is &quot;rejected&quot;, but a plot of the data gives you a clue as to the problem" CreationDate="2011-01-24T16:13:05.220" UserId="2392" />
  <row Id="10148" PostId="6474" Text="I would also add that I bought &quot;An Introduction to State Space Time Series Analysis (Practical Econometrics)&quot;, which was a very gentle introduction to the topic, and it too, used SsfPack/Ox. Makes it much less useful, in one sense, though it does have the benefit of once you implement it in dlm (in R) yourself, you can check your answer against an independent implementation." CreationDate="2011-01-24T18:00:07.060" UserId="1764" />
  <row Id="10149" PostId="6455" Text="@mpiktas, the questions are all the same, just different ways of looking at it.  (As pointed out, if p&gt;0.05 for all h, then we know how to interpret the smallest p; if we knew the optimal h--we don't--then we would not be concerned with choosing the smallest p.)" CreationDate="2011-01-24T18:43:59.767" UserId="2875" />
  <row Id="10150" PostId="6493" Text="Normals have conjugate priors: http://en.wikipedia.org/wiki/Normal-gamma_distribution .  You might find these much easier to use." CreationDate="2011-01-24T19:26:47.550" UserId="919" />
  <row Id="10151" PostId="6492" Score="2" Text="A t distribution might not be a good choice, because it is symmetric whereas asset returns tend to have strong skew.  At a minimum, consider modeling the *logarithms* of the returns rather than the returns themselves." CreationDate="2011-01-24T19:28:47.717" UserId="919" />
  <row Id="10152" PostId="6484" Score="1" Text="@lcl23 This procedure will likely converge, but the results might be useless.  Exactly what do they characterize?  Certainly not the population or process you are trying to understand, because you have thrown away so much of the data!  A solution developed specifically for this phenomenon is *robust regression,* which automatically downweights--but does not eliminate--unusual cases." CreationDate="2011-01-24T19:31:59.443" UserId="919" />
  <row Id="10153" PostId="6492" Text="Yeah, that's a good point, I was thinking about that in the back of my mind, but this question is still of interest to me." CreationDate="2011-01-24T19:34:44.440" UserId="1146" />
  <row Id="10154" PostId="6493" Text="Interesting. I am doing numerical stuff, is there an advantage to these distributions besides congugality?" CreationDate="2011-01-24T19:40:17.353" UserId="1146" />
  <row Id="10155" PostId="6492" Text="Do you have a truly *huge* amount of data? I think it's more common even in Bayesian modelling to fix the df and try different values as a sensitivity analysis." CreationDate="2011-01-24T20:11:03.437" UserId="449" />
  <row Id="10156" PostId="6493" Score="3" Text="Not really my area but this 'might' be relevant?&#xA;Gelman A. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis 2006; 1:515–533. http://dx.doi.org/10.1214/06-BA117A" CreationDate="2011-01-24T20:22:25.977" UserId="449" />
  <row Id="10157" PostId="5327" Text="Huge mistake; I realized I meant *Tobit* regression the whole time, not *Truncated* regression. I just changed the question to reflect this error." CreationDate="2011-01-24T20:25:42.133" UserId="1583" />
  <row Id="10159" PostId="5327" Text="The Wooldridge reference is still the correct reference; i.e., it refers to Tobit regression." CreationDate="2011-01-24T20:31:27.620" UserId="1583" />
  <row Id="10160" PostId="6492" Text="I do have a pretty large quantity of data, but it may be that this is the best approach. Submit as an answer and I'll vote you up and accept if no one provides a better solution." CreationDate="2011-01-24T21:07:19.150" UserId="1146" />
  <row Id="10162" PostId="6332" Text="@Rob, thanks for the suggestion, I'll start to move in that direction.  In the case where my holdout equals zero what if I introduced some kind of penalty for parameters?" CreationDate="2011-01-24T23:23:11.503" UserId="2817" />
  <row Id="10163" PostId="6332" Text="As I keep saying, use out-of-sample performance on a large set of series. You can't easily compare in-sample performance of the two model classes." CreationDate="2011-01-24T23:47:09.340" UserId="159" />
  <row Id="10164" PostId="6363" Text="@ richiemorrisroe - i agree, worth keeping in mind.  &#xA;&#xA;@ Andy W - Just confirmed in SPSS that, using forced entry only, missing pairwise and missing listwise give different results in every respect, including different df." CreationDate="2011-01-25T00:13:52.197" UserId="2669" />
  <row Id="10165" PostId="6484" Text="@ICI23 and whuber - I'm going to look up robust regression.  But one question is, 44 out of how many?  If out of 50,000, then I would feel less squeamish about excluding at least that first set of influential or poorly-fitting cases.  Beyond that,  normally (and forgive me if this is basic stuff you already know) you'd want to examine all your distributions and see whether some outliers are actually implausible values, like age being coded as 999 for 'missing.'  Also you'd often want to transform any very skewed distribution to pull outliers in." CreationDate="2011-01-25T00:34:31.370" UserId="2669" />
  <row Id="10166" PostId="6481" Text="@onestop, the independent variable is ordinal, but not the outcome variable, outcome variable is whether the patient gets an infection or not" CreationDate="2011-01-25T03:20:18.120" UserId="588" />
  <row Id="10167" PostId="6363" Text="@rolando2 , I still think your confused, how can SPSS return different sets of results by declaring missing pairwise unless it makes up values for the missing data? Here is an example using simulated data I have posted in a text file, http://dl.dropbox.com/u/3385251/SPSS_missing_Listwise_vs_Pairwise.txt . I have currently downvoted your answer, as all this talk about how the regression command handles missing data is confusing, has nothing to do with the OP's original question and is likely to be misleading. &#xA;" CreationDate="2011-01-25T04:40:59.407" UserId="1036" />
  <row Id="10168" PostId="6486" Text="Just one note on p-values (in relation to my bracketed comment), it could very well be that in the case that you have &quot;unusual&quot; data (say p-value of 0.0001).  BUT it may also be the case that it is *even more unusual* under the alternative hypothesis (say a p-value of $10^{-30}$ when you switch the null and alternative hypothesis around).  I believe this can happen when the statistic $T$ is not a sufficient statistic for the  hypothesis test." CreationDate="2011-01-25T05:18:23.260" UserId="2392" />
  <row Id="10169" PostId="6486" Text="..continuing...It may also be the case that you have very &quot;good&quot; data (say p-value of 0.5).  BUT the alternative hypothesis may be *better* (or more consistent) with this data (say p-value of 0.99999 when the null and alternative hypothesis are switched around)." CreationDate="2011-01-25T05:21:11.980" UserId="2392" />
  <row Id="10170" PostId="6466" Text="So, what you are saying is that AIC doesn't sufficiently punish models for parameters so using it as a criteria may lead to overparametrization.  You recommend the use of AICc instead.  To put this back in the context of my initial question, since BIC already is more stringent than AIC is there a reason to use AICc over BIC?" CreationDate="2011-01-25T05:41:24.113" UserId="196" />
  <row Id="10172" PostId="6483" Score="1" Text="I would speculate that, in restricting the algorithm to combining clusters you may get a poorer result than if you started from scratch, because you disallow &quot;interactions&quot; between the geo-spatial and temporal effects." CreationDate="2011-01-25T06:14:32.527" UserId="2392" />
  <row Id="10173" PostId="6484" Text="@chl &amp; rolando2 - 44 out of 797. All ZResid &gt; 2.58, so i deleted all 44 cases &amp; re-run. As mentioned, still appear 11 outliers. By the way, I thought logistic doesn't require normal distribution?" CreationDate="2011-01-25T07:03:51.967" UserId="2793" />
  <row Id="10174" PostId="6497" Text="@bill_080, note that using t-distribution for GARCH returns is quite popular in financial econometrics. So the question is pretty valid and does not imply that the link applies to the poster." CreationDate="2011-01-25T07:09:07.563" UserId="2116" />
  <row Id="10175" PostId="6507" Text="@probabilityislogic, Delta method does not give the approximation $E[g(X)]\approx g(EX)$. If $g$ is convex(concave) then you even have the inequality. See http://stats.stackexchange.com/questions/5782/variance-of-a-function-of-one-random-variable/5790#5790 and http://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables" CreationDate="2011-01-25T07:17:08.073" UserId="2116" />
  <row Id="10176" PostId="6494" Text="Hi John.  I believe he might want it on a bootstrap sample." CreationDate="2011-01-25T07:48:23.630" UserId="253" />
  <row Id="10177" PostId="6505" Score="2" Text="@lokhearrt your accept rate is very low, here and on SO. Please do something about it otherwise you may find people won't answer your questions. Accepting a supplied answer indicates to others with similar problems that the answer worked and solve the problem. If the answers provided don't solved your problem, then engage with the providers of those answers to reach a solution. Be a good citizen here!" CreationDate="2011-01-25T07:55:49.153" UserId="1390" />
  <row Id="10179" PostId="6511" Text="thanks, and I just found that i can use glm(output ~ NULL, data=z, family=binomial(&quot;logistic&quot;)) for creating a NULL model, and so i can use the lrtest afterwards. FYI, thanks again" CreationDate="2011-01-25T08:04:01.237" UserId="588" />
  <row Id="10180" PostId="6511" Text="@lokheart `anova(model1, model0)` will work too." CreationDate="2011-01-25T08:05:15.037" UserId="930" />
  <row Id="10181" PostId="6511" Score="1" Text="@lokheart `glm(output ~ 1, data=z, family=binomial(&quot;logistic&quot;))` would be a more natural null model, which says that `output` is explained by a constant term (the intercept)/ The intercept is implied in all your models, so you are testing for the effect of `a` after accounting for the intercept." CreationDate="2011-01-25T08:13:47.563" UserId="1390" />
  <row Id="10182" PostId="6513" Score="2" Text="@Grega, you try very general models for specific problem. Did you any research before trying these models? Google search on &quot;forecasting electricity loads&quot; gives plenty of links." CreationDate="2011-01-25T08:45:22.490" UserId="2116" />
  <row Id="10183" PostId="6505" Text="@lokheart thanks for dealing with your accept rate." CreationDate="2011-01-25T08:49:50.420" UserId="1390" />
  <row Id="10184" PostId="6515" Text="+1 It's good to know (and it seems I forgot about that package)." CreationDate="2011-01-25T09:28:59.057" UserId="930" />
  <row Id="10185" PostId="6481" Score="1" Text="Ah, in that case you can definitely ignore ordinal logistic regression, as that's for an ordinal *outcome* variable." CreationDate="2011-01-25T09:46:37.833" UserId="449" />
  <row Id="10186" PostId="6514" Text="Could you clarify what problem this is posing?" CreationDate="2011-01-25T09:52:39.477" UserId="449" />
  <row Id="10187" PostId="6510" Text="for &quot;nested mixed random effects model&quot;, do you mean multi-level modelling? if I don't use this for hierarchical variable (my variable are some kind of risk index that higher the number, more risk the patient has, does it suit your case?)" CreationDate="2011-01-25T10:29:23.133" UserId="588" />
  <row Id="10188" PostId="6513" Score="1" Text="There is loads of literature on this problem. Check out http://www.forecasters.org/ijf/journal-issue/320 for a few papers published in the IJF." CreationDate="2011-01-25T11:57:09.553" UserId="159" />
  <row Id="10189" PostId="6514" Text="Do you suspect multicollinearity or something? Are your parameter estimates not stable when you include/exclude additional observations? Could you suggest latent variable for the combination of two valence and self-relevance?" CreationDate="2011-01-25T12:44:54.763" UserId="2645" />
  <row Id="10190" PostId="6520" Text="If you want an answer to include **R** code, it would probably help if you include your current **R** code. Just the `lmer()` call should be enough." CreationDate="2011-01-25T12:45:52.203" UserId="449" />
  <row Id="10191" PostId="6507" Text="Yes it does. expand $g(X)$ about $\mu=E(X)$.  Then we have $g(X)\approx g(\mu) + (X-\mu)g'(\mu)$  where $g'(\mu)$ is the derivative of $g(.)$ evaluated at $\mu$.  Taking expectations of both sides gives $E[g(X)]\approx E[g(\mu)] + E[(X-\mu)g'(X)]=g(\mu)$.  So my approximation is valid.  However, it is not the only way to get an approximation.  Perhaps this is not called the &quot;delta method&quot;, but it is a valid approximation.  Can be improved by adding an extra term in the Taylor series $E[g(X)]\approx g[\mu + \frac{1}{2}\sigma^2 g''(\mu)]$ (which is on wiki page)." CreationDate="2011-01-25T12:52:56.450" UserId="2392" />
  <row Id="10192" PostId="6507" Text="I thought the &quot;delta method&quot; only took the first term in a taylor series, but no extra terms.  And in the &quot;large N&quot; case $\sigma$ will be small, and basically negligible. so my approximation should give an adequate answer.  Only asked for the &quot;tail behaviour&quot; w.r.t $N$." CreationDate="2011-01-25T12:54:55.310" UserId="2392" />
  <row Id="10193" PostId="6521" Text="@onestop, the OP observes percentages, so the dependent variable is not binomial. Can you use it in glm anyways?" CreationDate="2011-01-25T12:55:10.677" UserId="2116" />
  <row Id="10194" PostId="6521" Text="The OP said the vacancy rate is computed as the ratio of vacant houses over the total number of houses, which seems to imply he has these data. Vacant houses would then be a binomial outcome, with total houses as the denominator." CreationDate="2011-01-25T13:05:37.670" UserId="449" />
  <row Id="10195" PostId="6519" Text="Whatever you end up doing, don't ever overwrite your raw data.  Just in case you change your mind, or want to compare methods." CreationDate="2011-01-25T13:20:24.210" UserId="2392" />
  <row Id="10196" PostId="6513" Text="Do you know why your linear model is not successful?  And you've put a massive amount of predictors into your model - without much thought? - I would find the best 4 or 5 predictors first, and see how they go (in the linear model)." CreationDate="2011-01-25T13:29:43.237" UserId="2392" />
  <row Id="10197" PostId="6524" Score="1" Text="@Rachey, this site supports latex, please reformat your question. I am reluctant to do it myself, since the notation is not very clear." CreationDate="2011-01-25T13:41:43.693" UserId="2116" />
  <row Id="10198" PostId="6332" Text="Ok, thanks for all the advice.  I'll use out-of-sample performance on the whole ensemble to choose a model class." CreationDate="2011-01-25T13:42:48.180" UserId="2817" />
  <row Id="10199" PostId="6461" Text="@onestop - I hadn't considered the issue of different signs in the effects. Good call." CreationDate="2011-01-25T14:04:36.027" UserId="364" />
  <row Id="10200" PostId="6521" Text="@onestop @mpiktas I have the house numbers. Actually, what I need is to simulate/forecast the probability for a house in a region to be vacant..." CreationDate="2011-01-25T14:08:36.887" UserId="1443" />
  <row Id="10201" PostId="6527" Text="This solution is a lot nicer than my &quot;hackish&quot; try!" CreationDate="2011-01-25T14:13:19.370" UserId="2714" />
  <row Id="10202" PostId="6527" Text="+1, did not know about curve :) Always used plot directly." CreationDate="2011-01-25T14:13:25.990" UserId="2116" />
  <row Id="10203" PostId="6525" Text="@Saneef, the legend in your graph does not correspond to the graph of function $y=10^6/x$, is this intentional?" CreationDate="2011-01-25T14:15:39.640" UserId="2116" />
  <row Id="10204" PostId="6528" Score="1" Text="@daroczig, why use loop? df &lt;- within(df,y&lt;-10^6/x) is much nicer and more efficient. Also OP asks for line, so you need to supply type=&quot;l&quot; in your plot function." CreationDate="2011-01-25T14:17:10.200" UserId="2116" />
  <row Id="10205" PostId="6529" Text="@Dmitrij, +1, this is what I was trying to express but was not able to :)" CreationDate="2011-01-25T14:20:10.853" UserId="2116" />
  <row Id="10206" PostId="6527" Text="For completeness, perhaps add `text(2000, 30000, expression(paste(&quot;where xy=&quot;, 10^6)))`?" CreationDate="2011-01-25T14:22:52.960" UserId="8" />
  <row Id="10207" PostId="6475" Text="I suggest Maxwell &amp; Delaney, 2004, Designing Experiments and Analyzing Data as a thorough introduction to ANOVA. Chapter 8 discusses the 3-factor case, especially the different types of interaction: http://books.google.de/books?id=h-bMhmQMifsC&amp;lpg=PA423&amp;ots=mMCqiVQXd-&amp;dq=maxwell%20delaney%202004&amp;pg=PA354#v=onepage&amp;q&amp;f=false." CreationDate="2011-01-25T14:23:32.610" UserId="1909" />
  <row Id="10209" PostId="6527" Text="@mpiktas: Good point ;)" CreationDate="2011-01-25T14:38:33.287" UserId="8" />
  <row Id="10210" PostId="6525" Text="Thanks @mpiktas. My mistake. &#xA;And my intention is not y=10^6/x. I want to make a graph with x &amp; y values and the mark the points(curve) where the values of x &amp; y satisfies (x*y=10^6). Is it possible? Not sure if this is mathematical correct or not. " CreationDate="2011-01-25T14:54:49.643" UserId="1706" />
  <row Id="10211" PostId="6527" Text="But I want to make graph with x &amp; y values and the mark the points(curve) where the values of x &amp; y satisfies (x*y=10^6)" CreationDate="2011-01-25T14:56:26.380" UserId="1706" />
  <row Id="10212" PostId="6528" Text="@mpiktas: true! It is terrible, that I fall into a habbit always using loops :( I edit my answer based on your suggestions." CreationDate="2011-01-25T14:58:31.407" UserId="2714" />
  <row Id="10213" PostId="6527" Text="@Saneef, my comment above is irrelevant now, since you fixed the graph. I will delete it. Now with @csgillespie comment it will be replicated." CreationDate="2011-01-25T15:01:22.300" UserId="2116" />
  <row Id="10214" PostId="6525" Text="For reference: http://en.wikipedia.org/wiki/File:Vector_Video_Standards2.svg I want to plot the line which acts as the boundary for yellow on the lower right (&gt;1M pixels)" CreationDate="2011-01-25T15:02:05.527" UserId="1706" />
  <row Id="10215" PostId="6527" Text="@Saneef: to add this curve to an existing plot, use the option `add=TRUE`. See ?curve." CreationDate="2011-01-25T15:13:03.887" UserId="449" />
  <row Id="10216" PostId="6513" Text="@mpiktas: I did some articles, however they are very implementation-scarce. One I found very instructive is http://www.sciencedirect.com/science?_ob=ArticleURL&amp;_udi=B6V92-3SWYBTY-1&amp;_user=10&amp;_coverDate=06%2F30%2F1997&amp;_rdoc=1&amp;_fmt=high&amp;_orig=search&amp;_origin=search&amp;_sort=d&amp;_docanchor=&amp;view=c&amp;_searchStrId=1619298249&amp;_rerunOrigin=google&amp;_acct=C000050221&amp;_version=1&amp;_urlVersion=0&amp;_userid=10&amp;md5=5f5ef0511a06f6c4e324fdf1faab7b5c&amp;searchtype=a. In this article, they model each hour by a different time series. However, I may not have enough data to do that. " CreationDate="2011-01-25T15:17:40.040" UserId="2889" />
  <row Id="10217" PostId="6513" Text="Also, they got very good results with Regression with autocorrelated errors. I tried to do that with arima() function in R. A lot of the predictors are from that article.&#xA;&#xA;@Rob Hyndman: Thank you for the link, I will look at those articles. Is there an article that you would suggest to start with? I would like it to be as implementation-close as possible.&#xA;&#xA;@probabilityislogic: Yes, my linear model is not succesful probably because the dynamics of my system is not linear. That's why I tried to model the system with arima() - autoregression with autocorrelated errors. " CreationDate="2011-01-25T15:18:40.227" UserId="2889" />
  <row Id="10218" PostId="6505" Text="@Gavin thanks for reminding me, as comparing with stackoverflow, I need to spend more time to &quot;digest&quot; the answer before deciding whether the answer is appropriate or not, anyway, thanks again." CreationDate="2011-01-25T15:32:49.853" UserId="588" />
  <row Id="10219" PostId="6532" Text="Byt the way, for more information about hierarchical agglomerative clustering:&#xA;&#xA;Izenman, Modern Multivariate Statistical Techniques. Springer. 2008." CreationDate="2011-01-25T16:19:25.943" UserId="2902" />
  <row Id="10220" PostId="6506" Text="A really great answer. Thanks. " CreationDate="2011-01-25T16:53:52.607" UserId="1146" />
  <row Id="10221" PostId="6412" Text="No special needs. When you say &quot;simply reimplementing&quot; .. that is work.  I want easy." CreationDate="2011-01-25T17:28:39.597" UserId="2849" />
  <row Id="10222" PostId="4684" Text="It's a little late, but I finally got time to sit down and re-create your argument. The key was &quot;multinomial coefficient&quot;. I had tried figuring it out using plain old binomial coefficients and I was getting all balled up. Thanks again for a nice answer." CreationDate="2011-01-25T17:29:09.907" UserId="1270" />
  <row Id="10223" PostId="6415" Text="Does it report fit statistics well (e.g., p values, variance inflation factors, ...)?" CreationDate="2011-01-25T17:29:44.403" UserId="2849" />
  <row Id="10224" PostId="6497" Text="@mpiktas I am not sure, but bill_080's point might have been that at the very beginning of the article they use n = 6 for their t-distribution. It can be useful to know what people consider reasonable values." CreationDate="2011-01-25T17:57:28.560" UserId="1146" />
  <row Id="10225" PostId="6519" Text="That's defenitly a valid point. We are *ver* reluctant to throw away data." CreationDate="2011-01-25T18:10:01.990" UserId="2907" />
  <row Id="10226" PostId="6538" Text="This should be a community wiki." CreationDate="2011-01-25T19:34:58.903" UserId="2116" />
  <row Id="10227" PostId="4684" Text="@Mike Thank you for getting back to us with your reactions.  I hope this has been helpful." CreationDate="2011-01-25T19:35:58.480" UserId="919" />
  <row Id="10228" PostId="6538" Text="@John, what field of mathematics you received your doctorate? This might be relevant." CreationDate="2011-01-25T19:42:36.740" UserId="2116" />
  <row Id="10229" PostId="6497" Text="@John, you are correct, n=6 was my point.   I tried several schemes that you might use in fitting n for your data and a t distribution (using fitdist from package fitdistrplus), but I couldn't configure a way that I thought was worth posting." CreationDate="2011-01-25T19:47:22.270" UserId="2775" />
  <row Id="10230" PostId="6535" Text="What software are you using? There are different types of 'balance' in ANOVA, and the precise requirements can get rather complicated. Some packages may be able to deal with designs that others can't, but any decent package should give an error, or at least a warning, if the data don't fulfil its requirements." CreationDate="2011-01-25T19:50:35.697" UserId="449" />
  <row Id="10231" PostId="6538" Score="3" Text="@John Could you share with us *why* you want to learn stats?  Curiosity?  Needed for a project or research?  Wanting to change jobs?  Need to teach some courses?  Want to collaborate with statisticians as the theoretical person?" CreationDate="2011-01-25T19:51:04.603" UserId="919" />
  <row Id="10232" PostId="6535" Text="@onestop In the past I have used SAS or SPSS. Currently, I'm trying to transition to mostly using R" CreationDate="2011-01-25T19:54:04.313" UserId="2322" />
  <row Id="10233" PostId="6537" Text="@B R I don't know if it will change your answer, but I have edited the question to be more specific." CreationDate="2011-01-25T19:57:03.717" UserId="2322" />
  <row Id="10234" PostId="6507" Text="@probabilityislogic Although you claim your result holds for &quot;any&quot; set of iid RVs, it is obviously incorrect for any distribution with appreciable chance of being negative (such the standard normal), for then the expectation of the minimum must be negative.  Even worse, if the left tail is heavy enough, the expectation of the minimum must diverge with $N$." CreationDate="2011-01-25T19:58:55.463" UserId="919" />
  <row Id="10235" PostId="1875" Text="This is related to a past question: http://stats.stackexchange.com/q/2275/495" CreationDate="2011-01-25T20:19:06.560" UserId="495" />
  <row Id="10237" PostId="6545" Text="Hi Dan, Good initiative, I'll try to contact you. " CreationDate="2011-01-25T21:07:24.870" UserId="223" />
  <row Id="10238" PostId="5791" Text="@glassy that is a nice simple suggestion. i will look into it. Thanks" CreationDate="2011-01-25T22:01:43.610" UserId="1381" />
  <row Id="10239" PostId="6507" Text="I did not claim my *final results* hold for any iid RVs.  I said the *distribution of the minimum* is of the form I specified.  And the &quot;danger&quot; you speak of (minimum diverging), that clearly does not happen with my answer, it converges to 0 as $N\rightarrow\infty$.  And I am not using the &quot;standard normal&quot; I am using a normal centered at $np$ with which is where $F(np)\approx\frac{1}{2}$ comes from.  The normal approx. to the binomial is good, with poor approximations for extreme p (same as the delta method I have used)." CreationDate="2011-01-25T22:19:31.337" UserId="2392" />
  <row Id="10240" PostId="6415" Text="I don't know logistic regression terribly well, so I cannot give a precise answer. However, I can see that the object implements a predict_proba method: http://scikit-learn.sourceforge.net/modules/generated/scikits.learn.linear_model.LogisticRegression.html#scikits.learn.linear_model.LogisticRegression.predict_proba . With regards to variance inflation factors, I am pretty sure that they are not implemented, though they could be computed as a post-processing step." CreationDate="2011-01-25T22:19:43.877" UserId="1265" />
  <row Id="10241" PostId="6521" Text="@onestop is binomial &quot;model&quot; not problematic, given that the economic indicators are at country level so the same for all the 100 regions?" CreationDate="2011-01-25T22:24:51.270" UserId="1443" />
  <row Id="10242" PostId="6507" Text="The *only* place where the normal approximation comes in is the approximation $F(np)\approx\frac{1}{2}$, so someone dissatisfied with that, just take $Nnp[1-F(np)]^{N-1}$ instead (which uses the Taylor series linear approximation only).  The approximation &quot;goes&quot; in all the &quot;right&quot; directions for the problem - $0$ for $N\rightarrow\infty$ or $np\rightarrow 0$ and $\infty$ for $np\rightarrow \infty$ (for fixed $N$)." CreationDate="2011-01-25T22:34:36.630" UserId="2392" />
  <row Id="10244" PostId="6548" Text="Nice answer, better than all the others, because you have *bounded* the expectation. +1 from me!" CreationDate="2011-01-25T22:44:25.963" UserId="2392" />
  <row Id="10247" PostId="6499" Text="Although the answers below differ in their exact form, they all point to *exponential* in $N$ for fixed $n$ and $p$" CreationDate="2011-01-25T23:12:50.853" UserId="2392" />
  <row Id="10250" PostId="6507" Text="I would conjecture that the approximation $E[min(X)]\approx \frac{N}{2^{N-1}} E[x_1]$ works pretty well for *any non-negative* RV which *is not too skewed* and has *finite variance*.  This is because the delta method any normal approx. are pretty good in these situations." CreationDate="2011-01-25T23:16:31.523" UserId="2392" />
  <row Id="10251" PostId="6539" Text="Dear bill_080, thank you for your elaborate answer. I know that more data would help, but unfortunately I have only 11 months of data. Is it possible to construct a model on some other data and use this model for my data? Then, how should one go about tweaking the model when applying it to the new data? Is that feasible at all? Am I looking in the right direction, or am I just doomed because of too little data that I have in my specific case? Thank you." CreationDate="2011-01-25T23:23:17.250" UserId="2889" />
  <row Id="10254" PostId="6539" Text="@Grega: You're somewhat doomed in that you need more data.  However, the electrical demand data that you need may be easier to find than you think.  For instance, in Texas here's a link: http://www.ercot.com/gridinfo/load/index     What area of the world are you trying to forecast?" CreationDate="2011-01-26T00:12:06.707" UserId="2775" />
  <row Id="10255" PostId="6551" Text="I am aware of te invariance property. My question refered to the fact, that the _likelihood_ at the MLEs does not change. If you for example consider a trivial transform like $x \mapsto 0.1 * x$, then this transform will _reduce_ variance because all the points move _closer_ together. Hence the likelihood will _increase_, because of the factors of the form $1/ \sqrt(2 \pi \sigma^2)$." CreationDate="2011-01-26T00:16:03.593" UserId="2916" />
  <row Id="10256" PostId="6542" Score="1" Text="I would suggest It's a must read for *anybody* who wants to be a good statistician, Frequentist, Bayesian or anything else." CreationDate="2011-01-26T00:16:46.337" UserId="2392" />
  <row Id="10257" PostId="6539" Text="@bill_080: I'm trying to forecast an electricity load of a factory in central Japan. &#xA;&#xA;My question still stays. Is it possible to devise a model on some other data and apply it to my specific case? But wouldn't dynamics of the system be different then?" CreationDate="2011-01-26T00:24:55.000" UserId="2889" />
  <row Id="10258" PostId="6537" Text="It won't! I don't know enough about advanced ANOVA techniques to say if that changes anything. My answer could really be summarized as &quot;There may be ways to overcome these difficulties, but why not just use mixed models?&quot;" CreationDate="2011-01-26T00:49:43.337" UserId="2739" />
  <row Id="10259" PostId="6526" Text="You can get this article for free (although may be a slightly different version) here  http://www.isds.duke.edu/~berger/papers/interplay.pdf .  Okay to give a &quot;free&quot; link, because its on James Berger's Website." CreationDate="2011-01-26T01:05:58.543" UserId="2392" />
  <row Id="10260" PostId="6551" Score="1" Text="Nope, the likelihood will remain *exactly the same* because a jacobian factor will come out of the transformation to make them equal (dx to 0.1 dx)" CreationDate="2011-01-26T01:09:33.093" UserId="2392" />
  <row Id="10262" PostId="6557" Text="Could you show an example image from an other package/software or give a more detailed description what you want to plot?" CreationDate="2011-01-26T01:38:14.350" UserId="2714" />
  <row Id="10263" PostId="5454" Score="1" Text="Just don't read the ggplot2 source. It will make your brain bleed" CreationDate="2011-01-26T02:01:58.727" UserId="46" />
  <row Id="10264" PostId="5419" Score="1" Text="I wouldn't recommend reshape - even I don't understand how it works. Reshape2 is much much better and follows many more good development principles. " CreationDate="2011-01-26T02:03:14.717" UserId="46" />
  <row Id="10265" PostId="6558" Text="Just one more thought, I don't know if it's obvious from my initial post. I am not predicting many days (or months/years) in advance. I am trying to predict maximum 16 hours in advance (with 30 minutes sampling, this means 32 points). Does the approach change with that fact?" CreationDate="2011-01-26T02:49:02.787" UserId="2889" />
  <row Id="10266" PostId="6554" Text="Have you read the manual and worked through the tutorial?  http://www.mrc-bsu.cam.ac.uk/bugs/winbugs/manual14.pdf" CreationDate="2011-01-26T03:36:38.900" UserId="919" />
  <row Id="10267" PostId="6558" Text="@Grega, I added an answer to your comment as an edit in the above answer." CreationDate="2011-01-26T04:21:53.023" UserId="2775" />
  <row Id="10268" PostId="6559" Text="@Zach, your answer indicates that your question is more about R implementation, than about statistics. So you better ask it at stackoverflow.com. " CreationDate="2011-01-26T05:07:11.187" UserId="2116" />
  <row Id="10269" PostId="6508" Text="Wow! Thanks for finding a viewable reference—I hadn't thought to look on Google Books when looking for a copy of Wooldridge's book." CreationDate="2011-01-26T05:15:48.880" UserId="1583" />
  <row Id="10270" PostId="6559" Text="@Zach, what is the purpose of parseCall function? In function getDepVar you call parseCall and then access formula object from parseCall result, why not accessing it directly from the call? When you have formula object you can get the response name using the all.vars function, try all.vars(formula(y~x)[[2]])." CreationDate="2011-01-26T05:19:39.200" UserId="2116" />
  <row Id="10271" PostId="6559" Score="1" Text="@Zach, you probably want lag(y,-1), not lag(y,1). Try cbind(y,lag(y,-1),lag(y,1)) to see the difference." CreationDate="2011-01-26T05:20:29.997" UserId="2116" />
  <row Id="10272" PostId="6559" Score="1" Text="@Zach, and final comment. R has a powerful formula interface, use it.  Look at the code of lm, and functions model.matrix, model.frame, model.response. Using strsplit and parse on formulas and call objects is a bit of trying to fit a square peg into the round hole." CreationDate="2011-01-26T05:22:42.520" UserId="2116" />
  <row Id="10273" PostId="6539" Text="@bill_080, +1 for detailed and insightfull answer" CreationDate="2011-01-26T06:51:28.617" UserId="2116" />
  <row Id="10274" PostId="6507" Text="@probabilitiyislogic, approximating $Eg(X)$ by $g(EX)$ in general will not work. Pick zero mean variable and $g(x)=x^2$, then $Eg(X)=var(X)$, but $g(EX)=0$. Pick centered random variable without second moment and you will get that $Eg(X)$ does not exist, but $g(EX)$ will still be zero. So if you want to approximate $Eg(X)$ by $g(EX)$ you should be very careful to check properties of $g$ and $X$." CreationDate="2011-01-26T06:58:05.910" UserId="2116" />
  <row Id="10275" PostId="6565" Text="&quot;additional binary dummy&quot;, meaning I need to create another variable, whereby 1=with loan &amp; 0=no loan? At the same time, still putting in the original Loan variable? If so, 80% of the cases will be treated as missing &amp; only 20% left for analysis. 20% of total sample is too little for me to run logistic regression.  (sorry, i'm still in the learning stage)" CreationDate="2011-01-26T07:01:08.287" UserId="2793" />
  <row Id="10276" PostId="6548" Text="@Didier, +1 for nice proof. It should be noted however, that for $c^N$ to dominate term $(n-1)d^N$ by a factor of $m$, $N$ should be larger than $\frac{\log m(n-1)}{-\log d}$. For binomial distribution with $n=100$ and $p=0.1$, and the factor $m=10$ this works out at $N\approx 3500$. So this estimate should be used carefully for not so large $N$. On the other hand you can use estimate $c^N+(n-1)d^N$ directly in that case." CreationDate="2011-01-26T07:32:05.587" UserId="2116" />
  <row Id="10277" PostId="6561" Score="1" Text="@daroczig, you can get predictions using predict.&#xA;dat[,&quot;y0&quot;]&lt;-predict(res,newdata=data.frame(x1=dat[,&quot;x1&quot;],x2=0))&#xA;Saves a bit of typing." CreationDate="2011-01-26T07:38:14.097" UserId="2116" />
  <row Id="10278" PostId="6562" Text="@lokheart, why haven't you tried it yourself? This of course works. There is a typo in your code, capital C instead of c." CreationDate="2011-01-26T07:45:15.153" UserId="2116" />
  <row Id="10279" PostId="6548" Text="@mpiktas Thanks for your comment. It seems you forgot $c$ in your estimation of the minimal $N$. Note that the OP asked for the *scale* of $E(X_N)$, meaning (presumably) its asymptotics when $N\to+\infty$. Re an actual estimation in the binomial case and for moderately large values of $N$, I would recommend going one step further, that is, using $c^N+d^N\le E(X_N)\le c^N+d^N+(n-2)f^N$ with $f=P(x_1\ge 3)$." CreationDate="2011-01-26T08:16:17.550" UserId="2592" />
  <row Id="10280" PostId="6552" Text="@bill_080 You need to shuffle `listfil` somewhere in your code." CreationDate="2011-01-26T08:17:49.213" UserId="930" />
  <row Id="10281" PostId="6514" Text="Actually, there is multicollinearity between the two variables. I'm not sure how I would be able to construct a latent variable crossing Valence and Self-relevance especially when I did a correlation, it was significant at r=.755. Thanks for any input!" CreationDate="2011-01-26T08:44:36.677" UserId="2934" />
  <row Id="10282" PostId="6562" Text="I tried already, but since I have no idea of how to counter-check the result, so I wonder if anyone can enlighten me here, thanks anyway." CreationDate="2011-01-26T08:54:14.377" UserId="588" />
  <row Id="10283" PostId="6548" Text="@Didier, $m$ accounts for $c$, but it is not that important. " CreationDate="2011-01-26T09:03:21.967" UserId="2116" />
  <row Id="10284" PostId="6565" Score="1" Text="They won't be treated as missing, they'll go into estimating the value for no loan.  Maybe you've made no loan 'NA' in which case you need to recode those to 0." CreationDate="2011-01-26T09:10:07.450" UserId="601" />
  <row Id="10285" PostId="6558" Text="@bill_080: sorry nothing related to the original question, but do you know anything about the australian electricity market? Seams quite stable, especially with all their coal easily available?" CreationDate="2011-01-26T09:57:23.623" UserId="1709" />
  <row Id="10286" PostId="6561" Text="@mpiktas: thank you, I did not know about `predict`, but seems useful." CreationDate="2011-01-26T10:12:06.087" UserId="2714" />
  <row Id="10288" PostId="6523" Text="Thanks. That's a nice article. It's a start at what I have in mind, but as you say it's very brief." CreationDate="2011-01-26T13:52:40.467" UserId="319" />
  <row Id="10289" PostId="6526" Text="I look forward to reading that one. Jim Berger has deep insights." CreationDate="2011-01-26T13:53:47.753" UserId="319" />
  <row Id="10292" PostId="6507" Text="@mpiktas- For the first example, how can a *non-negative* random variable have a mean of 0?  Only if it is a constant (=0), then the approx. is exact!  and I said *finite variance* case, which rules out the other example.  I did say when the approximation was good.  Find an example which is in the class I specified in the comment above (non-negative, finite variance, low skewness).  And besides, the approximation works well *in the particular case being described*." CreationDate="2011-01-26T14:07:36.507" UserId="2392" />
  <row Id="10295" PostId="6565" Text="@John Thank you, that's exactly what I am recommending.  The point is to express the loan values ($X$) in any way appropriate (such as log(amount+1)) and set $X=0$ and $I=1$ for any case without a loan.  This is a standard technique in regression, including logistic regression." CreationDate="2011-01-26T15:10:15.423" UserId="919" />
  <row Id="10296" PostId="6570" Score="2" Text="This is not a professional opinion, so I'm putting it in a comment.  As a responder to questionnaires, I always wonder about the validity of results based on questions where no answer corresponds to my opinion.  If you force me to answer arbitrarily, you will get arbitrary results.  If many respondents have the same problem with a question, the *collective* response will be arbitrary, too.  (This can lead to awful decision making if you have no independent way of checking that these results are meaningless.)" CreationDate="2011-01-26T15:16:31.310" UserId="919" />
  <row Id="10297" PostId="6572" Text="@KrishKalyan - are you saying that four options is better for a **new product**. Can you elaborate - why do you say that?" CreationDate="2011-01-26T15:37:52.803" UserId="1259" />
  <row Id="10298" PostId="6572" Text="The new product is in its initial stage and so there may not be time enough to form an opinion; four options are not good for this case." CreationDate="2011-01-26T15:53:08.647" UserId="2926" />
  <row Id="10299" PostId="6551" Text="I am not talking about the likelihood of $x$ under the new distribution obtained by plugging in the transformation, but about the likelihood of $t(x)$! That _is_ something different." CreationDate="2011-01-26T15:59:26.873" UserId="2916" />
  <row Id="10300" PostId="6570" Text="As is I don't think this is a question of statistics. If the question were changed to how do you assess the reliability between the two separate options, that will be more within the domain of the site. Any statement treating one or the other as correct will be based on domain specific knowledge." CreationDate="2011-01-26T16:01:32.813" UserId="1036" />
  <row Id="10301" PostId="6558" Text="@fRed: I don't know much about Australia.   However, I do know that the power market is changing all over the world.  Coal is cheap and stable, but the CO2 guys are after them.   Combined Cycle plants (gas fired turbines with waste heat recovery) can now achieve fairly high efficiencies.  Things are changing fast." CreationDate="2011-01-26T16:06:59.730" UserId="2775" />
  <row Id="10303" PostId="6552" Text="@chl: I assumed that getting the filenames into a variable was his main question." CreationDate="2011-01-26T16:15:42.523" UserId="2775" />
  <row Id="10304" PostId="6570" Score="2" Text="I think OK is very different from No opinion in most situations. If you want to include &quot;no opinion&quot; then put it as a 6th option (or remove the OK part)" CreationDate="2011-01-26T16:19:01.680" UserId="582" />
  <row Id="10308" PostId="6485" Score="1" Text="Really - deserves far more votes than it has." CreationDate="2011-01-26T17:21:08.997" UserId="71" />
  <row Id="10309" PostId="6548" Text="@mpiktas I cannot reconcile your &quot;$m$ accounts for $c$&quot; with your &quot;for $c^N$ to dominate term $(n-1)d^N$ by a factor of $m$&quot;, which, for me, asks that $c^N\ge m(n-1)d^N$, that is, that $N\ge N(m)$ with $N(m)=\log(m(n-1))/\log(c/d)$. But, as you say, *it is not that important*." CreationDate="2011-01-26T17:24:43.973" UserId="2592" />
  <row Id="10310" PostId="6552" Text="@bill_080 Yes, I just re-read the question and it makes sense as filenames are assumed to follow no particular order. " CreationDate="2011-01-26T18:33:13.260" UserId="930" />
  <row Id="10311" PostId="6580" Text="Hey @tomek, do you mean parameter $b_p$ or parameter estimate $\hat{b_p}$?" CreationDate="2011-01-26T18:34:17.973" UserId="1307" />
  <row Id="10312" PostId="6582" Text="What do you mean by 'follow-up comparisons'? Looking at the same three-way interaction in other data sets?" CreationDate="2011-01-26T19:03:15.757" UserId="449" />
  <row Id="10313" PostId="6580" Text="@suncoolsu: I menat parameter estimate $\hat{b_{p}}$" CreationDate="2011-01-26T19:05:01.067" UserId="1643" />
  <row Id="10314" PostId="6573" Text="@bernd thanks for the explanation. If I understand correctly, what I was missing is that my my priors needed hyperpriors?" CreationDate="2011-01-26T19:30:45.493" UserId="2750" />
  <row Id="10315" PostId="6573" Text="@Abe Your BUGS code was syntactically wrong and you didn't define any priors." CreationDate="2011-01-26T19:38:07.433" UserId="307" />
  <row Id="10316" PostId="6573" Text="@Bernd I see that the syntax was wrong, but is it not possible to have a single $N(\mu,\sigma)$ prior? or do $\mu$ and $\sigma$ need priors? " CreationDate="2011-01-26T19:59:53.930" UserId="2750" />
  <row Id="10317" PostId="6573" Text="@Abe What do you mean by &quot;single $N(\mu, \sigma^2)$ prior&quot;? Assuming that you want to estimate both parameters, $\mu$ and $\sigma^2$, one has to assign appropriate prior distributions, e.g. dnorm and dgamma. " CreationDate="2011-01-26T20:08:34.620" UserId="307" />
  <row Id="10318" PostId="6591" Text="Oh, I see. ZTNB seems viable. Or you could model visits *including* unobserved non-purchases, as you originally noted. That would basically be a free parameter, but if Sichel is describing some other aspect of the underlying behavior, it might be worth it. " CreationDate="2011-01-26T20:45:23.153" UserId="6" />
  <row Id="10319" PostId="6591" Text="Incidentally, you should have responded to my answer, not in a separate answer. And if you like the answer, please up-vote it, and if it actually answers the question to your satisfaction, accept it. (Likewise, your follow-up should have been a comment on your question, not an answer.) Welcome to Stack Exchange! :)" CreationDate="2011-01-26T20:46:39.953" UserId="6" />
  <row Id="10321" PostId="6582" Text="@Joanna I have merged your account, but you should rather register it -- this way it won't get lost. You can do it here: http://stats.stackexchange.com/users/login" CreationDate="2011-01-26T20:59:35.617" UserId="88" />
  <row Id="10322" PostId="6591" Text="Oops--sorry. Thanks for the tips!" CreationDate="2011-01-26T21:06:28.573" UserId="2730" />
  <row Id="10323" PostId="6573" Text="@Bernd sorry. I think I lost this in the translation between paper and code." CreationDate="2011-01-26T22:15:08.003" UserId="2750" />
  <row Id="10324" PostId="6593" Text="@Tal, I don't use `rpart` and I will leave the answer to more experienced members of this forum." CreationDate="2011-01-26T22:19:48.603" UserId="1307" />
  <row Id="10325" PostId="6595" Text="@mbq is there another method that you would reccomend?" CreationDate="2011-01-26T22:22:21.473" UserId="2750" />
  <row Id="10326" PostId="6595" Text="@Abe Well, this depends on what do you want to do..." CreationDate="2011-01-26T22:26:23.420" UserId="88" />
  <row Id="10327" PostId="6551" Text="Okay, here's how it goes, start with CDF $F_{T}(t_0)=Pr(T&lt;t_0)$, then we know $T=t(X)$ sub this in to get $F_{T}(t_0)=Pr(t(X)&lt;t_0)=Pr(X&lt;t^{-1}(t_0))=F_{X}(t^{-1}(t_0))$  To get the pdf you have to differentiate *with respect to $t_0$, which leaves us with $f_{T}(t_0)=f_{X}(t^{-1}(t_0)) \times |\frac{\partial t^{-1}(t_0)}{\partial t_0}|$.  It is this Jacobian I was taking about, which makes the &quot;maximum&quot; of the likelihood equal, whenever $t(x)$ is a 1-to-1 function." CreationDate="2011-01-26T23:29:42.133" UserId="2392" />
  <row Id="10328" PostId="6593" Text="I think I've got the idea...&#xA;But rpart prints deviance even for regression trees O.o&#xA;" CreationDate="2011-01-26T23:46:28.543" UserId="2902" />
  <row Id="10329" PostId="6551" Text="...In your example we have $T=0.1X$ so $t(x)=0.1x$ and $t^{-1}(x)=10x$ with derivative 10, and $f_{T}(t)=f_{X}(10t)10$  Which, in the normal case you will have $f_{T}(t)=\frac{10}{\sigma \sqrt{2\pi}}exp(-\frac{1}{2\sigma^2}(10t-\mu)^2)$.  So it would appear you're right.  My thinking now, is that in fixing the variance, you are also fixing the Jacobian to be equal to 1 for certain functions $t(.)$ (i.e. functions who's inverse have derivative 1).  This may explain why it doesn't always work." CreationDate="2011-01-26T23:50:11.723" UserId="2392" />
  <row Id="10330" PostId="6555" Text="Its not a slow thing to do it this way.  In excel it wouldn't take more than 5 minutes (and you wouldn't need to manually work out the days, just use a date format, and it will have a &quot;number&quot; hiding behind it, which you can use in the calculations).  And you should clarify in the question precisely what it is that you want (which I believe you have done, and thank you for doing it).  Otherwise you get people like me telling you something you already know. :)" CreationDate="2011-01-27T00:31:30.360" UserId="2392" />
  <row Id="10331" PostId="6538" Text="It would be useful in my work which involves a heavy amount of machine learning. It would also be useful at times as my (very small) company occasionally could use a genuine statistician." CreationDate="2011-01-27T00:58:41.247" UserId="2912" />
  <row Id="10332" PostId="6565" Text="Thanks! I get your point. Anyway, is imputation work in my case?" CreationDate="2011-01-27T02:54:29.613" UserId="2793" />
  <row Id="10333" PostId="6565" Text="@lcl23 If I understood the situation correctly, imputation makes no sense: your &quot;missing&quot; data aren't missing; they indicate no loan has been taken out." CreationDate="2011-01-27T03:44:40.800" UserId="919" />
  <row Id="10334" PostId="6555" Score="1" Text="You *cannot* use the google function (FV (..,..,)) to update, because it requires *evenly spaced* intervals, and *equal payments*, both of which you do not have (at least not in the example).  So you could &quot;in theory&quot; apply the function individually, but it is just as quick to &quot;reinvent the wheel&quot; (especially as you don't understand the syntax of the FV function).  If you *know* how to calculate FV, then you don't *need* a special function to do it for you, unless it would save you a considerable amount of time." CreationDate="2011-01-27T04:21:17.337" UserId="2392" />
  <row Id="10335" PostId="6564" Text="Ohmigosh, you are so right!  I'm embarrassed I missed this.  Thank you so much for your help: it is much appreciated." CreationDate="2011-01-27T05:46:37.403" UserId="2921" />
  <row Id="10336" PostId="6568" Text="Measure theory for dummies - that sounds like its written at the right level for me, I'll definitely check it out. Thanks!" CreationDate="2011-01-27T06:19:56.390" UserId="1913" />
  <row Id="10337" PostId="6558" Text="@bill_080: yes very interesting topic" CreationDate="2011-01-27T06:27:52.947" UserId="1709" />
  <row Id="10338" PostId="6605" Score="3" Text="@shabbychef: Most of the people grasp it in the _worst_ possible way i.e. probability of making Type I error." CreationDate="2011-01-27T07:19:05.397" UserId="1307" />
  <row Id="10339" PostId="6564" Text="@D.W. There's no cause for embarrassment.  In fact, your question was worded in an unusually clear manner." CreationDate="2011-01-27T07:24:57.213" UserId="919" />
  <row Id="10340" PostId="6605" Score="2" Text="I think that's mostly related to how p-values are explained in classes (i.e.: just by giving a quick definition and without specifying what p-values are NOT)" CreationDate="2011-01-27T07:34:43.957" UserId="582" />
  <row Id="10341" PostId="6602" Text="@whuber: As usual great answer, I was looking exactly for this." CreationDate="2011-01-27T08:43:49.560" UserId="1643" />
  <row Id="10342" PostId="6585" Text="@onestep: This is very good answer to the problem described in the motivation, I haven't heard about it earlier and it seems to be really useful. Thanks!" CreationDate="2011-01-27T08:45:28.687" UserId="1643" />
  <row Id="10343" PostId="6593" Text="@deps_stats that deviance is node residual sums of squares summed over the terminal nodes of the tree." CreationDate="2011-01-27T08:48:07.547" UserId="1390" />
  <row Id="10344" PostId="6601" Text="(I don't know enough to put this as an answer, hence adding a comment.) I always thought it was strange that PI crops up in statistical equations. I mean - what's PI got to do with statistics? :)" CreationDate="2011-01-27T09:16:32.087" UserId="1259" />
  <row Id="10345" PostId="6600" Text="Thank you for your comments but I am not sure whether or not I understand. Perhaps the best thing to ask is how to determine the significance of my interactions in my mixed model other than just looking at the t-statistic. " CreationDate="2011-01-27T09:35:32.247" UserId="2934" />
  <row Id="10346" PostId="6608" Text="@Gavin (+1) Sorry, my post came later and I didn't notice yours. As I think they don't overlap too much, I will leave mine if you don't mind." CreationDate="2011-01-27T10:02:42.947" UserId="930" />
  <row Id="10347" PostId="6589" Text="Rotation is not limited to FA models: You can use it on any loadings or pattern matrix, provided it makes sense. In fact, there is a `promax()` function in base R which takes as input the loadings matrix, and other oblique rotations methods can be found in the [GPArotation](http://cran.r-project.org/web/packages/GPArotation/index.html) package. This is basically what `principal()` from the [psych](http://cran.r-project.org/web/packages/psych/index.html) package uses." CreationDate="2011-01-27T10:56:41.070" UserId="930" />
  <row Id="10348" PostId="6601" Score="2" Text="I'd agree (In my surprisal) - I think its that $\pi$ pops up in many mathematical analysis.  Just a note you can write $\pi$ by with Latex commands as $\text{\pi}$ enclosed within $ signs.  I use the wiki page to get the syntax http://en.wikibooks.org/wiki/LaTeX/Mathematics . Another trick is to &quot;right click&quot; on an equation you see on this site, and select &quot;show source&quot; to get the commands that were used." CreationDate="2011-01-27T10:59:43.320" UserId="2392" />
  <row Id="10349" PostId="6605" Text="I think this is mainly to do with how it is introduced.  For me, it was an &quot;add-on&quot; to the classical hypothesis test - so it appears as though its just another way to do a hypothesis test.  The other problem is that it is usually only taught with respect to a normal distribution, where everything &quot;works nice&quot; (e.g. p-value *is* a measure of evidence in testing a normal mean).  Generalising the p-value is not easy as there is no specific principles to guide the generalisation (e.g. there is no general agreement on how a p-value should vary with the sample size &amp; multiple comparisons)" CreationDate="2011-01-27T11:12:31.860" UserId="2392" />
  <row Id="10350" PostId="6602" Text="@whuber: The book that You mentioned is available for free in google books, well at least most of it. Nevertheless, have You heard about differentiation with respect to a single attribute? I'm asking because I was 'playing' with matrices lately and I have found that it's quite simple to calculate and I was courious if that is anything new..." CreationDate="2011-01-27T11:18:24.740" UserId="1643" />
  <row Id="10351" PostId="6606" Score="1" Text="The more I think about confidence intervals, the harder it is for me to think of what kind of question they can answer at a conceptual level that cannot be answered by asking for &quot;the chance a true value is within an interval, given one's state of knowledge&quot;.  If I were to ask &quot;what is the chance (conditional on my information) that the average income in 2010 was between 10,000 and 50,000?&quot; I don't think the theory of confidence intervals can give an answer to this question." CreationDate="2011-01-27T11:28:22.983" UserId="2392" />
  <row Id="10352" PostId="6610" Text="+1 nice answer chl, which, as you say, complements mine so no problems there. Very nicely put." CreationDate="2011-01-27T11:54:55.350" UserId="1390" />
  <row Id="10353" PostId="6601" Text="@Wiki If you accept that $\pi$ crops up when you go from measuring the length of a straigh piece of line to the length of a piece of circle, I don't see why it would not appear while going from measuring a probability to fall down on a segment to measuring the probability to fall down in a piece of circle ? " CreationDate="2011-01-27T12:19:01.487" UserId="223" />
  <row Id="10354" PostId="6605" Text="@shabbychef +1 though student often have difficulties with p-values (roughly because the concept in testing is a bit more subtle than a binary decision process and be cause &quot;inverting a function&quot; is not easy to aprehend). When you say  &quot;for some reason&quot; do you mean it is unclear for you why people have difficulties ? PS: If I could, I would try to make statistics on this site about the relation between &quot;being a top answer&quot; and &quot;talking about p-value&quot; :) . I also even ask myself if the hardest statistical concept to grasp can have the most upvote (if it is difficult to grasp ... :) )  " CreationDate="2011-01-27T12:28:33.483" UserId="223" />
  <row Id="10355" PostId="6609" Score="2" Text="I don't use SPSS, so I can't help with that, but I think you want to at least explore an ordinal logistic model.  If the assumptions are met, you will have a much simpler model to deal with." CreationDate="2011-01-27T13:18:58.457" UserId="686" />
  <row Id="10356" PostId="2369" Text="The question as posed is a bit ambiguous, because it does not stated clearly what *information* the 100 people have.  Do they know the distribution in the bag? for if they do, they &quot;experiment&quot; is useless, one would just give the interval $[0.1,0.5]$ or even just the two values $0.1$ and $0.5$ (does give required $\text{100%} \geq \text{95%}$ coverage).  If we only know that there are a bag of coins to be drawn from, the Bayesian would specify the whole [0,1] interval, because false positives is *all* that matters in this question (and the *size* of the interval does not)." CreationDate="2011-01-27T13:24:03.727" UserId="2392" />
  <row Id="10357" PostId="6615" Text="Looks good, except (of course) we would need to scale(x1) and scale(x2) first." CreationDate="2011-01-27T13:54:31.767" UserId="196" />
  <row Id="10358" PostId="6619" Text="Thanks for the &quot;TeXation&quot; mpiktas :))" CreationDate="2011-01-27T13:55:02.123" UserId="1833" />
  <row Id="10359" PostId="6619" Score="1" Text="@Queops, your are welcome. Note that right clicking on formula will present you pop-up menu with a choice to see the source." CreationDate="2011-01-27T13:57:49.877" UserId="2116" />
  <row Id="10360" PostId="6620" Text="You confirm the answer is aprox. 0.735 ? Seems the problem solution might be wrong." CreationDate="2011-01-27T14:03:36.170" UserId="1833" />
  <row Id="10361" PostId="2369" Text="I would have thought the above argument holds just as valid for the frequentist as well.  The argument above (as far as I can tell) does not invoke any specifically Bayesian or Frequentist principles (although it does invoke the principle of *sanity*)." CreationDate="2011-01-27T14:04:16.780" UserId="2392" />
  <row Id="10362" PostId="6620" Text="@Queops, R code `pbinom(5,20,1/5)-pbinom(1,20,1/5)` gives 0.7350325, so yes. " CreationDate="2011-01-27T14:07:37.510" UserId="2116" />
  <row Id="10363" PostId="6620" Text="@Queops, maybe the problem is to calculate the probability that exactly questions numbered 2 to 5 are answered correctly? Then probability should be calculated differently." CreationDate="2011-01-27T14:08:55.607" UserId="2116" />
  <row Id="10365" PostId="6620" Text="No no, it's just the number, doesn't matter which ones. And here I was trying to figure out what was wrong. Thanks so much for the help." CreationDate="2011-01-27T14:10:56.393" UserId="1833" />
  <row Id="10366" PostId="6619" Text="@mpiktas: Nice to know that it is possible to see the source." CreationDate="2011-01-27T14:11:58.943" UserId="1643" />
  <row Id="10367" PostId="6613" Text="@Gireesh, in its present form the question is unanswerable, since it is not known what failed. One advice I can give is to reduce your data set and try again. Also look for NA values." CreationDate="2011-01-27T14:21:10.387" UserId="2116" />
  <row Id="10368" PostId="6613" Text="Is there a word missing in the note at the end? &quot;1st column is ... and first row are names&quot;" CreationDate="2011-01-27T14:26:50.437" UserId="449" />
  <row Id="10370" PostId="6615" Text="@drknexus Yes, of course (in my initial tests, I used standardized N(0;1) variates, instead of yours)." CreationDate="2011-01-27T14:36:37.733" UserId="930" />
  <row Id="10371" PostId="6619" Text="@Queops If you feel @mpiktas's answer helped you to solve your problem, then the best way to thank him is probably to upvote his response." CreationDate="2011-01-27T14:41:10.147" UserId="930" />
  <row Id="10372" PostId="6561" Text="I'd alway recommend using predict instead of calculating the slopes yourself - it's much simpler especially when you have interactions or non-linear components." CreationDate="2011-01-27T14:55:26.817" UserId="46" />
  <row Id="10373" PostId="6610" Text="One minor thing - probably not best to use `lm.fit` in your example, as this is the name of the workhorse behind `lm`." CreationDate="2011-01-27T14:57:31.730" UserId="1390" />
  <row Id="10374" PostId="6610" Text="@Gavin Oh, yes, that's right. Thx for reminding me that, I often make this mistake..." CreationDate="2011-01-27T15:03:25.750" UserId="930" />
  <row Id="10375" PostId="6621" Text="Ack, sorry, I totally messed up the shape of my data when constructing the fake example (the real data is HIPAA-sensitive).  My true observations aren't paired, there are a different number of x and y observations in each group.  Maybe I should kill this question and start a new one." CreationDate="2011-01-27T15:11:53.430" UserId="1434" />
  <row Id="10376" PostId="6622" Text="Since the error states `too many elements specified`, rather than `cannot allocate vector`, it seems that its an address space issue rather than a memory issue (though this would follow). This message kicks in if you attempt to allocate more than $2^31-1$ elements. While this doesn't seem to be the case here, there may be more elements in use than specified in the question above." CreationDate="2011-01-27T15:42:00.453" UserId="229" />
  <row Id="10378" PostId="6500" Text="Thanks very much, Rob." CreationDate="2011-01-27T16:02:00.733" UserId="2669" />
  <row Id="10379" PostId="6503" Text="Thank you, Andy." CreationDate="2011-01-27T16:02:58.563" UserId="2669" />
  <row Id="10380" PostId="2281" Text="@svadalli - the Bayesian approach does not take the view that $\theta$ *is random*.  It is not $\theta$ that is distributed ($\theta$ is fixed but unknown), it is the *uncertainty about* $\theta$ *which is distributed, conditional on a state of knowledge about* $\theta$.  The actual probability statement that $f(\theta)$ is capturing is $Pr(\theta\text{ is in the interval } (\theta,\theta+d\theta)|I)=f(\theta)d\theta$.  In fact, the exact same argument applies to $X$, it too can be considered fixed, but unknown." CreationDate="2011-01-27T16:14:56.637" UserId="2392" />
  <row Id="10381" PostId="6602" Text="@Tomek Differentiation w.r.t. a single attribute ought to give its coefficient (when there are no interactions).  Of course, since the solutions are functions of all the data, you could differentiate w.r.t any particular $x_{ij}$.  It should be straightforward to work out.  (In fact, finding the solution for OLS would be sufficiently general.)  I haven't read B,K,&amp; W in 20 years, so my memory is dim, but I don't recall that they went down to this level of detail and they definitely don't do this computation in the section I quoted." CreationDate="2011-01-27T16:22:42.387" UserId="919" />
  <row Id="10382" PostId="2373" Text="The issue with confidence intervals in repeated experiments, is that in order for them to work, the conditions of the repeatable experiment need to stay the same (and who would believe that?), whereas the Bayesian interval (if used properly) conditions on the data observed, and thus provides allowances for changes which occur in the real world (via data).  I think it is the *conditioning rules* of Bayesian statistics which make it so hard to outperform (I think it is impossible: only equivalence can be achieved), and the automatic machinery which it achieves this that make it seem so slick." CreationDate="2011-01-27T16:40:03.073" UserId="2392" />
  <row Id="10383" PostId="2372" Text="@svadali -  confidence intervals evaluate *data* for a fixed hypothesis.  Thus when changing the &quot;fixed&quot; part of the equation, if you fail to take account of the probability of the hypothesis prior to observing your data, then you are bound to come up with inconsistencies and incoherent results.  Conditional probability is not &quot;constrained&quot; when changing the conditions (e.g. by changing the conditions you can change a conditional probability from 0 to 1). The prior probability takes account of this arbitrariness. Conditioning on X is done because we are certain X has occured-we did observe X!" CreationDate="2011-01-27T17:11:49.710" UserId="2392" />
  <row Id="10384" PostId="6610" Text="+1 great answer!" CreationDate="2011-01-27T17:17:35.603" UserId="1307" />
  <row Id="10385" PostId="6611" Text="This solution requires no less calculation than the one I gave (unless $e^{rt}$ is less calculation than $(1+r)^{t_0-t}$, I hardly think so).  Another thing is that the interest rate in the PV and FV calculations are not consistent in *r* for in one place you are annually compounding, and another place you are continuously compounding.  your equations imply that $(1+r)^{t}=e^{rt}$.  $r$ in the exponential should be adjusted to $log(1+r)$." CreationDate="2011-01-27T17:25:36.903" UserId="2392" />
  <row Id="10386" PostId="6618" Score="3" Text="I think John Tukey might disagree http://en.wikipedia.org/wiki/Exploratory_data_analysis ;o)" CreationDate="2011-01-27T17:32:28.670" UserId="887" />
  <row Id="10387" PostId="6611" Text="@hhh - how does the continuous compounding give *proper bounds* when there are both positive and negative terms?  I understand how it does when there are only positive terms. But including negative terms means you have *minimum negative* (max absolute) plus *maximum positive* cash flows, how does this put a &quot;bound&quot; on the sum?" CreationDate="2011-01-27T17:36:25.693" UserId="2392" />
  <row Id="10388" PostId="6596" Text="@Rob Thank you for your answer, but I am confused- you say 'estimate the spike using the proportion of zeros' but plot it with no bounds. does the spike have a discrete height or is it infinite, if discrete, is it $P(X=0)$?" CreationDate="2011-01-27T17:45:54.737" UserId="2750" />
  <row Id="10389" PostId="6611" Text="@hhh - another question, in your answer, why did you use an example which was different to the example data given in the question?  &quot;100EUR with 5% after 3 years&quot; is an odd example, given that there is *already example data in the question*.  I think if you try to use your method on the example data, and then mine (using the same degree of accuracy in time, whole years in both, decimal years in both,etc.) you will find that the take the same time to do, except that my method gives the exact FV, yours an approximate one." CreationDate="2011-01-27T17:54:30.843" UserId="2392" />
  <row Id="10390" PostId="6622" Text="Oh, good point." CreationDate="2011-01-27T17:58:30.450" UserId="46" />
  <row Id="10391" PostId="6618" Score="2" Text="I would partially disagree here.  I think the caveat that people miss is that *the appropriate conditioning operations are easy to ignore* for these kinds of issues.  Each of these operations change the conditions of the inference, and hence, they change the conditions of it applicability (and therefore to its generality).  These is definitely only applicable to &quot;confirmatory analysis&quot;, where a well defined model and question have been constructed.  In exploratory phase, not looking to answer definite questions - more looking to build a model and come up with hypothesis for the data." CreationDate="2011-01-27T18:05:20.880" UserId="2392" />
  <row Id="10392" PostId="6610" Score="1" Text="Just one remark: For linear regression Deviance is equal to the RSS because the normality assumption of errors implies RSS is same as LR test statistic which further implies Deviance is normally distributed, _irrespective of asymptotics_. This is just an expansion of (I think) terse comment of chl. " CreationDate="2011-01-27T18:11:46.980" UserId="1307" />
  <row Id="10393" PostId="6621" Text="I've gone back &amp; revised the data example here to make it unpaired." CreationDate="2011-01-27T19:54:25.950" UserId="1434" />
  <row Id="10394" PostId="2356" Text="I think, I have seen such an example in the book named &quot;All of Statistics&quot; by Larry Wasserman in which he gives an example where using Bayesian CI is _not_ the sensible thing to do. However, it is a pathological example." CreationDate="2011-01-27T20:00:06.043" UserId="1307" />
  <row Id="10395" PostId="6610" Text="@suncoolsu If it is about my comment about ML and OLS estimates, yes I meant &quot;assuming a gaussian distribution&quot; for the $\varepsilon_i$ (i.e., in linear models). Your comment is welcome." CreationDate="2011-01-27T20:39:07.633" UserId="930" />
  <row Id="10396" PostId="6642" Score="3" Text="Whether $E(X_1)=\pm E(Y_1)$ or not would yield quite different results. And a hypothesis more precise than &quot;$X_i$ and $Y_i$ are dependent&quot; might be needed. (Duplicate from math.stackexchange... You might want to choose one of the two sites and stick to it.)" CreationDate="2011-01-27T21:38:57.457" UserId="2592" />
  <row Id="10397" PostId="6596" Text="This is a mixture of a discrete distribution and a continuous distribution. When plotted as a density, the spike is infinite (actually a Dirac delta function). Sometimes people plot the discrete part as a probability mass function (so then the spike has height $P(X=0)$) and the continuous part as a density function. That probably makes a better visual, but it does involve two different scales." CreationDate="2011-01-27T23:10:30.477" UserId="159" />
  <row Id="10398" PostId="6622" Text="I'm running on 4gb RAM MacBook. Anyways thanx for the comments." CreationDate="2011-01-27T23:12:18.987" UserId="2895" />
  <row Id="10399" PostId="6645" Score="5" Text="Use the trigonometric method for this one and then perform a little algebraic simplification.  About the shortest way to express the solution is $u(z) = 2 \sin \left(\frac{1}{3} \arcsin(2 z-1)\right)$." CreationDate="2011-01-27T23:28:21.850" UserId="919" />
  <row Id="10400" PostId="6647" Score="2" Text="&quot;90% confident&quot; is only half of a question.  The other half is, &quot;with what precision&quot;?  E.g., do you want to be 90% confident that the rate is less than 1%?  That it's within a factor of 2 of your estimate?  Something else?  In your circumstance it's impossible to be 90% confident that the rate is *exactly* equal to your estimate (except in a psychological sense :-)." CreationDate="2011-01-27T23:37:51.117" UserId="919" />
  <row Id="10401" PostId="6649" Text="Brilliant. Thanks for the clarification." CreationDate="2011-01-27T23:48:13.857" UserId="1833" />
  <row Id="10402" PostId="6644" Text="Thanks for the quick response. All of the stores are included in this analysis and I do wonder what kind of conclusions can be made &quot;without probability statements or statistics.&quot; For example, if the average sales are higher for Group A but the st. dev. seems high then I would think I might need some sort of statistics to say whether or not the promotion truly &quot;worked&quot; for that group of stores.&#xA;&#xA;I've used the t-test but usually in the context of a a true sample of a population and I'm probably just confusing myself since this scenario isn't really a sample of a population?" CreationDate="2011-01-28T00:32:16.653" UserId="2951" />
  <row Id="10403" PostId="6599" Score="1" Text="@AmV great question. I have wondered as well about how to construct the calibration even if you do use 1 versus the rest sort of scheme. If you create k models using 1 versus the rest (and there are k classes) do you have to / should you normalize them somehow so that they sum to 1 (e.g. divide each calibrated probability by the sum of all k)?" CreationDate="2011-01-28T01:10:36.833" UserId="2040" />
  <row Id="10404" PostId="6645" Text="I'm afraid this is a bit overwhelming.  In your u(z) the z refers to what?  Some transformation?  " CreationDate="2011-01-28T02:04:26.947" UserId="2808" />
  <row Id="10405" PostId="6600" Text="@ joanna - apologies for not tying the end back to the significance side of things.  Once the procedure is finished, you have 1 model, and will act *as if this model is true*.  Hence any interactions remaining in this model can be declared &quot;significant&quot; compared to those which don't make it into model.  Because the procedure is for the whole model, this automatically adjusts for the multiple comparisons you want to do.  And you can state that the relationships in this model are the ones which are able to reproduce the data as best as you can out of the alternatives you had." CreationDate="2011-01-28T03:05:17.270" UserId="2392" />
  <row Id="10406" PostId="6652" Score="2" Text="This post has some good discussion of the issue of confidence intervals http://stats.stackexchange.com/questions/2356/are-there-any-examples-where-bayesian-credible-intervals-are-obviously-inferior-t/6373#6373 .  The article referred to in the post, I think, helps shed some light one precisely why the above definitions are correct for confidence intervals.  It is often when viewing how CIs break down that one is able to understand them better." CreationDate="2011-01-28T03:21:31.217" UserId="2392" />
  <row Id="10407" PostId="6636" Text="Pretty well posed question (+1 from me), I like the background context, it makes it easier to judge which assumptions are plausible to make.  I'll have a go at answering, post my answer at some point in the near future" CreationDate="2011-01-28T03:32:16.670" UserId="2392" />
  <row Id="10408" PostId="6645" Text="@rmarimon It's the inverse CDF: that's what your title asks for, right?  It is *identical* to the formula @Erik gives in the range $0 \le z \le 1$, which are the only values that matter." CreationDate="2011-01-28T03:50:36.273" UserId="919" />
  <row Id="10409" PostId="6651" Text="+1, even though I cannot fathom what the OP is looking for either." CreationDate="2011-01-28T04:11:43.620" UserId="919" />
  <row Id="10410" PostId="6623" Text="The Sumatra project is one that may be of help too: http://neuralensemble.org/trac/sumatra/wiki.  You can use it's command line interface to run your code, be in R or something else. There is a Python API to it too. There is a nice blog post on R-bloggers discussing R-centric tools for reproducible research, and it mentions using Sumatra as well.         &#xA;                                                                        http://www.r-bloggers.com/managing-a-statistical-analysis-project-–-guidelines-and-best-practices/" CreationDate="2011-01-28T04:59:48.927" UserId="1080" />
  <row Id="10411" PostId="6629" Text="@Erik, don't think your solution will work because most of the abnormal points have wrong $z$ values, not wrong $x,y$ values" CreationDate="2011-01-28T06:06:30.043" UserId="175" />
  <row Id="10412" PostId="6632" Text="`some fixed multiple of the standard error`. This is pretty ambiguous. How do I know what is the correct coefficient?" CreationDate="2011-01-28T06:07:23.057" UserId="175" />
  <row Id="10413" PostId="6634" Text="I think that's the problem; I don't know what value of $\hat{\gamma}(k,k) = \frac{1}{\#\mathcal{N}(k,k)} \sum_{(i,j), (p,q) \in \mathcal{N}(k,k)} | z_{i,j} - z_{p,q} |^2$ would qualify as &quot;outlier&quot;" CreationDate="2011-01-28T06:08:29.527" UserId="175" />
  <row Id="10414" PostId="6656" Text="But what is the question precisely? Can you give more details?" CreationDate="2011-01-28T08:02:57.507" UserId="88" />
  <row Id="10415" PostId="6658" Text="Just to be clear, you've got 4 tables, each one if of 2 by 2, with number of &quot;Fatality&quot; for each population (from the people who got and didn't get treatment).  Is that correct, or are you having it in any way different then what I described?" CreationDate="2011-01-28T08:16:41.187" UserId="253" />
  <row Id="10416" PostId="6660" Score="2" Text="[Odds ratio versus relative risk](http://www.childrens-mercy.org/stats/journal/oddsratio.asp) offers a good summary of those two measures. Another caveat that is frequently seen is people that apply a logistic regression model, but interpret the results with RR instead of OR." CreationDate="2011-01-28T09:05:52.983" UserId="930" />
  <row Id="10417" PostId="6632" Text="That will be a trade-off. Setting a lower value will give you more false-positives but fewer false-negatives, i.e. between flagging some points that are fine, and missing some points that aren't. In practice, a suitable value will depend on the number of points in your data set. I'd suggest maybe trying 4 to start with." CreationDate="2011-01-28T09:13:38.890" UserId="449" />
  <row Id="10418" PostId="6191" Text="You're right, whuber; nevertheless I can't resist sharing another quip I've just stumbled across: &quot;Those with a taste&#xA;for foundational questions are referred to measure theory, an excursion from which few return.&quot; —James Franklin http://dx.doi.org/10.1007/BF02985802" CreationDate="2011-01-28T10:16:51.673" UserId="449" />
  <row Id="10419" PostId="6655" Text="Quick clarification: minimising orthogonal errors rather than perpendicular ones isn't really an estimation strategy decision but a decision about the model that needs estimating - typically one that assumes the existence of measurement error in X rather than one that doesn't. (And you could estimate *its* parameters various ways too.)" CreationDate="2011-01-28T11:49:31.727" UserId="1739" />
  <row Id="10420" PostId="6655" Score="1" Text="I like your idea of a 'best fitting' estimation strategy, but what is being fitted best?  Usually the issue is about the fit of the strategy to the state of the researcher's confidence in her knowledge of the various parametric assumptions she'd like to make, and the level of anxiety she has about them being quite wrong.  The econometrics literature is quite explicit about this as a motivation ML vs GMM vs Robust, etc.  " CreationDate="2011-01-28T11:57:44.380" UserId="1739" />
  <row Id="10421" PostId="6654" Score="2" Text="Thank you for providing an answer to an excellent question. May I bring up the following analogy for further discussion? Suppose I flip a fair coin over and over. Then, $P(Head) = .50$. Now, I flip the coin once, but don't show you what I flipped, and I ask: &quot;What is the probability that heads are up?&quot;. How would you answer that question?" CreationDate="2011-01-28T12:46:51.633" UserId="1934" />
  <row Id="10422" PostId="6642" Text="@Ian Langmore, I've answered the question in math.SE (http://math.stackexchange.com/questions/19274/limiting-distribution-of-a-squared-sum-of-random-variables/19351#19351)" CreationDate="2011-01-28T12:53:29.073" UserId="2116" />
  <row Id="10423" PostId="6642" Text="hm, what is the usual practice for questions duplicated in two SE sites? Should I post the answer in both sites, and then gather the feedback from two different communities?" CreationDate="2011-01-28T13:01:58.673" UserId="2116" />
  <row Id="10425" PostId="6642" Text="@mpiktas Just flag it for mod attention (which is what I've done)" CreationDate="2011-01-28T13:17:47.553" UserId="930" />
  <row Id="10426" PostId="6634" Text="@dep_stats , while this is a good answer and I have upvoted, it will be very hard for anyone not already familiar with what a variogram is to follow your notation. I would suggest either expanding upon the definition of the variogram or linking to other references that define how to estimate variograms in more detail.I think this article is a good one on introducing variograms , http://www.geog.ucsb.edu/~chris/readings/Variogram.Interpretation.Modeling.Aide.pdf ." CreationDate="2011-01-28T13:29:02.220" UserId="1036" />
  <row Id="10427" PostId="6642" Text="@chl, judging from the number of votes this question is more interesting to statisticians :)" CreationDate="2011-01-28T13:33:19.823" UserId="2116" />
  <row Id="10428" PostId="2647" Text="You can interpret a continuous density the same as the discrete case if $O$ is replaced by $dO$, in the sense that if we ask for $Pr(O\in(O',O'+dO') |\theta)$ (i.e. probability that the data $O$ is contained in an infinintesimal region about $O'$) and the answer is $f(O'|\theta)dO'$ (the $dO'$ makes this clear that we are calculating the area of an infinintesimaly thin &quot;bin&quot; of a histogram)." CreationDate="2011-01-28T13:40:08.867" UserId="2392" />
  <row Id="10429" PostId="6654" Text="Another way to phrase it: for non-Bayesians, the only &quot;things&quot; that can have a probability are possible events - in the sense of future outcomes of a random experiment. Given that the parameter has a fixed true value, once you have an interval with specific values, it's not a possible event anymore whether or not the parameter is included in the interval. As a result, you can have confidence in the process that generates the interval, but not in two specific numbers." CreationDate="2011-01-28T13:50:45.490" UserId="1909" />
  <row Id="10430" PostId="2645" Text="Interesting answer, I actually thought that the &quot;likelihood school&quot; was basically the &quot;frequentists who don't design samples school&quot;, while the &quot;design school&quot; was the rest of the frequentists.  I actually find it hard myself to say which &quot;school&quot; I am, as I have a bit of knowledge from every school.  The &quot;Probability as extended logic&quot; school is my favourite (duh), but I don't have enough practical experience in applying it to real problems to be dogmatic about it." CreationDate="2011-01-28T13:53:58.340" UserId="2392" />
  <row Id="10431" PostId="6666" Score="2" Text="+1 for the effort" CreationDate="2011-01-28T14:00:56.353" UserId="2116" />
  <row Id="10432" PostId="6645" Score="1" Text="@whuber thanks.  The u(z) you provided seemed so simple in comparison to the other that it was hard for me to understand that they are the same.  Just plotted both and indeed they are the same." CreationDate="2011-01-28T14:01:10.383" UserId="2808" />
  <row Id="10433" PostId="6654" Text="@caracal - just some food for thought, is a &quot;coin flip&quot; every truly &quot;random&quot;? If you say &quot;yes&quot; then you would reject the idea that whether a coin comes up heads is a deterministic (but complicated) function of many things (say- wind,altitude,force and angle of flip,weight of coin,etc.etc.).  I think this shows the *double standard* of &quot;randomness&quot; that applies to CI-based thinking, The *data* are fixed but we are uncertain about its value (ergo **data are random**), while the *parameters* are fixed but we are uncertain about its value (ergo **parameters are not random**)." CreationDate="2011-01-28T14:33:23.813" UserId="2392" />
  <row Id="10434" PostId="6654" Text="Very good analogy @wolfgang (+1 from me).  This is exactly the way a CI should be interpreted." CreationDate="2011-01-28T14:40:02.810" UserId="2392" />
  <row Id="10435" PostId="6645" Text="Thanks @whuber." CreationDate="2011-01-28T14:42:28.727" UserId="2898" />
  <row Id="10436" PostId="6629" Text="I think it's a moot point because others have suggested better techniques, but I'm not sure I understand your objection -- I would think the fact that the $z$ coordinates are off is *essential* to why my method would work." CreationDate="2011-01-28T14:47:16.020" UserId="2898" />
  <row Id="10437" PostId="6618" Text="I edited my answer a bit to take into account the comments of Dikran and probabilityislogic. Thanks." CreationDate="2011-01-28T14:52:12.347" UserId="25" />
  <row Id="10438" PostId="6642" Text="@chl, mpiktas This question is certainly in-scope here, even if it is an exact duplicate on another site.  In fact, it's probably more relevant to our community than the other one.  I'm inclined to suggest that we leave it open?" CreationDate="2011-01-28T14:54:23.890" UserId="5" />
  <row Id="10439" PostId="6642" Text="@Shane @mpiktas @Didier It seems more appropriate here (IMHO). Maybe @Ian Langmore could close it on math.SE so that @mpiktas can move his response there?" CreationDate="2011-01-28T15:03:47.230" UserId="930" />
  <row Id="10440" PostId="6654" Text="@probabilitylogistic I agree that the waters become more murky once we are talking about empirical experiments, and not anymore about the statistical concept of random experiments and variables." CreationDate="2011-01-28T15:13:43.607" UserId="1909" />
  <row Id="10441" PostId="6644" Text="Statistics is still appropriate even if you have data on the entire population if you consider there to be a &quot;random&quot; element to the process, both from truly random factors and from other errors from unrecorded characteristics that are randomly distributed in the characteristics you are modelling." CreationDate="2011-01-28T15:31:11.300" UserId="229" />
  <row Id="10442" PostId="6654" Text="Great discussion. I am not sure though if I understand all the points being made. I brought up the analogy, because I think one could very well answer with &quot;The probability that heads are up on that particular flip is .5&quot;. I guess under a frequentist framework, this statement is technically not correct (since the event has already occured, so either heads are really up or not), so the only way I can make a probability statement is by reference to the long-run relative frequency. But I just can't help thinking: Well, on \bold{that} flip, the probability is .5 (before I see the actual outcome)." CreationDate="2011-01-28T15:47:53.483" UserId="1934" />
  <row Id="10443" PostId="6644" Text="I'd say it depends on what your goal is: 1)I want to describe *the population I observed* or 2)I would like to describe *a population like the one I observed*.  For 1) you have no uncertainty at all, so statistics just reduces to describing differences and similarities.  But beware: the &quot;standard deviation&quot; in this case does not describe the *accuracy* of a difference in means.  In case 1) if the $\overline{x}_A$ and $\overline{x}_B$ are different, then the means *are* different, regardless of what the standard deviation is." CreationDate="2011-01-28T15:50:03.280" UserId="2392" />
  <row Id="10444" PostId="6666" Text="I think I just had a minor heart attack. I am digesting this right now, thank you so much. I wish I could upvote more than once!" CreationDate="2011-01-28T15:59:50.753" UserId="2950" />
  <row Id="10445" PostId="6644" Text="... just one point of clarification on my comment above, while the means *are* different, this alone tells us *nothing* about **why** they are different, apart from the labels of $A$ and $B$ (or equivalent synonyms of them).  Anything beyond that is not answered by the means, but by the interpretation it is given (which will depend on your prior beliefs about what the potential causes of a difference may be - the obvious stand out being the promotion)." CreationDate="2011-01-28T16:01:19.727" UserId="2392" />
  <row Id="10446" PostId="6656" Text="What equation do you use to calculate the entropy?  Like a &quot;frequency&quot; entropy?  What constraints are the maximised entropy based on?  agree with @mbq, this question needs more details." CreationDate="2011-01-28T16:13:52.633" UserId="2392" />
  <row Id="10447" PostId="6654" Text="@Wolfgang I do not see how your example pertains to confidence intervals.  You do not ask for anything related to a distributional parameter.  Your situation is most closely related to *prediction intervals.*  I think this entire discussion may have some interest in that context, but it doesn't belong in a thread about confidence intervals." CreationDate="2011-01-28T16:36:12.177" UserId="919" />
  <row Id="10448" PostId="6668" Score="2" Text="If you mean that you have a density function expressed as a linear combination of Epanechnikov densities and you wish to compute its quantiles directly using the inverse CDF, the answer is no in general.  (Conceivably some special configurations of data--equal spacing comes to mind--could be amenable to such an approach.)  You're best off integrating your density using the CDF of the kernels and numerically finding its quantiles." CreationDate="2011-01-28T16:45:18.043" UserId="919" />
  <row Id="10449" PostId="6654" Text="@whuber The question whether one can make a probability statement about a particular 95% CI capturing the true unknown parameter is very similar to the question whether one can make a probability statement about a specific flip where the outcome is still unknown. In the long run, 95% of the CIs will capture the parameter. In the long run, 50% of the flips are heads. Can we say that there is a 95% chance that a particular CI captures the parameter? Can we say that there is a 50% chance that heads are up before looking? I would say yes to both. But some people may disagree." CreationDate="2011-01-28T16:55:44.500" UserId="1934" />
  <row Id="10450" PostId="6635" Score="1" Text="I rule I use to remember: Use probabilities to predict frequencies.  Once the frequencies have been observed, use them to evaluate the probabilities you assigned.  The unfortunately confusing thing is that, often the *probability* you assign is equal to a *frequency* you have observed.  One thing I have always found odd is why do *frequentists* even use the word probability? wouldn't it make their concepts easier to understand if the phrase &quot;the frequency of an event&quot; was used instead of &quot;the probability of an event&quot;?" CreationDate="2011-01-28T16:58:17.103" UserId="2392" />
  <row Id="10451" PostId="6605" Text="@robin girard - in the spirit of the last part of your comment, one quote I like, I heard it from a comedian Bill Bailey is: *I analyse things too much...or maybe not enough?*.  Another one I like (from me, not Bill Bailey): *I used to be uncertain, but now I'm not so sure...* :).  It does look like p-value is going to win" CreationDate="2011-01-28T17:06:49.267" UserId="2392" />
  <row Id="10452" PostId="6672" Score="2" Text="Whether you get the same result using frequentist confidence intervals and Bayesian credible intervals depends on the problem, and in particular on the prior distribution used in the Bayesian approach.  It also important in maths and science that when you are right you are right for the correct reason!" CreationDate="2011-01-28T17:20:56.287" UserId="887" />
  <row Id="10453" PostId="6666" Text="let me know if I need to clarify anything.  It would be interesting to see how the results change if we assumed the rate of failure was non-constant, but sinusoidal - i.e. system is cyclical, with &quot;good&quot; and &quot;bad&quot; streaks.  You would probably need more info to make this kind of improvement work, such as the &quot;transition times&quot; - the time points when the system changed from &quot;up&quot; to &quot;down&quot; and &quot;down&quot; to &quot;up&quot;." CreationDate="2011-01-28T17:27:00.543" UserId="2392" />
  <row Id="10455" PostId="6246" Text="This site supports LateX commands, you just enclose them in dollar signs.  For example to get $(-\infty,\infty)$ write &quot;(-\infty,\infty)&quot; enclosed in $ signs. see [the wiki page](http://en.wikibooks.org/wiki/LaTeX/Mathematics) for syntax, but ignore the 'begin{math}' and 'end{math}' parts, just use dollar sign instead." CreationDate="2011-01-28T17:33:01.927" UserId="2392" />
  <row Id="10458" PostId="6600" Text="@probabilityislogic - Thanks again for your input. However, I feel like I'm still not being very clear. I thought the initial model would be held true and thus probing the significant interactions of the fixed effects in the model, I could calculate the significance of the multiple comparisons stemming from that significant interaction. If I understand correctly, is what you suggest something along the lines of the Monte Carlo Markov Chain sampling? I apologize for being so dense...MEM is not my strong suit!" CreationDate="2011-01-28T17:39:08.137" UserId="2934" />
  <row Id="10459" PostId="6377" Score="1" Text="You need to state what software package you are using." CreationDate="2011-01-28T17:54:53.793" UserId="8" />
  <row Id="10460" PostId="6642" Text="My initial thought was to close this question, but as @Shane points out, it is particularly relevant for statisticians. @mpiktas do you want to answer the question (again?)" CreationDate="2011-01-28T17:56:50.313" UserId="8" />
  <row Id="10461" PostId="6642" Text="@mpiktas: It's usual for a question to only appear on a single site. But I think their should be some flexibility to this rule." CreationDate="2011-01-28T17:57:54.073" UserId="8" />
  <row Id="10462" PostId="6654" Text="@Wolfgang You are discussing the meaning of probability rather than the interpretation of confidence intervals.  This is interesting, but in the interest of keeping these comments relevant, I suggest opening a new thread if you want to continue." CreationDate="2011-01-28T18:04:45.497" UserId="919" />
  <row Id="10463" PostId="6672" Score="2" Text="@Ringold If you &quot;use classical statistics to calculate the 95% confidence interval for [a parameter],&quot; then, if you are reasoning consistently, it is *meaningless* to refer to a &quot;probability that the parameter lies in that interval.&quot;  The moment you mention that probability, you have changed your statistical model of the situation.  In the new model, where the parameter is random, it is incorrect to compute a CI using frequentist methods.  Obtaining the right answer this way in some situations is interesting but does not justify the conceptual confusion that underlies it." CreationDate="2011-01-28T18:13:46.527" UserId="919" />
  <row Id="10464" PostId="6634" Text="I am downvoting primarily because the suggestion doesn't work.  Outliers in a set of &quot;variograms&quot; (I use quotes because the formula is not the usual empirical variogram), as indexed by $(k,k)$ (why the double subscript?) will identify *neighborhoods,* not points.  One truly outlying point will contribute to many *large* (not small) outlying values in this set of variograms.  A sharp fault in the slope, such as a cliff (or building, for LIDAR datasets) will also create large numbers of large outliers." CreationDate="2011-01-28T18:19:34.053" UserId="919" />
  <row Id="10465" PostId="6629" Score="1" Text="@Erik The main problem is that the size of an extreme variation in the $z$ values may be--and usually is--quite a bit smaller than typical spacing between points in $(x,y)$.  Your method would tend to thin the data without necessarily finding any outlying elevations at all. " CreationDate="2011-01-28T18:22:36.950" UserId="919" />
  <row Id="10466" PostId="6632" Text="@onestop The idea is good but I think it can fail where slopes are abrupt.  Such areas won't be fit well by local linear fits (which is what LOESS does) and therefore will seem to be outlying.  Rejecting those points will grossly smooth the sharpest features of the terrain.  To make it work you could apply cross-validation methods of Kriging, such as a jackknife calculation (leave-one-out) with residuals standardized by the kriging estimation error.  That's usually a big effort." CreationDate="2011-01-28T18:26:46.490" UserId="919" />
  <row Id="10467" PostId="6601" Text="@Wiki Whenever you have trigonometric funcions (sine, cosine, tangent etc.) you risk having $\pi$ pop up. And remember that whenever you derive a function you're actually finding a tangent. What is surprising is that $\pi$ doesn't appear *more* often." CreationDate="2011-01-28T18:36:58.580" UserId="666" />
  <row Id="10468" PostId="6642" Text="@csgillespie, ok, I am reposting my answer here. Maybe we should notify moderators of math.SE to close the question there?" CreationDate="2011-01-28T18:42:15.013" UserId="2116" />
  <row Id="10469" PostId="6677" Score="2" Text="@shabbychef The two formulas are essentially the same: 1.34 is 1.349 rounded down and 0.9 has been increased to 1.06.  But that change makes little difference in the appearance of smoothness, so we might just as well recognize what's going on and provide a simpler formula: **divide the smaller of two estimates of the standard deviation by the fifth root of the number of observations.**  The two estimates are the sample standard deviation, *s*, and a rescaled IQR (which agrees with *s* asymptotically for normal distributions)." CreationDate="2011-01-28T19:14:25.477" UserId="919" />
  <row Id="10470" PostId="6670" Text="What is Silverman's formula, exactly?" CreationDate="2011-01-28T19:15:11.387" UserId="919" />
  <row Id="10471" PostId="6641" Text="@whuber, +1, excelent answer!" CreationDate="2011-01-28T19:22:43.263" UserId="2116" />
  <row Id="10472" PostId="6677" Text="@whuber: yes indeed, the two answers are the same up to a factor of about 15% for any set of data. I am not sure the constant factor in front is irrelevant, however!" CreationDate="2011-01-28T19:49:46.457" UserId="795" />
  <row Id="10473" PostId="6680" Score="1" Text="@Jana, table B is not a table, and probably you have the letters mixed up." CreationDate="2011-01-28T19:55:28.933" UserId="2116" />
  <row Id="10474" PostId="6680" Text="@Jana, the best you can do is get a list: `tapply(tb$value,tb$name,function(x)x)`, where `tb` is the your table A with 1st and 2nd columns named `name` and `value`." CreationDate="2011-01-28T20:09:30.893" UserId="2116" />
  <row Id="10475" PostId="6680" Text="yeah its not a table but I need the answer in 2 columns, first one should have the alphabets and the second one should have the corresponding values either in comma separated or tab delimited." CreationDate="2011-01-28T20:10:36.220" UserId="2725" />
  <row Id="10476" PostId="737" Text="Rolf Sundberg attributed this quote to J.M. Hammersley in a 1994 article: http://dx.doi.org/10.1016/0169-7439(93)E0041-2" CreationDate="2011-01-28T20:41:06.567" UserId="449" />
  <row Id="10480" PostId="6682" Text="Thanks mpiktas! As you said the question was not correct in the way I wanted, I agree with you but you answer is what I needed.Thanks again" CreationDate="2011-01-28T21:06:11.307" UserId="2725" />
  <row Id="10481" PostId="6656" Text="I use this: http://upload.wikimedia.org/math/a/2/f/a2f05485301595188046d986c8cdd705.png. It is probability-based, as it compares the frequency of a state to the probability of a state. The maximum entropy is simply the highest possible entropy for the list, and occurs when the frequency for the two states is equal. The minimum occurs when it is all one state. The specific question is whether it would be &quot;correct&quot; to set the frequencies in the initial condition to produce an entropy that is an average between the minimum and maximum, to allow for equal change in entropy in either direction." CreationDate="2011-01-28T21:10:10.057" UserId="2976" />
  <row Id="10482" PostId="6611" Text="prob.*: updated." CreationDate="2011-01-28T21:15:50.940" UserId="2914" />
  <row Id="10483" PostId="6682" Text="@Jana, you're welcome. If this is the answer you need I suggest you accept it, this is how this site works. Look at the faq how to do that. You might want to accept answers to your other questions." CreationDate="2011-01-28T21:21:29.900" UserId="2116" />
  <row Id="10484" PostId="6677" Score="1" Text="@Shabbychef Given that these formulas are approximate and have optimal properties only (a) asymptotically and (b) for Normally distributed data, a 15% difference is immaterial.  Remember, too, that they are &quot;optimal&quot; only in the sense of reducing mean squared error, which tends to impose a lot of smoothness.  They are not necessarily optimal for other purposes.  That's why the OP should feel free to depart from such recommendations: they should be considered initial values only." CreationDate="2011-01-28T21:36:51.203" UserId="919" />
  <row Id="10485" PostId="6682" Text="I'm fairly new to this community. How do I accept your answer ?" CreationDate="2011-01-28T21:50:21.673" UserId="2725" />
  <row Id="10487" PostId="6540" Text="Just emulate http://en.wikipedia.org/wiki/Net_present_value#Example" CreationDate="2011-01-28T22:20:53.020" UserId="919" />
  <row Id="10488" PostId="6629" Text="I see. In my mind the variation in $z$ was pretty big. Thanks for both of your comments." CreationDate="2011-01-28T22:31:18.363" UserId="2898" />
  <row Id="10489" PostId="6682" Text="@Jana: click on the checkmark just below the vote up/down buttons." CreationDate="2011-01-28T22:51:25.547" UserId="582" />
  <row Id="10490" PostId="4559" Text="@chl: upps- today(end of january) I came back to this question/conversation of mid-november.... Sorry I didn't check earlier - there was something with my courses and then I forgot the stat.exchange... If I've something worth I'll come back next days." CreationDate="2011-01-28T23:23:04.190" UserId="1818" />
  <row Id="10491" PostId="6686" Text="If I use names(dat) instead of c(xVar,traceVar) when renaming newdata prior to using predict, then predict doesn't do what I'd expect. I'm sorry that my previous example code was unclear.  Now, I've provided a more clear example above.  What I want is to find the predicted y value at the mean, 1 SD above of the mean, and 1 SD below the mean for each predictor." CreationDate="2011-01-29T01:46:37.320" UserId="196" />
  <row Id="10492" PostId="6687" Score="2" Text="What's the matter with the built-in LINEST function?  It works exactly like the LINEREG function you describe.  For even more automation, use the regression procedure in the &quot;Data analysis&quot; add-in that comes with Excel.  The best solution, though, is to dump Excel and use a real statistical program.  " CreationDate="2011-01-29T03:54:36.153" UserId="919" />
  <row Id="10493" PostId="4560" Text="I'm working on a project that uses stepwise regression.  The reason is because I have D &gt;&gt; N, where D is dimensionality and N is sample size (thus ruling out using one model with all the variables), subsets of the features are highly correlated with each other, I want a statistically principled way of selecting maybe 2-3 &quot;best&quot; features, and I don't intend to report the P-values, at least without some kind of fairly conservative correction." CreationDate="2011-01-29T04:16:14.197" UserId="1347" />
  <row Id="10494" PostId="6618" Text="For me, the &quot;excluding outliers&quot; is not as clearly *wrong* as your answer implies.  For example, you may only be interested in the relationships at a certain range of responses, and excluding outliers actually helps this kind of analysis.  For example, if you want to model &quot;middle class&quot; income, then excluding the super rich and impoverished outliers is a good idea.  It is only the outliers within your frame of inference (e.g. &quot;strange&quot; middle class observations) were your comments apply" CreationDate="2011-01-29T06:10:08.530" UserId="2392" />
  <row Id="10495" PostId="6682" Text="@Jana, you might also want to read the [faq](http://stats.stackexchange.com/faq). It gives a pretty good summary how this site works." CreationDate="2011-01-29T06:20:50.847" UserId="2116" />
  <row Id="10496" PostId="6690" Text="I would like to preface the question by saying, I know I should have gone to r-help for questions like these. However, I know that some of the gurus are members of this forum as well. Also, I fear total destruction on r-help and r-dlevel for asking simple questions like these.  " CreationDate="2011-01-29T06:51:29.437" UserId="1307" />
  <row Id="10497" PostId="6691" Text="Ok - so probably, I should have made it clearer, x is supposed to `integer`, right? Any specific reason it's double?" CreationDate="2011-01-29T07:03:49.603" UserId="1307" />
  <row Id="10499" PostId="6691" Text="Thanks @mpiktas! You are my hero!" CreationDate="2011-01-29T07:14:52.457" UserId="1307" />
  <row Id="10500" PostId="6691" Text="@suncoolsu, you are welcome." CreationDate="2011-01-29T07:19:11.437" UserId="2116" />
  <row Id="10504" PostId="6666" Text="@probabilityislogic It took me awhile, but I am pretty sure I grasped the $\hat{F}$ calculations, however, I am fairly stumped with the $F_{obs}$ data column. Let's say we are looking at $F$ failures of 1 second or less (total downtime of 500,000 and 1,000,000 &quot;switches&quot;). According to the first formula, to calculate the probability, the result is $$1-\frac{31056926}{31556926}\Bigg(\frac{31056926}{(1,000,000*30)+31056926}\Bigg)^{(500,000/1,000,000)+1}$$ but that yields a probability of $0.643$ for me vs. $0.625$ found in your table. Am I missing something obvious? And also is $S = F * 30$?" CreationDate="2011-01-29T07:44:53.733" UserId="2950" />
  <row Id="10506" PostId="6666" Text="To add to the comment above, I do not think I understand the $F_{obs}$ quite well. If $F_{obs}$ stands for &quot;number of down periods&quot;, then for $F = 1$ the $F_{obs}$ *should* be 1 as well. Unless of course $F_{obs} = F / T_D$ in which case we match at the $F = 1$ but not $F = 1,000,000$ (my P yielding 0.87)" CreationDate="2011-01-29T07:59:55.753" UserId="2950" />
  <row Id="10507" PostId="6684" Text="@drknexus, these types of questions might get better answers in http://stackoverflow.com. " CreationDate="2011-01-29T08:16:27.137" UserId="2116" />
  <row Id="10508" PostId="6689" Text="@shabbychef: Have You ever used LARS? I'm asking because I have never heard about it earlier and it sounds really interesting. The orginal article is a bit long (93 pages) so I'd like to get some opinion before I get deep into it." CreationDate="2011-01-29T11:26:27.570" UserId="1643" />
  <row Id="10509" PostId="6684" Text="@drknexus If `xxA` and `xxB` are standardized before calling `lm()`, it seems to work. E.g., `dat &lt;- cbind.data.frame(dat, xxAstd=scale(dat$xxA), xxBstd=scale(dat$xxB)); lm.res.scale &lt;- lm(out ~ xxAstd*xxBstd,data=dat); names(newdata) &lt;- c(&quot;xxAstd&quot;,&quot;xxBstd&quot;); newdata$Y &lt;- predict(lm.res.scale,newdata)`. Not sure it's what you seek, though." CreationDate="2011-01-29T11:50:42.633" UserId="930" />
  <row Id="10510" PostId="6666" Text="@terry - for your first question, $S=30$ in your case, because you want the probability of $\theta$ (actual time of first failure) being between $0$ and $S$.  The $(1,000,000*30)$ should be replaced by $30$.  And the power should be raised to $F_{obs}+1=1,000,001$, not $\frac{500,000}{1,000,000}+1=\frac{3}{2}$.  So correct form is: $1-\frac{31056926}{31556926}\Bigg(\frac{31056926}{\textbf{30}+31056926}\Bigg)^{\textbf{1,000,000}+1}$.  Which equals 0.625 (I double checked just to be sure)." CreationDate="2011-01-29T12:44:04.710" UserId="2392" />
  <row Id="10512" PostId="6666" Text="@terry - clarification about $\hat{F}$ and $F_{obs}$.  The difference is that $F_{obs}$ indicates you actually know the number of failures which occurred (i.e. you know that the $T_D=500,000$ seconds of down time was caused by *exactly* $F_{obs}$ failures).  When using $\hat{F}$, you are *uncertain* about the precise number of failures, but you expect it to be $\hat{F}$.  Mathematically, it is $E(F_{obs}|\hat{F})=\hat{F}$, where the expectation is taken over your *uncertainty* about $F_{obs}$ (it is the *uncertainty* which is distributed, $F_{obs}$ is a fixed unknown constant)." CreationDate="2011-01-29T13:07:22.333" UserId="2392" />
  <row Id="10513" PostId="6600" Text="@joanna - you are considering different models, therefore you must be uncertain about what the true model is.  If you believe the initial model is *already true*, then by definition, everything in the initial model is significant.  I don't think this is what you believe though.  I think that the initial model specifies a *class* of models (i.e. all variables and their interactions), one of which you believe to be true (or at the very least, you are prepared to act *as if* the true model is in the class you specified)...see next comment for more..." CreationDate="2011-01-29T13:16:28.597" UserId="2392" />
  <row Id="10514" PostId="6600" Text="...cont'd... Thus you are *partially* uncertain about the true model.  The procedure above gives you a coherent way to choose a model from this class (p-values give you a quick way).  And yes it is an Monte Carlo algorithm, but it is not a markov chain, because each cycle is independent of the other (more like a bootstrap than MCMC).  Note it need not be a sampling based method, you could also do a &quot;jacknife&quot; based evaluation (split data into $G$ groups, and predict the $gth$ group based on the model fit to remaining $G-1$ groups)" CreationDate="2011-01-29T13:28:48.913" UserId="2392" />
  <row Id="10516" PostId="6666" Text="@terry-just to add to your comment, yes when $\hat{F}=1$ this implies $F_{obs}=1$, and my equations reflect that the probability is the same regardless of which one you use:&#xA;$$1-\frac{T_U}{T_U+T_D}\Bigg(\frac{T_U}{T_U+S}\Bigg)^{1+1}=1-\frac{\Bigg(\frac{T_U}{T_U+T_D}\Bigg)\Bigg(\frac{T_U}{T_U+S}\Bigg)^2}{1-(1-1)\Bigg(\frac{T_U}{T_U+S}\Bigg)}$$&#xA;This is a good &quot;common sense&quot; check that the calculations are sound.  Also another example of the coherence of Bayesian methodology (when things should be equal, they are equal *automatically*, a property which never ceases to amaze me)." CreationDate="2011-01-29T13:43:15.750" UserId="2392" />
  <row Id="10517" PostId="6684" Text="Yeah chl, I noticed that, but was disappointed I couldn't call scale from within lm and have predict work." CreationDate="2011-01-29T16:34:56.097" UserId="196" />
  <row Id="10518" PostId="6668" Text="[This paper](http://www.uv.es/~bernardo/Kernel.pdf)  may be useful as a &quot;structural&quot; guide to what you want to do, but you will need to adapt it to the particular distribution you used.  It uses a normal mixture, rather than a Epanechnikov mixture" CreationDate="2011-01-29T16:41:52.833" UserId="2392" />
  <row Id="10519" PostId="6694" Text="Thanks for the advice.  There are three problems with this, though:  &#xA;&#xA;1.  I care about quantifying the uncertainty in my predictions and the contribution of each variable, not just binary prediction accuracy.&#xA;&#xA;2.  Given the nature of my dataset, it's way too computationally intensive.&#xA;&#xA;3.  From domain knowledge, I believe that the local optima issue is not important." CreationDate="2011-01-29T17:18:40.337" UserId="1347" />
  <row Id="10520" PostId="6656" Text="Hang on, to have maximum entropy of $50$, this means you have $n=e^{50}=gozillian$ categories? is that right? and how many times will you observe the system?" CreationDate="2011-01-29T17:44:45.573" UserId="2392" />
  <row Id="10521" PostId="6672" Score="2" Text="@whuber - your premise &quot;...if you are reasoning consistently...&quot; has a consequence from the good old Cox's theorem.  It says that if you are reasoning consistently, then your solution *must* be mathematically equivalent to a Bayesian one.  So, given this premise, a CI will necessarily be equivalent to a credible interval, and its interpretation as a probability is a valid one.  And in Bayes, it is not the parameter which has a distribution, it is the uncertainty about that parameter which has a distribution." CreationDate="2011-01-29T17:58:00.630" UserId="2392" />
  <row Id="10522" PostId="6672" Score="1" Text="...cont'd...So one can play the silly game of I'm a Bayesian &quot;Prob that parameter is in the interval&quot;, I'm a frequentist &quot;prob that interval covers parameter&quot;, I'm a Bayesian..., I'm a frequentist,..., I'm a Bayesian..., I'm a frequentist,..... all the while the numbers of the actual calculation never change" CreationDate="2011-01-29T18:01:32.313" UserId="2392" />
  <row Id="10523" PostId="6694" Text="How can it be computationally intensive, when you have less than $100$ observations? The drop-one-jacknife is very quick for small $n$.  And the fraction $F$ is a measure of the uncertainty of your predictions, namely the probability that you will classify correctly given your training data, and your selected model." CreationDate="2011-01-29T18:16:42.813" UserId="2392" />
  <row Id="10524" PostId="6694" Text="@probabilityislogic:  Actually you're right.  This procedure would be part of a larger codebase and I forgot that some of the rest of the code wouldn't need to be re-run for every jackknife iteration.  The other two points still apply, though." CreationDate="2011-01-29T18:31:19.093" UserId="1347" />
  <row Id="10525" PostId="6689" Text="@Tomek Tarczynski: I have used it a small amount. There is a package in Matlab (I am sure there is one or more in R), which I have used. It also provides a sparse PCA, which I was more interested in. I admit I only skimmed the paper. ;)" CreationDate="2011-01-29T18:35:01.660" UserId="795" />
  <row Id="10526" PostId="6688" Text="Is there a reason not to go with a penalized regression approach (lasso, elasticnet, etc.)?" CreationDate="2011-01-29T18:36:10.113" UserId="2126" />
  <row Id="10527" PostId="6656" Text="No, a list of length 100 with 2 states is enough. $50=-\frac{50*0.5*\log{0.5}}{\log{2}}-\frac{50*0.5*\log{0.5}}{\log{2}}$" CreationDate="2011-01-29T18:49:26.250" UserId="2976" />
  <row Id="10528" PostId="6618" Text="Ultimately the real problem with the issues raised in the initial answer is that they (at least partially) invalidate p-values.  If you are interested in quantifying an observed effect, one should be able to do any and all of the above with impunity." CreationDate="2011-01-29T19:24:22.703" UserId="196" />
  <row Id="10529" PostId="6697" Text="Hmm, the much-cited Benjamini-Hochberg JRSSB paper was published in 1995, so just *outside* the window I'm afraid! http://www.jstor.org/stable/2346101&#xA;Storey's paper that introduced $q$-values was 2002 though. http://dx.doi.org/10.1111/1467-9868.00346" CreationDate="2011-01-29T20:11:37.260" UserId="449" />
  <row Id="10530" PostId="6695" Text="@Dirk. Thanks. I was actually using Rf_dpois in a C++ program using `Rcpp` and testing it using `inline`. I have been been a user for about a week. But I very happy with the way I can test my C functions with `inline`. It turns out I am not smart enough to see this simple proof of concept. Thanks again! " CreationDate="2011-01-29T20:54:17.300" UserId="1307" />
  <row Id="10531" PostId="6693" Text="thanks @Gavin. I should have searched better. " CreationDate="2011-01-29T20:56:48.637" UserId="1307" />
  <row Id="10532" PostId="6514" Text="When you say &quot;two-dimensional,&quot; &quot;interactions,&quot; and even &quot;mixed effects,&quot; I'm having trouble telling whether you're using the jargon in standard ways or more impressionistically.  This makes it hard to give you a helpful response." CreationDate="2011-01-29T20:57:40.740" UserId="2669" />
  <row Id="10533" PostId="6676" Text="Thanks for the response!" CreationDate="2011-01-29T21:16:00.103" UserId="2952" />
  <row Id="10534" PostId="6642" Text="Sorry for the duplicate question...I'm claiming newbie over-enthusiasm...In any case, if people agree that the question is better on stats.SE, I happy with it being closed on math.SE.  Not sure if I can do this though...I could vote to delete it." CreationDate="2011-01-29T21:24:51.103" UserId="2952" />
  <row Id="10535" PostId="6661" Text="Onestop: This is certainly the spirit of the question I was aiming for. And thank you for the clarification between criteria of evaluating estimators and methods for deriving them!" CreationDate="2011-01-29T21:38:17.273" UserId="1118" />
  <row Id="10536" PostId="6699" Text="So Dirk, in this example the annual standard error of stock returns would simply be: 2% times SQRT(12) = 6.93%.  Can you confirm that is correct.  It seems so straightforward, I am afraid I missed something.  " CreationDate="2011-01-29T21:45:58.250" UserId="1329" />
  <row Id="10537" PostId="6699" Score="1" Text="Correct. See the edit I just made to my initial answer." CreationDate="2011-01-29T21:54:20.183" UserId="334" />
  <row Id="10538" PostId="6601" Text="@Carlos I suspect the prevalence of $2\pi$ is mostly due to the use of the $\ell^2$ metric, leading to n-spheres. In the same vein, I would expect it's $e$ whose prevalence is due to analysis." CreationDate="2011-01-29T22:00:38.143" UserId="2456" />
  <row Id="10540" PostId="6607" Score="1" Text="This was the question I found most distracting after statistics 101. I would encounter many distributions with no motivation for them beyond &quot;properties&quot; that were relevant to topics at hand. It took unacceptably long to find out what any represented." CreationDate="2011-01-29T22:12:34.160" UserId="2456" />
  <row Id="10541" PostId="6617" Text="Measure theory is neither a field of statistics nor hard. Some types of integration are hard, but, once again, that isn't statistics." CreationDate="2011-01-29T22:12:59.570" UserId="2971" />
  <row Id="10542" PostId="6701" Text="I was interested in binary because it was easiest to analyze. I have found a very broad sufficient condition in works of Lauritzen -- you get closed form if a corresponding log-linear model is decomposable" CreationDate="2011-01-29T22:16:24.243" UserId="511" />
  <row Id="10543" PostId="6605" Text="If the null hypothesis is &quot;your girlfriend hasn't been cheating on you with your best friend,&quot; and you catch them hugging, cuddling and holding hands in a dark place, then the p-value is **really** low. Unless she is ugly, in that case, the p-value might still be high. I'm completely sure even high-school students could understand p-values that way." CreationDate="2011-01-29T22:16:33.857" UserId="2971" />
  <row Id="10544" PostId="6699" Score="1" Text="This is the right approach, but for stock returns (and for many other time series) serial correlation has an important influence.  Ignoring it (by using $\sqrt{N}$) is going to be too optimistic." CreationDate="2011-01-29T22:16:51.843" UserId="919" />
  <row Id="10545" PostId="6661" Text="Thanks. I'm a bit surprised no-one else has chipped in though - that was written pretty much off the top of my head, and i'm certainly no expert in estimation theory." CreationDate="2011-01-29T22:40:44.193" UserId="449" />
  <row Id="10546" PostId="6699" Text="Dirk, thanks much for the confirmation and the real data application.  To whuber, I know Benoit Mandelbrot argues there is a long term serial correlation of returns.  I remember looking at real data, and I saw no evidence of serial correlation using lags from - 1mth to - 12 mth.  Thus, I am not convinced that Dirk's approach is incorrect.  In any case, my main concern was understanding how to convert a monthly standard error into an annual one.  And, Dirk did a good job at giving me the straightforward answer to that. " CreationDate="2011-01-29T23:07:25.997" UserId="1329" />
  <row Id="10547" PostId="6700" Text="Thank you. I got the answer. I request you for one more advise. Now I have some more statistics. I want to compare number of deaths and number of cases of a disease from three different populations Each population received a different treatment. I have year wise data from 2003 to 2010. I want to compare the effect of these two treatments for each year and also total eight years and know whether the difference is significant.&#xA;Kindly advise which is the best statistical test.&#xA;" CreationDate="2011-01-29T23:27:57.297" UserId="2956" />
  <row Id="10548" PostId="6694" Text="@dsimcha - if all you're doing is building a binary classifier, why would you care about anything other than the predictions it makes?  Surely this has to be the number 1 priority, with the contribution from each variable only needed to help understand *why* it is the best classifier.  A way you can get the contribution for each variable, is to re-calculate $F$ removing each variable, one at a time, and check how this compares to the $F$ for the chosen model ($F_{chosen}-F_{(-j)}$ is like a marginal effect for the *jth* variable on the predictive accuracy due to each variable)" CreationDate="2011-01-30T01:10:52.200" UserId="2392" />
  <row Id="10549" PostId="6635" Text="Interestingly, cross validation can be seen as a Monte Carlo approximation to the integral of a loss function in Decision Theory.  You have an integral $\int p(x) L(\textbf{x}_{n},x) dx$ and you approximate it by $\sum_{i=1}^{i=n} L(\textbf{x}_{[n-i]},x_i)$ Where $\textbf{x}_{n}$ is data vector, and $\textbf{x}_{[n-i]}$ is the data vector with the *ith* observation $x_i$ removed" CreationDate="2011-01-30T01:30:56.060" UserId="2392" />
  <row Id="10550" PostId="6618" Text="@drknexus - I wouldn't say they invalidate the p-value per say, rather that the null hypothesis that it is based on is implicitly changed" CreationDate="2011-01-30T01:38:37.953" UserId="2392" />
  <row Id="10551" PostId="6605" Text="@Eduardo - a reasonable alernative hypothesis &quot;your girlfriend was not chased by a rapist into a dark place, and your best friend found her and got the rapist to leave&quot;.  If you see them hugging, cuddling, and holding hands in that dark place, this is inconsistent with the above hypothesis (unless your best friend or your girlfriend are uncomfortable about touching people).  Hence it too will have a small p-value" CreationDate="2011-01-30T01:52:45.893" UserId="2392" />
  <row Id="10552" PostId="6605" Text="@probabilityislogic: Well. At least we both understand p-values." CreationDate="2011-01-30T01:59:20.543" UserId="2971" />
  <row Id="10553" PostId="6605" Text="@Eduardo - I don't really get the &quot;ugly&quot; argument.  &quot;Ugly&quot; people have sex, and for some people the saying &quot;forbidden fruit is the sweetest&quot; holds true.  Another thing, is that it is your best friend who needs to think she is ugly for it to apply, and you cannot be certain of this (unless you can read his/her mind)." CreationDate="2011-01-30T02:02:15.147" UserId="2392" />
  <row Id="10554" PostId="6605" Text="@probabilityislogic: I was just trying to use an example even the most mentally-challenged high-school student could understand." CreationDate="2011-01-30T02:09:04.250" UserId="2971" />
  <row Id="10555" PostId="6700" Text="Interesting.  And more complicated.  So now your outcome variable has 3 values:  healthy, ill, or died?  It seems as if you could run chi-square tests separately within each year and compare the residuals or the measures of association across years.  Another level of sophistication would involve, for each year, a multinomial logistic regression, and finding a way to compare the coefficients across years.  Someone out there may be able to suggest another way using something like loglinear modeling." CreationDate="2011-01-30T02:09:23.617" UserId="2669" />
  <row Id="10556" PostId="6605" Text="@eduardo - I think the problem that comes about is that the data are insufficient for the problem.  If you replaced observing &quot;hugging, cuddling, and holding hands&quot; with &quot;walking in on your best friend and girlfriend having sex&quot;, then it would be much harder to come up with alternatives (although having thought about it, an alternative could be assault or blackmail)" CreationDate="2011-01-30T02:10:00.007" UserId="2392" />
  <row Id="10557" PostId="6605" Text="@probabilityislogic: This is rapidly going off-topic. We might argue elsewhere." CreationDate="2011-01-30T02:11:02.530" UserId="2971" />
  <row Id="10558" PostId="6605" Text="@eduardo - I'd say the example is off topic, but the analogy isn't.  This is because you can always come up with the hypothesis of &quot;honesty&quot; - i.e. you can get a small p-value by saying the hypothesis is &quot;these data were not changed to be consistent with a different hypothesis&quot;.  Because the probability of observing the particular data you observed is always small, the p-value will also be small.  hence it is only by using your *prior information* about the integrity of the data, that you can eliminate these kinds of hypothesis." CreationDate="2011-01-30T02:18:22.710" UserId="2392" />
  <row Id="10559" PostId="6605" Text="@probabilityislogic: I know. I never claimed that the p-value would be zero. But a small enough p-value is sufficient to claim &quot;Bullcrap!&quot; when you are (1 minus p-value) sure they lying to you. Of course, how small does &quot;small&quot; mean to you is a matter of policy, and statistics doesn't deal with that." CreationDate="2011-01-30T02:32:42.013" UserId="2971" />
  <row Id="10560" PostId="6656" Text="I would have thought that if a list can only have 2 states, then its maximum entropy (in nits) is $log(2)=0.693$ (or 1 bit).  Or is it that each element in the list can have 2 states for a total of $2^{100}$ possible lists?  This has maximum entropy of $100 log(2)$, which is not equal to $50$ (in bits or nits).  Where does the $50$ come from in your equation, and why did you divide by $log(2)$?  neither of the $50$ or the division by $log(2)$ is in the formula in your link." CreationDate="2011-01-30T02:47:23.767" UserId="2392" />
  <row Id="10561" PostId="6708" Text="@Aaron: +1 Thank you very much. That does the trick. Sorry about not specifying the `fitdistr` package. I'm quite new to R so I will keep that in mind for the next time. I have accepted this as the answer but as a final note, can you please tell me if there is a numerical measure that tells me how well the data could be fitted using the distribution?" CreationDate="2011-01-30T03:12:38.407" UserId="2164" />
  <row Id="10562" PostId="6708" Text="@Aaron: In addition, is there a way to limit the upper limit of the sampling to the maximum value in the data samples? Otherwise, the Q-Q plot seems to be showing some numbers way out of the data sample range." CreationDate="2011-01-30T03:31:27.780" UserId="2164" />
  <row Id="10564" PostId="6700" Text="Thank you. There is some communication gap. I will clarify. I am getting more data from 3 to 4 regions. Now I have survived and died cases from different regions. One region reported more deaths because of not giving treatment thoroughly. one region reported zero deaths. some regions had only 1 case and no deaths. I have data from 2003 to 2010 year wise and region wise. I am a medical doctor with limited knowledge of statistics." CreationDate="2011-01-30T04:40:04.043" UserId="2956" />
  <row Id="10565" PostId="6607" Text="Maximum entropy &quot;thinking&quot; is one method which helps understand what a distribution is, namely a state of knowledge (or a description of uncertainty about something).  This is the only definition that has made sense to me in all situations" CreationDate="2011-01-30T04:57:47.983" UserId="2392" />
  <row Id="10566" PostId="6700" Text="It's difficult to visualize the data that you have.  I suggest you post a table with the number who had each outcome, organized by region and by year (not necessary to label the regions of course).  That way one of us can give you the most informed recommendation." CreationDate="2011-01-30T05:10:13.660" UserId="2669" />
  <row Id="10567" PostId="6605" Score="1" Text="@eduardo - yes a small enough p-value is sufficient to cast doubt on the null hypothesis: but it is calculated *in complete isolation* to an alternative.  Using p-values alone, you can never formally &quot;reject&quot; $H_0$, because *no alternative has been specified*.  If you formally reject $H_0$, then you must also reject the calculations which was based on the assumption of $H_0$ being true, which means you must reject the calculation of the p-value that was derived under this assumption (it messes with your head, but it is the only way to reason *consistently*)." CreationDate="2011-01-30T05:14:36.433" UserId="2392" />
  <row Id="10568" PostId="6605" Text="@eduardo - using p-values in hypothesis testing is similar to the logical case of requiring a set of axioms to prove their own consistency.  It just can't be done.  However you can assess a given *theorem*, and work out which sets of axioms are consistent with the theorem. (axiom=hypothesis, theorem=data)" CreationDate="2011-01-30T05:19:48.720" UserId="2392" />
  <row Id="10569" PostId="6700" Text="@rolando2: sorry about that. I have edited my original post to include a table. Thank you for your time" CreationDate="2011-01-30T05:21:38.517" UserId="2956" />
  <row Id="10570" PostId="6708" Text="@Legend: As to your first comment, it should really be a separate question.  I can point you towards the Kolmogorov-Smirnov test (https://secure.wikimedia.org/wikipedia/en/wiki/Kolmogorov_Smirnov_Test), but I encourage you to ask this question, both because you may get better advice and because the answer will be more findable by others)." CreationDate="2011-01-30T06:06:31.890" UserId="2975" />
  <row Id="10571" PostId="6708" Text="@Legend: As to the second point, I'm not sure what you're really asking.  (This usage of) a QQ plot compares the distribution of some data to a mathematical function -- so there's no &quot;sampling&quot; involved.  The fact that the high end of the plot has values higher than those in the data shows you exactly what you are using the QQ-plot to test for: that the shape of the tail of the distribution of the observed data does not match the tail of the theoretical distribution.  (Also, I don't see any &quot;out of range&quot; values with this code and your data from the question...how did you generate that data?)" CreationDate="2011-01-30T06:09:48.770" UserId="2975" />
  <row Id="10572" PostId="6605" Text="@probabilityislogic: Again, I know. If the null hypothesis is wrong, then the test statistic is meaningless." CreationDate="2011-01-30T06:18:25.767" UserId="2971" />
  <row Id="10573" PostId="6715" Text="Nice question, its a good example to expose to different procedures, because we basically know without any maths or formality, that Environment 4 and 6 are different to the rest (and Environment 1 is a little different from 2, 3, and 5).  Thus any good procedure should be able to produce the obvious result, only difference coming from quantifying *how different* in a mathematical sense.  The obvious question is &quot;is there any other way the experiment could have actually produce these results, besides an error?&quot;" CreationDate="2011-01-30T06:28:48.883" UserId="2392" />
  <row Id="10574" PostId="6708" Text="@Aaron: Thank You. I will look into the Kolmogorov-Smirnov test and post another question if it is not clear. Regarding my second question, the highest data point in my sample is 170 and there are 89 points. Now, if I use the function you specified, it is producing 89 points but some points are above 170 (181.38, 207.32...). While this can happen, I was just wondering if there is a way to have some upper limit so that it will sample only within a range (0,170) so that the Q-Q plot will contain equal limits on the X and Y axis. " CreationDate="2011-01-30T06:31:42.337" UserId="2164" />
  <row Id="10575" PostId="6715" Text="@probabilityislogic: Thank You. What you say is useful: if I can somehow quantify the effectiveness of the experiment in each environment, then may be I can say something but am still not sure what to say or how to say. Ah.. (...feeling stupid typing in puzzles) :) Regarding your question: the experiment was quite controlled in the sense that, it was made sure that the environment did not change. However, the procedure could have gone wrong. May be the procedure was not executed properly according to the guidelines (perhaps?)" CreationDate="2011-01-30T06:35:04.597" UserId="2164" />
  <row Id="10576" PostId="6716" Text="@robin girad: Actually, this question came up from someone over the testing team. Properly means that they were given a set of instructions to execute to get a final value. The experiment will complete even if one of the instructions is skipped but will result in an incorrect observation. I will check out the `student test` that you mentioned. But if the test relies on mean, isn't mean supposed to be a bad measure due to its sensitivity to change in the data values? Thank you for your time." CreationDate="2011-01-30T06:52:45.907" UserId="2164" />
  <row Id="10577" PostId="6694" Text="The other issue is that my answer is generic in the criterion, because I don't know what the consequences of using this classifier are.  If you know more background information, then $F$ should be changed to reflect what is important *when you will actually be using the classifier* and **not** what's important when you are fitting the model.  I believe I made this clear in my answer." CreationDate="2011-01-30T07:10:34.043" UserId="2392" />
  <row Id="10578" PostId="6605" Text="@Eduardo - but if you think about it, Say we reject the null hypothesis, due to a small p-value.  Once we have rejected the null; this means that the decision to reject the null hypothesis was based upon a *meaningless statistic*.  Therefore, this invalidates the rejection we just made!" CreationDate="2011-01-30T07:20:37.633" UserId="2392" />
  <row Id="10579" PostId="6605" Text="@probabilityislogic: No it doesn't invalidate the rejection we just made. We have already concluded that the null hypothesis can't be true. The p-value is meaningless because it is defined in terms of something that isn't true. From a strictly logical point of view, the rejection of the null hypothesis (on the premise that p-values shouldn't be that low) just forces you to come up with another null hypothesis or stay content with being ignorant about what's going on." CreationDate="2011-01-30T07:31:41.147" UserId="2971" />
  <row Id="10580" PostId="6605" Text="I think the above discussion between @eduardo and myself shows quite clearly why p-values are a conceptual nightmare.  even in a seemingly obvious example, they do not appear to be what we think they are." CreationDate="2011-01-30T07:43:26.077" UserId="2392" />
  <row Id="10581" PostId="6676" Text="@Ian Langmore, you are welcome, I had some fun in answering it. If this answer was useful, you might want to accept it, this is the way this site works. To cite the [faq](http://stats.stackexchange.com/faq) you can accept the answer by clicking on the check box outline to the left of the answer." CreationDate="2011-01-30T07:53:42.223" UserId="2116" />
  <row Id="10583" PostId="6667" Text="Thank you for the response. I had initially used linear regression to examine this, with which I tested the change in mean between tow time points. However, the challenge comes when testing for the general trend over all the time periods. That means having one significant (or non-significant) value. I believe there are several assumptions that have to be incorporated. I have also seen that non-parametric tests (Mann-Kendall) can be used to test for trend. I am not quite sure how feasible this will be with the data I have maybe for non-normal data.Will look at the packages suggested. Thanks." CreationDate="2011-01-30T09:57:15.220" UserId="2961" />
  <row Id="10584" PostId="6719" Text="Don't the $c_t$ need to form an absolutely convergent sum? i.e. $\sum_{t=0}^{\infty}|c_t| &lt; \infty$ as well" CreationDate="2011-01-30T10:03:11.050" UserId="2392" />
  <row Id="10585" PostId="6702" Score="2" Text="@Zach Did you read the accompagnying vignette, [Estimation of multinomial logit models in R : The mlogit Packages](http://cran.r-project.org/web/packages/mlogit/vignettes/mlogit.pdf)? It seems to me you just have to apply the fitted coefficients on new data, isn't it?" CreationDate="2011-01-30T10:14:24.220" UserId="930" />
  <row Id="10586" PostId="823" Score="1" Text="(-1) also (concur with @walkytalky), and @lese - note that &quot;...consult the consensus of experts in that field of research than to misinterpret the data first-hand&quot;  can be cheekily rebutted by &quot;I consult the experts so that I can mis-interpret the data second-hand&quot;." CreationDate="2011-01-30T10:38:08.183" UserId="2392" />
  <row Id="10587" PostId="1064" Text="If I was a betting man I'd say Richard Feynman was an agnostic" CreationDate="2011-01-30T10:48:28.700" UserId="2392" />
  <row Id="10588" PostId="1406" Score="1" Text="I think this quote means that the results of the experiment should be &quot;obvious&quot;, and statistics just lets one put a precise figure as to &quot;just how obvious&quot;.  The word *needs* is the key." CreationDate="2011-01-30T10:56:58.613" UserId="2392" />
  <row Id="10589" PostId="4302" Text="I think its because physics has a similar level of pedantry required for statistics, and the physicist has the huge benefit of wanting to get rid of uncertainty, the statistician just wants to describe it." CreationDate="2011-01-30T11:04:59.273" UserId="2392" />
  <row Id="10590" PostId="1360" Text="I would go one step further, within statistics.  Those who ignore Bayesian statistics are condemned to reinvent it." CreationDate="2011-01-30T11:08:03.870" UserId="2392" />
  <row Id="10591" PostId="6605" Text="@eduardo - So you would consider second order logic to be fallicious reasoning?  because that is what your answer implies.  If you reject the *conditions* ($H_0$ is a condition of the p-value) then you must reject the conclusions which were based on those conditions.  The same thing goes for axioms: you cannot simultaneously reject an axiom, but keep the theorem that depends on it for its proof." CreationDate="2011-01-30T11:29:12.037" UserId="2392" />
  <row Id="10592" PostId="6719" Text="@probabilityislogic, I intentionaly wrote non-rigorous mathematical text, since strict definition of singular and regular processes involves Hilbert spaces, which I do not think are really necessary to get the point accros. I will update the answer to reflect that. Concerning $c_t$, it does not need to be summable. The equation must hold in $L^2$ sense, hence summability of $c_t^2$ is sufficient. " CreationDate="2011-01-30T11:58:42.623" UserId="2116" />
  <row Id="10593" PostId="2365" Text="I think something which is perhaps been overlooked in the discussion above (including by me) is that the ML solution is exactly equal to the maximum of the joint posterior density using a uniform prior (so $p(\theta|X)\propto p(X|\theta)$ ($\theta$ is the vector of parameters).  So you *cannot* claim that ML is good and Bayes is not, because ML is mathematically equivalent to a Bayesian solution (flat prior, and 0-1 loss function).  You need to find a solution which *cannot* be produced using Bayesian methods." CreationDate="2011-01-30T12:55:53.090" UserId="2392" />
  <row Id="10594" PostId="2356" Score="2" Text="pathology or not, it would probably be the first of its kind.  I am very keen to see this example, for these &quot;pathologies&quot; usually have a good learning element to them" CreationDate="2011-01-30T13:18:52.187" UserId="2392" />
  <row Id="10595" PostId="6720" Score="1" Text="@Tomek, if there are zeroes in your data, exponential regression might not be appropriate, since the model as you stated it cannot allow zero values to be observed." CreationDate="2011-01-30T14:00:34.440" UserId="2116" />
  <row Id="10596" PostId="2356" Text="@suncoolsu - is the example in Wasserman where the supposed &quot;coverage&quot; drops to zero?" CreationDate="2011-01-30T14:21:44.513" UserId="2392" />
  <row Id="10597" PostId="6715" Text="@legend - I'm talking more along the lines of &quot;is $32.1$ a physically meaningful quantity?  What would happen in the real world if this were correct.&quot;  It may also be useful to speak to someone who actually did the experiment 4 or 6 (preferably the person who recorded the data)." CreationDate="2011-01-30T14:31:06.483" UserId="2392" />
  <row Id="10598" PostId="6723" Text="@Biogeek, you have 6 independent variables, is there a particular reason you need only a subset of them in your model? Why not including all of them?" CreationDate="2011-01-30T15:40:35.273" UserId="2116" />
  <row Id="10599" PostId="6618" Text="@probablityislogic: Agreed.  In practice, since we assume a straightforward null and the underlying probabilities governing various choices the experimenter made are probably undefined, do we have any realistic hope of converting a mangled p value into our standard frame of reference?  I guess the difficulty of this task is what prompted me to call them invaldated." CreationDate="2011-01-30T16:22:30.383" UserId="196" />
  <row Id="10600" PostId="6718" Text="mean(newdata[,&quot;Y&quot;]); mean(dat[,&quot;out:).  If predict was working as expected these values should be near one another.  They aren't." CreationDate="2011-01-30T16:24:25.437" UserId="196" />
  <row Id="10601" PostId="2356" Text="@probabilityislogic. I think so, but I need to check. Will get back to you soon on this soon!" CreationDate="2011-01-30T16:24:40.333" UserId="1307" />
  <row Id="10602" PostId="6605" Text="@probabilityislogic: That's where you're wrong. The p-value isn't a _conclusion_ of the null hypothesis, but a _means to determine its believability_. Thus, if the p-value is too low, I can say, &quot;Hahaha! Bullcrap!&quot;" CreationDate="2011-01-30T16:46:49.277" UserId="2971" />
  <row Id="10603" PostId="6700" Text="Ok, so each row in the table is a different region--not 3 or 4, but 16?  also, have you assigned just 2 levels of the treatment variable (thorough and not thorough)?" CreationDate="2011-01-30T17:43:08.037" UserId="2669" />
  <row Id="10604" PostId="6725" Score="4" Text="Generate a random float _x_ in the interval (0,1) and perform a binary search of the cumulative fitness array, finding the first entry that equals or exceeds _x_.  Its index denotes the chosen fitness." CreationDate="2011-01-30T18:02:24.730" UserId="919" />
  <row Id="10605" PostId="6726" Text="+1, for excellent answer. In your second point the statement in bold should probably read: the stated model (A) is both wrong and inadequate." CreationDate="2011-01-30T18:08:22.460" UserId="2116" />
  <row Id="10606" PostId="6716" Text="@Legend A test of difference of means may be inappropriate, but that is not the fault of @robin, as pointed out in the second half of his response, which is apt: the test to use is determined by which characteristic of a suite of results signals an &quot;improper&quot; experiment.  You could conduct an F-test for a difference of standard deviations; you could conduct multiple-outlier tests; you could conduct a Kruskal-Wallis test; etc., depending on what kind of differences you're looking for." CreationDate="2011-01-30T18:08:32.130" UserId="919" />
  <row Id="10607" PostId="6725" Text="Thank you.. Make that an answer!" CreationDate="2011-01-30T18:18:03.300" UserId="1205" />
  <row Id="10609" PostId="6723" Text="@Biogeek, isn't this a homework assignment?" CreationDate="2011-01-30T20:14:25.390" UserId="2116" />
  <row Id="10610" PostId="6371" Text="I'm asking the moderators to close this as no longer relevant. " CreationDate="2011-01-30T22:39:08.547" UserId="1144" />
  <row Id="10611" PostId="6657" Text="it is possible to estimate the probability of absorption in a given state given any initial distribution if you have the transition matrix. The entries of the transition matrix are estimated as you say, but you forget that in the presence of $m$ absorbing states you don't have $n(n-1)$ parameters but $(n-1)(n-m)$ (at least by my rough count), so for 4 states and 2 absorbing states, you have exactly enough parameters to estimate the transition matrix (and in my case there are further constraints which cut down the number of parameters needed - this is immaterial)" CreationDate="2011-01-30T22:56:48.470" UserId="1144" />
  <row Id="10612" PostId="6657" Text="See e.g. http://math.furman.edu/~tlewis/math110/maki/chap8/sec4handout.pdf for a quick discussion of calculating absorption probabilities." CreationDate="2011-01-30T23:04:05.967" UserId="1144" />
  <row Id="10613" PostId="2356" Text="@suncoolsu - this example of Wasserman is not an example of &quot;defective Bayes&quot;.  Because $\theta\sim N(0,1)$, and the coverage  for $\theta&lt;2$ is good, note that $Pr(|\theta|&lt;2)\approx 0.977$, so the supposed &quot;poor coverage&quot; is obtained only in a small fraction of possibilities, if the prior is true.  If you were to average this coverage taken with respect to the posterior of $\theta$, it would be about 95%, because most of the posterior probability would be in the $\theta&lt;2$ range.  (more later)" CreationDate="2011-01-30T23:15:37.087" UserId="2392" />
  <row Id="10614" PostId="2356" Text="@suncoolsu - I would say that it is a good example of the non-robust properties of conjugate priors though.  Because if the true value of $\theta$ is say $4$, but your prior says $\theta\sim N(0,1)$, then almost surely, the prior and the data will come into conflict.  If the prior is conjugate, then what you are basically saying is that the prior information is *just as cogent as the data*.  If the prior was instead $\theta\sim St(0,1,10)$ (T with 10 df), then because the likelihood is normal, you are saying that the data is more cogent than the prior...(more still)" CreationDate="2011-01-30T23:20:09.093" UserId="2392" />
  <row Id="10615" PostId="2356" Text="...cont'd... and that in a case of conflict, you want the data to &quot;win&quot;.  If the situation was reversed (Student likelihood and normal prior), then if the data conflicts with the prior, the prior would &quot;win&quot;.  See [This post](http://stats.stackexchange.com/questions/6493/mildly-informative-prior-distributions-for-scale-parameters/6506#6506) for some links to how this works.  I would suspect that the coverage would be better for large $\theta$ (but possibly worse for small $\theta$) if a student-t distribution was used as prior instead of normal." CreationDate="2011-01-30T23:25:45.700" UserId="2392" />
  <row Id="10616" PostId="6723" Text="@mpiktas It is not a homework assignment, but the R code does indeed come from a course I'm following. The course text is very short on how to interpret the derived results and I posted here trying to get a better grasp on the material while stuyding." CreationDate="2011-01-30T23:49:54.920" UserId="2980" />
  <row Id="10617" PostId="720" Score="1" Text="@whuber: if the data are continuously distributed (which they might be, depending on whether the original poster really means &quot;general&quot; or &quot;generalized&quot; LMM) then the probability density has an implicit &quot;delta-x&quot; term in it, which is indeed affected by changing units. See also &lt;http://emdbolker.wikidot.com/faq&gt;" CreationDate="2011-01-31T00:03:02.937" UserId="2126" />
  <row Id="10618" PostId="2356" Text="I think I will put this in more detail, as an answer, because it is a prime example of what Jaynes talks about in his paper.  Wasserman shows a problem, shows that the Bayesian way gives an apparently counter-intuitive result, warns about the &quot;danger of Bayesian methods&quot;, without any investigation as to *why* the Bayesian solution gives the result it does.  Secondly the *Frequentist Confidence Interval is not given in the equivalent problem!* I will show in my answer, that the example can be formulated in equivalent frequentist terms *which give exactly the same answer as the Bayesian one!*" CreationDate="2011-01-31T00:24:05.393" UserId="2392" />
  <row Id="10621" PostId="6718" Text="What does, &quot;this happens since R is forced to forget the original scaling using formula&quot;, mean?  What is meant by saying that R knows or does not know the original scaling?" CreationDate="2011-01-31T01:05:28.740" UserId="196" />
  <row Id="10622" PostId="6718" Text="Ah... I see - you mean that the predictors in new data themselves are subjected to Z scaling prior to the generation of the prediction?" CreationDate="2011-01-31T01:29:49.560" UserId="196" />
  <row Id="10624" PostId="6718" Text="... so I think the theme here that there is no way to use the scale command within the call to lm and pass predict the Z values I want and have everything turn out smoothy in one step the way I might hope." CreationDate="2011-01-31T01:37:22.963" UserId="196" />
  <row Id="10625" PostId="6719" Text="Thanks to all responders." CreationDate="2011-01-31T02:00:30.773" UserId="2669" />
  <row Id="10626" PostId="6718" Text="I'm going to mark your answer as accepted.  A learned a lot - thanks." CreationDate="2011-01-31T02:00:45.733" UserId="196" />
  <row Id="10627" PostId="6700" Text="Thank you for the prompt reply. Yes. Each row is a different region. There are 16 regions with different populations. The 2 levels of treatment variable are thoroughly implemented and not so thoroughly implemented. Some regions implemented the treatment thoroughly and so the deaths were less. Other regions did not implement the treatment thoroughly and so the deaths were more. We have to show statistically that in regions where treatment was inadequate had significantly higher number of deaths. If the table is too big, we can use 3 or 4 regions data from 2006 to 2010 (5 years) for analysis. " CreationDate="2011-01-31T03:12:38.863" UserId="2956" />
  <row Id="10628" PostId="6735" Text="Thank you for your answer. I see what you mean (kind of). Maybe you can comment on this: I did not mean that the &quot;ordering&quot; of the list changes -- of course it does. Imagine that you sort the list anyway after you did all changes. But would the sorted list then look (kind of) the same , i.e. have the same distribution of values, in the second scheme compared to the first one?" CreationDate="2011-01-31T03:31:08.170" UserId="2440" />
  <row Id="10629" PostId="6732" Text="Thank you! That did the trick.  " CreationDate="2011-01-31T03:45:36.030" UserId="2984" />
  <row Id="10630" PostId="6735" Text="@Felix Dombek, the answer is &quot;yes&quot;. In each of the three schemes, if you sorted each of the three vectors at the end, into ascending or descending order, then the resulting three vectors would all have the same joint distribution. These would be the order statistics of a multinomial random vector. But, the ordering at the end is important. Otherwise, each of the vectors have different distributions (though I think first and last would still be the same)." CreationDate="2011-01-31T03:57:05.713" UserId="2970" />
  <row Id="10631" PostId="6732" Text="@Will, good. Glad it worked. You might consider accepting the answer then. Regards." CreationDate="2011-01-31T03:58:58.553" UserId="2970" />
  <row Id="10632" PostId="6735" Text="@cardinal, +1, great answer." CreationDate="2011-01-31T04:12:47.233" UserId="2116" />
  <row Id="10633" PostId="6732" Text="@cardinal, +1 great answer." CreationDate="2011-01-31T05:04:40.520" UserId="2116" />
  <row Id="10634" PostId="2369" Text="A confidence interval does not bound the rate of false positives - see my answer below for a counter-example to back up my claim." CreationDate="2011-01-31T07:11:53.130" UserId="2392" />
  <row Id="10635" PostId="6716" Text="@Legend There is also another difficulty that is shadowed by your question because here you guessed that 4,6 was the different samples. But what if you don't know in advance... you will have to test all configuration and probably introduce a multiple hypothesis criterion. In this case this looks like outliers detection and many question have already dealt with that here. " CreationDate="2011-01-31T07:35:54.183" UserId="223" />
  <row Id="10636" PostId="6714" Score="1" Text="I do not see why the probability of a proposition according to definition 1 should be a rational number. *Long run proportion* seems to refer to the limit of the proportions of times such that the proposition is observed to be true. Each proportion is a rational number but their limit might not be. (Fortunately, this parenthesis of yours seems tangential at best to the rest of your answer.)" CreationDate="2011-01-31T07:55:30.563" UserId="2592" />
  <row Id="10638" PostId="6657" Text="David: I was not aware that the situations you were interested in involved such small values of $n$ (and you are right, the number of parameters is $(n-1)(n-m)$). May I ask why you now want this question closed? Did you solve the question? I note that there is nothing in the handout you refer to about the *estimation* problem you asked." CreationDate="2011-01-31T08:18:40.743" UserId="2592" />
  <row Id="10639" PostId="6737" Text="Thanks for the analysis.  AFAICS this is just an example of a problem caused by an incorrect (informative) prior assumption and says nothing about the internal consistency of the Bayesian approach?" CreationDate="2011-01-31T08:25:36.813" UserId="887" />
  <row Id="10640" PostId="6727" Text="@Ralph, +1 for looking up the reference. I should note however that if this is truly how the authors are dividing stochastic part it is a bit misleading. Suppose you have autoregressive model $Y_t=\rho Y_{t-1}+\varepsilon_t$. The division is that $\rho Y_{t-1}$ is systematic and $\varepsilon_t$ is non-systematic part. But if we substitute $Y_{t-1}=\rho Y_{t-2}+\varepsilon_{t-1}$ into the first equation and repeat the process indefinitely  we will get that $Y_t=\sum_{s=0}^{\infty} \rho^{s}\varepsilon_{t-s}$, so there is actually no systematic part according to the original definition." CreationDate="2011-01-31T08:28:03.037" UserId="2116" />
  <row Id="10643" PostId="6721" Text="@onestop: Shame I havent been taught about it at the university :/ It looks that is will be helpful in this case, but I need some time to get deep into details. Thanks!" CreationDate="2011-01-31T09:53:05.337" UserId="1643" />
  <row Id="10644" PostId="6726" Text="@whuber: There is a lot of great thoughts in Your answer. I really appreciate it!" CreationDate="2011-01-31T10:00:40.227" UserId="1643" />
  <row Id="10645" PostId="6722" Text="@mpiktas: What about initial values of the parameters? What is good way to choose them? As I stated in an update it may be assumed that there are no continuous variables." CreationDate="2011-01-31T10:02:52.487" UserId="1643" />
  <row Id="10647" PostId="6738" Text="@Didier, I feel rephrasing this part of the question will be the hardes part of the job, maybe the question is asking for pairs of (purposes,best distance). But you are right, maybe @Felix could tell us more about why he is asking his question. " CreationDate="2011-01-31T10:34:06.403" UserId="223" />
  <row Id="10648" PostId="6371" Text="@David Are you sure you want to remove it? I think it should stay because it still may be useful for other people." CreationDate="2011-01-31T10:38:24.713" UserId="88" />
  <row Id="10649" PostId="6736" Text="What is this &quot;given purpose&quot;?" CreationDate="2011-01-31T10:40:43.727" UserId="88" />
  <row Id="10650" PostId="6736" Text="@mbq: This here, for example: http://www.lscp.net/persons/peperkamp/Peperkamp_Le_Calvez_Nadal_Dupoux_%282006%29_The_acquisition_of_allophonic_rules.pdf (i.e. probability distributions for speech sounds in context of other adjacent sounds) ... but my question aims for a general answer of what I have to take into account when I rate each measure's applicability for any use case." CreationDate="2011-01-31T10:44:16.913" UserId="2440" />
  <row Id="10651" PostId="6738" Text="Actually, the first and foremost reason was to get some reputation to be able to answer a question on meta: http://meta.stats.stackexchange.com/questions/707/could-r-tag-be-displayed-as-a-capital-r . Also, I don't understand much of the wikipedia pages and I have of course read them first. I'm extremely lousy in statistics and maths." CreationDate="2011-01-31T10:47:22.030" UserId="2440" />
  <row Id="10652" PostId="6738" Score="2" Text="Dumbfounded by this comment. This is priceless and should be kept for posterity: &quot;Actually, the first and foremost reason [for me to ask this  question] was to get some reputation to be able to answer a question on meta&quot;.--Felix Dombek, January 31st, 2011." CreationDate="2011-01-31T10:53:17.487" UserId="2592" />
  <row Id="10653" PostId="6738" Text="Good that only five are needed ... I don't understand why they don't use the total rep on all SE sites for some actions =)" CreationDate="2011-01-31T10:57:48.137" UserId="2440" />
  <row Id="10654" PostId="6721" Text="Note that $y_i$ can always be rescaled to integer values when it is rational, eg measure pence/cents rather than pounds/dollars. Though you may want to round to the nearest pound/dollar anyway since the distribution of the pence/cents part of the price of goods will likely be very uneven (ie mostly 99)." CreationDate="2011-01-31T11:13:22.703" UserId="229" />
  <row Id="10655" PostId="6738" Text="I understand that if more points were needed, you would have asked several other fake questions. Is this what you mean?" CreationDate="2011-01-31T11:38:47.180" UserId="2592" />
  <row Id="10656" PostId="4712" Text="This is particularly important. I remember reading a study about whether Ramadan was bad for babies whose mothers were fasting. It looked plausible (less food, lower birth weight), but then I looked at the appendix. Thousands of hypotheses, and a few percent of them were in the &quot;significant&quot; range. You get weird &quot;conclusions&quot; like &quot;it's bad for the kid if Ramadan is the 2nd, 4th or 6th month&quot;." CreationDate="2011-01-31T13:06:21.333" UserId="2813" />
  <row Id="10657" PostId="6722" Text="@Tomek, I think there is no one good way of choosing them. Usually it depends on the data. I suggest mean for the intercept and zero for other coefficients. " CreationDate="2011-01-31T13:12:07.333" UserId="2116" />
  <row Id="10658" PostId="6739" Text="@mpiktas. I found this link helpful for deciphering how lm.ridge is calculating the coefficients: http://www.mail-archive.com/r-help@r-project.org/msg81115.html  But still puzzled how different the results are from the text I referenced and SAS, given that each is supposedly back to the original scale." CreationDate="2011-01-31T13:13:45.020" UserId="2040" />
  <row Id="10659" PostId="6739" Text="@user2040, check that the same data is used by SAS and R. If it is the same, then the only conclusion is that the algorithms are different. What SAS help page is saying?" CreationDate="2011-01-31T13:22:21.210" UserId="2116" />
  <row Id="10660" PostId="6745" Text="Thank you, I clarified the question: I need to know *σ* as well, is this possible for every *y*?" CreationDate="2011-01-31T13:25:38.730" UserId="2989" />
  <row Id="10661" PostId="6426" Text="EP does seem cool (although it still hurts my head).  Does it still lack general convergence guarantees?" CreationDate="2011-01-31T13:27:49.333" UserId="1739" />
  <row Id="10662" PostId="6739" Text="@user2040, I've replicated [SAS ridge regression](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_reg_sect046.htm) on R with your data. So we know for sure that the algorithms are different. " CreationDate="2011-01-31T13:59:21.810" UserId="2116" />
  <row Id="10663" PostId="6738" Text="This is not a fake question. I do indeed not understand what the different formulas have as implications, and from my comment for the question, it should be evident that I encountered them in real life. If more points were needed, I would either have just dropped it, or possibly just told someone who answered on a previous question of mine what I wanted to answer on meta and asked him to do it for me. Also, I intended to get the points only from accepting an answer (I already had 3 previously) and not from upvotes, but I guess my question isn't bad after all with 3 upvotes so far." CreationDate="2011-01-31T14:07:06.327" UserId="2440" />
  <row Id="10664" PostId="6738" Text="Also, you must admit that my answer on meta is pretty useful." CreationDate="2011-01-31T14:11:47.197" UserId="2440" />
  <row Id="10665" PostId="6743" Text="@Robin maybe you could add a word on what specifically make InvivoStat relevant to this kind of analysis?" CreationDate="2011-01-31T14:24:54.013" UserId="930" />
  <row Id="10666" PostId="6730" Text="I'ts also in Knuth's Graphbase book, p. 392. For a Python recipe, see http://code.activestate.com/recipes/576564-walkers-alias-method-for-random-objects-with-diffe/" CreationDate="2011-01-31T14:35:04.663" UserId="557" />
  <row Id="10667" PostId="6726" Text="@mpiktas Thank you for suggesting the clarification." CreationDate="2011-01-31T14:48:25.800" UserId="919" />
  <row Id="10668" PostId="6704" Text="+1; could you add an example or two, favourite or current ?" CreationDate="2011-01-31T14:53:20.263" UserId="557" />
  <row Id="10669" PostId="720" Score="1" Text="@Ben Thank you.  When I wrote this I was confused between the AIC and the difference of AICs, thinking the latter was the former.  It is correct that the choice of units introduces a multiplicative constant into the likelihood.  Thence the *log* likelihood has an *additive* constant which contributes (after doubling) to the AIC.  The difference of AICs unchanged." CreationDate="2011-01-31T14:57:32.243" UserId="919" />
  <row Id="10670" PostId="6714" Text="@probability This reply seems to be taking us off on a tangent in a not very constructive way.  Equating probability and proportion is a form of ontological confusion, akin to equating a temperature with the level of mercury in a thermometer: one is a theoretical construct and the other is a physical phenomenon used to measure it.  There is some discussion of this at http://stats.stackexchange.com/questions/1525/whats-the-difference-between-a-probability-and-a-proportion/4850#4850 ." CreationDate="2011-01-31T15:02:49.777" UserId="919" />
  <row Id="10671" PostId="4083" Text="A note on why this was downvoted would be appreciated." CreationDate="2011-01-31T15:10:19.283" UserId="1036" />
  <row Id="10673" PostId="6738" Text="Felix: I simply read what you wrote, nothing else. Now if you'll excuse me, I will take my leave from here." CreationDate="2011-01-31T15:33:32.163" UserId="2592" />
  <row Id="10674" PostId="6739" Text="@mpiktas. You just beat me to that! :) I was looking at the SAS help page you referenced. I compared the RMSE (in-sample data only, did not validate with a CV or bootstrap yet) and the R result was superior.So, do you think ridge regression is best suited for prediction and not interpretting the coefficients (since the results can be so different by algorithm)? I know already that regular linear model theory (CI's, contrasts etc.) are out for ridge regression (bootstrapping gets closer but still can be misleading due to bias)." CreationDate="2011-01-31T16:06:28.597" UserId="2040" />
  <row Id="10675" PostId="6256" Text="Thank you very much for the answer. What can i do, if i have higly screwed numerical(!) data? Does weighting work there, too? " CreationDate="2011-01-31T16:14:22.427" UserId="2781" />
  <row Id="10676" PostId="6753" Text="I really wonder how an equal frequency histogram might look like. I have the intuition that it is really flat. Can you give an example?" CreationDate="2011-01-31T16:58:57.840" UserId="442" />
  <row Id="10677" PostId="6753" Text="@Henrik: See e.g. this question (http://stats.stackexchange.com/questions/5573/how-to-build-an-equilibrated-histogram). Yes it is flat, so it clearly cannot be used for density estimation ;). However, since the equal-width approach is so generic, it seems that it can be applied in every situation equal-freq can be applied. So when to favor equal-freq ?" CreationDate="2011-01-31T17:09:40.383" UserId="264" />
  <row Id="10678" PostId="6671" Text="Thank you for your answer!" CreationDate="2011-01-31T17:13:10.393" UserId="2953" />
  <row Id="10679" PostId="6677" Text="Thanks for answering so fast" CreationDate="2011-01-31T17:13:32.890" UserId="2953" />
  <row Id="10680" PostId="6683" Text="I'll consider that, thank you" CreationDate="2011-01-31T17:13:48.707" UserId="2953" />
  <row Id="10681" PostId="6712" Text="Thanks for the answer" CreationDate="2011-01-31T17:14:03.717" UserId="2953" />
  <row Id="10682" PostId="6754" Score="1" Text="@baris_a Could you clarify what you mean by a &quot;common entity&quot;. It's not clear to me because it seems you are describing two features (for me, a feature or attribute = a variable) with different units (meters and seconds) rather than a single one?" CreationDate="2011-01-31T18:25:33.280" UserId="930" />
  <row Id="10683" PostId="6750" Score="1" Text="Thanks a lot for such a descriptive answer. Thanks to StackExchange, I think it's time I'm not ashamed of asking stupid questions because they trigger such beautiful answers." CreationDate="2011-01-31T18:39:53.290" UserId="2989" />
  <row Id="10684" PostId="6751" Text="Thanks, I think I got it. You can have infinite different normal distribution curves with same maximum, right?" CreationDate="2011-01-31T18:41:08.810" UserId="2989" />
  <row Id="10685" PostId="6751" Score="2" Text="@gaearon There may be a potential for confusion here due to terminology: &quot;maximum&quot; can refer to a maximum value attained by the distribution function (which, for the normal distribution, occurs at the *average* value of the distribution!) or it can refer to the largest possible number that has any chance of occurring (which for any normal distribution is infinitely large).  @Aniko's response appears to interpret your &quot;worst/best&quot; cases as specifying extreme values in the *second* sense, whereas your comment appears to interpret the maximum in the *first* sense." CreationDate="2011-01-31T19:19:28.687" UserId="919" />
  <row Id="10687" PostId="6754" Text="@chl with common entity, I mean an &quot;instance&quot;. As I tried to explain above, there are several measurements with different criteria(s), I need to normalize(?) them to further use in regression algorithms. But, I am not sure how to make these numerical measurements to carry same meaning for each instance.  " CreationDate="2011-01-31T19:27:41.217" UserId="2170" />
  <row Id="10688" PostId="6630" Text="You question is rather tricky... I'm thinking in it." CreationDate="2011-01-31T19:27:54.120" UserId="2902" />
  <row Id="10689" PostId="6753" Score="4" Text="@Henrik No, an equal frequency histogram generally is *not* flat. Histograms are commonly confused with bar charts, which display values by means of the *heights* of bars. However, by definition, a histogram displays frequencies by means of *areas*. Consider (*e.g.*) the data {0,1,2,4,8,16,32,64}, to be shown in the range [0,100] with two bins. The break for an equal-frequency histogram has to be between 4 and 8.  If we put it at 6, the height of the left bar *multiplied* by (6-0) = 6 equals 4, whence the height is 4/6. The height of the right bar equals 4/(100-6) = 4/94. Not flat at all!" CreationDate="2011-01-31T19:29:30.327" UserId="919" />
  <row Id="10691" PostId="6753" Score="1" Text="(Continued) See a Wikipedia example of a variable-width histogram at http://en.wikipedia.org/wiki/File:Travel_time_histogram_total_n_Stata.png , which is an illustration for its article on &quot;Histogram.&quot;" CreationDate="2011-01-31T19:31:33.787" UserId="919" />
  <row Id="10692" PostId="6753" Score="2" Text="@steffen Your second question has already been asked and answered at http://stats.stackexchange.com/q/798/919 .  More formulas appear at http://en.wikipedia.org/wiki/Histogram#Number_of_bins_and_width ." CreationDate="2011-01-31T19:32:46.297" UserId="919" />
  <row Id="10695" PostId="6760" Text="Thanks Bernd. I thought unique can be applied only for a specific column. I didn't know that it can be used for the entire data frame as well. thanks again" CreationDate="2011-01-31T20:25:13.270" UserId="2725" />
  <row Id="10696" PostId="6753" Text="@whuber wow. It is weird to see that one still mixes up pretty basic things. Thanks a lot for pointing my mistake out!" CreationDate="2011-01-31T20:50:40.490" UserId="442" />
  <row Id="10697" PostId="6714" Text="@Didier - you're right, in fact the sequence of $x_n=\frac{r}{2x_{n-1}}+\frac{x_{n-1}}{2} \rightarrow \sqrt{r}$, which is rational terms with irrational limit.  I have removed this remark.  Thanks for bringing this up." CreationDate="2011-01-31T21:03:04.197" UserId="2392" />
  <row Id="10698" PostId="6559" Text="@mpiktas: I posted my second draft, what do you think?" CreationDate="2011-01-31T21:08:22.670" UserId="2817" />
  <row Id="10699" PostId="6714" Score="1" Text="@whuber - The point is relevant to bring up because it is precisely this misunderstanding that leads people to interpret CIs in the wrong way.  Confusing probability with &quot;rational degree of belief&quot; is not consistent with the frequentist paradigm. This is what happens when you take CIs to mean &quot;probability of true value being in interval&quot;, which is what @dsimcha is doing in the question." CreationDate="2011-01-31T21:10:33.077" UserId="2392" />
  <row Id="10700" PostId="6737" Text="Nope, the prior is not necessarily incorrect, unless one didn't actually observe a value of $0$ prior to conducting the experiment (or obtain some equivalent knowledge).  It basically means that, as the true $\theta$ becomes arbitrarily large, the probability of observing these implicit observations becomes arbitrarily small (like getting a &quot;unlucky sample&quot;)." CreationDate="2011-01-31T21:22:33.277" UserId="2392" />
  <row Id="10701" PostId="6737" Text="you can see by noting that the sample consists of an observation at $0$ and another one at $X$.  $0$ is fixed (because it has been observed), but $X$ will be &quot;close&quot; to $\theta$ in most cases.  So as $\theta$ becomes large, the sample average gets further and further away from both $X$ and $0$, and because the variance is fixed, the width of the CI is fixed, so it will eventually not contain either $X$ or $0$, and hence not be near either of the two likely values of $\theta$ (for one of them is an outlier when they become far apart, for fixed $\theta$)" CreationDate="2011-01-31T21:27:17.950" UserId="2392" />
  <row Id="10702" PostId="6714" Text="@probability Thank you for the explanation.  I had understood your reply as being in accord with a definition of &quot;probability = proportion.&quot;  In fact, a close rereading still suggests this is what you are saying in the third paragraph, even though your comment now characterizes this as a misunderstanding.  You might want to clarify this point." CreationDate="2011-01-31T21:42:44.163" UserId="919" />
  <row Id="10703" PostId="6754" Score="1" Text="@baris_a Could you give an example of sth that is recorded as a single entity in meters *and* seconds?" CreationDate="2011-01-31T21:48:48.033" UserId="930" />
  <row Id="10704" PostId="6605" Text="if the p-value isn't a conclusion of the null hypothesis, they why does it appear as one of the conditions required when calculating the null hypothesis.  presumably, your response implies that the p-value could be constructed without the null hypothesis.  And you cannot determine the *believability of the null* without referring to the prior probability of the null.  Otherwise, you are to believe incompatible null hypothesis simulatneously (the data is fake vs the data is not fake can give the same p-value)" CreationDate="2011-01-31T22:11:23.977" UserId="2392" />
  <row Id="10705" PostId="6605" Text="@eduardo - suppose *every* p-value for every null you could think of (including faking the data) was smaller than $10^{-5}$, what are you to believe?" CreationDate="2011-01-31T22:14:55.220" UserId="2392" />
  <row Id="10706" PostId="6758" Text="Using only `dynlm` package will not provide forecasts for your dependent variables. Providing forecasts for your dependent variables will require a model to explain them and probably additional data. I suggest you to read something about multivariate regression such as &quot;Applied Multivariate Statistical Analysis&quot; by Johnson and Wichern. or a course on forecasting: http://www.duke.edu/~rnau/411home.htm" CreationDate="2011-01-31T22:32:43.857" UserId="2902" />
  <row Id="10707" PostId="6371" Text="I don't want to delete it, but close it (on MathOverflow there's a difference, but they don't run SE2.0). I'm happy to leave it as is for the sake of posterity, but I've realised my problem is not what I was asking in this question." CreationDate="2011-01-31T22:42:41.093" UserId="1144" />
  <row Id="10708" PostId="6757" Score="1" Text="Yes I found it interesting that there was a write function but no read function.  What would be ideal is if there was a read function and then you could write it as a sparse matrix." CreationDate="2011-01-31T23:01:12.367" UserId="2310" />
  <row Id="10709" PostId="6657" Text="I've realised that what the paper cited shows is not really what I need. The paper deals with absorption probabilities calculated from samples, whereas I have complete information, and the uncertainty is not in how well I can guess the true values of the model given incomplete information, but the fit of the model to reality. " CreationDate="2011-01-31T23:24:00.790" UserId="1144" />
  <row Id="10711" PostId="6758" Text="@deps_stats The dependent variable is what I want to forecast.  I'm assuming that I already have forecasts for my independent variables.  In my example code, y is the dependent variable I am trying to forecast, and A,B,C are the independent variables, which I already have forecasts for. If you run the example code I posted, you will understand the nature of my problem." CreationDate="2011-01-31T23:31:52.407" UserId="2817" />
  <row Id="10712" PostId="6715" Text="@probabilityislogic: I see. I get your point. The data is question is a response time variable. My take on your question would be that the value does make sense in a physical world but its just too unusual enough to be called a rare case. The person I talked to said he did not do anything differently. Actually, the data that I put here is just a sample from the entire data and there are some cases like this spread here and there." CreationDate="2011-02-01T00:17:27.080" UserId="2164" />
  <row Id="10713" PostId="6716" Text="@whuber: I did not intend to see it is anyone's fault. I am a novice here so I apologize if sounded so. @robin girard: That is a very interesting take. Thanks. I was just thinking about outlier detection. Will you be able to point me to some relevant material for this particular case? All I have used before are simple ones like k-means etc." CreationDate="2011-02-01T00:20:07.857" UserId="2164" />
  <row Id="10715" PostId="6763" Text="Which software are you working with?" CreationDate="2011-02-01T00:36:40.853" UserId="71" />
  <row Id="10716" PostId="6652" Text="Part of me applauds the question (+1).  A competing part wants to point out that 1. The vast majority of statistics consumers, people who use statistics pragmatically but not philosophically in order to make their point in chemistry or market research, will never grasp the niceties of the issues, and we will often be at a loss to explain results.  2. Even some purist statisticians can fall into the trap of making supposedly probabilistic statements like those involving confidence intervals when they are not working with random samples.  A much bigger issue." CreationDate="2011-02-01T01:33:49.450" UserId="2669" />
  <row Id="10718" PostId="6715" Text="@legend - so it would appear that the most likely result is an error, but interesting discoveries can be made if you &quot;dig deeper&quot; so to speak.  Could possibly be a new finding of some sort!  but don't get too excited, it probably is nothing, but it may be worthwhile to entertain the possibility, and see where it leads you." CreationDate="2011-02-01T02:29:35.833" UserId="2392" />
  <row Id="10720" PostId="6770" Text="1/4 1/8 1/16 ...." CreationDate="2011-02-01T02:39:49.593" UserId="1585" />
  <row Id="10721" PostId="6770" Text="so... you're saying you don't see any consistent relationship between the amounts at the various lengths?" CreationDate="2011-02-01T02:44:26.813" UserId="601" />
  <row Id="10722" PostId="6770" Text="each one is 1/2 of the one before and  1/2 + 1/4 +1/8 ... =1" CreationDate="2011-02-01T02:45:54.597" UserId="1585" />
  <row Id="10724" PostId="6770" Text="Well yes, that is why I said that the pattern is obvious. But I need a formula, preferably with an explanation as to WHY each one is 1/2 of the one before, and how do I get the first value. I need to calculate this independently of the simulation, and then compare these two results. Don't get me wrong, I'm not looking for someone to solve my homework for me, I just don't know what to do anymore." CreationDate="2011-02-01T03:57:08.850" UserId="2996" />
  <row Id="10726" PostId="6371" Text="So you could accept my answer if it fits your question as you asked it, then open a new question on stats.SE with what you really are interested in." CreationDate="2011-02-01T06:24:32.800" UserId="2592" />
  <row Id="10727" PostId="6605" Text="@probabilityislogic: Setting the alpha of a test is a matter of &quot;policy,&quot; if you want to give it a name. Given that the null hypothesis is true, the p-value could be _anything_ from 0 to 1, even a small value. So, from a strictly logical point of view, you cannot _decide_ (in a deductive way) whether the null hypothesis is true or not. So, at the risk of being wrong, you establish that the p-value shouldn't be lower than alpha, otherwise, you _don't believe_ that the null hypothesis is true, regardless of whether it is actually true, which you will never know. There is no paradox." CreationDate="2011-02-01T07:11:33.463" UserId="2971" />
  <row Id="10728" PostId="6753" Text="@whuber: Thank you for your explanation. Your answer to the second question indicates, that all this rules are not necessarily require an equal-width-histogram. I did not know that." CreationDate="2011-02-01T07:13:23.617" UserId="264" />
  <row Id="10729" PostId="6748" Text="Thanks for the insight and references! The Gary King piece is indeed very interesting though not directly on topic." CreationDate="2011-02-01T07:31:20.270" UserId="1283" />
  <row Id="10730" PostId="6770" Text="@Davor, [this](http://mathforum.org/library/drmath/view/56637.html) might give you some ideas." CreationDate="2011-02-01T08:11:40.273" UserId="2116" />
  <row Id="10731" PostId="6761" Text="@Zach, did you look at the results? Your function still does not predict the future values. In the loop you are basicaly repeating the same thing you done before the loop. Also note, that your function will fail if the dependent variable is not in the first column of `newdata`." CreationDate="2011-02-01T08:22:43.217" UserId="2116" />
  <row Id="10732" PostId="6763" Score="2" Text="@Paul, transpose your data. You sample 8 measurements and the each batch is an element in your sample. In this case PCA expects batches in rows and measurements in the columns. " CreationDate="2011-02-01T08:27:46.303" UserId="2116" />
  <row Id="10733" PostId="6753" Text="@whuber: Although it seems &quot;strange&quot; that one can use FD to calculate bin-width, then the number of bins and then the equal-freq-bin-width." CreationDate="2011-02-01T08:37:48.587" UserId="264" />
  <row Id="10734" PostId="6653" Text="Tough question! My guess is that it would help to know more than I do about counting processes (and more than Wikipedia does: http://en.wikipedia.org/wiki/Counting_process )" CreationDate="2011-02-01T09:30:44.077" UserId="449" />
  <row Id="10735" PostId="6749" Text="Thanks. So let's say that the $K$ functions for processes $x$ and $y$ are given by $K_x(t)$ and $K_y(t)$. How do I test the hypothesis that $x$ is independent of $y$? Under the assumption of stationarity I can simply look at $L_{xy}(t) = \sqrt{K_{xy}(t)/\pi}$ and look for departures from $L_{xy}(t)-t=0$. For non-stationary distributions I see that $K_{xy}(t)$ will in general be different from $K_{yx}(t)$, but I don't see what the impact of this on my hypothesis testing should be." CreationDate="2011-02-01T10:07:20.120" UserId="2425" />
  <row Id="10738" PostId="6605" Text="@Eduardo - but if you don't believe the null is true, then you are acting *as if* $P(H_0)=0$.  And when this occurs, the conditional probability based on $H_0$ is *undefined* $P(T&gt;T_{obs}|H_0)=\frac{P(T&gt;T_{obs},H_0)}{0}$.  By putting your *prior probability* of $H_0$ in addition to the p-value $P(H_0)\times P(T&gt;T_{obs}|H_0)$ the $P(H_0)$ in the denominator cancels out, and you are no longer dividing by zero, and you are just left with the joint probability.  This is a more sound way to reason because you are reject the null AND the data as a combination." CreationDate="2011-02-01T10:49:04.697" UserId="2392" />
  <row Id="10739" PostId="6754" Text="@chl assume there is a runner, who runs 100 m in 15 secs, and 200 m in 32 secs, other runner runs 500 m 40 secs and 1000 m 83 secs, who will run 1 m (or a common) distance better? is this process related with standardization or normalization? It may seem simple but we can also assume that running 1000 m is harder than 100 m." CreationDate="2011-02-01T11:39:46.643" UserId="2170" />
  <row Id="10740" PostId="6780" Text="@Diegolo, according to [wikipedia page](http://en.wikipedia.org/wiki/Zipf's_law) Zipf's law has two parameters. Number of the elements $N$ and $s$ the exponent. What is $N$ in your case, 10? And frequencies can be calculated by dividing your supplied values by the sum of all the supplied values?" CreationDate="2011-02-01T13:44:00.337" UserId="2116" />
  <row Id="10741" PostId="6780" Text="let it's ten, and frequencies can be calculated by dividing your supplied values by the sum of all the supplied values.. how can I estimate?" CreationDate="2011-02-01T13:51:02.840" UserId="2998" />
  <row Id="10742" PostId="6761" Text="@mpiktas: what I am trying to do is this: 1. Starting with the first NA value, use all previous data to construct a 1-step forecast using dyn.  Replace the NA value with this forecast, and repeat.  I thought this would iteratively step through all future NA values... what am I missing?" CreationDate="2011-02-01T14:21:24.843" UserId="2817" />
  <row Id="10743" PostId="6761" Text="@Zach, the problem is with one-step forecast. Apparently it does not work. " CreationDate="2011-02-01T14:30:42.963" UserId="2116" />
  <row Id="10744" PostId="6749" Text="You have to redo the calculations for 1D and your particular assumptions, Chris: the formulas you quote are for 2D and a homogeneous process.  Dixon's references provide the theory." CreationDate="2011-02-01T14:37:16.527" UserId="919" />
  <row Id="10745" PostId="6778" Text="This is the point of departure in Dixon's survey I referenced." CreationDate="2011-02-01T14:38:11.220" UserId="919" />
  <row Id="10746" PostId="6753" Text="@Steffen Sorry, I was mistaken.  Those rules are for equal-width histograms.  For equal-frequency histograms the theory is different, because you determine the *area* of each bar in advance.  Thus, variation in the area is proportional to the square root of the (common) bin population.  Choosing that population is therefore a tradeoff between horizontal precision (number of bars) and areal precision; where to come down in that tradeoff is your decision." CreationDate="2011-02-01T14:45:23.627" UserId="919" />
  <row Id="10747" PostId="6781" Score="1" Text="There's also an R package zipfR http://cran.r-project.org/web/packages/zipfR/index.html&#xA;I haven't tried it though." CreationDate="2011-02-01T14:51:30.560" UserId="449" />
  <row Id="10748" PostId="6766" Score="1" Text="+1 You can obtain the distribution with an (efficient) dynamic program.  Let $p(n,j,k,m)$ be the probability of $j$ isolated actors and $k$ non-isolated actors choosing from $m$ actions.  Adding one more actor gives the recursion $p(n+1,j,k,m)$ = $p(n,j,k,m)(k/m)$ + $p(n,j+1,k-1)((j+1)/m)$ + $p(n,j-1,k)(m-j-k+1)/m$.  Of course $p(n,1,0,m) = 1$, letting us start the recursion.  Computations simplify when you base them on $m^{n-1}/(m-1)^{[j+k-1]}p(n,j,k,m)$: the values are integers independent of $m$.  ($x^{[j]}$ is the factorial power $x(x-1)\cdots(x-j+1)$.)" CreationDate="2011-02-01T15:03:25.000" UserId="919" />
  <row Id="10749" PostId="6781" Text="@onestop, thanks for the link. It would be nice if someone would answer this question using this package. My solution definitely lacks depth, though it gives some kind of answer." CreationDate="2011-02-01T15:09:56.830" UserId="2116" />
  <row Id="10750" PostId="6770" Text="Well you said a formula and 1/2 1/4 1/8 etc is certainly a formula. It seem to me it indicates a rather simple proof.  Suppose you start throwing the coin and record the length of each run.   the first throw is either H or T.  the second throw can be H or T and the probability is  0.5 that they are it is different from the first throw. thus the probability is 0.5 that the length of the string is 1.  Similarly it is 0.25 that the length of the string is 2 and so on." CreationDate="2011-02-01T15:53:50.503" UserId="1585" />
  <row Id="10751" PostId="6761" Text="@It seems to work with the dyn function, rather than the dynlm function." CreationDate="2011-02-01T16:17:50.203" UserId="2817" />
  <row Id="10752" PostId="6772" Score="2" Text="Assuming A and B are uncorrelated, use the variance propagation formula on Wikipedia at http://en.wikipedia.org/wiki/Fieller%27s_theorem#Case_1 .  Because your CIs are so small, we know the sampling distributions are approximately normal and the sample sizes are fairly large.  Therefore you can back-calculate Var(A) and Var(B) from your CIs in the usual way, for input in the variance propagation formula.  (BTW, your edit is mysterious.  Perhaps you are confusing the coverage of a CI with the interval itself?)" CreationDate="2011-02-01T16:23:52.373" UserId="919" />
  <row Id="10753" PostId="6778" Text="Yes, I think I understand now. I didn't grasp the full range of your answer when I read it the first time!" CreationDate="2011-02-01T16:54:59.887" UserId="2425" />
  <row Id="10754" PostId="6783" Score="1" Text="What do you means by &quot;proportionally&quot;? Proportional to what?" CreationDate="2011-02-01T16:56:05.753" UserId="449" />
  <row Id="10755" PostId="6605" Text="@probabilityislogic: No. Not believing it is plausible that the null hypothesis be null doesn't mean acting as if the probability of the null hypothesis were zero. It means &quot;it might be true, but I won't _bother_ taking that possibility into consideration anymore.&quot; Remember that, in real life, we use statistical inference to make decisions when we don't have enough information to use the strictly deductive approach." CreationDate="2011-02-01T17:39:19.397" UserId="2971" />
  <row Id="10756" PostId="6785" Text="Thank you very much for your answer. All the preidictor you mentioned are meaningful. To simply the problem, lets suppose I am trying to find out the number of library visit and how many book borrowed, how long each book kept etc realted to a student final mark of this course." CreationDate="2011-02-01T17:46:35.207" UserId="2454" />
  <row Id="10757" PostId="6786" Text="@whuber, +1 nice answer!" CreationDate="2011-02-01T17:46:38.417" UserId="2116" />
  <row Id="10758" PostId="6786" Text="@mpiktas Thanks.  And +1 to you especially for the R code." CreationDate="2011-02-01T17:50:23.637" UserId="919" />
  <row Id="10759" PostId="6785" Text=" if I use predictor a1, b1, c1 to build network(a1 is avearge number of library visit per week, b1 is average book borrowed weekly etc).  and I build another network with a2, b2, c2 (a2 is total number of libary visit, b2 is total book borrowed). ...  what if I want to find out combination (a1, b2, c1)??   Does that mean I have to include all these variation in the network? if yes, which one is parents? a1 or a2 ... and I think the network will be very compliate.  there must be smart way to do it I think?  " CreationDate="2011-02-01T17:53:55.960" UserId="2454" />
  <row Id="10760" PostId="6786" Score="1" Text="@whuber, I might humbly suggest a little caution with the formulation given above. Zipf's law is usually stated as a relative-frequency result. It is not (normally considered) the distribution from which an iid sample is drawn. An iid framework is probably not the best idea for these data. Perhaps I'll post more on this later. " CreationDate="2011-02-01T18:09:15.460" UserId="2970" />
  <row Id="10761" PostId="6785" Text="My main goal is to find something like:  if a student go to libary total number &gt;50,and  average weekly borrowed books &gt;3, or last week borrow book.2 ....  then the probability of the student get good mark is 80%.      as you can see, not only the structure of these rules will be different combination of different predictor,  I also need to find the threshold for each one, such as 50 for total book borrowed in the example.   as Bayesian network normally do not take contiunous values,  thus I possibly could try different thresholds to discretize these continuous value.   " CreationDate="2011-02-01T18:09:37.830" UserId="2454" />
  <row Id="10763" PostId="6786" Score="2" Text="@cardinal I look forward to what you have to say.  If you don't have time for a thorough response, even a sketch of what you think might be the &quot;best idea for these data&quot; would be most welcome.  I can guess where you're going with this: the data have been ranked, a process which creates dependencies and should require me to defend a likelihood derived without recognizing the potential effects of the ranking.  It would be nice to see an estimation procedure with sounder justification.  I am hopeful, though, that my analysis can be rescued by the sheer size of the dataset." CreationDate="2011-02-01T19:20:06.183" UserId="919" />
  <row Id="10764" PostId="6786" Text="@whuber, there is an error in your log-likelihood function, the first term should be $-s\sum_{j=1}^nf_j\log(j)$. You used correct formula in the calculations, that is how I could check :)" CreationDate="2011-02-01T19:20:27.383" UserId="2116" />
  <row Id="10765" PostId="6786" Text="@mpiktas Thank you!  I double-checked: my code indeed has the correct formula and what you found is a typo, so I'll just go ahead and change it." CreationDate="2011-02-01T19:21:42.343" UserId="919" />
  <row Id="10766" PostId="6605" Text="@Eduardo - but rejecting the null hypothesis *does* mean you are acting *as if it is false*.  If you are using p-values as a heuristic guide to your inference, then there is no quams with that.  But a low p-value on its own is *not a direct measure of the evidence* against the null( [Bernardo and Rueda](http://www.stat.duke.edu/research/conferences/valencia/IntStatRev.pdf) ).  [This paper](http://predictive.files.wordpress.com/2010/03/binder1.pdf)  also shows some of the logical flaws of p-values" CreationDate="2011-02-01T20:02:08.143" UserId="2392" />
  <row Id="10767" PostId="6605" Text="@probabilityislogic: Who said statistical inference was a logical process? Basically, the situation is this: Someone would like to know something about a distribution (usually to make a decision), however, all he has is a limited subset of data drawn from that distribution. **From a strictly logical point of view, he simply does not have enough information to do anything at all. But he must do something anyway.** Thus, he introduces &quot;premises&quot; (such as, if a p-value were calculated, it shouldn't be lower than alpha) in order to make up for the information he doesn't have." CreationDate="2011-02-01T21:32:25.937" UserId="2971" />
  <row Id="10768" PostId="6782" Text="Great, that works - thanks DWin." CreationDate="2011-02-01T21:49:17.903" UserId="2595" />
  <row Id="10769" PostId="6781" Text="@mpiktas (+1) You're really impressive. So many good contributions in so many different statistical fields!" CreationDate="2011-02-01T22:22:30.587" UserId="930" />
  <row Id="10770" PostId="6786" Text="@whuber +1 for this detailed explanation of the modeling approach, and ensuing comments." CreationDate="2011-02-01T22:26:38.070" UserId="930" />
  <row Id="10771" PostId="6792" Score="3" Text="@AJB If you have a specific question that differs from this one, please feel free to post it as a new question (and consider providing a link to this one, since it seems to have some bearing on it).  Otherwise a remark like this is best provided as a comment rather than a reply.  BTW, you will need to provide more information, because you haven't specified the payoffs for the loaded die.  This makes it difficult to tell whether your question is a genuinely different one or has already been answered here." CreationDate="2011-02-02T00:18:59.627" UserId="919" />
  <row Id="10772" PostId="6786" Text="@whuber, unfortunately, I fear the margins of this comment are a bit small, yet I don't think my thoughts on this would qualify as an &quot;answer&quot;. Can you think of a sensible &quot;generative&quot; model from which your likelihood would arise and which would match the structure of this data?" CreationDate="2011-02-02T01:17:07.677" UserId="2970" />
  <row Id="10773" PostId="6783" Text="1000 institutions, or 1000 points?" CreationDate="2011-02-02T01:19:16.063" UserId="2817" />
  <row Id="10774" PostId="6792" Text="@AJB I agree with whuber.  Please delete your answer and post a separate question with more details." CreationDate="2011-02-02T01:23:29.930" UserId="5" />
  <row Id="10775" PostId="6371" Text="Yes, I know what my options are :) Unfortunately the real problem is not a statistical one...." CreationDate="2011-02-02T02:28:30.380" UserId="1144" />
  <row Id="10776" PostId="6653" Text="Do you have access to the ASA journal Chance?  Seems to me there was a relevant article appearing in the last year or so, whether about hockey or another sport." CreationDate="2011-02-02T04:33:14.670" UserId="2669" />
  <row Id="10777" PostId="2369" Text="Hi -- yes, a confidence interval's coverage probability is bounded below by the confidence parameter. So a 95% confidence interval will have coverage of at least 95%, irrespective of the true value of the parameter. A credibility interval does not make this guarantee, and can have coverage lower than its probability -- it can even have 0% coverage for some values of the parameter, as in the &quot;king&quot; example. See http://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval for a fuller explanation." CreationDate="2011-02-02T05:37:39.203" UserId="1122" />
  <row Id="10778" PostId="6781" Text="@chl, thanks! I certainly feel that I am not the only one with such characteristics in this site though ;)" CreationDate="2011-02-02T07:37:26.307" UserId="2116" />
  <row Id="10779" PostId="6786" Text="@cardinal, do not do Fermat on us :) If you have some different insight than other answerers feel free to express it in the separate answer, even if it will not constitute a valid answer per se. In [math.SE](http://math.stackexchange.com) for example such situations arise quite frequently." CreationDate="2011-02-02T07:43:41.933" UserId="2116" />
  <row Id="10780" PostId="6761" Text="@Zach, I thought dyn$lm was a typo of dynlm. I did not know about package dyn. With it everything works. For the code, my other comment about first column of `newdata` still holds. Also you might want to look into using `call` objects. They make the passing of default arguments much more easier. You can ask about it in stackoverflow. " CreationDate="2011-02-02T07:52:27.290" UserId="2116" />
  <row Id="10781" PostId="6761" Text="@Zach, as this answer fully solves the problem, I suggest deleting your other answer (with `parseCall`), and rephrasing this answer to stress the usage of function `dyn$lm`. I will delete my answer also, since `dyn$lm` solves the problem I was refering in it. This way it will be clearer for people who will stumble upon this question (using Google search or other ways), how to deal with dynamic regression in `R`." CreationDate="2011-02-02T07:55:54.760" UserId="2116" />
  <row Id="10782" PostId="6761" Text="@Zach, +1 for the `dyn$lm` function." CreationDate="2011-02-02T07:57:10.517" UserId="2116" />
  <row Id="10783" PostId="6784" Text="@Zach, +1 for reference to arima." CreationDate="2011-02-02T07:57:33.263" UserId="2116" />
  <row Id="10784" PostId="6796" Text="Yes, I realize I'm not &quot;supposed&quot; to be looking at some of these things for ordinal data, but really, it's the only tool I could think of to compare the two years. Really, I was looking at things that could compare the distributions. But, I guess testing means could be plausible - but a confidence interval may not necessarily include my mean as there have been many structural changes to the industry for which this question reviews YoY." CreationDate="2011-02-02T09:11:28.210" UserId="776" />
  <row Id="10785" PostId="6797" Text="As a note to your second comment. The anchors are the same as they were in the previous year of the survey. Essentially, the granularity of the scale was reduced." CreationDate="2011-02-02T09:16:59.027" UserId="776" />
  <row Id="10787" PostId="6785" Text="You may think a decision tree will work best for this kinda problem. however, there are a lot unknown factors will affect the result, thus I think probability based model like Bayesian network might be more suitable" CreationDate="2011-02-02T10:18:57.480" UserId="2454" />
  <row Id="10788" PostId="6791" Text="why do you say Likert scale and then Categorial/Ordinal? Likert means interval scaled. Can you clarify this a little?" CreationDate="2011-02-02T10:36:48.803" UserId="442" />
  <row Id="10789" PostId="6785" Text="@Zhang: 1. decision trees are also based &quot;on counting&quot; and calculating approximate probabilities (similar to bayes), so this is no argument. The question is whether you want to have rules at the end (as stated in one of your comments) or not. 2. If you do not want to have rules, I'd try a naive bayes first and if the results are not satisfying, switch to bayesian network 3. a) Bayesian Networks do take continuous variables, it is just more difficult b) finding the correct/best network architecture is an NP-problem. I suggest to learn/read more how to build BN." CreationDate="2011-02-02T10:53:21.430" UserId="264" />
  <row Id="10790" PostId="6791" Text="To be more specific, the title should be changed to Likert &quot;item&quot;. On your second point, I think a lot of people would disagree as to whether or not a Likert item presents interval or ordinal data. For my question, it's an agreement scale, from strongly disagree to strongly agree. Each level of agreement being a &quot;category&quot; and the distance between being &quot;ordinal&quot;. But let's not get tied up in semantics! " CreationDate="2011-02-02T11:04:50.117" UserId="776" />
  <row Id="10791" PostId="6653" Text="I try to reformulate the problem (to stimulate discussion ?): Let's say we have a set of discrete states in a game (e.g. in tic-tac-toe). Now it is reasonable to create one model per state (maybe using logistic regression) to predict the outcome. Now HERE we have also a game, but with continuous states (i.e. game time). The question now of the OP is: How to a) discretize the time into finite set states or b) how to build a model whose parameters vary depending(!) on the current game-time. There must be someone who has already solved this &quot;general&quot; problem." CreationDate="2011-02-02T11:08:53.867" UserId="264" />
  <row Id="10792" PostId="6788" Text="@user2376, shouldn't $P(Y_i=1)$ be $P(X_{i-1}=0,X_{i}=1,X_{i-1}=0)$?" CreationDate="2011-02-02T13:08:44.360" UserId="2116" />
  <row Id="10793" PostId="6801" Score="1" Text="Note that $Y&gt;a$ almost surely, hence the interval $[y_l,y_u]$  contains the parameter $a$ with probability zero. In fact your argument works if what you are estimating is $\theta=a+\frac12$." CreationDate="2011-02-02T13:58:17.127" UserId="2592" />
  <row Id="10794" PostId="6795" Score="2" Text="@Belmont, what is $u(\alpha)$? Could you provide more context about your problem? Link to article with standard properties of LAR for example would help a lot." CreationDate="2011-02-02T14:00:46.373" UserId="2116" />
  <row Id="10795" PostId="6806" Score="1" Text="@Biorelated. May be I am doing something wrong, but your I _don't_ get any error if df$pos is a character vector. Can you pls `dput` your data.frame?" CreationDate="2011-02-02T14:29:09.213" UserId="1307" />
  <row Id="10796" PostId="6785" Text="Yes, I do want the rules.  I am quite new to Bayesian network, I plan to build a network,get all possible combination of predictors then calcuate probability.then rank them. However, I do not know how to include those predictor with time related values. one solution is like u said, build many networks with different possible preidictor, the problem is there are quite a few similar predictors, thus total number will be big. ...  another solution is to include all predictors in one network, but I do not know how to define their relationship, which one is parents which one is child etc" CreationDate="2011-02-02T14:36:01.527" UserId="2454" />
  <row Id="10797" PostId="6806" Score="1" Text="@suncoolsu : The transform call uses the as.data.frame convention of stringsAsFactors=TRUE so the goal of getting a character column was defeated. It was another factor column. Better would have been to use as.character around the argument passed to strsplit. Full code below." CreationDate="2011-02-02T14:57:16.033" UserId="2129" />
  <row Id="10798" PostId="6805" Text="@onestop: I know that they will be uncorrelated, but still it would be useful to know what is a good approach to such experiments." CreationDate="2011-02-02T15:27:57.813" UserId="1643" />
  <row Id="10799" PostId="6788" Text=" @mpiktas  If $b_i$ is the first bit in a run of ones,  why should be $b_{i+1}=0$? ( I think you intended to write $(Y_i=1)=(X_{i-1}=0, X_i=1, X_{i+1}=0)$)" CreationDate="2011-02-02T15:51:09.540" UserId="2376" />
  <row Id="10800" PostId="6808" Text="Thanks for your attention on the matter. It's nice to see that I'm not the only one getting confused over the stuff. That being said, I think you've overestimated my situation; while I have taken a number of courses and am familiar with the existence of a number of different ways of statistical analysis; they never stick with me after the courses. A couple of months after the exams, I keep finding myself wondering; &quot;I have seen/heard this somewhere, but how did it work really?&quot; This to me suggests that I need to tear it all down and start building it over with a stronger foundation." CreationDate="2011-02-02T16:08:45.790" UserId="3014" />
  <row Id="10801" PostId="6814" Text="I don't have the answer... but I am not sure that, in this context, it does really make sense to consider an inverse gaussian distribution with mean 1 and variance 3 or 5..." CreationDate="2011-02-02T16:14:12.183" UserId="3016" />
  <row Id="10802" PostId="6805" Text="I've just added a ref to my answer above." CreationDate="2011-02-02T16:17:53.030" UserId="449" />
  <row Id="10803" PostId="6786" Text="@cardinal Easily.  For example, you collect frequencies and identify and rank the ten highest.  You hypothesize Zipf's Law.  You collect a new set of frequencies and report them according to the *previous* ranking.  That's the iid situation, to which my analysis is perfectly suited, *contingent* upon the new ranks agreeing with the old ones." CreationDate="2011-02-02T16:27:10.130" UserId="919" />
  <row Id="10804" PostId="6817" Text="this seems more like a question for the or-exchange." CreationDate="2011-02-02T16:29:42.843" UserId="795" />
  <row Id="10805" PostId="6814" Score="1" Text="Careful!  The KL divergence is not a true &quot;distance&quot; because it is *asymmetric*.  In each case, which of the two possible values have you computed?" CreationDate="2011-02-02T17:21:27.687" UserId="919" />
  <row Id="10806" PostId="6761" Text="@mpiktas Thanks for the suggestion. I re-wrote the version I'm using to make use of &quot;call&quot; objects." CreationDate="2011-02-02T17:26:41.907" UserId="2817" />
  <row Id="10807" PostId="6805" Score="1" Text="+1 Good advice and good references.  It's difficult to think of what use a &quot;random&quot; model would have: no method is universal, so one needs to test it on data that are characteristic of the intended areas of application." CreationDate="2011-02-02T17:28:08.750" UserId="919" />
  <row Id="10808" PostId="6819" Score="3" Text="+1.  Maybe the ranking depends on the order in which H was computed.  E.g., one could be KL(green, red) and the other could be KL(red, blue).  Morever, tail behavior can have a profound effect on the value: what we really need to see are plots of the *logarithms* of the densities, not the densities themselves." CreationDate="2011-02-02T17:32:59.087" UserId="919" />
  <row Id="10809" PostId="6814" Score="1" Text="What distributions are these?  Both the Gamma and the Inverse Gaussian take two parameters.  The red one clearly is *not* a Gamma with a shape parameter of 0.85.  Through trial and error it looks like the Gamma has a scale of 1 and shape of 1/0.85 while the Inverse Gaussians have means of 1 and scale parameters as given.  Is this correct?" CreationDate="2011-02-02T17:39:11.183" UserId="919" />
  <row Id="10810" PostId="6786" Text="@whuber, good example. I think that last sentence is key. There is actually some recent work in this vein where it is shown that the number of words in the correct order for a model very similar to the one you posit grows quite slowly. For example, for the entire British National Corpus the number of correctly ranked words is about 75. Bruce Hill's estimator seems tailor made for similar and more general models. More to follow (hopefully). " CreationDate="2011-02-02T18:04:18.707" UserId="2970" />
  <row Id="10811" PostId="6786" Text="@whuber, also, if there is any dependence in how words are generated, then the frequencies are not iid. For the simplest example, consider multinomial sampling. Also, if with very high probability the frequency counts were in a consistent order, how could the associated random variables be iid?" CreationDate="2011-02-02T18:16:20.470" UserId="2970" />
  <row Id="10812" PostId="6786" Text="@cardinal These potential problems seem easy to overcome.  The model I used is appropriate for frequencies observed by randomly sampling the text with replacement.  That's an appropriate model for testing Zipf's law.  If you view the data (hypothetically) as an exhaustive census of a work, then statistical testing is irrelevant: the frequencies (in this example) manifestly do not follow the law.  If the ranks of the words in the dataset differ from their anticipated ranks, then it looks like sorting them makes the distribution look *more*, not *less*, like Zipf: thus my test is generous." CreationDate="2011-02-02T18:25:19.670" UserId="919" />
  <row Id="10813" PostId="6817" Score="1" Text="Yes.  It's also vague: whose &quot;standard method&quot; are we talking about?  Whose &quot;third equation&quot;?  Moreover, these issues are discussed extremely well in Numerical Recipes, www.nr.com , chapter 10." CreationDate="2011-02-02T18:29:31.677" UserId="919" />
  <row Id="10814" PostId="6786" Text="@whuber, thanks for your patience. Now I'm fully clear on your line of reasoning. Under the sampling model you've now fully fleshed out, I agree with your analysis. Perhaps your very last statement is still a bit slippery. If the sorting does not induce strong dependence than your method would be conservative. If the induced dependence were moderately strong, it might become anticonservative. Thx for you patience in the face of my pedantry. " CreationDate="2011-02-02T19:16:56.853" UserId="2970" />
  <row Id="10815" PostId="6794" Text="This question has hardly anything to do with applied or theoretical statistics, and I would have voted to close if no (thorough) answers were already given." CreationDate="2011-02-02T19:21:07.703" UserId="930" />
  <row Id="10816" PostId="6791" Text="@Henrik @Brandon There were already some discussions, headed under the [scales](http://stats.stackexchange.com/questions/tagged/scales) tag, about the nature and the way to treat Likert scale/item." CreationDate="2011-02-02T19:23:47.953" UserId="930" />
  <row Id="10817" PostId="6794" Score="1" Text="@ chl where would these questions go though? I found it quite hard to find references on setting up R, sweave and latex when i started, so thats why i answered." CreationDate="2011-02-02T19:30:31.113" UserId="656" />
  <row Id="10818" PostId="6786" Text="@cardinal Not pedantry; correctness.  My analysis blithely overlooked the important issues that you raise.  I agree that the last argument is a bit of hand-waving, too.  It would still be interesting (hint) to see an approach to the original question that is directly suited to how such frequency data are typically collected." CreationDate="2011-02-02T19:48:48.037" UserId="919" />
  <row Id="10819" PostId="6794" Text="@richiemorrisroe Well, at least a quick check on SO and Google would be helpful (IMHO). I'm not criticizing the question itself, just the fact that such questions are not really in line with CV [FAQ](http://stats.stackexchange.com/faq) and the other questions here, but I may be wrong. I've upvoted your response as well as @PaulHurleyuk's one, though. I guess this question will be kept alive because of your answers (although there's already a vote to close, which was also the reason of my warning)." CreationDate="2011-02-02T20:25:24.303" UserId="930" />
  <row Id="10820" PostId="6822" Text="thanks for the links, I will try and go through them as much as I can in the coming weeks... I have been exposed to R once before, in survival analysis course in which we did a lot of multivariate regression (cox and aelen models) and a bunch of other stuff I can't really recall. My impression of R, as a person who's very used to MATLAB was quite negative, but I did it had a lot to do with the fact that we were more or less thrown to the deep end of the pool, and then expected to learn to swim on our own, which of course led to me hating the software from then on :) Time to change that perhaps" CreationDate="2011-02-02T20:37:28.497" UserId="3014" />
  <row Id="10821" PostId="6817" Score="1" Text="The second equation makes no sense and appears to contradict the first.  What are these equations supposed to be doing?  If we take $x_k$ to be a vector then the first equation looks like an update step.  The second and third equations are not.  The third equation is part of the usual termination criteria.  Another part of those criteria is that $|F(x_{k+1})-F(x_k)|$ should be small, where $F$ is the objective function (of which $f$ is its (scaled) gradient)." CreationDate="2011-02-02T21:45:22.893" UserId="919" />
  <row Id="10822" PostId="6817" Text="I think the OP meant the second equation to be $k = k+1$. We should also not take $f(x)$ to be the function to be minimized, I believe." CreationDate="2011-02-02T21:58:21.003" UserId="795" />
  <row Id="10823" PostId="6808" Text="I would add a resounding &quot;agree&quot; for Harrell's (note spelling) text. It is excellent as is the two-package combination of the R code that accompanies it. I also think &quot;Modern Applied Statistic with S&quot; by Venables and Ripley would be a good acquisition. I had a masters level background (with an undergrad degree in physics)  before using MASS to learn R. There is a wealth of application-wisdom in that text." CreationDate="2011-02-02T22:44:04.500" UserId="2129" />
  <row Id="10824" PostId="6817" Text="I agree that this is off-topic, but possible relevant sites are in production... It could work on SO, but I'm eager to let it stay while it was answered." CreationDate="2011-02-02T23:21:26.660" UserId="88" />
  <row Id="10825" PostId="6820" Text="It is also a pretty good topic for machine learning, especially if linear methods would fail/be too general." CreationDate="2011-02-02T23:26:22.083" UserId="88" />
  <row Id="10826" PostId="6808" Text="Thanks @DWin! corrected" CreationDate="2011-02-02T23:37:37.220" UserId="1307" />
  <row Id="10827" PostId="6827" Score="1" Text="What is the final product supposed to look like? Have you already tried something that doesn't work or takes too long? Where does your &quot;table&quot; above come from?" CreationDate="2011-02-02T23:52:57.627" UserId="696" />
  <row Id="10828" PostId="6795" Text="@Belmont, This looks like a problem from Hastie, et al., *Elements of Statistical Learning*, 2nd. ed. Is this homework? If so, you might add that tag." CreationDate="2011-02-03T01:50:41.550" UserId="2970" />
  <row Id="10829" PostId="6605" Text="@Eduardo - I think you would have a hard time convincing someone that abandoning logic is the best thing to do.  You simply move from *deductive* logic to *inductive* logic when you have insufficient information to do *deductive* logic.  And of course he must do something - but the something that he does should be consistent, based on a proper test statistic.  A proper test statistic has to be defined when the null is true and when it is false. A p-value is not, as my previous comments show." CreationDate="2011-02-03T02:56:56.893" UserId="2392" />
  <row Id="10830" PostId="6814" Text="Information-theoretic interpretation of KL divergence is in answer here -- http://stats.stackexchange.com/questions/1028/questions-about-kl-divergence/1569#1569" CreationDate="2011-02-03T03:21:25.423" UserId="511" />
  <row Id="10831" PostId="6821" Text="can you give link to Mathematica source?" CreationDate="2011-02-03T03:24:34.983" UserId="511" />
  <row Id="10832" PostId="6814" Text="Also, KL-divergence is not symmetric, so to remove ambiguity it is better to say KL(A,B) or KL(B,A) instead of &quot;distance between A and B&quot;" CreationDate="2011-02-03T03:28:55.180" UserId="511" />
  <row Id="10833" PostId="6605" Text="@probabilityislogic: Inductive &quot;logic&quot; is not logic at all; it is just (sometimes educated) guessing. Fortunately for us, some forms of induction (such as statistical inference) are _reliable enough_, so the risk associated to the possibility that the induction be wrong is overshadowed by the benefit of getting more information, even if it is somewhat uncertain. We don't seek to have perfect information, just enough information to make good decisions. Remember that both deduction and induction are just tools." CreationDate="2011-02-03T05:02:54.053" UserId="2971" />
  <row Id="10834" PostId="6835" Score="5" Text="You will need much more explanation. What do you mean by &quot;trend detection&quot; and how are you defining &quot;trend&quot;? It would help if you explained your specific problem rather than ask a general and vague question." CreationDate="2011-02-03T05:04:47.763" UserId="159" />
  <row Id="10835" PostId="6821" Text="@Yaroslav I added it to the end." CreationDate="2011-02-03T05:05:03.457" UserId="919" />
  <row Id="10836" PostId="6817" Text="@shabbychef Thank you.  Yes, $f$ clearly is some multiple of the gradient of the objective function." CreationDate="2011-02-03T05:07:01.270" UserId="919" />
  <row Id="10837" PostId="6828" Text="@deps_stats: +1 on Nocedal &amp; Wright!" CreationDate="2011-02-03T05:18:19.110" UserId="795" />
  <row Id="10838" PostId="6821" Text="@Whuber : Wahou ! Thank you very much. I am going to prepare a coffee and then I focus on your answer !" CreationDate="2011-02-03T06:59:27.540" UserId="3016" />
  <row Id="10839" PostId="6785" Text="@zhang: I can only repeat myself. If you want to have rules, I'd try a decision tree approach first. BN does not give you rules without application of an additional algorithm (don't know which one, but I am pretty sure one exists). See additionally the field of feature subset selection. E.g. the tool rapidminer has a lot of functions to search the predictor space efficiently." CreationDate="2011-02-03T07:25:31.487" UserId="264" />
  <row Id="10840" PostId="672" Score="1" Text="I also want to suggest this link as some kind of low-level-explanation: http://yudkowsky.net/rational/bayes" CreationDate="2011-02-03T07:52:40.477" UserId="264" />
  <row Id="10841" PostId="6821" Text="@whuber +1, excellent and detailed answer as always :)" CreationDate="2011-02-03T07:56:36.260" UserId="2116" />
  <row Id="10842" PostId="6821" Text="@Whuber: I have read carefully your answer. As far as I understand, the blue curve is indeed closer to the red curve than the green one even if my eyes suggested me the opposite. The way to realize this is to plot the logarithms instead of the densities themselves. Right? Thank you!" CreationDate="2011-02-03T08:09:41.937" UserId="3016" />
  <row Id="10843" PostId="6812" Score="1" Text="@Dwin +1, did not know that you can pass `[[` as a function." CreationDate="2011-02-03T09:00:07.717" UserId="2116" />
  <row Id="10844" PostId="6808" Text="The Gelman regression book is wonderful, he explains it all very well, and provides R code which is really useful to check your understanding of the material." CreationDate="2011-02-03T10:04:07.910" UserId="656" />
  <row Id="10845" PostId="2742" Text="I asked a question that illustrates the need of factor rotation after PCA since PCA gives the biased result. See http://stats.stackexchange.com/questions/6575/how-to-do-primary-component-analysis-on-multi-mode-data-with-non-orthogonal-prima" CreationDate="2011-02-03T10:08:29.517" UserId="2820" />
  <row Id="10846" PostId="6841" Text="@teucer, what do you mean by simulate? You want to get the forecasts?" CreationDate="2011-02-03T10:28:11.450" UserId="2116" />
  <row Id="10847" PostId="6812" Text="@mpiktas: Yes, and one can also pass &quot;[&quot; as a function, including sometimes adding an extra comma with an empty argument followed by an second index expression in order to work on columns in dataframes or to get to parts of other lists. Probably &quot;[&lt;-&quot; as well, but I don't have a ready made example in mind, and I don't know if there might be some sort of environment issue." CreationDate="2011-02-03T12:19:36.640" UserId="2129" />
  <row Id="10848" PostId="6841" Text="@mpiktas I want  to plug in new values for x1, x2 and  get the probability distribution for trvr (which should be the sum of two normals, right?). So it is a simulation and not a  forecast..." CreationDate="2011-02-03T12:53:19.227" UserId="1443" />
  <row Id="10849" PostId="6841" Text="@teucer, in general Arrelano-Bond estimator does not assume normality. Furthermore it is used to estimate conditional expectation $E(y|x)$, so it is not a good idea to use it for simulating the probability distribution of $y$." CreationDate="2011-02-03T13:10:21.887" UserId="2116" />
  <row Id="10850" PostId="6818" Text="@shabbychef: Good to know, it also might be helpful. Thanks!" CreationDate="2011-02-03T13:11:20.513" UserId="1643" />
  <row Id="10851" PostId="6844" Text="Both books seem to have good reviews on amazon, can anyone add any opinions (perhaps in a bit more detail) on these? btw; by casella &amp; berger do you mean &quot;Statistical Inference&quot;?" CreationDate="2011-02-03T13:11:59.253" UserId="3014" />
  <row Id="10852" PostId="6449" Text="@onestop, @probabilityislogic, I've updated my answer to take your comments into account." CreationDate="2011-02-03T13:36:12.130" UserId="2116" />
  <row Id="10853" PostId="6841" Text="@mpiktas thx for the explanation. I tried a mixed effect model (lme4), the problem is somehow the fitted values (lme4::fitted) are strangely close to trvr(-1) instead of trvr. Is that normal?what can be the problem?  " CreationDate="2011-02-03T13:54:47.077" UserId="1443" />
  <row Id="10855" PostId="6841" Text="@teucer, mixed effect models are not appropriate when lagged dependent variables are used. This is a standard result which is discussed in all books which describe dynamic panel regression. The problem is that lagged variable correlates with the random effect (in mixed effect model parlance, it is individual effect in panel data terminology)." CreationDate="2011-02-03T14:00:33.497" UserId="2116" />
  <row Id="10856" PostId="6841" Text="@teucer, you will need to specify the error distribution also. Even then it will not be trivial if normality is not assumed. For normality assumption you will need to test whether your residuals are normal." CreationDate="2011-02-03T14:02:27.910" UserId="2116" />
  <row Id="10857" PostId="6841" Text="@mpiktas how can I model such systems then? The goal is to simulate next year's values. (I have mistakenly deleted my previous comment: can't I assume distributions for x1,x2 and derive a distribution for trvr in Arellano-Bond (AB) setup? Btw, the AB fitted values are also close to trvr(-1), is that normal?)" CreationDate="2011-02-03T14:05:19.763" UserId="1443" />
  <row Id="10858" PostId="6841" Score="1" Text="@teucer, why do you insist on simulating? If you want to get the feel how model works for different future values of `x1` and `x2`, simply plug them into model and calculate `trvr`. What is the purpose of your simulation?" CreationDate="2011-02-03T14:09:43.037" UserId="2116" />
  <row Id="10859" PostId="6821" Score="1" Text="@Marco Yes.  It's a good idea also to plot the pdf for the density against which you are integrating (which you had already done for this question).  You might find it helpful to read the section on &quot;Graphical Moments&quot; at http://www.quantdec.com/envstats/notes/class_06/properties.htm ." CreationDate="2011-02-03T14:09:52.123" UserId="919" />
  <row Id="10860" PostId="6839" Text="Great!  Thanks for the help.  I was going to use R, and was going to setup the data similar to how you suggested, interaction effects and all.  Glad to see I was on the right track, and I do really appreicate your time." CreationDate="2011-02-03T14:53:53.487" UserId="569" />
  <row Id="10861" PostId="6841" Text="@mpiktas OK, I can do that. But why are the AB fitted values close to trvr(-1)?" CreationDate="2011-02-03T15:15:04.043" UserId="1443" />
  <row Id="10862" PostId="684" Text="-1: does *not* take new evidence into account??  This bears no resemblance to actual frequentist theory." CreationDate="2011-02-03T16:51:14.417" UserId="919" />
  <row Id="10863" PostId="6841" Score="1" Text="@teucer, look at the coefficients, lag coefficient is 0.9, this is very large. This means that almost everything is explained by the lagged value. If the values of x1 and x2 are small they contribute little, since the corresponding coefficients for the them and the constant are small." CreationDate="2011-02-03T16:59:01.347" UserId="2116" />
  <row Id="10864" PostId="1696" Text="Wish I could vote up more than once. Great idea in the link!" CreationDate="2011-02-03T18:39:16.057" UserId="2770" />
  <row Id="10866" PostId="6850" Text="Understood ! Thank you very much, again !&#xA;I have computed the KL divergence using the integrate() function of R and I have obtained the same result as yours." CreationDate="2011-02-03T18:45:43.383" UserId="3019" />
  <row Id="10867" PostId="6853" Text="@user862, hint: use chracteristic function of bivariate normal." CreationDate="2011-02-03T18:52:59.743" UserId="2116" />
  <row Id="10868" PostId="6855" Text="Are you that lazy that you couldn't copy the Q text here? Fail." CreationDate="2011-02-03T19:05:58.473" UserId="1390" />
  <row Id="10869" PostId="6855" Text="I have thought it is not very nice, but if you wish..." CreationDate="2011-02-03T19:07:42.650" UserId="3041" />
  <row Id="10870" PostId="6856" Score="3" Text="Sounds a bit like data dredging to me. Shouldn't the focus be on what you plausibly think is an appropriate model, what covariates, transformations etc *before* you start modelling. R doesn't know you did all that model fitting to find a good model." CreationDate="2011-02-03T19:08:48.297" UserId="1390" />
  <row Id="10871" PostId="6836" Text="Thanks for the response!" CreationDate="2011-02-03T19:11:26.503" UserId="3025" />
  <row Id="10872" PostId="6836" Text="Whoa. That posted prematurely… the table structure didn't account for year since I was assuming that I'd have a separate table for each year… so the year was inherent. I think, like you suggested, I'll have separate tables while I input the data (which I have to mangle from existing Excel files), but I'll combine them into a wide-structured final analytical database." CreationDate="2011-02-03T19:18:04.903" UserId="3025" />
  <row Id="10873" PostId="6836" Text="Now I just need to figure out how JMP works with long and wide structures…" CreationDate="2011-02-03T19:18:42.297" UserId="3025" />
  <row Id="10874" PostId="6856" Score="3" Text="@Gavin - I can see this getting horrendously off-topic very quickly, but the short answer is no, I am not advocating data dredging or finding spurious relationships between random variables in a dataset. Consider a regression model that includes income. Is it not reasonable to test transformations on income to see their impact on the model? Log of income, log of income in 10s of dollars, log of income in 100s...?Even if this is data dredging - a function / summary tool that can aggregate the output from many model runs would still be very helpful, no? " CreationDate="2011-02-03T19:32:15.963" UserId="696" />
  <row Id="10875" PostId="6859" Score="1" Text="@Eduardo, +1, nice graph. It should be used with care though when the different transformation of dependent variable is used in different regressions." CreationDate="2011-02-03T19:39:58.247" UserId="2116" />
  <row Id="10876" PostId="6859" Text="mpiktas, that´s true in a table as well. Graphs just make it more compact, at the expense of precision. " CreationDate="2011-02-03T19:44:02.530" UserId="375" />
  <row Id="10877" PostId="6853" Score="1" Text="See equation (11) in http://www.stuart.iit.edu/shared/shared_stuartfaculty/whitepapers/thomopoulos_some.pdf (but watch out for the awful typesetting)." CreationDate="2011-02-03T19:44:59.237" UserId="919" />
  <row Id="10878" PostId="6858" Text="This is the right approach.  But how did the *variances* (instead of the SDs) get into the numerator of the answer?" CreationDate="2011-02-03T19:46:32.727" UserId="919" />
  <row Id="10879" PostId="6825" Text="@shabbychef I agree, this question needs another exchange site :)" CreationDate="2011-02-03T19:54:28.580" UserId="2902" />
  <row Id="10880" PostId="6858" Text="@whuber, sd is a square root from the variance, I do take the square root. Or I do not understand your question?" CreationDate="2011-02-03T20:01:13.483" UserId="2116" />
  <row Id="10881" PostId="6858" Text="@whuber, also numerator is above or below? I always mix-up those even in my native language. " CreationDate="2011-02-03T20:09:37.980" UserId="2116" />
  <row Id="10882" PostId="6858" Text="@whuber, ok I've got it. Thanks. The reason is that it is late :)" CreationDate="2011-02-03T20:14:11.607" UserId="2116" />
  <row Id="10883" PostId="6858" Text="@mpiktas Looks good (+1)." CreationDate="2011-02-03T20:31:10.097" UserId="919" />
  <row Id="10885" PostId="6855" Score="1" Text="This is quite strange since those are implementation of the same algorithm... Are you sure you are not comparing OOB error estimation with error on train (this one is usually near 0 for RF, and this does not mean overfitting)?" CreationDate="2011-02-03T22:44:49.153" UserId="88" />
  <row Id="10886" PostId="6855" Score="1" Text="RFs are generally not prone to overfitting issues... With default settings, 500 trees will be grown, considering $\sqrt{9}=3$ variables each time. As @mbq said, the `cparty` package relies on the `randomForest` package, but add some convenient way to assess &quot;conditional importance&quot;. It's hard to tell anything without knowing how data are structured (No. classes, sample size/class, etc.). Of note, if you're trying all parameters combinations for your model without using a cross-validation scheme, then you're likely to break the control exerted by bagging etc." CreationDate="2011-02-03T22:54:17.707" UserId="930" />
  <row Id="10887" PostId="6855" Text="No, I compare both train errors and for RF it is usually equal to zero, but for cforest (party) it is much bigger and not expected... I know RF errors on additional test data are typically about 10-20%. " CreationDate="2011-02-03T23:12:31.627" UserId="3041" />
  <row Id="10888" PostId="6855" Score="1" Text="If so, I'm clueless... As @chl wrote, some code (or, ideally, a reproducible example) would be helpful. It may be also a bug/strange feature of party itself, so it can be a good idea to consult package maintainer. " CreationDate="2011-02-03T23:53:55.543" UserId="88" />
  <row Id="10889" PostId="6855" Text="@mbq I found that randomForest is based on CART trees and cforest - on unbiased conditional inference trees. " CreationDate="2011-02-04T00:50:59.023" UserId="3041" />
  <row Id="10890" PostId="6844" Text="Yes 'statistical inference'. For me a big step was going from understanding probability models to understand how to use data to test models and estimate parameters of models. Especially the Davison book really focuses on this point." CreationDate="2011-02-04T01:02:32.227" UserId="3033" />
  <row Id="10891" PostId="6865" Text="Hey @caracal, I think the way you are using &quot;Split-Plot Design&quot; is not the way as Casella, George defines it in his book, Statistical Design. Split Plot definitely talks about nesting, but is a special way of imposing the correlation structure. And most of the time you will end up using `lme4` package to fit the model AND NOT `lm`. But this may be a very specific book-based view. I will let other's comment on it. I can give an example based on how I interpret it which is different from yours." CreationDate="2011-02-04T01:31:24.907" UserId="1307" />
  <row Id="10892" PostId="1639" Score="1" Text="F-test is the generalization of t test. t-test is for 2 treatment comparison and the F test is for multiple treatments. The derivation is in Casella's Statistical Design, Chapter 3 and 4. However, as Prof. Hyndman points out, with unequal variances, it is not a t-test anymore. It is the Fisher Behren's problem. We generally don't use the Fisher's solution, instead use Welch's Test or a Bayesian approach." CreationDate="2011-02-04T01:35:03.200" UserId="1307" />
  <row Id="10893" PostId="6859" Text="@Eduardo can you please share the code for graphs?" CreationDate="2011-02-04T01:40:00.670" UserId="1307" />
  <row Id="10896" PostId="6870" Score="2" Text="To truncate, use T(-,-), but read the users manual for info on censuring and truncationq" CreationDate="2011-02-04T04:06:11.787" UserId="1381" />
  <row Id="10898" PostId="6799" Text="Thank you for the thorough respose. KPSS is having trouble assessing my output, but I think that this is the sort of analysis that I was trying to do. I will keep the question open a bit longer, in case there are some other suggestions." CreationDate="2011-02-04T04:25:38.940" UserId="3010" />
  <row Id="10899" PostId="6799" Text="@Marshall Ward, KPSS has a problem if the non-stationary part is only a small part of the all series. If there are new observations coming up all the time, I suggest using `monitor` from the same **strucchange** package." CreationDate="2011-02-04T06:48:17.210" UserId="2116" />
  <row Id="10900" PostId="4280" Text="@mariana +1, I can confirm that this works and it is very simple. I used it in my pretty complicated project: jQuery+PHP+R." CreationDate="2011-02-04T06:53:41.290" UserId="2116" />
  <row Id="10901" PostId="6859" Score="2" Text="@suncoolsu R code is available on the first link given in @Eduardo's response. He he, it's `grid`, not `lattice` :)" CreationDate="2011-02-04T07:58:48.097" UserId="930" />
  <row Id="10902" PostId="5257" Text="@mpiktas I found a potential bug in fitted.pgmm and left an &quot;issue&quot; in github..." CreationDate="2011-02-04T08:33:31.233" UserId="1443" />
  <row Id="10903" PostId="5257" Score="1" Text="@teucer, thanks I've replied in github." CreationDate="2011-02-04T08:41:30.840" UserId="2116" />
  <row Id="10904" PostId="6838" Text="@robin girard, this is a very interesting question. Is it possible to play at least with a subset of anonymized data? And how were the forecasts generated, what kind of model was used?" CreationDate="2011-02-04T08:49:02.957" UserId="2116" />
  <row Id="10905" PostId="6838" Score="1" Text="@mpiktas thanks, you can concider it has been generated with a suitable AR modeling (one for each wind farm), it won't change the problem very much. Sorry, there are too much confidenciality issues with these data, can't provide you anything, even anonymized... " CreationDate="2011-02-04T09:57:43.237" UserId="223" />
  <row Id="10906" PostId="6865" Text="@suncoolsu The terminology in the social sciences might be different, but both Kirk (1995, p512) and Maxwell &amp; Delaney (2004, p592) call models with one between- and one within-factor &quot;split-plot&quot;. The between-factor provides the &quot;plots&quot; (analogous to the agricultural origin)." CreationDate="2011-02-04T10:46:45.363" UserId="1909" />
  <row Id="10907" PostId="6869" Text="@suncoolsu I appreciate the intro to analyzing split-splot designs with mixed-effects models and further background info! It certainly is the preferred way to carry out the analysis. I've updated my question to emphasize that I'd still like to know how to do this &quot;the old way&quot;." CreationDate="2011-02-04T10:51:53.373" UserId="1909" />
  <row Id="10908" PostId="6605" Text="@Eduardo - The problem with most forms of statistical inference is that they are hard to generalise, becuase they are often based on intuition rather than solid foundations.  The p-value is such an example (how to adjust for multiple comparisons, how to deal with nuisance parameters, arbitrary choice of the statistic on which the p-value is based).  Without a set of more broad axioms or desiderata to guide the generalisation, p-values are basically restricted to the realm of 1 parameter problems, or where pivotal quantities exist." CreationDate="2011-02-04T10:59:36.213" UserId="2392" />
  <row Id="10909" PostId="6839" Score="1" Text="Be careful with the non-independence generated by including multiple time-slices. A random effects (multi-level) model could help. " CreationDate="2011-02-04T11:52:17.223" UserId="375" />
  <row Id="10910" PostId="6714" Text="Is (1) really the commonly accepted definition of classical probability? I find it circular, and use a different one myself. Specifically, the long-run proportion is a random variable, and only approaches the probability in probability. The definition I use is a many-worlds interpretation, in which probability is the proportion of relevant possible futures. My understanding is that quantum physics isn't at war with this interpretation, but that's difficult to ascertain with coevolutionary fields." CreationDate="2011-02-04T12:48:01.187" UserId="2456" />
  <row Id="10912" PostId="6877" Text="I've edited with your suggestion to clarify, if you'd like to add anything." CreationDate="2011-02-04T13:51:35.503" UserId="3048" />
  <row Id="10913" PostId="6877" Text="It's interesting that you don't feel any specific courses are good indicators. In comparison, I've found that when interviewing programmers, great programmers have often studied any combination of Advanced Algorithms II/III, Cryptography, Paradigms of Programming languages, or Compiler Construction. If their grades in those are good they seem more likely to be good programmers." CreationDate="2011-02-04T13:55:55.797" UserId="3048" />
  <row Id="10914" PostId="6876" Text="If someone wants to re-tag this question I'd be happy. I couldn't with my low rep." CreationDate="2011-02-04T14:00:01.423" UserId="3048" />
  <row Id="10915" PostId="6605" Text="@probabilityislogic: I don't think it's desirable to generalize p-values to n-parameter problems. I find it conceptually simpler to reduce those problems to 1-parameter goodness of fit problems. It might not fit your definition of beautiful (it certainly doesn't fit mine), but it works." CreationDate="2011-02-04T14:17:08.843" UserId="2971" />
  <row Id="10916" PostId="6839" Score="1" Text="@ Eduardo: I agree that the dependence is not modelled and that this is somewhat problematic, thanks for pointing it out. I'm not sure how random effects would help -- since the binary outcome `win_home` is constant at the level of grouping (i.e. for all time slices for any given match it's either 0 or 1), including, e.g. a random intercept, for the matches will just result in huge problems with separation in this context." CreationDate="2011-02-04T14:44:05.380" UserId="1979" />
  <row Id="10917" PostId="4551" Score="10" Text="possible duplicate of [What are common statistical sins?](http://stats.stackexchange.com/questions/4551/what-are-common-statistical-sins)" CreationDate="2011-02-04T14:46:45.127" UserId="919" />
  <row Id="10918" PostId="6839" Text="You might also want to consider including a parameter for total goals scored, as leads tend to be given away more easily in high scoring games." CreationDate="2011-02-04T14:51:14.607" UserId="229" />
  <row Id="10919" PostId="6340" Text="You maybe meant to say the gamma, beta, etc. functions are &quot;transcendental.&quot;" CreationDate="2011-02-04T15:15:15.643" UserId="2971" />
  <row Id="10920" PostId="6867" Score="1" Text="See http://stats.stackexchange.com/q/5504/919 ." CreationDate="2011-02-04T15:41:18.847" UserId="919" />
  <row Id="10921" PostId="6862" Score="2" Text="@gd047 - Thanks for posting this, very insightful on how to approach issues like this. +1!" CreationDate="2011-02-04T16:35:37.617" UserId="696" />
  <row Id="10922" PostId="6859" Text="@Eduardo - Thanks for the detailed answer, I wasn't aware of `memisc` previously, looks like a very handy package to have in one's quiver!" CreationDate="2011-02-04T16:36:40.470" UserId="696" />
  <row Id="10923" PostId="6873" Text="+1 Thanks Fabians, You are totally right, the problem is not that I don't have enough data. Note that my question is especially about the interdependence structure. Scatterplots, heatmaps and hexbin plot are good tools if they are used for the good purpose.  I think general additive model can also be very powerfull there is a wonderfull paper of Brillinger providing good hints on how to use GAM. " CreationDate="2011-02-04T16:45:06.010" UserId="223" />
  <row Id="10924" PostId="6879" Text="This is very helpful, thanks. Your answer will help me get further. I understand the need for more specific questions, which I shall do in the future. Now that I have a starting point, I can get into more specific areas. Thanks again." CreationDate="2011-02-04T17:07:03.633" UserId="3040" />
  <row Id="10925" PostId="5581" Text="Very, very important!" CreationDate="2011-02-04T17:18:08.080" UserId="2902" />
  <row Id="10926" PostId="6886" Score="1" Text="I think it may be a good idea to migrate it to Math.SE... " CreationDate="2011-02-04T18:11:30.440" UserId="88" />
  <row Id="10927" PostId="2952" Text="Hello there, &#xD;&#xA;&#xD;&#xA;Just to let you know that Gephi cannot handle weighted undirected graph to identify community through modularity. &#xD;&#xA;thanks. &#xD;&#xA;&#xD;&#xA;-Gautam" CreationDate="2011-02-04T18:24:21.047" UserId="3038" />
  <row Id="10928" PostId="2952" Text="@Gautam Good to know, thanks. I am not really familiar with Gephi, but I thought it was in active development." CreationDate="2011-02-04T19:11:29.053" UserId="930" />
  <row Id="10929" PostId="6876" Score="1" Text="@John this question is off-topic, so I will comment instead of answering. Ask about M-estimators. Or pick any problem from Assymptotic Statistics by van der Vaart. The only one problem with this question is that you cannot say that the interviewee is not excelent if he(she) did not answer. If on the other hand he (she) does answer, excellency is pretty much guaranteed. This is of course IMHO." CreationDate="2011-02-04T19:15:14.277" UserId="2116" />
  <row Id="10930" PostId="6859" Text="Memisc has a bunch of not very much related but nonetheless very useful functions. I don't use it as often as I should." CreationDate="2011-02-04T19:23:45.687" UserId="375" />
  <row Id="10931" PostId="6891" Text="@Dave To be sure, your design is really 3x2x3 (3 factors, all with 3 levels except the second one), with three replicates, that is 54 values for each statistical unit?" CreationDate="2011-02-04T19:34:19.207" UserId="930" />
  <row Id="10932" PostId="6891" Text="there are 3 factors, each of which have 3 more factors, each of which has 2 more factors (so like DrugA/B/C + DrugX/Y + ConditionA/B), for a total of 18 measurement groups" CreationDate="2011-02-04T19:38:03.073" UserId="1327" />
  <row Id="10933" PostId="6855" Text="@42n4 My fault! You're right: `cparty` does not depend on the `randomforest` package. The authors have implemented their own stuff on RFs (I just browse the R and C code). This does not solve your problem, though. Could you add example data or more context about your analysis?" CreationDate="2011-02-04T19:42:11.553" UserId="930" />
  <row Id="10934" PostId="6876" Score="2" Text="@mpiktas Are you saying that knowledge of asymptotics will assure that one is excellent at visualizing data?  There seems to be little to connect the two.  Indeed, that's the problem with this entire question: its premise is that excellence in data mining, stats modeling, and data visualization requires &quot;great mathematical statisticians&quot; who have taken lots of courses.  Neither one of those criteria--being mathematical or taking courses--seems to be closely related to succeeding in such positions." CreationDate="2011-02-04T21:08:13.470" UserId="919" />
  <row Id="10935" PostId="6877" Text="@John Things have changed.  When I was hiring programmers in the late '80s, two tests emerged as the most reliable indicators of actual performance.  The first was typing ability(!) and the second was any ability at all to do trigonometry.  The former was related (quite crudely but effectively) to experience and the latter was related to capacity for abstract thinking.  I suspect your list of courses might be measuring something similar to the latter." CreationDate="2011-02-04T21:16:28.763" UserId="919" />
  <row Id="10936" PostId="6896" Text="what is your motive behind &quot;determining the closeness&quot; of these data sets? What you are trying to do may be Ok.., but may inference can be improved if you are interested in other specifics like `mean configuration` or equivalent." CreationDate="2011-02-04T21:58:55.417" UserId="1307" />
  <row Id="10937" PostId="6896" Score="3" Text="As a side note: I don't think you mean $|\theta_1 - \theta_2| \mod \pi$. Rather something like $\min\{|\theta_1 - \theta_2|, 2 \pi - |\theta_1 - \theta_2|\}$." CreationDate="2011-02-04T23:16:03.117" UserId="2898" />
  <row Id="10938" PostId="6896" Score="3" Text="Rather than working with angles, I suggest converting to (x, y)-coordinates on the unit-circle first. You can then calculate normally (distances and the like), and averaging isn't a problem like with angles." CreationDate="2011-02-04T23:19:48.547" UserId="1909" />
  <row Id="10939" PostId="6896" Text="@suncoolsu The motive is to determine if my training set has enough similar examples to infer the poses of my test set. I am using a learning-based approach to pose estimation, and I am beginning to suspect that the data I am using is too different across data sets. I want to ensure that for each pose in a test set, there is at least a &quot;reasonably similar&quot; pose in the training set. Visually (reanimating the poses) it appears that my suspicion is well-founded, but I want to analyse it quantitatively." CreationDate="2011-02-05T01:36:53.777" UserId="3052" />
  <row Id="10940" PostId="6896" Text="@Erik Thanks, well spotted." CreationDate="2011-02-05T01:49:00.567" UserId="3052" />
  <row Id="10941" PostId="6896" Text="@caracal Can you please explain what problem exists when averaging angles? Is it specific to a particular representation, or a general problem regardless of representation? Thanks." CreationDate="2011-02-05T02:06:38.133" UserId="3052" />
  <row Id="10942" PostId="6896" Score="2" Text="@Josh Erik P.'s suggestion is a good one.  Alternatively, consider each angle $\theta$ to be a point $\left(\cos(\theta), \sin(\theta)\right)$ on the unit circle and compute the Euclidean distances between them using the usual (Pythagorean) formula.  The difference between these distances and the angular distances shouldn't matter.  (I believe this may be what Caracal suggested, too.)" CreationDate="2011-02-05T02:34:02.427" UserId="919" />
  <row Id="10943" PostId="6896" Text="@Erik, I just corrected what @Josh had and I think @whuber is right." CreationDate="2011-02-05T03:38:46.297" UserId="1307" />
  <row Id="10944" PostId="6891" Text="@Dave. Lets see: you are saying -- for each unit, you have the treatment like (., ., .) and the dots represent one of ABC, one of XY, on of AB? and you have 18 such units? For eg: a treatment may look like (A, Y, A) for _a_ particular unit?" CreationDate="2011-02-05T04:06:46.697" UserId="1307" />
  <row Id="10945" PostId="6859" Text="@Chl .. Thanks .. for the humor. Eduardo, your website is great! Thank you for sharing awesome code :-) I hope I can do it in `lattice`. :D" CreationDate="2011-02-05T04:37:30.403" UserId="1307" />
  <row Id="10946" PostId="6889" Text="What you say makes sence. We use the information on courses and grades as one of many data points to make sense of a greater picture." CreationDate="2011-02-05T07:50:34.517" UserId="3048" />
  <row Id="10947" PostId="6876" Text="@whuber - I'm not claiming it to be the only way, nor the best way. I'm hoping that it'll be a nice addition to an already extensive and exhaustive recruiting process that hopefully is more effective than asking the man on the street." CreationDate="2011-02-05T07:54:10.473" UserId="3048" />
  <row Id="10948" PostId="6896" Score="1" Text="@Josh The average of, e.g.,  $\pi/4$ and $7\pi/4$ is $\pi$. In many circumstances, this doesn't make sense, and it should be $0$ instead. In your specific situation, this might not be an issue since maybe human joints don't have a range of motion past $\pi$. Also, in your case, maybe you want the aforementioned average to be $\pi$ since the joint motion is uni-directional. @whuber's suggestion is exactly what I meant." CreationDate="2011-02-05T10:45:18.023" UserId="1909" />
  <row Id="10949" PostId="6714" Text="@sesqu- but a &quot;many worlds&quot; interpretation is useless, because we only ever observe one of them!  How do know that the &quot;other worlds&quot; obey the same set of probabilities that this particular world does? (perhaps logic does not exist in these other worlds - how could we ever know?)  It is completely arbitrary (at least to us in &quot;this world&quot;), and based entirely on assumption.  (obviously, I am not a subscriber to the &quot;many worlds&quot; - I think it is just as demonstrable as the existence or non existence of a God)." CreationDate="2011-02-05T10:46:27.257" UserId="2392" />
  <row Id="10950" PostId="6714" Text="@sesqu - in reply to the rest of your comment, you may have a point about circularity.  I don't like the definition of probability as frequency anyway, but this is not the point of the answer to the question.  If you replaced my definition with yours, I don't think it would change the conclusions, because it is something which can be confused with &quot;rational degree of belief&quot; (which I think is the best way to define probabilities - they are only known to exist in your mind, so why not define them as such?)" CreationDate="2011-02-05T10:55:46.883" UserId="2392" />
  <row Id="10951" PostId="2419" Text="Rpy is a really nasty dependency. R has a huge set of features, and thus it is nice to be able to dig in them using Rpy, but if you have to share that work, you might be in trouble, even if it is across different computers of a same lab, if your lab is in a heterogeneous computing environment. This is due to the fact that Rpy depends on having the right minor versions of Python, numpy and R. For instance, it keeps being broken in the major Linux distributions." CreationDate="2011-02-05T10:59:47.673" UserId="1265" />
  <row Id="10952" PostId="2421" Text="I am a developer of the scikit-learn. I must say that it does not including boosting or decision trees yet. Some people are working on it, so I hope it will be included in 6 months." CreationDate="2011-02-05T11:02:39.820" UserId="1265" />
  <row Id="10953" PostId="2421" Text="@Gael That's good to know! I wish you nice coding sprints :)" CreationDate="2011-02-05T11:12:48.787" UserId="930" />
  <row Id="10954" PostId="6859" Text="@suncoolsu Let's go with a `lattice` implementation then :) It should not be too hard." CreationDate="2011-02-05T11:14:19.347" UserId="930" />
  <row Id="10955" PostId="2369" Text="@Keith - if what you say is true, then you should point out the mistake I have made in my answer (relating to Wasserman's example).  Because the CI in that case does not have the 95% coverage for all values of the parameter.  So if you are correct, then logically, I must have made a mistake somewhere in the calculations." CreationDate="2011-02-05T11:25:18.147" UserId="2392" />
  <row Id="10956" PostId="2287" Text="Good answer - just one minor point, you say &quot;....Instead of saying the parameter has one true value, a Bayesian method says the value is chosen from some probability distribution.....&quot;  This is not true.  A Bayesian fits the probability distribution to express the uncertainty about the true, unknown, fixed value.  This says which values are plausible, given what was known before observing the data.  The actual probability statement is $Pr[\theta_0\in (\theta,\theta+d\theta)|I]$, where $\theta_0$ is the true value, and $\theta$ the hypothesised one, based on information $I$. " CreationDate="2011-02-05T11:34:13.310" UserId="2392" />
  <row Id="10957" PostId="2287" Text="...cont'd... but it is much more convenient to just write $p(\theta)$, with the understanding of what it means &quot;in the background&quot;.  Clearly this can cause much confusion." CreationDate="2011-02-05T11:38:26.533" UserId="2392" />
  <row Id="10958" PostId="6868" Score="1" Text="Sounds right to me.  But remember that this is basically a prior probability, because it is based on expectations (aka beliefs) about what will happen, rather than a description of what will actually happen to the price.  The price in reality is not random, because it will be set by the managers of the various companies.  Probability provides a way to describe the uncertainty due to not knowing what the other managers will actually do when setting their prices." CreationDate="2011-02-05T12:59:50.673" UserId="2392" />
  <row Id="10959" PostId="744" Text="This reminds me of a quote made by Edwin Jaynes.  It roughly goes &quot;...a mathematician came to me and said 'I found a brilliant solution, all I need now is the problem'...&quot;" CreationDate="2011-02-05T13:09:13.720" UserId="2392" />
  <row Id="10960" PostId="6714" Text="@probabilityislogic - if probability is defined as state of mind, we lose derived results (except maybe as conjugate posteriors from a description length perspective). Consider the binomial distribution - if the probability $P(K=k|p,N)$ is defined as belief, what is the $p$ parameter of that belief? Not a probability. The binomial would have to be seen as the limiting case of a beta-binomial. While such complications may prove manageable, I don't think the *rational belief* can be. As for the other worlds behaving similarly: that is explicit in the conditional iid assumption always made." CreationDate="2011-02-05T13:09:19.883" UserId="2456" />
  <row Id="10961" PostId="732" Text="I would say that the key to cracking the meaning of this quote is to recognise that the word &quot;strange&quot; is relative to what your model of &quot;normal&quot; is." CreationDate="2011-02-05T13:11:45.797" UserId="2392" />
  <row Id="10962" PostId="1280" Text="And 45.8% of people don't believe that statistic" CreationDate="2011-02-05T13:12:23.133" UserId="2392" />
  <row Id="10963" PostId="6903" Score="6" Text="Simulation of what? &quot;There is no single Monte Carlo method; instead, the term describes a large and widely used *class* of approaches.&quot; http://en.wikipedia.org/wiki/Monte_Carlo_method#Overview" CreationDate="2011-02-05T13:26:51.363" UserId="449" />
  <row Id="10964" PostId="6900" Text="You are still not presenting your problem clearly. If what you want is the local maximum of $\|f(\vec x)\|_d$, there are a variety of numerical optimization algorithms for that that are merely tangential to statistics and neural networks. Your computational environment of choice is likely to have an implementation of one, such as R's *optimize*." CreationDate="2011-02-05T13:36:17.327" UserId="2456" />
  <row Id="10965" PostId="6891" Text="yes, but maybe that was a bad choice of letters.  It would be more like ABC/XY/NM for an example of A,X,M " CreationDate="2011-02-05T13:37:09.930" UserId="1327" />
  <row Id="10966" PostId="6896" Score="2" Text="Your problem will probably become much easier to solve if you can specify the consequences of &quot;getting it wrong&quot;.  So if you say the data sets are the same or similar, but they in fact aren't, what will happen to you?  Will it depend on &quot;how wrong&quot; your decision was?  What will happen if you declare the data/poses different, but they are in fact the same or similar?  What is lost?  answering these questions will help determine *what matters* for the comparison *you* want to make.  This ensures that you are answering the right question." CreationDate="2011-02-05T13:39:26.290" UserId="2392" />
  <row Id="10967" PostId="6714" Text="@sesqu - It is a belief, conditional on *knowing* $p$ and *knowing* $N$, and *knowing* that they are parameters of a binomial distribution.  But this probability is either $0$ or $1$ once we actually observe the value $K$ (it either is $k$ or it isn't).  What changed? nothing but our state of mind" CreationDate="2011-02-05T13:49:26.667" UserId="2392" />
  <row Id="10968" PostId="6714" Text="@sesqu - on the &quot;iid&quot; comment: this may be true, but it is unverifiable if you consider &quot;alternate worlds&quot; view.  In the &quot;single world&quot; view, you can compare observations and verify the consistency of the iid assumption, but in &quot;alternate worlds&quot; how can you do this?  You have to be able to look into the other worlds, to see what results they give.  How is one supposed to do that?" CreationDate="2011-02-05T13:55:41.537" UserId="2392" />
  <row Id="10969" PostId="6889" Text="Thanks! Of course, I figured you would not base your selection process on grades, I merely tried to present an alternative perspective. I have a number of friends that are devoted programming geeks, that never cherished under the rigidity of university education. Anyways, as long as you stay open minded about the strengths and weaknesses of the candidates, I am sure you'll get great team members eventually. :) " CreationDate="2011-02-05T13:56:31.897" UserId="3014" />
  <row Id="10970" PostId="6714" Text="@sesqu - a better way to think of iid assumption is that it is one of the least restrictive assumptions you can make about data.  Each data point is &quot;free&quot; to roam without being restricted to what other data points are." CreationDate="2011-02-05T13:59:39.487" UserId="2392" />
  <row Id="10971" PostId="6714" Text="@probabilityislogic - the question I attempted to pose is the nature of $p$. What is this known thing your faith in $k$ is so specifically conditional on? Why is it conditional on $p$ in that particular manner?" CreationDate="2011-02-05T14:17:52.250" UserId="2456" />
  <row Id="10972" PostId="6714" Text="@sesqu - because that is the information that was put into the probability (i.e. the conditions on the right hand side of the $|$, plus the statement you made about it being a binomial distribution).  Probability only exists conditionally, you must specify what conditions the probability is to be calculated (i.e. you must specify your state of mind)" CreationDate="2011-02-05T15:25:50.440" UserId="2392" />
  <row Id="10973" PostId="6714" Text="@sesqu - it seems like you are asking something along the lines of &quot;how could one come to hold such a particular kind of belief/knowledge&quot;?  would I be correct in this assumption?" CreationDate="2011-02-05T15:31:33.223" UserId="2392" />
  <row Id="10974" PostId="6909" Text="Even the &quot;intrinsic discrepancy&quot; will be infinite when $P$ is zero with positive probability for $Q$ and *vice versa,* even if $P$ and $Q$ are otherwise identical." CreationDate="2011-02-05T15:56:01.810" UserId="919" />
  <row Id="10975" PostId="6907" Text="In what &quot;natural sense&quot; are these curves &quot;not that distinct&quot;?  How is this intuitive closeness related to any statistical property?  (I can think of several answers but am wondering what you have in mind.)" CreationDate="2011-02-05T15:57:36.690" UserId="919" />
  <row Id="10976" PostId="6913" Score="2" Text="Why would having a 5 star rating be fairer? There are 6/16 reviews that gave a lower rating... " CreationDate="2011-02-05T17:00:43.903" UserId="582" />
  <row Id="10977" PostId="6913" Text="Ok, than you think Mean is the right average? the majority said its 5. 60% more the rest 6/16 said so." CreationDate="2011-02-05T17:08:31.137" UserId="778" />
  <row Id="10978" PostId="6915" Text="because just one person in your example can change the result dramatically. if the person believed the book has a different topic his fault will change the rating" CreationDate="2011-02-05T17:11:31.337" UserId="778" />
  <row Id="10979" PostId="6714" Text="@probabilityislogic - yes. If probability is redefined, one can no longer use the inductive and abductive justifications for various distributions. Distributions like the binomial need to be re-derived within this belief system, and it's my understanding that this is done merely by grafting the necessary axioms onto the system, under the pseudonym &quot;rationality&quot;. These axioms are intrinsic to the nondeterministic interpretation." CreationDate="2011-02-05T17:30:24.293" UserId="2456" />
  <row Id="10980" PostId="6896" Text="@whuber Using the unit circle seems like the best approach. I can then convert each distance between two angles back into an angle ( theta = cos^-1((2-dist^2)/2) ), given the properties the unit circle and the law of cosines, right?" CreationDate="2011-02-05T17:40:33.870" UserId="3052" />
  <row Id="10981" PostId="6896" Text="@caracal Thanks for the example and also the suggestion of the unit circle representation; I understand what you mean now." CreationDate="2011-02-05T17:42:03.260" UserId="3052" />
  <row Id="10982" PostId="6858" Text="I got confused with finding expectation at a step. This solves it. Thanks." CreationDate="2011-02-05T17:43:21.137" UserId="862" />
  <row Id="10983" PostId="6907" Text="Well... they are pretty close to each other in the sense that both are defined on positive values; they both increase and then decrease; both have actually the same expectation; and the Kullback Leibler distance is &quot;small&quot; if we restrict to a portion of the x-axis... But in order to link these intuitive notions to any statistical property, I would need some rigourous definition for these features..." CreationDate="2011-02-05T17:47:28.210" UserId="3019" />
  <row Id="10984" PostId="6909" Text="Yes... I am afraid that the intrinsic discrepancy does not fulfil the requirement. But thank you for the suggestion. Any other suggestion would be appreciated." CreationDate="2011-02-05T17:49:24.697" UserId="3019" />
  <row Id="10985" PostId="6914" Text="Hi onestop.  I imagined this was the case, but couldn't quite see that from the wikipage (http://en.wikipedia.org/wiki/Tukey's_range_test), any suggestions on references for how this is performed?" CreationDate="2011-02-05T17:54:34.067" UserId="253" />
  <row Id="10986" PostId="6896" Text="@probabilityislogic The consequence is simply a pose that is not representative of the ground truth pose. It would depend on the estimation approach used as to what would occur. If knn was used then the nearest neighbour would be too distant. What is considered &quot;too distant&quot;? It depends on the application, but as a general rule, it should at least be indicative of the action being performed (e.g. in a walking sequence, you would always want to have the person in an upright position). However, in ridge regression, the learnt mapping is wrong and likely to result in output that is rubbish." CreationDate="2011-02-05T17:56:49.283" UserId="3052" />
  <row Id="10987" PostId="6896" Text="@probabilityislogic Adding to my previous comment, the learnt mapping isn't necessarily wrong (of course you have to be careful of overfitting, and other considerations... perhaps you can't even use a linear regressor given the problem statement), but applying a learnt mapping for doing push-ups to a walking sequence is going to give very wrong output poses in every case. Whereas, a jogging sequence mapping, whilst still not perfect, would yield much better results, even though it is still not the best mapping to use." CreationDate="2011-02-05T18:02:28.843" UserId="3052" />
  <row Id="10988" PostId="6913" Score="2" Text="If I had to give a discrete evaluation, seeing those 16 reviews I would give 4, not 5, as -to me- 5 would mean that all (or the great majority) of the votes are 5. 6/16 is ~40%, which is not exactly negligible. " CreationDate="2011-02-05T18:22:04.600" UserId="582" />
  <row Id="10989" PostId="6913" Score="3" Text="So, in essence, I think neither mean or median are good. Showing (as Amazon does) a bargraph with the different votes is the best option. Also, it is interesting to point out that online 1-5 ratings are not always so fair... http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html" CreationDate="2011-02-05T18:27:05.797" UserId="582" />
  <row Id="10990" PostId="6922" Text="maybe I'm confused but this appears to refer to a time series model? my model is actually simpler in that the samples are not a time series (effectively they are independent (x-&gt;y) samples, they are just accumulated in large volumes over time)" CreationDate="2011-02-05T19:26:36.793" UserId="2942" />
  <row Id="10991" PostId="6913" Text="Median is &quot;less sensitive&quot; to data. It is less affected by outliers, but also it contains less information about the original distribution" CreationDate="2011-02-05T19:43:03.487" UserId="511" />
  <row Id="10992" PostId="6922" Text="Yes, in the general case this is used for time series with non independent observations; but you can always assume incorrelation between successive observations, which gives the special case of interest to you." CreationDate="2011-02-05T19:46:18.147" UserId="892" />
  <row Id="10993" PostId="6913" Score="1" Text="@nico: scoring is full of traps, what you point out is one of the arguments of my article here: http://objektorient.blogspot.com/2010/09/scoring-you-doing-it-wrong.html" CreationDate="2011-02-05T19:56:45.703" UserId="778" />
  <row Id="10994" PostId="5966" Text="@whuber, @venkasub, you might also look at Muller's method for root-finding. It's simpler than Brent's method and &quot;almost&quot; as efficient as Newton's method, but without needing to know the derivatives of the target function." CreationDate="2011-02-05T20:24:19.533" UserId="2970" />
  <row Id="10995" PostId="6898" Text="@caracal, Thanks for your answer. It has been very useful." CreationDate="2011-02-05T21:18:33.130" UserId="221" />
  <row Id="10996" PostId="6923" Text="Is the method Maindonald describes related to or the same as Gentleman's algorithm? http://www.jstor.org/stable/2347147" CreationDate="2011-02-05T21:46:26.153" UserId="449" />
  <row Id="10997" PostId="6923" Text="@onestop It must be related.  Maindonald uses Gentleman &amp; Wilk (*Biometrics* **31**, 1975) as a reference." CreationDate="2011-02-05T21:49:58.503" UserId="919" />
  <row Id="10998" PostId="6909" Text="It does fulfil the requirement, if you restrict the support of the blue density to be where it has strictly positive support, just as you have for the red one (&gt;0)" CreationDate="2011-02-05T22:01:53.117" UserId="2392" />
  <row Id="10999" PostId="6927" Score="1" Text="@BR whats the bottle neck here? Basically what's taking time in `Rprof`." CreationDate="2011-02-05T22:14:50.443" UserId="1307" />
  <row Id="11000" PostId="6714" Text="@sesqu - One example is as a limiting form of urn sampling $N$ balls without replacement, where I know the number of &quot;red&quot; and &quot;blue&quot; balls; and they both tend to infinity in such a way that $\frac{R}{R+B}=p$ stays fixed in the limit. I then let $K$ be the number of Red balls drawn in $N$ draws.  But you always need to be a bit careful about how the limit is carried out, because a different limiting process may give different results.  This discussion has become sidetracked of course, so I suggest you start a new post if you wish to continue your line of thinking down this path." CreationDate="2011-02-05T22:17:08.980" UserId="2392" />
  <row Id="11001" PostId="6927" Score="1" Text="One way is to just ignore the &quot;mixed&quot; part of the GLMM.  Just fit an ordinary GLM, the estimates won't change much, but their standard errors probably will" CreationDate="2011-02-05T22:40:01.813" UserId="2392" />
  <row Id="11002" PostId="6923" Score="1" Text="In that case see also the extensions by Alan Miller http://www.jstor.org/stable/2347583.&#xA;An archive of his Fortran software site is now at http://jblevins.org/mirror/amiller/" CreationDate="2011-02-05T22:44:25.940" UserId="449" />
  <row Id="11003" PostId="6923" Text="@onestop Your first reference obviously solves the entire problem: at the bottom of the first page he even states it can be used for weighted least squares.  And Alan Miller's very first Fortran link (lsq.f90) contains the code.  Why don't you combine your comments into a reply so they are more prominently available?" CreationDate="2011-02-05T22:51:38.457" UserId="919" />
  <row Id="11004" PostId="6314" Score="2" Text="@Denis, Beyer et al. seem to be addressing a little bit different aspect of the NN problem. But, for (binary) classification purposes, under mild conditions, it is a classical result that 1-NN classification has, *in the worst case*, twice the probability of error of the Bayes (i.e., optimal) classifier asymptotically. In other words, the first nearest neighbor contains &quot;at least half the information&quot; about the label of the target as the best classifier does. In this sense, the 1-NN seems quite relevant. (See Cover &amp; Hart (1967) for more. I'm surprised Beyer et al. doesn't cite it.)" CreationDate="2011-02-05T23:13:04.333" UserId="2970" />
  <row Id="11005" PostId="6923" Score="2" Text="An explicit algorithm appears at the bottom of p. 4 of http://saba.kntu.ac.ir/eecd/people/aliyari/NN%20%20files/rls.pdf .  This can be found by Googling &quot;recursive least squares.&quot;  It does not look like an improvement on the Gentleman/Maindonald approach, but at least it is clearly and explicitly described." CreationDate="2011-02-05T23:20:49.330" UserId="919" />
  <row Id="11006" PostId="6295" Text="Additionally, since the interaction has the additional parameter, technically rather than a simple Likelihood-ratio test, shouldn't some information criterion (AIC, BIC, etc.) be used?" CreationDate="2011-02-05T23:25:27.153" UserId="2800" />
  <row Id="11007" PostId="6927" Text="@probabilityislogic. In addition to your remark, I also think, whether answer would differ much depends on the group size and individual behavior in group. As Gelman and Hill say: mixed effects model results would be between pooling and no pooling. (Obv. this is for Bayesian hierarchical models, but mixed models are a classical way of doing the same.)" CreationDate="2011-02-05T23:29:22.513" UserId="1307" />
  <row Id="11009" PostId="6927" Text="@probabilityislogic: That works for LMM, but it seems to fail for GLMM (meaning that I ran models with and without the extra M on the same data and ended up with significantly different results). Unless, of course, there is an error in the implementation of glmer." CreationDate="2011-02-05T23:47:59.693" UserId="2739" />
  <row Id="11010" PostId="6927" Text="@suncoolsu: the bottleneck is the estimation of the GLMM, which may a couple of seconds (especially with several random effects). But do that 1000*1000 times, and that's 280 hours of computation. Fitting a GLM takes about 1/100 of the time." CreationDate="2011-02-05T23:55:39.670" UserId="2739" />
  <row Id="11011" PostId="6927" Text="In my specific situation, there are about 50 individuals and each is measured about 10 times." CreationDate="2011-02-05T23:57:26.767" UserId="2739" />
  <row Id="11012" PostId="6929" Text="Thanks for the answer, it'll take me a bit to digest it (and I already have plans for my Saturday night). It is different enough that it is not clear to me if it gives the same answer as the GLMM approach, but I need to think about it more." CreationDate="2011-02-06T00:16:51.413" UserId="2739" />
  <row Id="11013" PostId="6923" Score="1" Text="The last link looks like the method I was going to suggest. The matrix identity they use is known in other places as the Sherman--Morrison--Woodbury identity. It is also quite numerically efficient to implement, but may not be as stable as a Givens rotation." CreationDate="2011-02-06T00:28:25.313" UserId="2970" />
  <row Id="11014" PostId="5671" Text="You would use Gibbs sampling when you know $p(y|x)$ and $p(x|y)$, but not if you know the marginal $p(x)$" CreationDate="2011-02-06T01:05:06.040" UserId="2392" />
  <row Id="11015" PostId="6909" Text="Yes indeed, restricting the support is the simplest idea but I guess that the resulting KL distance should be reduced in some way in order to take that trick into account..." CreationDate="2011-02-06T08:10:04.337" UserId="3019" />
  <row Id="11016" PostId="6919" Text="Very nice! I am going to try to find &quot;A Probabilistic Theory of Pattern Recognition&quot; and to understand its chapter 3!" CreationDate="2011-02-06T08:11:21.543" UserId="3019" />
  <row Id="11017" PostId="6924" Text="Thank you for your suggestion about the Kolmogorov distance. Can you make your comment about the monotonic transformation a little bit more explicit? Thx" CreationDate="2011-02-06T08:13:12.147" UserId="3019" />
  <row Id="11018" PostId="4551" Text="@whuber There was some good answers, so I've merged the both." CreationDate="2011-02-06T11:02:41.973" UserId="88" />
  <row Id="11019" PostId="4095" Text="Comments for downvote are always welcome!" CreationDate="2011-02-06T11:08:55.717" UserId="930" />
  <row Id="11020" PostId="4083" Text="I'll second @Andy and suggest leaving a comment when downvoting: it's always helpful to learn what's wrong, if any. Especially in this case: Andy W came back with CW extended comments which, in my opinion, add further support to the original question. There's no need to downvote anything here!" CreationDate="2011-02-06T11:10:14.870" UserId="930" />
  <row Id="11021" PostId="3917" Text="Comments for downvote are always welcome!" CreationDate="2011-02-06T11:14:15.290" UserId="930" />
  <row Id="11022" PostId="6916" Score="4" Text="No. If you lived in Glasgow, Scotland, then most people who wear a &quot;blue t-shirt&quot;, would probably be a [Rangers](http://en.wikipedia.org/wiki/Rangers_F.C.) supporter. You would be missing out on [Celtic](http://en.wikipedia.org/wiki/Celtic_F.C.) supporters. In Glasgow the [football](http://en.wikipedia.org/wiki/Soccer) team would be a proxy for religion." CreationDate="2011-02-06T13:28:29.933" UserId="8" />
  <row Id="11023" PostId="6909" Text="Actually now that I think of it, you don't need to restrict the support, because the $0$ in the log gets canceled out by the $0$ in the pdf multiplying it.  So you actually get $Lim_{P\rightarrow 0} P log(\frac{P}{Q})= Lim_{P\rightarrow 0} P log(P)=0$, and the intrinsic discrepancy is defined after all!" CreationDate="2011-02-06T14:19:13.673" UserId="2392" />
  <row Id="11024" PostId="6877" Score="1" Text="The problem with looking at courses studied is that courses with the same name at different universities can vary widely. Knowing that someone has studied 'Advanced Algorithms' tells me very little about what they actually know, even less about whether they can apply that knowledge in their job, and less still about whether they can effectively explain what they're doing to someone with a non-technical background." CreationDate="2011-02-06T15:12:29.203" UserId="2425" />
  <row Id="11025" PostId="6909" Score="1" Text="@probabilityislogic: I do not unerstand your last remarks. First, let us give their proper names to the notions involved and say that $P$ is absolutely continuous with respect to $Q$ (denoted $P\ll Q$) if, for every measurable $A$, $Q(A)=0$ implies $P(A)=0$. Now, notwithstanding your somewhat mysterious (to me) limit considerations, your $\delta(P,Q)$ is finite iff $P\ll Q$ or $Q\ll P$. .../..." CreationDate="2011-02-06T15:25:29.493" UserId="2592" />
  <row Id="11026" PostId="6909" Score="1" Text=".../... A way out of the conundrum you seem to be dug into might be to introduce the mid-point measure $P+Q$. Since $P\ll P+Q$ and $Q\ll P+Q$, the quantity $\eta(P,Q):=\kappa(P|P+Q)+\kappa(Q|P+Q)$ is always finite. Furthermore $\eta(P,Q)=0$ iff $P=Q$ and $\eta$ is symmetric. Hence $\eta(P,Q)$ indeed measures a kind of &quot;distance&quot; between $P$ and $Q$." CreationDate="2011-02-06T15:26:42.720" UserId="2592" />
  <row Id="11027" PostId="6909" Text="Edit: the midpoint is $\frac12(P+Q)$ and $\eta(P,Q):=\kappa(P|P+Q)+\kappa(Q|P+Q)+2\log2$." CreationDate="2011-02-06T15:33:39.387" UserId="2592" />
  <row Id="11028" PostId="6909" Text="@probabilityislogic, @Didier Piau, A simple example to show what Didier is saying is to take $p(x) = 1_{(x \in (0,1))}$ and $q(x) = 1_{(x \in (1,2))}$. Then $\delta(p,q)$ is infinite. If you want a more extreme example, for any $\epsilon &gt; 0$, let $q(x) \equiv q_{\epsilon}(x) = 1_{(x \in (\epsilon, 1+\epsilon))}$." CreationDate="2011-02-06T15:43:53.430" UserId="2970" />
  <row Id="11029" PostId="6909" Text="@cardinal: Indeed. I now wrote an answer to the OP explaining this &quot;midpoint+KL&quot; stuff." CreationDate="2011-02-06T15:50:04.820" UserId="2592" />
  <row Id="11030" PostId="6909" Text="@probabilityislogic, A simple example which I believe shows where your limiting argument fails is to consider $p(x) = 1_{(x \in (0,1/2)\cap\mathbf{Q})} + 2 1_{(x \in (1/2,1))}$. Then $p(x)$ is zero on a subset of $(0,1/2)$ with positive measure, but $p(x) \not\to 0$ for any $x \in (0,1/2)$. So your limit argument needs further conditions, even if it were to otherwise work. A sort of work-around is not to focus on the limits of $p$, but instead to define $x \log x$ on $[0,\infty]$ via an extension by continuity. The other problems with your development would still remain, however." CreationDate="2011-02-06T15:50:39.413" UserId="2970" />
  <row Id="11031" PostId="4553" Text="This one really bugs me.  My clients do this all the time.  And when I point out the error, they understand, but insist on doing it anyway" CreationDate="2011-02-06T15:54:50.273" UserId="686" />
  <row Id="11032" PostId="6926" Score="4" Text="This is correct, but how would you know if it was orthogonal unless you had a truly random sample?" CreationDate="2011-02-06T15:57:20.923" UserId="686" />
  <row Id="11033" PostId="6900" Text="-1: if you do not agree with an answer, post a comment to it instead of creating a new one. Didn't you read the popup ?" CreationDate="2011-02-06T16:53:45.833" UserId="264" />
  <row Id="11034" PostId="6937" Score="1" Text="@Marco, @Didier Piau, it might be noted that @Didier's suggestion is another special case of an $f$-divergence where $f(x) = x \log x - (1+x) \log( \frac{1+x}{2} )$." CreationDate="2011-02-06T17:26:26.103" UserId="2970" />
  <row Id="11035" PostId="6937" Text="@Marco, @Didier Piau, an alternative formulation which has some evocative nature is $\eta(P, Q) = \int P \log P + \int Q \log Q - 2 \int R \log R = 2 H(R) - (H(P) + H(Q))$ and so $\eta(P,Q) = 2 ( H(\mu(P,Q)) - \mu(H(P), H(Q))$ where $\mu(x,y) = \frac{x+y}{2}$. In other words, $\frac{1}{2} \eta(P,Q)$ is &quot;difference between the entropy of the average measure and the average entropy of the measures&quot;." CreationDate="2011-02-06T17:30:17.130" UserId="2970" />
  <row Id="11036" PostId="6941" Text="I must remark -- this is one the reasons why p-values are so popular and easy to be misunderstood. (IMHO)" CreationDate="2011-02-06T18:47:30.633" UserId="1307" />
  <row Id="11037" PostId="6941" Text="Ok, this makes sense. I do have what I believe is a good estimation of the null distribution. Any hints on how to implement this in R? Thanks!" CreationDate="2011-02-06T19:07:01.243" UserId="52" />
  <row Id="11038" PostId="4551" Text="@mbq Good move." CreationDate="2011-02-06T19:09:36.150" UserId="919" />
  <row Id="11039" PostId="6924" Score="1" Text="@Marco I don't understand how one could be any more explicit.  Do you mean restating what I wrote in terms of a formula such as $\arctan(KL(P,Q))$ or $f(KL(P,Q))$ for $f:\mathbb{R_+} \to [0,C]$ with $x \ge y$ implies $f(x) \ge f(y)$ for all $x,y \ge 0$?" CreationDate="2011-02-06T19:12:37.300" UserId="919" />
  <row Id="11040" PostId="6923" Score="1" Text="@cardinal Thank you for the reference and for the advice.  I was mainly concerned about stability.  In a long sequence of updates, there may be circumstances where the data are nearly collinear and there could also be a lengthy accumulation of numerical errors, so stability seems to be of paramount concern.  That is the justification for the Givens rotations (instead of Householder reflections, which take less computation)." CreationDate="2011-02-06T19:16:27.013" UserId="919" />
  <row Id="11041" PostId="6941" Text="@Alan - Do you know how to generate random values from your Null Distribution? If yes, suppose - T= c(T1, ..., TN) are draws from the null distribution - p-value = sum(T &gt; T_obs)/N. If you don't know how to generate, may you need to use Metropolis Sampling or Gibbs Sampling to get T1...TN, but it is very do-able." CreationDate="2011-02-06T19:30:44.183" UserId="1307" />
  <row Id="11042" PostId="6941" Text="Ah! I get it. Thanks again." CreationDate="2011-02-06T19:37:18.890" UserId="52" />
  <row Id="11043" PostId="6940" Score="1" Text="It's a silly question if I can answer it." CreationDate="2011-02-06T22:04:51.700" UserId="2775" />
  <row Id="11044" PostId="6923" Text="@whuber. Please don't take it otherwise - but the reference(s) that you cite are classic, like - Freedman, Maindonald, etc.. I haven't seen a lot of people cite these books. (It's my personal observation, I may be wrong. In my graduate studies, I have seen a lot of references to Freedman (and Diaconis), but never his books.)" CreationDate="2011-02-06T22:18:28.643" UserId="1307" />
  <row Id="11045" PostId="6938" Score="1" Text="Your point about a measure of spread is a key here. That's one of the issues that keeps coming up in this discussion, under other names, and it also ties in with Erik P's discussion of weighting schemes." CreationDate="2011-02-06T22:47:21.017" UserId="1764" />
  <row Id="11046" PostId="6935" Text="Thanks for the answer. Somehow, I've overlooked the fact that I have two cores (my computer isn't very new). I should've looked at multicore a long time ago." CreationDate="2011-02-06T23:16:04.723" UserId="2739" />
  <row Id="11047" PostId="6936" Text="That makes sense. Thanks." CreationDate="2011-02-06T23:23:43.713" UserId="2739" />
  <row Id="11048" PostId="6946" Text="@Queops, are the three consecutive pages assumed to be predetermined or is it the probability of finding errors on any set of three consecutive pages out of a total of $n$?" CreationDate="2011-02-07T02:00:13.397" UserId="2970" />
  <row Id="11049" PostId="6947" Text="Very detailed. I loved it. Thank you and I'll consider it as an answer unless someone comes up with something else." CreationDate="2011-02-07T02:58:03.247" UserId="1833" />
  <row Id="11050" PostId="6946" Score="2" Text="I realize this question will be answered by someone wanting to rack up points, but I want to point out that answering trivial homework question dilutes the usefulness of this forum to professionals." CreationDate="2011-02-07T03:00:23.033" UserId="3080" />
  <row Id="11051" PostId="6946" Score="1" Text="@Leo, I, for one, am not looking to &quot;rack up&quot; points. I am interested in learning from others and also good pedagogy. My understanding is that this forum's target audience is fairly broad and students are welcome. Probability and statistics can both seem daunting at first. That said, when looking at homework questions, it's always nice to see that the OPs have already spent some time on their own thinking about it." CreationDate="2011-02-07T03:19:00.770" UserId="2970" />
  <row Id="11052" PostId="6946" Text="@cardinal, I don't know what you mean by predetermined, but the problem indicates it's not some specific set of them, just the probability of finding 5 errors in 3 consecutive pages. @Leo, are you suggesting there isn't space for everyone? Sure most professionals probably can figure out this by themselves but we have to start somewhere right? I for one, am very new to this and I find it hard to follow most articles on the internet and I find this community most helpful and friendly." CreationDate="2011-02-07T03:23:44.097" UserId="1833" />
  <row Id="11053" PostId="6946" Text="@Queops, I think the problem probably means given a set of three prespecified pages, e.g., pages 13-15, what's the probability of five errors on these three pages. This is very different from the asking, e.g., what's the probability of finding 5 errors on some set of three consecutive pages between, say, pages 1--50. See the difference?" CreationDate="2011-02-07T03:27:50.203" UserId="2970" />
  <row Id="11054" PostId="6946" Text="@cardinal, yes completely. And no, it's just some set, doesn't matter which one. (Which I believe it's essential in a Poisson process?). The problem doesn't specify how many pages are in total, though one could assume there's more than 3." CreationDate="2011-02-07T03:32:05.213" UserId="1833" />
  <row Id="11055" PostId="6948" Text="I understand now! You were most helpful. Much appreciated. I believe I'm ready for some real &quot;Poissonic&quot; action hehe." CreationDate="2011-02-07T03:38:02.883" UserId="1833" />
  <row Id="11056" PostId="6923" Text="@whuber, I agree that stability is the main concern. That's why it was very nice to see the cute Givens rotation approach. I can't imagine that a full update wouldn't be necessary every so often, too. @mikera, for an exponential-decay-like approach, you might look at the LMS algorithm from signal processing. It's normally geared toward time-series, but I suspect can be adapted to a more general setting." CreationDate="2011-02-07T03:49:17.157" UserId="2970" />
  <row Id="11057" PostId="6924" Text="Yes, that's what I meant :-) I was not sure on what to apply the transformation. Now, it is clear, thx" CreationDate="2011-02-07T07:17:45.943" UserId="3019" />
  <row Id="11058" PostId="6950" Score="1" Text="You may find helpful those related posts http://stats.stackexchange.com/questions/6856/aggregating-results-from-linear-model-runs-r http://stats.stackexchange.com/questions/1812/fa-choosing-rotation-matrix-based-on-simple-structure-criteria" CreationDate="2011-02-07T07:21:00.930" UserId="339" />
  <row Id="11059" PostId="6924" Text="@Marco: I am lost. Do you settle for the Kolmogorov distance (which is always finite but has nothing in common with KL divergence)? Or for a bounded monotone transform of KL divergence (such as $\arctan$)? In the example of your post (and in any other *not absolutely continuous* example), the latter produces the supremum of the transform ($\pi/2$ if you settle for $\arctan$). In effect, this abandons any idea of estimating a *distance* between such probability measures more precisely than saying they are *far far away* (whether you encode this by $\pi/2$ or by $+\infty$ is irrelevant)." CreationDate="2011-02-07T08:47:27.167" UserId="2592" />
  <row Id="11060" PostId="6924" Text="Yes, you are right!" CreationDate="2011-02-07T09:03:18.990" UserId="3019" />
  <row Id="11061" PostId="6900" Text="sorry, because I logged with different computers, not intend to do that" CreationDate="2011-02-07T09:12:33.007" UserId="2454" />
  <row Id="11062" PostId="6949" Text="I can't actually remember if such a specific application is covered in *Biometrics: Theory, Methods, and Applications*, by Boulgouris et al. (Wiley, 2009), but it may be worth checking on [Google Books](http://j.mp/gSKX7K)." CreationDate="2011-02-07T09:51:07.683" UserId="930" />
  <row Id="11063" PostId="6940" Score="1" Text="@bill_080 As you can see, it is more complex in practice." CreationDate="2011-02-07T10:49:38.550" UserId="88" />
  <row Id="11064" PostId="6900" Text="The problem is, normally Neural network will not give you a formula how the output related to input. Most optimization approach I know all need a specific function first.     Even use Genetic Algorithm, I need fitness function like x=w1m+w2n+w31o+w41p+w51q,  however, the function should be related to input  a,b,c,d,f,g instead of output m,n,o,p,q" CreationDate="2011-02-07T10:50:47.883" UserId="2454" />
  <row Id="11065" PostId="4144" Text="Am I correct in thinking that if the variances did vary significantly, you should weight inversely by the variances? In the case of equal variance this reduces to weighting by the number of observations since the variance of n IID observations is (variance of one observation / n)." CreationDate="2011-02-07T12:20:08.753" UserId="2425" />
  <row Id="11067" PostId="6924" Text="@Didier Yes, the transformed KL divergence (when symmetrized, as you describe) might not satisfy the triangle inequality and therefore would not be a distance, but it would still define a topology (which would likely be metrizable).  You would thereby give up little or nothing.  I remain agnostic about the merits of doing any of this: it seems to me this is just a way of papering over the difficulties associated with infinite values of the KL divergence in the first place." CreationDate="2011-02-07T13:22:54.080" UserId="919" />
  <row Id="11068" PostId="6783" Text="Each installation have a different power capacity. E.g. Install1 = 3kW, Install2 = 6kW. One particular day, Install2 will produce twice more than Install1" CreationDate="2011-02-07T13:40:37.453" UserId="3000" />
  <row Id="11069" PostId="6923" Text="@suncoolsu Is there a problem with classic references?  (BTW, it's likely that all the stats software people respect--SAS, SPSS, S-Plus/R, Systat, Stata, etc.--still uses code based on the algorithmic references from the 60's through the 80's, because the core parts of all these programs were written by the early 90's.)" CreationDate="2011-02-07T13:45:13.873" UserId="919" />
  <row Id="11070" PostId="6952" Text="I would entirely agree with Andrej. This is one of those rare cases where principal components will probably do a better job than factor analysis. " CreationDate="2011-02-07T13:46:21.580" UserId="656" />
  <row Id="11071" PostId="6924" Text="@whuber Topology and the (defect of) triangle inequality are not my point here. I simply wondered why Marco suddenly seemed happy with the $\arctan$ artefact you suggested, although this artefact leaves every pair of not absolutely continuous probability measures as infinitely *far far away* from each other as they were with respect to KL divergence. Note also that $\eta$ is not only (and not mainly) a *symmetrized* version of KL, more importantly $\eta$ is *finitized*. .../..." CreationDate="2011-02-07T13:52:20.507" UserId="2592" />
  <row Id="11072" PostId="6924" Text=".../... Example: for every $t&gt;0$, let $P_t$ denote the uniform probability distribution on the interval $(0,t)$. For every $t\ne s$, $P_t$ and $P_s$ are at infinite distance for KL but $\eta(P_t,P_s)$ is positive and finite and depends non trivially on $(t,s)$. Inter alia, $\eta$ *quantifies* the fact that, if $t&lt;s&lt;u$, then $P_t$ is closer to $P_s$ than to $P_u$. I understood this is what Marco was after but maybe I am mistaken." CreationDate="2011-02-07T13:53:12.597" UserId="2592" />
  <row Id="11073" PostId="6924" Text="@Didier Could you clarify what you mean by &quot;finitized&quot;?  After all, two distributions can be infinitely far apart w.r.t. $\eta$, even when their supports overlap.  The family of uniforms you discuss is special in that in each pair, one of them is absolutely continuous w.r.t the other.  (Even that doesn't in general guarantee $\eta$ is finite, but it suffices in this case.)  However, we are moving off onto a tangential topic here: what we are missing is any sense of *why* there is a concern about the possibility of an infinite divergence (or infinite distance)." CreationDate="2011-02-07T13:58:02.307" UserId="919" />
  <row Id="11074" PostId="6924" Text="@whuber Why is there a concern about the possibility of infinite distance? Dunno... maybe because infinite distances is the feature the OP asked a remedy for in the first place... Re $\eta$, let me stress once again that $\eta(P,Q)$ is finite for **every** probability measures $P$ and $Q$." CreationDate="2011-02-07T14:09:52.570" UserId="2592" />
  <row Id="11075" PostId="6951" Text="Small correction: 1.6^5 should be (3*1.6) ^5. Thanks!" CreationDate="2011-02-07T14:18:46.470" UserId="1833" />
  <row Id="11076" PostId="6924" Text="@whuber Here is another example to let you better understand the behaviour of $\eta$. Let $U_t$ denote the uniform distribution on the interval $(t,1+t)$. Then $\eta(U_t,U_s)=\min(1,|t-s|)2\log(2)$. No $U_t$ and $U_s$ are absolutely continuous with respect to each other unless $s=t$. Finally, note that $\eta\le2\log(2)$ uniformly, and that $\eta(P,Q)=2\log(2)$ iff $P$ and $Q$ are mutually singular." CreationDate="2011-02-07T14:23:27.553" UserId="2592" />
  <row Id="11077" PostId="6783" Text="@user3000, if the proportions should be kept over time, they should be already known. Then you only need to test whether proportion for the day deviates from the true value. " CreationDate="2011-02-07T14:28:35.740" UserId="2116" />
  <row Id="11078" PostId="6924" Text="@Didier Thank you: I misread the formula for $\eta$ until just this moment; I had confused it with the $\delta$ defined in another answer.  Too hasty.  I appreciate your patient explanations as well as the clarifications provided by @cardinal in comments to your answer." CreationDate="2011-02-07T14:59:18.543" UserId="919" />
  <row Id="11079" PostId="6783" Text="The proportion is 30% approximative... That's why I want to use statistics to be sure to detect failure and not false positive ;)" CreationDate="2011-02-07T15:01:02.953" UserId="3000" />
  <row Id="11080" PostId="4144" Text="@Chris (This replaces an earlier reply, which mistakenly referenced a different question.)  Yes, that's correct.  One must always weight in direct proportion to the number of observations, though (assuming the observational errors in each group are independent).  I don't think there's an easy or general rule concerning whether or not one should use variance-weighted regression when the variances don't differ by much.  (The range of SD's of 6:2 in this case, with only 2 - 8 degrees of freedom, is not wide enough to provide convincing evidence of heteroscedasticity.)" CreationDate="2011-02-07T15:09:02.317" UserId="919" />
  <row Id="11081" PostId="6939" Text="@Tom, how did you uncover the obvious pattern? For 4 time periods and 4 different values for each period there are $4^4=256$ possible combinations, why did you post only 4 of them? I am asking because I suspect that there might be another variable which indicates how to group the values." CreationDate="2011-02-07T15:11:16.157" UserId="2116" />
  <row Id="11082" PostId="6961" Text="I agree it's a bit inept to recreate the ADF-test by hand. It was more an exercise for me to get used to the language of ADF-tests. Thanks for the help, very much appreciated." CreationDate="2011-02-07T15:14:00.810" UserId="3086" />
  <row Id="11083" PostId="6962" Score="1" Text="This is spooky." CreationDate="2011-02-07T15:16:41.523" UserId="88" />
  <row Id="11084" PostId="6962" Text="@mbq, seconded :)" CreationDate="2011-02-07T15:19:55.777" UserId="2116" />
  <row Id="11085" PostId="6915" Score="1" Text="Is someone's opinion a fault? I'd argue that the failure is trying to draw meaningful conclusions based on a single statistic from only a few data points. As noted by @nico above, Amazon does show a bar graph of all the ratings." CreationDate="2011-02-07T16:14:18.357" UserId="597" />
  <row Id="11086" PostId="5150" Text="Have a look at the answers to [this question](http://stackoverflow.com/questions/2018178/finding-the-best-trade-off-point-on-a-curve) on Stackoverflow to find the 'elbow' in the ROC." CreationDate="2011-02-07T17:32:48.593" UserId="198" />
  <row Id="11087" PostId="6966" Text="[Duplicate?](http://stats.stackexchange.com/questions/1164/why-havent-robust-and-resistant-statistics-replaced-classical-techniques/1185)" CreationDate="2011-02-07T18:09:35.373" UserId="8" />
  <row Id="11088" PostId="6966" Text="@csgillespie, seconded, although this might be one of those good duplicates. " CreationDate="2011-02-07T18:21:14.983" UserId="2116" />
  <row Id="11089" PostId="6940" Score="1" Text="@mbq In theory, even I could be correct" CreationDate="2011-02-07T18:21:17.720" UserId="2775" />
  <row Id="11090" PostId="6939" Text="@mpiktas, I probably missed a lot of patterns too in the above example. I just gave a simple example (it is not my actual data). I made them up while writing this question. I am trying to find patterns between individual fields in a dataset." CreationDate="2011-02-07T18:44:49.883" UserId="3040" />
  <row Id="11091" PostId="6942" Text="It is clear to me I should learn R. Do you recommend any books that I could read to learn about pattern recognition in R? I am not just looking for a pattern in my distribution. I'm looking for all patterns that multiple fields of my distribution make up together." CreationDate="2011-02-07T18:47:44.037" UserId="3040" />
  <row Id="11092" PostId="6942" Text="@Tom: you definitely should :) R has a really great potential if you plan to make complex computations in the future. To learn R, you can find a lot of great books and blogs also in the issue, just to pick up a free one, look for *Statistics with R* by Vincent Zoonekynd (http://zoonek2.free.fr/UNIX/48_R/all.html) and look for  Ch. 11-12 for different kind of models. I have also made up a small collection of useful links in the subject, check it if you wish: http://r.snowl.net/interesting-urls-dealing-with-r/" CreationDate="2011-02-07T18:55:04.693" UserId="2714" />
  <row Id="11093" PostId="6942" Text="@Tom: a better approach could be reading through the answers of *Books for learning the R language* question on SO: http://stackoverflow.com/q/192369/564164" CreationDate="2011-02-07T18:57:11.417" UserId="2714" />
  <row Id="11094" PostId="6968" Text="+1, Thanks. Well argued. But in introductory courses, there is no model selection, in the strict sense. You could cite other contexts that are appropriate for the introduction of hypothesis testing? It is acceptable to report the outcome of a test without an estimate of power? " CreationDate="2011-02-07T19:09:45.527" UserId="523" />
  <row Id="11095" PostId="6939" Text="@Tom, so what is your real data look like then?" CreationDate="2011-02-07T19:20:56.477" UserId="2116" />
  <row Id="11096" PostId="6950" Text="Mallows Cp http://en.wikipedia.org/wiki/Mallows'_Cp  Here's a paper on Cp http://mlrv.ua.edu/2008/vol34_1/Lieberman-Morris.pdf" CreationDate="2011-02-07T19:23:07.853" UserId="2775" />
  <row Id="11097" PostId="6967" Text="@Zach, why do the approaches described in the question you mention not work for you?" CreationDate="2011-02-07T19:26:17.403" UserId="2116" />
  <row Id="11098" PostId="6967" Text="@mpiktas - it looks like the OP tried the methods mentioned previously has has copied the corresponding errors in the code above." CreationDate="2011-02-07T20:08:58.877" UserId="696" />
  <row Id="11099" PostId="6967" Text="@Chase, when I commented, there was no R code example." CreationDate="2011-02-07T20:25:45.713" UserId="2116" />
  <row Id="11100" PostId="6967" Text="@mpiktas - I blame the [R gremlins](http://theshoegame.com/wp-content/uploads/2009/01/gizmo-gremlins-picture.jpg)" CreationDate="2011-02-07T20:26:59.533" UserId="696" />
  <row Id="11101" PostId="6970" Text="Maybe you could elaborate?" CreationDate="2011-02-07T20:43:32.827" UserId="88" />
  <row Id="11102" PostId="6966" Text="@mpiktas Yes I wasn't sure. We'll see if anyone votes to close." CreationDate="2011-02-07T20:47:54.510" UserId="8" />
  <row Id="11103" PostId="6939" Text="@mpiktas it's stock data by date, so basically multiple stocks with certain daily dates with stock prices of each date" CreationDate="2011-02-07T21:08:41.693" UserId="3040" />
  <row Id="11104" PostId="6967" Text="@mpiktas - I added a code example that lists what I tried to do and the errors I got.  I hit the 3 methods discussed in the previous question." CreationDate="2011-02-07T21:57:55.457" UserId="2817" />
  <row Id="11105" PostId="6923" Text="@whuber. Oh no! I am impressed actually. That's why I said don't take it otherwise. It was in praise that you can actually site literature that old. I agree with you on the algorithm part as well -- we still use Prof. Maindonald's code for optimization in R (if I am not mistaken)." CreationDate="2011-02-07T22:04:13.567" UserId="1307" />
  <row Id="11106" PostId="6953" Text="@mpiktas -- I think your method is way easier than what I suggested, especially if there are no restriction on the function that you are integrating. I am not aware of the numerical tech. in R." CreationDate="2011-02-07T22:12:08.190" UserId="1307" />
  <row Id="11107" PostId="538" Text="@Craig That's the point; it's not possible." CreationDate="2011-02-07T22:16:51.993" UserId="3093" />
  <row Id="11108" PostId="6859" Text="How would I plot just the intercept and unemployed?" CreationDate="2011-02-07T23:12:21.323" UserId="2817" />
  <row Id="11109" PostId="6953" Text="Yes, I think this is more in line with my current abilities. Thanks!" CreationDate="2011-02-07T23:19:14.810" UserId="52" />
  <row Id="11110" PostId="6972" Score="2" Text="@Queops, maybe you can edit the question and add in your thoughts and approach so far and where you are getting stuck. " CreationDate="2011-02-07T23:35:36.700" UserId="2970" />
  <row Id="11111" PostId="6972" Text="More likely is that 575 is the standard deviation. You should first ask yourself what would be the probability in a sample of size 1. (Your prof didn't make it a nice even multiple of sd's away from the mean, so I imagine he expected you to use a statistical system or calculator of some sort.)" CreationDate="2011-02-07T23:36:41.580" UserId="2129" />
  <row Id="11112" PostId="6972" Text="I suspect that 575 is the population standard deviation, both from the rest of the question and the real Norwegian data in figures 1 and 2 of http://eb.niehs.nih.gov/bwt/subcfreq.htm " CreationDate="2011-02-07T23:38:36.620" UserId="2958" />
  <row Id="11113" PostId="6968" Text="Having no model selection in introductory courses isn't a necessity.  If you're considering changing a course consider that as a good place to start." CreationDate="2011-02-07T23:42:16.570" UserId="601" />
  <row Id="11114" PostId="6972" Score="1" Text="While we are happy to help with homework questions, this site isn't really aimed at *doing* your work. Please see the relevant [meta](http://meta.stats.stackexchange.com/questions/12/how-should-we-deal-with-obvious-homework-questions) question. In particular, how homework questions should be formatted." CreationDate="2011-02-08T00:01:04.760" UserId="8" />
  <row Id="11115" PostId="6972" Text="The answer is of minimal importance to me, I'm studying and it's the how to I'm interested in. @DWin and Henry, I think it really is the standard deviation. Henry, I'll look into it, thanks. @Cardinal, I tried applying the probability density function but that was probably silly right?" CreationDate="2011-02-08T00:30:24.287" UserId="1833" />
  <row Id="11116" PostId="6972" Text="You might want to think, not about the density function, but about using the probability I suggested , coming from the CDF, as input into a binomial expansion." CreationDate="2011-02-08T00:37:22.573" UserId="2129" />
  <row Id="11117" PostId="6952" Text="Sorry I have one doubt, should I include in PCA the variable I want to predict?" CreationDate="2011-02-08T01:58:40.290" UserId="1808" />
  <row Id="11118" PostId="6974" Text="My logic so far: From stage 1 to stage 2, every person has 5 options to choose from for their stage 2 group (A, B, C, D, E). So, the chance that he/she will retain his/her own group is 20%. So, the expected number of people who will retain their own group is 20%. Comparing 65% with 20% using a 1-sample proportions test with continuity correction (in R: prop.test(34, 52, 0.2)), I get: Chi^2 = 54.14, df = 1, p &lt; 0.001, meaning that my claim 1 is statistically significant." CreationDate="2011-02-08T02:25:32.487" UserId="3097" />
  <row Id="11119" PostId="6976" Text="@pchalasani, Does LTCM count? ;)" CreationDate="2011-02-08T03:00:42.670" UserId="2970" />
  <row Id="11120" PostId="6976" Text="What about discrete choice modeling? I'd think most people would consider DCM to be a branch of econometrics and the entire travel demand forecasting industry relies on DCM to generate inputs for their forecasting models...that may not be what you had in mind though." CreationDate="2011-02-08T03:06:25.833" UserId="696" />
  <row Id="11121" PostId="6976" Text="@pchalasani, yes, how broadly *are* you interpreting the word &quot;econometric&quot;? Does, for example, Netflix's &quot;Cinematch&quot; system count (or any of the other successful &quot;recommender&quot; engines out there)?" CreationDate="2011-02-08T03:23:51.387" UserId="2970" />
  <row Id="11122" PostId="6975" Score="3" Text="Can you please be more specific about what kind of data set do you have? Answers may vary accordingly. `rpart` may be Ok - but may be something better is available." CreationDate="2011-02-08T03:28:20.627" UserId="1307" />
  <row Id="11123" PostId="6939" Text="@Tom, so you want to see if stock price has any obvious time trend?" CreationDate="2011-02-08T03:59:10.637" UserId="2116" />
  <row Id="11124" PostId="6510" Text="Yes, nested models are hierarchical in nature. In the case of a variable representing degree (i.e. degree of risk), you do not need a hierarchical model. Depending on how the risk index you're using is divided, you may even want to consider grouping the levels on the index (i.e. if it's 1-9 consider grouping as high [7-9], medium [4-6], and low [1-3]) as this may allow you to better represent the dominance effect where those in the reference group represent a substantially different outcome than those in the comparisons." CreationDate="2011-02-08T04:42:59.380" UserId="2166" />
  <row Id="11125" PostId="6979" Text="@JMS, &quot;how many intro stats students know the real definition of a confidence interval?&quot; Or, PhD stat graduates, for that matter." CreationDate="2011-02-08T05:11:26.377" UserId="2970" />
  <row Id="11126" PostId="6978" Text="@grg, I think the answer is &quot;yes&quot;. Of course, there is a bit of an issue of definitions here. What one person considers &quot;large-scale&quot; is sometimes very different from other's. My impression is that, e.g., many academic researchers consider the Netflix dataset &quot;large-scale&quot;,  while in many industrial settings it would be considered &quot;puny&quot;. As regards estimation techniques, usually with very large data, computational efficiency trumps statistical efficiency. For example, method of moments will, in many cases, perform (nearly) as well as MLE in these settings and can be *much* easier to compute." CreationDate="2011-02-08T05:25:35.443" UserId="2970" />
  <row Id="11127" PostId="6978" Score="2" Text="@grg, you might also look up the Workshop on Algorithms for Modern Massive Data Sets (MMDS). It's young, but draws a pretty impressive set of speakers at the interfaces of statistics, engineering and computer science as well as between academia and industry." CreationDate="2011-02-08T05:27:17.487" UserId="2970" />
  <row Id="11128" PostId="6979" Text="Quite! Incidentally, I meant no dig at students or practitioners of any stripes. But it's a little crazy to expect the mental gymnastics from someone who didn't sign up for advanced work in statistics. " CreationDate="2011-02-08T05:32:30.547" UserId="26" />
  <row Id="11129" PostId="6700" Text="@rolando2: Thank you for the prompt reply. Yes. Each row is a different region. There are 16 regions with different populations. The 2 levels of treatment variable are thoroughly implemented and not so thoroughly implemented. Some regions implemented the treatment thoroughly and so the deaths were less. Other regions did not implement the treatment thoroughly and so the deaths were more. We have to show statistically that in regions where treatment was inadequate had significantly higher number of deaths. If the table is too big, we can use 3 or 4 regions data from 2006 to 2010 for analysis." CreationDate="2011-02-08T06:00:50.797" UserId="2956" />
  <row Id="11130" PostId="6923" Score="1" Text="@suncoolsu Hmm... Maindonald's book was newly published when I started using it :-)." CreationDate="2011-02-08T06:02:11.280" UserId="919" />
  <row Id="11131" PostId="6972" Score="2" Text="@Queops I added tags to provide links to useful threads.  As a hint, consider that $Y$ is a random variable that counts a certain number of things out of 25.  What distribution do you suppose it might have?  What do you need to know about that distribution?  The assumptions about the distribution of $X$ are there so you can compute the chance that a *single* baby has low birth weight." CreationDate="2011-02-08T06:09:16.963" UserId="919" />
  <row Id="11132" PostId="6963" Score="1" Text="@Łukasz Can you say anything more about the parameters $n$, $\alpha_i$, and $\beta_i$?  It's possible to obtain exact expressions for $\sum_j{X_j}$ and thereby approximate the expectations of the ratios, but for certain combinations of the parameters one could exploit Normal or saddlepoint approximations with less work.  I don't think there will be a universal approximation method, which is why additional restrictions would be welcome." CreationDate="2011-02-08T06:17:59.163" UserId="919" />
  <row Id="11133" PostId="6909" Text="@cardinal - in the example you give, the distance *should* be infinite, because the supports are *disjoint* (and thus not comparable)." CreationDate="2011-02-08T06:55:59.350" UserId="2392" />
  <row Id="11134" PostId="6974" Text="@Mikhil Masli: I'm sorry, but when 65% stayed in their group it isn't very probable, that the chance for staying in your group is 20%." CreationDate="2011-02-08T06:59:10.953" UserId="3100" />
  <row Id="11135" PostId="6909" Text="@cardinal: I think the measure I made requires one measure to have a support of a subset of the other.  If there is a &quot;zero&quot; region in P, and not Q, and also a &quot;zero&quot; region in Q, and not in P, then it is infinite.  If the supports are nested, then it is always finite" CreationDate="2011-02-08T07:33:06.217" UserId="2392" />
  <row Id="11136" PostId="6976" Text="Arnold Zellner isn't a bad name to look up.  larry bretthorst's phd thesis on spectral estimation is also good" CreationDate="2011-02-08T07:36:30.097" UserId="2392" />
  <row Id="11137" PostId="6979" Text="How many *people* can say the real definition of CIs?  And how many people use them consistently with this definition?  Its just too hard not to think &quot;the parameter is likely to be in said interval&quot; - even if you *know* its not what a CI is." CreationDate="2011-02-08T07:51:38.433" UserId="2392" />
  <row Id="11138" PostId="6933" Score="2" Text="@cardinal, +1. The answer is magnitudes better than the question. " CreationDate="2011-02-08T07:56:51.240" UserId="2116" />
  <row Id="11139" PostId="6795" Text="@Belmont, now that @cardinal gave a complete answer, can you specify what LAR really is, for future reference? Judging from the answer this is standard manipulation of products of least squares regressions given some initial constraints. There should not be a special name for it without serious reason." CreationDate="2011-02-08T07:59:02.083" UserId="2116" />
  <row Id="11140" PostId="6984" Text="thanks: it really didn't occur to me that a transpose could make such a large difference." CreationDate="2011-02-08T08:46:00.980" UserId="603" />
  <row Id="11141" PostId="6984" Text="@user603, you might want to file a bug for Matrix package developers. It seems that this is a simple matter of forgetting to add appropriate method definitions." CreationDate="2011-02-08T08:49:16.033" UserId="2116" />
  <row Id="11142" PostId="6605" Text="@Eduardo - but then how do you adjust the p-value to account for the multiple 1-parameter comparisons you are making?  there is no principles to guide you, only your intuition." CreationDate="2011-02-08T08:50:12.800" UserId="2392" />
  <row Id="11143" PostId="6952" Text="No, leave your dependent variable (Y/N variable in your case) out of PCA dimension reduction. You can read my article about same topic at the http://goo.gl/lJh5s" CreationDate="2011-02-08T08:57:14.050" UserId="609" />
  <row Id="11144" PostId="6966" Text="@Washington, [These quotes](http://en.wikiquote.org/wiki/George_Box) are very appropriate. All models are wrong, but some are useful." CreationDate="2011-02-08T09:15:20.720" UserId="2116" />
  <row Id="11146" PostId="6939" Text="@mpiktas not in the whole dataset. I want it to combine all possible combinations of each field and test for patterns in the slope for consecutive days (so eg. x=1,2,3 or x=5,6,7), if there is one (or with a slight variation in pattern I can set) I want it to return that combination." CreationDate="2011-02-08T11:02:25.683" UserId="3040" />
  <row Id="11147" PostId="6984" Text="@mpiktas (+1) Nicely done. (oh, and good to know that you're on a Mac :-)" CreationDate="2011-02-08T11:15:18.117" UserId="930" />
  <row Id="11148" PostId="6978" Text="It's only a few decades since most datasets were too large to fit in main memory, and the choice of algorithms used in early statistical programs reflected that. Such programs didn't have facilities for mixed-effects models though." CreationDate="2011-02-08T11:38:48.657" UserId="449" />
  <row Id="11149" PostId="6984" Text="@chl, only because of hardware :) Mac OS X's woeful swap treatment is killing me. Better than Windows, but way worse than Linux." CreationDate="2011-02-08T11:41:32.743" UserId="2116" />
  <row Id="11150" PostId="6976" Text="@chase and @cardinal: yes Travel demand forecasting and Netflix Cinematch ( and other recommendation engines) would be great examples, if there were any documentation about how they work, or at least a description of the underlying techniques. Other examples could be from non-profit domains such as education or government. " CreationDate="2011-02-08T11:47:11.913" UserId="2544" />
  <row Id="11151" PostId="6976" Text="@probabilistic logic: thanks for the refs I will look them up." CreationDate="2011-02-08T11:47:40.117" UserId="2544" />
  <row Id="11152" PostId="6314" Text="@cardinal, the Cover-Hart bound seems not to depend on dimension at all, as you say a different aspect ?" CreationDate="2011-02-08T11:53:41.620" UserId="557" />
  <row Id="11154" PostId="6984" Text="@chl, if your swap partition is larger than 5GB sure :) If you know that you will need more swap than your current swap partition size, you can mount additional space as swap. This needs manual intervention of some kind, but on other hand it is doable via R." CreationDate="2011-02-08T12:08:01.913" UserId="2116" />
  <row Id="11155" PostId="6984" Text="@mpiktas Oups, I deleted my previous comment while editing... My point was about loading or processing large amount of (genetic) data in R which I found convenient under the Mac thx to its &quot;unlimited&quot; swapping facilities (I must admit I don't use Linux for some years now, and all that is certainly of marginal interest to the OP). Thx for sharing your thoughts anyway." CreationDate="2011-02-08T12:13:54.437" UserId="930" />
  <row Id="11156" PostId="6909" Text="@probabilityislogic, the second remark you make is essentially the concept of absolute continuity that @Didier was bringing to your attention. As regards your first comment, look again. I gave you the &quot;extreme&quot; example for a very good reason: the difference in support can be made *arbitrarily* small and yet your measure $\delta$ remains infinite. " CreationDate="2011-02-08T12:52:26.833" UserId="2970" />
  <row Id="11157" PostId="6795" Text="@mpiktas, LAR stands for *least angle regression* as coined by Efron et al. In their 2004 paper. It is closely related to both forward-stagewise regression and the lasso. Indeed it allows the *full* regularization path of each of these to be calculated with the same computational complexity of a single least-squares fit. (Note that both methods reduce to least-squares when the shrinkage goes to zero.) LAR was discovered and developed after the similarities in the regularization paths of the coefficients in forward-stagewise and the lasso were noted. " CreationDate="2011-02-08T13:02:09.860" UserId="2970" />
  <row Id="11158" PostId="6314" Text="@Denis, yes I believe this is true and this was, in large part, my point in bringing it up. 1-NN seems pretty relevant in that sense, i.e., the fact that it works (so) well (theoretically) uniformly in the dimension of the feature space seems to help it stand on it's own, regardless of what the behavior of the nearest and farthest neighbors is in a large dimensional space. It makes me wonder if Beyer was aware at all of this (classical) result. " CreationDate="2011-02-08T13:08:30.630" UserId="2970" />
  <row Id="11159" PostId="6795" Text="@cardinal, but if we regularize $\beta$ why $u(\alpha)=\alpha X\hat{\beta}$, with $\hat{\beta}$ LS estimate? " CreationDate="2011-02-08T13:13:35.087" UserId="2116" />
  <row Id="11160" PostId="6985" Text="Thanks to everyone." CreationDate="2011-02-08T13:16:23.440" UserId="1833" />
  <row Id="11161" PostId="6977" Text="@cardinal. Can you please give a little more information. For example, what are the hypothesis being tested with this model? Are there any R implementations using @Mikhil's data?" CreationDate="2011-02-08T13:19:01.753" UserId="2040" />
  <row Id="11162" PostId="6795" Score="1" Text="@mpiktas, it's a stagewise algorithm, so each time a variable enters or leaves the model on the regularization path, the size (i.e., cardinality/dimension) of $\beta$ grows or shrinks respectively and a &quot;new&quot; LS estimate is used based on the currently &quot;active&quot; variables. In the case of the lasso, which is a convex optimization problem, the procedure is is essentially exploiting special structure in the KKT conditions to obtain a *very* efficient solution. There are also generalizations to, e.g., logistic regression based on IRLS and Heine-Borel (to prove convergence in finite no. of steps.)" CreationDate="2011-02-08T13:19:41.953" UserId="2970" />
  <row Id="11163" PostId="6990" Score="8" Text="Please write something more about your problem -- this way this question will be closed." CreationDate="2011-02-08T13:21:37.067" UserId="88" />
  <row Id="11164" PostId="6977" Text="@user2040, ok. I'll try to post something more in a bit. I noticed the Lindsey book can be viewed on Google Books. A standard **glm** call using family=&quot;poisson&quot; can be used for the fitting and testing. All it requires is to get the symmetry covariates coded properly. " CreationDate="2011-02-08T13:27:59.757" UserId="2970" />
  <row Id="11165" PostId="6795" Text="@cardinal, thanks for trying to explain. I understand the idea, but some of the details are still not clear, I see though that I can figure them out myself using the references you gave." CreationDate="2011-02-08T13:37:38.657" UserId="2116" />
  <row Id="11166" PostId="6795" Text="@mpiktas, unfortunately I'm not aware of a good overview treatment of the algorithm. The Hastie et al. text has a handful of pages on it, but as with most of the methods discussed in the book, many significant details are missing. The original Efron et al. paper is at least 45 pages long, if I recall, which requires some determination to get through. Perhaps there is a gap in the literature to be filled. " CreationDate="2011-02-08T14:05:55.340" UserId="2970" />
  <row Id="11167" PostId="6973" Text="Thanks for the advice.  I'll look into that." CreationDate="2011-02-08T14:18:36.623" UserId="2817" />
  <row Id="11168" PostId="6605" Text="@probabilityislogic: I'm more often interested in the whole model than in individual 1-parameters. If the model works (i.e., helps me understand how a physical system works, and predict how it would behave given certain conditions), then each parameter has a good enough value." CreationDate="2011-02-08T14:52:20.317" UserId="2971" />
  <row Id="11169" PostId="6795" Text="@cardinal, ok I got the Efron paper, will spend few evenings reading it instead of answering questions here :)" CreationDate="2011-02-08T14:52:36.140" UserId="2116" />
  <row Id="11170" PostId="6975" Score="2" Text="@celenius Before you do any of this stuff, perform univariate exploration of each of your 40 attributes.  Many of them should be re-expressed as square roots or started logarithms before you go any further.  In practice this will improve subsequent results *immensely.*  You also want to identify outliers and make a plan for assessing their influence in your analyses, at least if you're going to use least-squares techniques (including PCA, which can be extremely sensitive to geometric [multivariate] outliers)." CreationDate="2011-02-08T16:25:16.140" UserId="919" />
  <row Id="11172" PostId="6987" Text="thanks for the details @Chase, DCM is an area I'm not familiar with, but looks interesting." CreationDate="2011-02-08T16:43:45.047" UserId="2544" />
  <row Id="11174" PostId="6975" Text="My preliminary visual survey of the data was plotting all parameters against each other, so I have a plot of every relationship, and can observe a few patterns and outliers. Should I apply the same transformations to every variable, and include them, or should I examine what the most likely transformations should be first." CreationDate="2011-02-08T16:53:46.873" UserId="2635" />
  <row Id="11176" PostId="6987" Text="@pchalasani - for a more general overview of discrete choice modeling, [Kenneth Train's book](http://www.econ.berkeley.edu/books/choice2.html) is a very good place to start for both theory and practical applications. It's a fairly dense read, but well worth the effort if you are interested." CreationDate="2011-02-08T16:58:48.317" UserId="696" />
  <row Id="11177" PostId="6983" Text="My goal with the issue was to collect expert opinion in order to enrich the debate on the revision of courses in statistics that is ongoing at the institute where I work in Brazil. The objective is being achieved, with opinions as well placed as of @cardinal, @Andrew Robinson, @probabilityislogic and @JMS. Clearly, hypothesis testing (via N-P, DT or Byes) should be very well taught, but the challenges to build courses as appropriate, given the universality of the teaching of statistics, is equally or more complex than the technique itself.&#xA;Thank you for your contribution." CreationDate="2011-02-08T17:19:24.877" UserId="523" />
  <row Id="11178" PostId="6981" Text="Good tip, thanks." CreationDate="2011-02-08T17:22:38.883" UserId="523" />
  <row Id="11179" PostId="6992" Text="@onestop +1, nice application of delta method." CreationDate="2011-02-08T17:24:22.470" UserId="2116" />
  <row Id="11180" PostId="6980" Text="Thanks for the interesting list of examples. Given the objective of the question: To contribute to the debate on the review of our statistics courses, we will try to get more details on the implementation of testing in modern devices, can be a great motivation for our engineering students." CreationDate="2011-02-08T17:28:41.140" UserId="523" />
  <row Id="11181" PostId="6979" Text="E sobre a prática usual de não reportar-se estimativas do " CreationDate="2011-02-08T17:30:20.187" UserId="523" />
  <row Id="11182" PostId="6979" Text="Sorry for the earlier comment in Portuguese. What about the usual practice of not reporting to estimate the power of a test and the difficulty in obtaining such an estimate? When is it (Is it?) acceptable to the absence of this estimate? Is it acceptable? " CreationDate="2011-02-08T17:35:29.497" UserId="523" />
  <row Id="11183" PostId="6959" Text="Thank you very much for such a nice answer!" CreationDate="2011-02-08T18:38:58.607" UserId="3019" />
  <row Id="11184" PostId="6923" Text="@whuber: Wow! Thanks for sharing years of experiences with us! I don't know, but I think, citing Maindonald's book may not be citing `Statistical Methods for Research Workers` - by `Fisher` but it comes pretty close it." CreationDate="2011-02-08T18:56:13.607" UserId="1307" />
  <row Id="11185" PostId="6959" Text="@Marco, you're welcome. " CreationDate="2011-02-08T19:19:08.057" UserId="2116" />
  <row Id="11187" PostId="6979" Text="I'm afraid I don't quite understand your question. Are you asking when/if it is appropriate to report the results of a test without an estimate of its power?" CreationDate="2011-02-08T21:09:35.753" UserId="26" />
  <row Id="11188" PostId="6310" Text="@Simon, why is the correction to minimize?&#xD;&#xA;&#xD;&#xA;Doesn't value of P found maximize the ratio?&#xD;&#xA;&#xD;&#xA;" CreationDate="2011-02-08T21:28:56.113" UserId="3115" />
  <row Id="11189" PostId="6865" Text="@caracal. I have a lot of things on my plate at the moment. I will expand my answer to be more specific to your question. I see you have invested a lot of effort in framing your question. Thanks for that." CreationDate="2011-02-08T22:01:05.867" UserId="1307" />
  <row Id="11196" PostId="6997" Text="When you say that ez was buggy for mixed ANOVA designs, are you basing this on my recent announcement of version 3.0 in which I note that there was a bug in the ezMixed() code? If so, I think you misinterpreted that note. ezMixed() isn't related to ezANOVA(). ezMixed() serves to help assess the influence of fixed effects in a mixed effects modelling context." CreationDate="2011-02-08T23:24:33.910" UserId="364" />
  <row Id="11198" PostId="6979" Text="JMS, sorry my English. I'll try again. Yes, I tried to challenge that hypothesis testing results without reporting estimates of power is highly questionable and that interval estimators do not have this additional source of sins, even agreeing that they are sin prone, but less sin prone." CreationDate="2011-02-08T23:55:58.010" UserId="523" />
  <row Id="11199" PostId="6996" Text="Formally speaking, *any* hypothesis test with alpha bound on the rate of Type I error can be turned into a confidence interval with coverage parameter (1-alpha) and vice versa, no? I don't think you have to be a hardcore frequentist to believe that this is entailed by the definitions. :-)" CreationDate="2011-02-09T00:08:45.013" UserId="1122" />
  <row Id="11200" PostId="6979" Text="We will ask for a wage increase, in addition to having to teach relatively complex topics for almost all courses in fields as diverse as engineering and management, we have to try to teach the approaches of different schools of thought. Construct good courses in statistics should be a line of research very well funded, it is very difficult to achieve an equilibrium between working hours and topics to be taught to students of courses that increasingly demanding the ability to analyze data" CreationDate="2011-02-09T00:14:49.787" UserId="523" />
  <row Id="11201" PostId="3790" Text="That should be &quot;mean is unique maximizer&quot;" CreationDate="2011-02-09T00:53:06.340" UserId="3116" />
  <row Id="11202" PostId="6977" Text="@B_Miner, @Mikhil, added model description and code. Hope this helps." CreationDate="2011-02-09T04:24:57.703" UserId="2970" />
  <row Id="11203" PostId="7010" Text="Ya, realized what I was doing.  Missed that whole exponential thing.  Thanks." CreationDate="2011-02-09T04:28:43.993" UserId="2387" />
  <row Id="11204" PostId="6979" Text="An interval estimate isn't necessarily (or usually) accompanied by a power calculation, so I'm not sure I agree with your point as I understand it. And as others have pointed out, classical hypothesis tests and confidence intervals are really two sides of the same coin (and as such share the same flaws). Certainly a CI is more informative than just a reject/fail to reject sort of statement, but I'm not sure that goes to the original point." CreationDate="2011-02-09T04:35:33.357" UserId="26" />
  <row Id="11205" PostId="7008" Text="@cardinal this is a very nice answer to my question. However, although I follow the calculation that you make for $n|\rho$, it is not exactly clear to me if the units for $rho$ is percent in the solution $n=65$ for $\rho&lt;1$; does this mean &quot;$\rho$ is less than $1\times s^2$&quot; or &quot;$\rho$ less than $1\%$ of $s^2$?" CreationDate="2011-02-09T05:58:50.607" UserId="2750" />
  <row Id="11206" PostId="595" Text="are you talking about MODWT (Maximum Overlap Discrete Wavelet Transform) ?" CreationDate="2011-02-09T06:59:17.013" UserId="1709" />
  <row Id="11207" PostId="595" Text="@fRed: nop, here is the paper, Coifman and Donoho: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.125.3682&amp;rep=rep1&amp;type=pdf" CreationDate="2011-02-09T07:17:09.490" UserId="223" />
  <row Id="11208" PostId="3790" Score="1" Text="Various points: $E|X-k|$ is minimised when $k$ is the median of $X$.  Von Hippel's article (linked above by chl) discuses exceptions and http://www.btinternet.com/~se16/hgb/median.htm shows the possible relationship between mean, median, mode and standard deviation, both for continuous and for discrete distributions. The 3 can in fact take any value: positive, negative, zero or infinite. " CreationDate="2011-02-09T08:18:13.503" UserId="2958" />
  <row Id="11209" PostId="6996" Text="@Keith  No argument over the definitions, but you do have to be a Frequentist to consider them to be more than interesting and perhaps handy bits of mathematics.  That is, *if* you think sampling theoretic properties are vital for statistical inference then you will (or should) be *equally* keen on confidence intervals and hypothesis tests since, as we agree, they have this symmetry.  Mine was a response to the questioners contrast between 'good' CIs and 'bad' HTs. By lumping them together I wanted to refocus on the contrasts brought up in other answers." CreationDate="2011-02-09T08:29:39.480" UserId="1739" />
  <row Id="11210" PostId="7007" Text="I guess you mean &quot;Strongly disagree&quot;. You can do all kinds of analysis on it, yet it depends on what type of arguments you want to make. " CreationDate="2011-02-09T08:39:09.160" UserId="2904" />
  <row Id="11211" PostId="6994" Text="Thanks!! I only recently started looking into bayesian analysis and i still find it a bit difficult to grasp. I guess this is an opportunity to learn a bit more about it. " CreationDate="2011-02-09T09:16:38.460" UserId="1871" />
  <row Id="11212" PostId="6992" Text="Hi, thanks very much. I was searching for nearly a week for a definitive answer and didn't find any." CreationDate="2011-02-09T10:17:47.863" UserId="3104" />
  <row Id="11213" PostId="6991" Text="I'm no statistician but a physicist trying to analyse some data, so thank you for this nice explanation." CreationDate="2011-02-09T10:29:45.667" UserId="3104" />
  <row Id="11214" PostId="595" Text="@robin girard: thanks, interesting" CreationDate="2011-02-09T10:40:37.527" UserId="1709" />
  <row Id="11215" PostId="6991" Text="@Askan, you're welcome." CreationDate="2011-02-09T11:59:14.950" UserId="2116" />
  <row Id="11216" PostId="6979" Text="What I tried to express is that hypothesis tests not accompanied by estimates of power are very questionable and that interval estimates do not have this additional source of complications." CreationDate="2011-02-09T12:33:12.177" UserId="523" />
  <row Id="11217" PostId="6605" Text="@Eduardo: I think we're talking at different levels here.  p-values usually give good indications in models (such as OLS regression) of the size of parameters.  This is because they are based on a sufficient statistic, and no nuisance parameters (T-test).  But once you leave this &quot;safe&quot; area (e.g. non-linearity), p-values do not necessarily have good properties. They can be useful as a guide, simply because the tend to be easy to calculate.  It makes more sense to calculate the prob of the null, given the data, because we know what data we saw, but do not know what hypothesis to accept" CreationDate="2011-02-09T12:53:29.303" UserId="2392" />
  <row Id="11218" PostId="7008" Text="@Abe, updated and hopefully clarified in the process. There was one particularly bad typo in the previous version. Sorry about that. " CreationDate="2011-02-09T12:55:32.890" UserId="2970" />
  <row Id="11219" PostId="6962" Text="@ Glen: I did it, but the warning keeps on coming as follows: &quot;your current security settings do not allow this file to be downloaded.&quot; Your help will be appreciated. Thanks a lot." CreationDate="2011-02-09T12:58:18.860" UserId="3085" />
  <row Id="11220" PostId="6605" Text="...cont'd... why do we calculate the probability of something which is certain (the data)?  And why do we condition on something which is uncertain (the null)?  This has always seemed backwards to me (although, if by prob of &quot;data&quot; we actually mean &quot;future data&quot;, then it makes a bit more sense).  To me, it makes more sense to condition on what you observe, because you cannot &quot;unobserve&quot; anything once it has been observed." CreationDate="2011-02-09T12:59:57.000" UserId="2392" />
  <row Id="11221" PostId="6909" Text="@cardinal - I take your point, but my answer does apply reasonably well to the particular densities being compared, as the supports are nested in this case." CreationDate="2011-02-09T13:15:26.367" UserId="2392" />
  <row Id="11222" PostId="6909" Text="@probabilityaslogic, I guess I viewed the exercise as an attempt to find a good distance measure in a general setting. If the objective is to find a good measure that matches one's intuition for a single particular example, you might as well pick your favorite number and assign that value to be the distance. " CreationDate="2011-02-09T13:31:47.880" UserId="2970" />
  <row Id="11223" PostId="6963" Text="$X_1$ and $\sum_j X_j$ are correlated so we have to approximate the integral itself. $\alpha_i$ is often a small number like 1 or 2 and sometimes as big as 10000. Similarly wih $\beta_i$ but it is usually 10 times bigger than $\alpha_i$." CreationDate="2011-02-09T13:50:07.193" UserId="217" />
  <row Id="11224" PostId="6963" Text="The problem is with small $\alpha_i$. If all $\alpha_i$ are big then the good approxmiation of the whole integral is: $\frac{\alpha_1/\beta_1}{\sum_j \alpha_j/\beta_j}$" CreationDate="2011-02-09T13:51:45.430" UserId="217" />
  <row Id="11225" PostId="6933" Text="@cardinal, you might want to change the link to amazon or some other site. I think that linking to the full book might raise some copyright issues." CreationDate="2011-02-09T14:09:41.390" UserId="2116" />
  <row Id="11226" PostId="6795" Text="@Belmont -1, as I recently bought the book of Hastie, I can confirm, that this is an exercise from it. So I am giving you a big -1, since you do not even manage to give all the definitions, I am not even talking about giving the reference. " CreationDate="2011-02-09T14:14:57.847" UserId="2116" />
  <row Id="11227" PostId="6933" Score="2" Text="@mpiktas, nope. No copyright issues. That is the official website for the book. The authors obtained permission from Springer to make the PDF freely available online. (See the note to this effect on the site.) I think they got the idea from Stephen Boyd and his *Convex Optimization* text. Hopefully such a trend will pick up steam over the next few years. Enjoy!" CreationDate="2011-02-09T14:30:10.093" UserId="2970" />
  <row Id="11228" PostId="6933" Text="@cardinal, ooh massive thanks! That is mighty generous from the authors. " CreationDate="2011-02-09T14:42:42.190" UserId="2116" />
  <row Id="11229" PostId="6933" Text="@mpiktas, it's by far the most popular book in the Springer Series in Statistics. It looks good on an iPad. Which reminds me---I should download Boyd's text onto it as well. Cheers. " CreationDate="2011-02-09T14:47:54.227" UserId="2970" />
  <row Id="11230" PostId="7021" Text="I can be more modeling specific, but I don't want to guess here. If you chose to modify your question, I will add more details.&#xA;" CreationDate="2011-02-09T14:52:17.317" UserId="1307" />
  <row Id="11231" PostId="6933" Text="@cardinal, it looks good on Kindle too, also Boyd's too :)" CreationDate="2011-02-09T14:52:20.573" UserId="2116" />
  <row Id="11232" PostId="6933" Text="@mpiktas, the Kindle must have gotten better at displaying books with math. When it first came out, I heard that it was useless for such a purpose. " CreationDate="2011-02-09T15:09:08.703" UserId="2970" />
  <row Id="11233" PostId="6310" Text="@statnovice: The original version of the answer had the numerator and denominator switched." CreationDate="2011-02-09T16:27:24.650" UserId="495" />
  <row Id="11234" PostId="6979" Text="I don't agree. If you're using a confidence interval to test a hypothesis (does it contain zero, for example) you have exactly the same problem, no? OTOH, if you're not worried about testing a hypothesis then the concept of statistical power, as it is traditionally developed, is meaningless." CreationDate="2011-02-09T17:07:09.897" UserId="26" />
  <row Id="11235" PostId="7022" Text="you should mention java environment in the title of your question this will maximize the chance to get an answer. Also is there a tag &quot;java&quot; ? " CreationDate="2011-02-09T17:09:25.053" UserId="223" />
  <row Id="11236" PostId="7022" Text="@robin Yes it is." CreationDate="2011-02-09T17:14:08.283" UserId="88" />
  <row Id="11237" PostId="7022" Text="@mbq: cheers! I wasnt too sure about tagging/mentioning java too much; my interest is primarily if it can be done relatively easily, then a Java implementation. But this works as well." CreationDate="2011-02-09T17:20:14.547" UserId="3014" />
  <row Id="11238" PostId="6933" Text="@cardinal, the new kindle displays pdf files just fine. There are some corner cases, but if pdf is nice, it looks pretty good." CreationDate="2011-02-09T17:50:41.237" UserId="2116" />
  <row Id="11239" PostId="7025" Score="2" Text="This question belongs on an R language server, not here." CreationDate="2011-02-09T18:56:32.690" UserId="919" />
  <row Id="11241" PostId="7025" Text="@Jana, read the file with read.csv and write with write.table, convert to appropriate format with melt. This question is similar to your most recent questions, it really does not belong here. Please ask in stackoverflow." CreationDate="2011-02-09T19:28:24.837" UserId="2116" />
  <row Id="11242" PostId="7025" Text="Oh ok. I'm new to R thought CV is an R community. Will follow it next time :)" CreationDate="2011-02-09T19:49:19.170" UserId="2725" />
  <row Id="11243" PostId="7025" Score="2" Text="@Jana - welcome to the site! With regard to CV vs SO, there is some overlap in terms of active members on each site and the types of questions that are appropriate on each site. See [this post](http://meta.stats.stackexchange.com/questions/584/vote-on-policy-for-r-questions) for some of that discussion. CV is certainly an R community (highest tag on the site) but the questions are generally more grounded in statistical questions or an R implementation of those questions. SO deals more with the programming with R side of the equation...though it's certainly shades of gray." CreationDate="2011-02-09T19:58:10.047" UserId="696" />
  <row Id="11244" PostId="7022" Text="@posdef, if it is normal, then simulation is [easy](http://acs.lbl.gov/software/colt/api/cern/jet/random/Normal.html), just calculate [mean](http://acs.lbl.gov/software/colt/api/cern/jet/stat/Descriptive.html#mean(cern.colt.list.DoubleArrayList)) and standard deviation. For motivation you will need to [test](http://www.vni.com/products/imsl/jmsl/v40/api/com/imsl/stat/NormalityTest.html) whether the sample is normal." CreationDate="2011-02-09T20:01:00.353" UserId="2116" />
  <row Id="11245" PostId="7005" Text="@Erik do you have a textbook reference for the unbiased estimator of variance? I don't know where to go from Wikipedia for more context." CreationDate="2011-02-09T20:02:27.690" UserId="2750" />
  <row Id="11246" PostId="7005" Text="I don't have my standard text [Rice](http://www.amazon.com/Mathematical-Statistics-Data-Analysis-John/dp/0534209343) with me here, so I can't check the page number for you, but I'm sure it's in there. Wikipedia suggests it should also be mentioned in: Montgomery, D.C. and Runger, G.C.: _Applied statistics and probability for engineers_, page 201. John Wiley &amp; Sons New York, 1994." CreationDate="2011-02-09T20:21:25.713" UserId="2898" />
  <row Id="11247" PostId="7028" Text="&quot;I made up a quite complicated and ugly solution which might work for you&quot;  Thanks for making me smile. ;-)" CreationDate="2011-02-09T20:25:14.817" UserId="1657" />
  <row Id="11248" PostId="7032" Score="3" Text="Context, details, or a real or hypothetical example would help focus this discussion greatly." CreationDate="2011-02-09T20:44:48.550" UserId="696" />
  <row Id="11249" PostId="7032" Text="@user3125. Pls be more specific. Also why did you tag it regression? One of the regression methods is related to least sqaure, but not necessarily the same. " CreationDate="2011-02-09T21:12:17.380" UserId="1307" />
  <row Id="11250" PostId="7028" Text="@Joshua Ulrich: I am happy that my answer had a nice effect :)" CreationDate="2011-02-09T22:52:50.180" UserId="2714" />
  <row Id="11251" PostId="6977" Text="@cardinal This is great, thanks so much. I need to study it and look at the preceding chapters in Agresti to truly understand." CreationDate="2011-02-09T23:53:06.527" UserId="2040" />
  <row Id="11253" PostId="6997" Text="HI Mike - you are quite right - it was your note on ezMixed() I read and misinterpreted this for ezANOVA(). " CreationDate="2011-02-10T01:15:19.780" UserId="3112" />
  <row Id="11254" PostId="7039" Score="1" Text="@whuber, note that $-2\log(\sin(\mathrm{arctan}(x))) = \log(1+x^{-2})$, which relates the form of the cdf closer to that of the pdf. It's also interesting to note that this pdf is asymptotic to one-half the pdf of a standard Cauchy. So, the main reason for its use would seem to have to be for its behavior around 0. " CreationDate="2011-02-10T03:41:27.290" UserId="2970" />
  <row Id="11255" PostId="7039" Text="@whuber, though I think I see where you are coming from with regard to your statement about cdfs having closed forms (hint: Louiville), I would urge caution with that remark. The Cauchy distribution itself is a &quot;counterexample&quot; in that respect. " CreationDate="2011-02-10T03:54:04.143" UserId="2970" />
  <row Id="11256" PostId="6977" Text="@B_Miner, I'm not sure what your familiarity with GLMs is, so forgive the suggestion if it is too elementary. But, if you want a good quick overview, look up the notes for log-linear models by G. Rodriguez---at Princeton, I believe. I came across them once and they seemed like a good way to get quickly up to speed for someone who hadn't had much previous exposure. " CreationDate="2011-02-10T04:12:00.307" UserId="2970" />
  <row Id="11257" PostId="6995" Text="thanks so much for your help. this is really helpful" CreationDate="2011-02-10T05:11:34.013" UserId="3107" />
  <row Id="11258" PostId="7039" Text="@cardinal I do not understand the point of your remark about the Cauchy distribution.  I am only using the form of the CDF as a heuristic for narrowing searches and as a target for searches.  The CDF is a little more convenient than the PDF because it is easier to see how it will change when the variable is transformed.  And yes, the relationship you noted is clear, but I chose to write the CDF in this form because of the presence of the arctangent in the other term (which suggests the substitution x = tan(u))." CreationDate="2011-02-10T05:47:02.240" UserId="919" />
  <row Id="11259" PostId="7026" Text="I want to measure the overall quality of service using the attributes like safety, convenience, comfort, etc. per age group, income level, occupation.Is it possible to sum the score for each attribute to obtain the overall quality of service? Is it possible to know how important these attributes to people when considering to ride the jeepney? Thanks." CreationDate="2011-02-10T07:58:01.450" UserId="3118" />
  <row Id="11260" PostId="7022" Text="@mpiktas: thanks for your reply, although I am not quite sure why you chose to comment instead of answering :) As you have mentioned, sampling is no problem but I need to test normality. It seems like the library you've mentioned is commercial though. Could I just do a $\chi^2$ test, or would I need to implement a Shapirov-Wilk, alternatively Lilliefors, test on my own?" CreationDate="2011-02-10T09:28:15.467" UserId="3014" />
  <row Id="11261" PostId="7030" Text="unfortunately I need to create samples as large as the original dataset, plus I need to do this process a large number of times... So picking random values doesn't seem like a viable option in my case." CreationDate="2011-02-10T09:30:28.863" UserId="3014" />
  <row Id="11264" PostId="7042" Text="@Andrej: Thank you for the prompt, clear and very useful answer. I have spent almost a day trying to search internet." CreationDate="2011-02-10T10:29:55.097" UserId="2956" />
  <row Id="11266" PostId="7042" Text="@DrWho So accept it by clicking tick mark." CreationDate="2011-02-10T10:31:57.530" UserId="88" />
  <row Id="11268" PostId="7022" Score="1" Text="@posdef, answers require better quality than comments, this is why I commented. As for commercial use, try google to find non comercial. [This seems free](http://www.numericalmethod.info/javadoc/suanshu/basic/com/numericalmethod/suanshu/stats/test/distribution/normality/ShapiroWilk.html). In worst case scenario you will need to reimplement this test, or another normality test. I suggest asking here or on stackoverflow , which of the normality tests is easiest to implement. " CreationDate="2011-02-10T10:48:46.137" UserId="2116" />
  <row Id="11269" PostId="3497" Text="Joshep, if I understand your question, the objective of your work would be to demostrate that running regression analysis on random samples you obtained simular results to those from the entire data set if the exchangeable assumption holds. My question is if someone knows any reference in which this method has been used." CreationDate="2011-02-10T10:54:47.067" UserId="221" />
  <row Id="11270" PostId="1539" Text="@robin girard, the difficult part actually is to show weak convergence. The convergence of supremums then follows directly from continuous mapping theorem. This result can be found in Billingsley's &quot;Convergence of Probability Measures&quot;. Van der Vaart and Wellner give more general result and their book is really, really tough :)" CreationDate="2011-02-10T10:58:16.300" UserId="2116" />
  <row Id="11271" PostId="684" Text="@whuber Fair point, have deleted the statement. " CreationDate="2011-02-10T11:42:00.793" UserId="81" />
  <row Id="11272" PostId="358" Text="Remember, the normal distribution is the most &quot;cautious&quot; one for known variance (because is has maximum entropy among all densities with fixed variance).  It leaves the most to be said by the data.  Or put another way, for &quot;large&quot; data sets with the same variance, &quot;you&quot; have to &quot;try&quot; *incredibly* hard to get a distribution which is different from a normal." CreationDate="2011-02-10T12:16:32.693" UserId="2392" />
  <row Id="11274" PostId="7039" Text="@whuber, well perhaps I would have been better off asking for clarification rather than assuming. What was your point regarding your comment that a closed form cdf severely limits the possibilities?" CreationDate="2011-02-10T12:27:05.630" UserId="2970" />
  <row Id="11275" PostId="7046" Text="@Bernd: I will try to rectify the table. I am new to this. I am trying" CreationDate="2011-02-10T12:35:01.837" UserId="2956" />
  <row Id="11276" PostId="7045" Text="@mariana, I don't understand. Both models you listed are linear in the coefficients. Are you looking for something that automatically does both feature extraction *and* model selection? Regards. " CreationDate="2011-02-10T12:37:36.823" UserId="2970" />
  <row Id="11277" PostId="2501" Text="+1 for a good and informative answer. I find it useful to see a good explanation for a common misunderstanding (which I have incidentally been experiencing myself: http://stats.stackexchange.com/questions/7022/parameter-estimation-for-normal-distribution-in-java). What I miss though, is an alternative solution to this common misunderstanding. I mean, if normality tests are the wrong way to go, how does one go about checking if a normal approximation is acceptable/justified? " CreationDate="2011-02-10T12:45:49.370" UserId="3014" />
  <row Id="11278" PostId="42" Score="2" Text="@shabbychef: Java programs doesn't have to be a slow monsters. I admit that well written C-code will almost always be faster than a similar implementation in Java but well written Java code will most likely not be super slow. Plus developing in Java has many significant advantages. " CreationDate="2011-02-10T12:52:21.620" UserId="3014" />
  <row Id="11279" PostId="7045" Text="Yes, but mostly for model selection." CreationDate="2011-02-10T12:54:24.423" UserId="1808" />
  <row Id="11280" PostId="7046" Text="@Bernd: Thank you for your help. The 1-tailed p-value was recommended by the software Comprehensive Meta Analysis v2.2. The Data are&#xA;Year Region 1	 Region 2	 Region 3&#xA;Cases	Dead     Cases	Dead	Cases	Dead Total cases	Total dead cases&#xA;2006	2320	528	1484	108	73	3	3877	639 2007	3024	645 1592	75	32	1	4648	721 2008	3012	537	1920	53	3	0	4935	590 2009	3073	556	1477	40	246	8	4796	604 2010	3540 494	1460	26	138	1	5138	521 Total	14969	2760	7933	302	492	13	23394	3075 I will try to rectify the table.– DrWho" CreationDate="2011-02-10T12:54:29.153" UserId="2956" />
  <row Id="11281" PostId="2498" Text="+1: great answer, very intuitive. Perhaps a bit off-topic but how would one go about implement the second method without qq-plots (due to lack of visualization)? What logical steps are taken here to get the p-values?" CreationDate="2011-02-10T13:04:34.003" UserId="3014" />
  <row Id="11282" PostId="2498" Text="@posdef : those are just the p-values of the shapiro-wilks test, to indicate that they contradict the qq-plots." CreationDate="2011-02-10T13:31:37.780" UserId="1124" />
  <row Id="11283" PostId="7046" Text="@Bernd: The data of a particular disease is&#xA;&#xA;Year	Region 1		Region 2		Region 3			&#xA;	Cases	Dead	Cases	Dead	Cases	Dead	Total cases	Total dead cases&#xA;2006	2320	528	1484	108	73	3	3877	        639&#xA;2007	3024	645	1592	75	32	1	4648	        721&#xA;2008	3012	537	1920	53	3	0	4935	        590&#xA;2009	3073	556	1477	40	246	8	4796	        604&#xA;2010	3540	494	1460	26	138	1	5138	        521&#xA;Total	14969	2760	7933	302	492	13	23394	       3075&#xA;" CreationDate="2011-02-10T13:41:06.337" UserId="2956" />
  <row Id="11284" PostId="6995" Text="My pleasure Jones :)" CreationDate="2011-02-10T13:48:07.293" UserId="253" />
  <row Id="11285" PostId="7039" Text="@cardinal I am performing a *wide* search in the sense of finding a named (or heretofore studied) distribution $G$ and a relatively simple re-expression $y$ (such as a power or logarithm etc.) such that $y(X)$ has cdf $G$ iff $X$ has pdf $f$.  If a distribution has been studied before, then it's highly likely its CDF has been obtained and, if it can be written in closed form, that form has also been published.  Therefore we need only look for functional forms $G$ that look like $u - \tan(u)\log(\sin(u))$ with $u = u(x)$.  Know of any?" CreationDate="2011-02-10T13:54:02.410" UserId="919" />
  <row Id="11286" PostId="684" Text="@Tony I removed the downvote :-)." CreationDate="2011-02-10T13:57:06.600" UserId="919" />
  <row Id="11287" PostId="7045" Score="1" Text="@Mariana Do you mean &quot;general linear model&quot; (http://en.wikipedia.org/wiki/General_linear_model ) or &quot;generalized linear model&quot; (http://en.wikipedia.org/wiki/Generalized_linear_model )?  The second one handles binomial dependent variables.  " CreationDate="2011-02-10T14:00:10.183" UserId="919" />
  <row Id="11289" PostId="7039" Text="@cardinal Exactly! Good analysis. That was the thought behind describing this as a &quot;perturbed Cauchy&quot; distribution. (There aren't too many algebraic manipulations to try. If the general-purpose substitution u = arctan(x/2) (which is essentially what we started with) does not work we're probably out of luck, because if an expression in terms of elementary functions is possible, this substitution should do it. Otherwise, start looking at elliptic functions...but at that point you know this is an unusual distribution anyway.) [It seems that the comment to which this is responding was deleted.]" CreationDate="2011-02-10T14:43:42.233" UserId="919" />
  <row Id="11290" PostId="7039" Text="@whuber, not off-hand. The distribution can be viewed as an additive mixture of a standard Cauchy with something else, though. That &quot;something else&quot; has thin tails, but doesn't seem to be obvious. It seems pretty implausible, but maybe finding the ch.f. and looking for some relationship with the Cauchy would yield something. " CreationDate="2011-02-10T14:44:28.367" UserId="2970" />
  <row Id="11291" PostId="7039" Text="@Cardinal The &quot;something else&quot; is not a distribution because it does not increase monotonically (its &quot;pdf&quot; would have some negative values).  It basically shifts some probability out of the region $|x| \gt$ 0.504976 into its complement (a neighborhood of 0)." CreationDate="2011-02-10T14:52:21.847" UserId="919" />
  <row Id="11292" PostId="7039" Text="@whuber, sorry, yes. My attempt to clarify something I was saying mucked things up. My apologies. And, yes, I noticed the fact that it couldn't be a mixture right before your most recent comment. But, you beat me to it!" CreationDate="2011-02-10T14:56:27.903" UserId="2970" />
  <row Id="11293" PostId="2498" Score="1" Text="@joris: I think there might have been a misunderstanding; Shapiro-Wilks give p_{n5000} = 0.87 while the second calculation yields p_{n5000} = 0.007. Or have I misunderstood something?" CreationDate="2011-02-10T14:58:06.390" UserId="3014" />
  <row Id="11294" PostId="2498" Text="Indeed. 0.87 is the proportion of datasets that give a deviation from normality, meaning that in 87% of the datasets from an almost normal distribution, Shapiro-Wilks will have a p-value smaller than 0.05. The second part is just an example of some datasets that illustrate this." CreationDate="2011-02-10T15:02:03.170" UserId="1124" />
  <row Id="11295" PostId="7049" Text="How do your empirical distributions differ from Gamma variates, then?" CreationDate="2011-02-10T15:15:40.280" UserId="919" />
  <row Id="11296" PostId="7050" Text="Hello Whuber.  Your example of ESDA is a great example, thank you!  If you (or others) can suggest other examples of when formal procedures are less relevant - this would be most helpful." CreationDate="2011-02-10T15:23:13.793" UserId="253" />
  <row Id="11297" PostId="7040" Text="Comparison of Region 1 and 2 on what metric? e.g. log odds ratio, log risk ratio, risk difference ...?" CreationDate="2011-02-10T15:27:57.543" UserId="449" />
  <row Id="11298" PostId="7056" Score="2" Text="Move to StackOverflow?" CreationDate="2011-02-10T16:23:20.497" UserId="597" />
  <row Id="11299" PostId="7047" Text="Do you strictly have to do it with Matlab? Otherwise I suggest using Hugin http://hugin.sourceforge.net/ . The software is released under GPL so you may also peek into the source code and see how the magic happens." CreationDate="2011-02-10T16:28:32.257" UserId="582" />
  <row Id="11300" PostId="7049" Text="I have used the gamma distribution for these data. If you use the gamma distribution with a log link you get almost the exact same result you get from an over-dispersed poisson model.However, in most of the statistical packages I am familiar with poisson regression is simpler and much more flexible. " CreationDate="2011-02-10T16:50:09.343" UserId="3136" />
  <row Id="11301" PostId="7005" Text="@Erik thanks for your help with this. This answer has been very useful and it has been informative to quantify variance uncertainty - I have applied the equation about 10 times in the last day. calculating $kappa$ is easy with the `moments` library: &#xA;`library(moments); k &lt;- kurtosis(x);&#xA; n &lt;- length(x); var(x)^2*(2/(n-1) + k/n)`&#xA;" CreationDate="2011-02-10T16:50:56.623" UserId="2750" />
  <row Id="11302" PostId="6963" Text="@Łukasz If you need to evaluate the expression of the expectation, why do you require an algebraic formula?&#xA;I am thinking in applying some numerical trick to get the expectation but I need some feedback :)" CreationDate="2011-02-10T16:58:26.490" UserId="2902" />
  <row Id="11303" PostId="7056" Score="1" Text="I think it is Ok to answer this question here. We do have a `R` tag." CreationDate="2011-02-10T16:58:42.963" UserId="1307" />
  <row Id="11304" PostId="7054" Score="1" Text="What is the aim of the analysis? What question(s) are you trying to answer?" CreationDate="2011-02-10T16:59:27.957" UserId="449" />
  <row Id="11305" PostId="7057" Score="1" Text="please provide an example illustrating your problem. Also how do you define power of algorithm?" CreationDate="2011-02-10T17:11:11.277" UserId="2116" />
  <row Id="11306" PostId="7048" Score="2" Text="At least in my case, I'm somewhat in the same boat. I appreciate Mondrian and keep it up to date, but when I actually explore a new dataset it tends to be in R, which is less interactive but more flexible overall. I started writing you a full answer and realized that I was speaking in the theoretical and not from actual experience." CreationDate="2011-02-10T17:14:08.807" UserId="1764" />
  <row Id="11307" PostId="7058" Text="it would help if you provided example of how do you define redundancy in the boolean case, and what kind of results you would expect in continuous case" CreationDate="2011-02-10T17:16:49.940" UserId="2116" />
  <row Id="11308" PostId="7058" Text="@mpiktas: Edit my question in response to your comment." CreationDate="2011-02-10T17:35:05.957" UserId="977" />
  <row Id="11309" PostId="7005" Text="Glad it's working well for you!" CreationDate="2011-02-10T18:51:20.840" UserId="2898" />
  <row Id="11310" PostId="7026" Text="You could revise your question to clarify this, or set up a new question. Of course you can build a &quot;scale&quot; by combining the three measures. But it's also good to look at the correlations (using a correlation matrix) between the three measures first. This way you can get an idea, whetter people differentiate between those measures, or if they all answer these items in a similar way." CreationDate="2011-02-10T19:27:24.610" UserId="1435" />
  <row Id="11311" PostId="7057" Score="1" Text="@Marina I've used glmnet and lars packages in a couple of projects. In my limited experience I've had A LOT of problems implementing glmnet. I think that glmnet needs some bug fixes regarding the type of variables used in the data frame. Besides, glmnet has confusing documentation. I ended up using lars, and I was very satisfied with the results. Nevermind the size of your problem, I think lars can handle it." CreationDate="2011-02-10T19:44:45.207" UserId="2902" />
  <row Id="11312" PostId="7061" Score="1" Text="The answer depends upon what prior knowledge you have.  You might get a different answer if you know John is the captain of the basketball team." CreationDate="2011-02-10T20:37:06.950" UserId="2485" />
  <row Id="11313" PostId="7059" Score="1" Text="@David this is not just a visualization problem, you may be interested in the discussion of split-apply-combine strategies facilitated by Hadley's plyr package http://had.co.nz/plyr/plyr-intro-090510.pdf" CreationDate="2011-02-10T20:37:50.013" UserId="2750" />
  <row Id="11314" PostId="7059" Text="@David how do you compute the &quot;amount&quot;? Is it the sum of actions? We would need the duration of each single action if you want the amount of time... " CreationDate="2011-02-10T20:38:25.917" UserId="1443" />
  <row Id="11315" PostId="7059" Text="@teucer I was thinking about the number of create/update/check actions entered per unit time for each table between the first login and last action. Some tables might have 1-2 creates/week, while others may have as many as 100 creates in a day or even more checks/day. " CreationDate="2011-02-10T20:42:36.470" UserId="1381" />
  <row Id="11316" PostId="7059" Text="@David could you provide the duration of each logging?" CreationDate="2011-02-10T20:49:38.747" UserId="1443" />
  <row Id="11317" PostId="7029" Text="@John D. Cook, out of curiousity, have you got a citation for the source where you saw this originally?&#xD;&#xA;" CreationDate="2011-02-10T20:49:48.430" UserId="26" />
  <row Id="11318" PostId="7059" Text="@David I guess when you say first login and last action, these are per day(?). I can easily compute the number of actions per user per week, but, with the given data, it is difficult to translate this in amount of time... " CreationDate="2011-02-10T20:56:35.747" UserId="1443" />
  <row Id="11319" PostId="7059" Text="@teucer, each logging event is instantaneous, but the employees are assumed to be working on the next logging event between one logging and the next. To a first approximation, for each day x user combination, the amount of total time worked could be estimated as the max(time) - min(time) + 30 min, and a continuous block of time could be considered any stretch of time during which $max(\Delta t)&lt;30\text{min}$." CreationDate="2011-02-10T21:01:28.247" UserId="1381" />
  <row Id="11320" PostId="7059" Text="@teucer I do have a separate time sheet, so that would probably be a good estimate of the total time spent, so really the number of actions per day or per week are more important." CreationDate="2011-02-10T21:06:26.993" UserId="1381" />
  <row Id="11321" PostId="7029" Text="JMS: &quot;The horseshoe estimator for sparse signals&quot; by Carvalho, Polson, and Scott. I saw it as a preprint, but it may have been published in Biometrika by now. They don't exactly use this prior, but the density above is an approximation to a special case of their prior." CreationDate="2011-02-10T21:11:56.840" UserId="319" />
  <row Id="11322" PostId="7017" Text="Thanks! I now have been able to perform the preacher and hayes test for mediation. I am not sure about the output though..I understand that ´boot´ is the estimation of the indirect effect, but is there also a significance p of this?" CreationDate="2011-02-10T21:18:37.187" UserId="3119" />
  <row Id="11323" PostId="6963" Text="I need to evaluate it many times in my program. It has to be very fast, i.e. no loops and preferably not too many divisions." CreationDate="2011-02-10T21:23:56.917" UserId="217" />
  <row Id="11324" PostId="7068" Text="@teucer thanks for walking me through the data manipulation, but how can I retain the variables 'action' and 'table' so that I can use something like `facet_grid(. ~ action)` or `facet_grid(. ~ action*table)`?  " CreationDate="2011-02-10T22:01:20.787" UserId="1381" />
  <row Id="11325" PostId="7061" Text="assume zero prior knowledge" CreationDate="2011-02-10T22:07:00.787" UserId="3143" />
  <row Id="11326" PostId="7066" Text="Aren't the probabilities for John and Mike to make their next shot simply respectively 0.76 and 0.8? Either way, that's not what I'm trying to determine. What I'd like to calculate is the probability (given the limited observations) that Mike has a higher free throw percentage than John.  " CreationDate="2011-02-10T22:21:56.110" UserId="3143" />
  <row Id="11327" PostId="7066" Text="+1.  The posterior probability that Mike, with distribution $B(\alpha,\beta)$, is better than John, with distribution $B(\gamma,\delta)$, is proportional to a generalized hypergeometric distribution.  This can be computed either as a polynomial or as a rapidly converging power series, giving far greater computational accuracy than any simulation. " CreationDate="2011-02-10T22:26:16.383" UserId="919" />
  <row Id="11328" PostId="7066" Text="@HawkEgg There is uncertainty about how good either John or Mike are.  E.g., if Mike makes 2/3 shots and John makes 8/11 shots, the estimated probability for each is 3/4 (assuming uniform priors for each), but the chance that John is better than Mike is only 165/364 = 45.3% because there's a good chance Mike is a superb shooter (and got unlucky once) whereas we have a fairly good idea how good John is (because he has taken more shots)." CreationDate="2011-02-10T22:32:33.033" UserId="919" />
  <row Id="11329" PostId="7061" Score="1" Text="Assuming a uniform prior, the answer is 848437653385740282130263/1167635946155025951609137 = about 72.66%." CreationDate="2011-02-10T22:35:52.450" UserId="919" />
  <row Id="11330" PostId="7071" Text="When you say correlation between values at different times... do you think all possible lag combinations or only t vs t-1?" CreationDate="2011-02-10T22:37:28.063" UserId="333" />
  <row Id="11331" PostId="7061" Text="@HawkEgg Michael McGowan's point is that you *must* make some such assumption in order to obtain an answer.  It's like asking whether a cannonball or a feather will hit the ground first without specifying how high each one was when dropped: if you are ignorant of that essential information, *you simply cannot provide a justified answer,* even though the problem makes sense and physical theory exists to solve it for any given pair of heights." CreationDate="2011-02-10T22:38:08.127" UserId="919" />
  <row Id="11332" PostId="7071" Text="@user333 *All* nonzero lags: that's the first equation in the Wikipedia link @onestop gave." CreationDate="2011-02-10T22:40:27.200" UserId="919" />
  <row Id="11333" PostId="7072" Text="@Onestop In what sense is logistic regression nonparametric?  Just wondering..." CreationDate="2011-02-10T22:42:27.073" UserId="919" />
  <row Id="11335" PostId="7072" Score="1" Text="I was trying to make the point that it isn't, or isn't normally thought of as such, but it's late and i'm too tired to word it well... " CreationDate="2011-02-10T22:47:01.933" UserId="449" />
  <row Id="11336" PostId="7049" Text="Wouldn't there be other distributions that are better, e.g. whuber's suggestion of gamma?" CreationDate="2011-02-10T22:49:12.943" UserId="686" />
  <row Id="11338" PostId="6820" Text="Thanks for the hint." CreationDate="2011-02-11T00:00:37.633" UserId="2909" />
  <row Id="11339" PostId="7041" Text="@whuber Isn't that the definition of statistics....I'm 99% sure that I don't know what 99% sure means, but I still have to eat....?  I've been chewing on your answer and I ran your code (there's a typo where t is calc'd, but I got it).  I follow the concept on the first 3 equations, but I ran into trouble on the last equation.  My main problem is where N is applied to both the dependent and independent variables. My 55% dependent wins number applies only when the independent variable is &quot;true&quot;.  That made me realize just how screwed up my original post really is.  I'll post more info later." CreationDate="2011-02-11T00:08:22.350" UserId="2775" />
  <row Id="11340" PostId="7074" Score="4" Text="@onestop, @mbq, close and merge with related question(s)? The OP has posted three similar questions in rapid succession. " CreationDate="2011-02-11T00:10:47.327" UserId="2970" />
  <row Id="11341" PostId="7038" Text="@RobH Wow, that's a surprise. Here I was molesting some Newtony-Raphsony thing and you were over there finding a solution.  It will be used in the future.  Thanks." CreationDate="2011-02-11T00:11:57.357" UserId="2775" />
  <row Id="11342" PostId="7062" Score="2" Text="@dsmcha, sorry to say this, but I don't think I like that example too much. The response is identical to two of the predictors? That's beyond pathological, in my view. " CreationDate="2011-02-11T00:31:11.457" UserId="2970" />
  <row Id="11343" PostId="7041" Text="@whuber From your calcs, doesn't the last equation come from aligning the alpha 1% tail of the 52.8% &quot;dumb luck&quot; distribution with the beta 99% tail of the 55% alternative?" CreationDate="2011-02-11T00:54:24.200" UserId="2775" />
  <row Id="11344" PostId="7066" Text="@whuber Are you saying that Mike, who made a lower % of shots than John, has a 54.7% chance of being better than John? That doesn't pass my sniff test. Could you give me some more details on how you calculated the 3/4 and 45.3% numbers? A reference?" CreationDate="2011-02-11T01:15:47.893" UserId="3143" />
  <row Id="11345" PostId="7017" Text="Hi Renee, you might want to ask a separate question about interpretation of bootstrapping for mediation. However, as a simple rule that is often applied, if the bootstrapped 95% confidence interval of the indirect effect does not include zero, then the indirect effect is deemed to be statistically significant. Preacher and Hayes (2004) explains this in great detail: http://www.comm.ohio-state.edu/ahayes/BRMIC2004.pdf" CreationDate="2011-02-11T01:20:28.920" UserId="183" />
  <row Id="11346" PostId="7066" Text="@HawkEgg if you'd like to know who has the higher free throw percentage than you can compare the posterior distributions instead of the predictive. That is, you can compare a Beta(a0 + shots made, b0 + shots missed) for each player." CreationDate="2011-02-11T02:17:35.343" UserId="1913" />
  <row Id="11347" PostId="7066" Text="@HawkEgg: Mike: 80/100 = 80%; John: 38/50 = 76%.  Sniff again :-)." CreationDate="2011-02-11T03:39:15.810" UserId="919" />
  <row Id="11348" PostId="7041" Text="@bill_080 I think you're right, but conventionally we would say we are matching the *99%* (i.e., upper) percentile of the 52.8% distribution to the *1%* (i.e., lower) percentile of the 55% alternative." CreationDate="2011-02-11T03:40:56.010" UserId="919" />
  <row Id="11349" PostId="7051" Text="@simon. I think, in addition to your point, even if @user3136 is not willing to make the assumption of mean = variance, he/she can use the `quasipoisson` family in `glm` ." CreationDate="2011-02-11T03:45:28.997" UserId="1307" />
  <row Id="11350" PostId="7051" Score="1" Text="@simon. But my problem is why would you want to transform continous data to discrete. It is loosing information essentially. Also when a simple `log` transform would have worked, why discretize your data? Using `glm` works, but every result is asymptotics based (which may or may not hold)" CreationDate="2011-02-11T03:47:56.763" UserId="1307" />
  <row Id="11351" PostId="7074" Text="@cardinal True.  I share your feelings.  However, this one is a legitimate question whose answer is not immediately found through an Internet search.  It would be better, IMHO, to close the preceding (trivial) questions and let this one stand." CreationDate="2011-02-11T04:11:08.767" UserId="919" />
  <row Id="11352" PostId="7072" Text="@onestop Thanks for clearing that up for me!  I was worried that I had missed something." CreationDate="2011-02-11T04:12:27.127" UserId="919" />
  <row Id="11353" PostId="7066" Text="@whuber I was referring to _&quot;Mike makes 2/3 shots and John makes 8/11 shots&quot;_ Not realizing that you were adding that to the previous totals (38/50 &amp; 80/100)" CreationDate="2011-02-11T04:12:43.250" UserId="3143" />
  <row Id="11354" PostId="7078" Text="This is correct!  The denominator of course is the product of two Beta functions.  The numerator is equivalent to the generalized hypergeometric function I referred to." CreationDate="2011-02-11T04:16:51.940" UserId="919" />
  <row Id="11355" PostId="7078" Text="@whuber Thank you for getting me started. I'm still new to stats so I had to scour wikipedia and my long forgotten calculus for help." CreationDate="2011-02-11T04:20:15.880" UserId="3143" />
  <row Id="11356" PostId="6240" Text="this question on equivalence testing also has some good suggestions http://stats.stackexchange.com/questions/3038/testing-hypothesis-of-no-group-differences" CreationDate="2011-02-11T04:46:23.047" UserId="183" />
  <row Id="11357" PostId="7054" Text="@onestop: Thank you for your time. The same treatment was given in all 3 regions but with different levels of dedication to the suffering patient due to various administrative reasons. The aim is to show that there is a significant increase in the number of deaths in 1 region due to poor implementation of the treatment guidelines." CreationDate="2011-02-11T05:04:53.667" UserId="2956" />
  <row Id="11358" PostId="7074" Text="@cardianl Working on it. @whuber I agree that this particular question should stay." CreationDate="2011-02-11T07:31:24.700" UserId="88" />
  <row Id="11359" PostId="7071" Text="@onestop, you forgot the constant variation. If variation varies, then the process is not white noise. " CreationDate="2011-02-11T07:37:03.927" UserId="2116" />
  <row Id="11360" PostId="7054" Text="@DrWho: I'm not sure if meta-analysis is the correct approach to analyze this dataset. It seems more like classic longitudinal design. I think you should reconsider your methodology." CreationDate="2011-02-11T07:56:39.277" UserId="609" />
  <row Id="11361" PostId="7017" Text="Thank you, that really helps!" CreationDate="2011-02-11T07:59:12.587" UserId="3119" />
  <row Id="11362" PostId="7017" Text="The last thing I am wondering about is what macro or test I can perform to assess moderation for non normal distributions. You mentioned bootstrapping, but I don´t know where to find it in SPSS or find a specific macro for that. It is very straightforward moderation I am looking for: 1 independent variable, 1 moderator and 1 dependent variable" CreationDate="2011-02-11T08:00:24.533" UserId="3119" />
  <row Id="11364" PostId="7068" Text="@David, `rr &lt;-records[records$action!=&quot;Login&quot;,]`&#xA;`ddply(rr,~user+table+action+week,nrow)`&#xA;" CreationDate="2011-02-11T08:41:53.260" UserId="2116" />
  <row Id="11365" PostId="2498" Text="@joris: I see, thanks for straightening it out for me :) " CreationDate="2011-02-11T08:46:10.490" UserId="3014" />
  <row Id="11366" PostId="7071" Text="@mpiktas: you're right, good point." CreationDate="2011-02-11T08:51:55.613" UserId="449" />
  <row Id="11367" PostId="7086" Text="true; and I do have permutation based statistics as well. Essentially I am implementing two different ways of analyzing the data, and I need comparable scores (in this case it'll be p-values). The motivation behind my approach is that I have reasons to believe: a) the underlying distribution essentially is very close to normal and, b) the measured values are not large enough to resample as many times, or will add bias to my analysis. But as I said above, there will be permutation based statistics as well." CreationDate="2011-02-11T10:35:14.710" UserId="3014" />
  <row Id="11368" PostId="7017" Text="re moderation, check out this discussion on Preacher and Hayes' facebook page: http://www.facebook.com/topic.php?uid=44574520333&amp;topic=15855 ; otherwise SPSS does have some bootstrapping support in the more recent versions, but I haven't tried it." CreationDate="2011-02-11T10:53:54.813" UserId="183" />
  <row Id="11369" PostId="7051" Text="@suncoolsu: 1) quasipoisson makes the assumption of mean proportional to variance. 2) I didn't mean transform to discrete, i meant transform (maintaining continuity) so you could use a different model." CreationDate="2011-02-11T10:58:21.940" UserId="495" />
  <row Id="11370" PostId="7082" Score="1" Text="Lamentably, in that document Crawley perpetuates the myth that ANCOVA can be applied to scenarios when the covariate is correlated with an IV, even when this explicitly violates an assumption of ANCOVA." CreationDate="2011-02-11T13:20:47.967" UserId="364" />
  <row Id="11371" PostId="7051" Text="@simon. yeah -- I understood agree with you. Sorry, I was talking about the question. Quasi-poisson, takes into account of overdisperson right? (if I remember correctly, cf Faraway 2006)" CreationDate="2011-02-11T13:51:15.453" UserId="1307" />
  <row Id="11372" PostId="7086" Score="2" Text="@posdef, without a little more information on your analysis, it's a little difficult to give advice. **But**, unless you are interested in some unusual quantities or have an unusual design, then using a parametric bootstrap in a Gaussian setting is suboptimal. The core of statistical inference is centered around assumptions related to normality (e.g., exponential families) and there is some very powerful optimality theory in these cases. Without good reason, one shouldn't try to avoid it." CreationDate="2011-02-11T13:51:23.877" UserId="2970" />
  <row Id="11373" PostId="7096" Score="1" Text="and `spline` does not work why?" CreationDate="2011-02-11T13:53:04.667" UserId="2116" />
  <row Id="11374" PostId="7096" Text="@Pantera, please clarify. Are you looking for packages that fit so-called *free-knot spline* models, i.e., regression splines where the number and location of knots must also be determined using the observed data?" CreationDate="2011-02-11T14:06:03.027" UserId="2970" />
  <row Id="11375" PostId="7087" Score="1" Text="@Marina, can you fit a nonregularized model to your data using **lm**? If so, then LARS should be able to handle it, too. " CreationDate="2011-02-11T14:26:23.663" UserId="2970" />
  <row Id="11376" PostId="7054" Text="@Andrej: can you kindly say a few more words about the design?" CreationDate="2011-02-11T14:32:27.543" UserId="2956" />
  <row Id="11377" PostId="7074" Text="against what alternative ?" CreationDate="2011-02-11T15:03:45.980" UserId="603" />
  <row Id="11378" PostId="7098" Score="1" Text="+1.  The remark about final presentations brings to mind, as a notable counterexample, Hans Rosling's famous 2006 TED talk (http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html ).  Re: the point about examining &quot;far more,&quot; I am reminded of how a lawyer in a deposition asked me about how I had examined the data that backed up my testimony and how her face fell when she learned the work had been done interactively and therefore nothing was printed or saved (which she could then subpoena, examine, and try to impugn). ;-)" CreationDate="2011-02-11T15:12:13.457" UserId="919" />
  <row Id="11379" PostId="7090" Score="3" Text="There are no computational issues.  Choose a time unit that is readily interpretable." CreationDate="2011-02-11T15:15:02.050" UserId="919" />
  <row Id="11380" PostId="7094" Text="@Onestop This formula has notational problems: there are $p$'s on the left and the right.  The right side makes no reference at all to the subscript $\mathbf{i}$.  Furthermore, still interpreting the $p_i$ as probabilities (as in the original question), the rhs clearly is positive whereas the lhs cannot be positive." CreationDate="2011-02-11T15:21:22.070" UserId="919" />
  <row Id="11381" PostId="7086" Text="@cardinal: can't really say I follow your reasoning, or rather, I am not sure I fully understand what you are saying. :) But I can try and explain my analysis in further detail, if necessary. " CreationDate="2011-02-11T15:31:54.003" UserId="3014" />
  <row Id="11382" PostId="7101" Score="4" Text="Maybe more suited for http://stats.stackexchange.com/" CreationDate="2011-02-11T09:15:38.933" UserId="434" />
  <row Id="11383" PostId="7086" Text="@posdef, any additional detail you could provide would be both helpful and interesting. My point was that **if** you make the leap to assuming normality, it's unlikely that a parametric bootstrap would be the desired approach to go about conducting your inference. But, maybe your situation is complex/unique enough that it does have a role to play. " CreationDate="2011-02-11T16:10:06.367" UserId="2970" />
  <row Id="11384" PostId="7051" Text="In this particular instance I was not satisfied that any transformation I tried (log, sqrt, box-cox) gave a good approximation to normality. Incidentally, if I use the normal score transformation method then I can transform most data to almost beautiful normality, but I have not seen this transformation widely used so I assume there's a catch (it is hard to back-transform)." CreationDate="2011-02-11T17:10:37.680" UserId="3136" />
  <row Id="11385" PostId="7096" Text="Hi - yes, you are perfectly right - I'm looking for an estimator of splines where the number and location of knots must be estimated and using the observed data (time serie data)." CreationDate="2011-02-11T17:43:00.707" UserId="3154" />
  <row Id="11386" PostId="7095" Text="remove 99% of your phones!" CreationDate="2011-02-11T17:46:49.147" UserId="795" />
  <row Id="11387" PostId="7097" Text="Hi - I'll check out the gam package. Thanks!" CreationDate="2011-02-11T17:50:47.753" UserId="3154" />
  <row Id="11388" PostId="7103" Score="4" Text="Some thoughts: (i) Gaussians won't do it.  You're interested in the right tail of the distributions (winds below a few km/h won't generate any power).  (ii) You don't need 100 quantiles; you can interpolate.  Probably 5-16 quantiles will do well.  (iii) You only need a few bits per quantile (8 will do) because economically useful wind speeds lie in a small range.  (iv) 1000 distributions / 15 min * 15 numbers/distribution * 5 E5 min/yr = 5 E8 values = 0.5 GB per year.  That's worth about $0.50 today: you can afford it!" CreationDate="2011-02-11T17:55:00.930" UserId="919" />
  <row Id="11389" PostId="7095" Score="1" Text="Stop paying the phone bill." CreationDate="2011-02-11T18:30:36.690" UserId="2310" />
  <row Id="11390" PostId="7102" Score="1" Text=" &quot;fails to converge&quot;.. &quot;do converge eventually&quot;. &quot;a fit that looks good&quot;...&quot;the visual fit is terrible&quot;. Sorry, but I'm confused!" CreationDate="2011-02-11T18:48:16.257" UserId="449" />
  <row Id="11391" PostId="7090" Score="1" Text="I believe it's advantageous to avoid ties when dealing with time as a continuous variable.  Therefore I think you'd want to base your results on the finest-grained measurements you can get (if that is part of what your question implies)." CreationDate="2011-02-11T18:53:28.233" UserId="2669" />
  <row Id="11392" PostId="7104" Text="Thanks - that seems to make some sense. Embarrassingly my maths isn't good enough to immediately see how you got to teh reparameterisation, but I'll try things out and report back." CreationDate="2011-02-11T20:06:02.160" UserId="3158" />
  <row Id="11393" PostId="7102" Text="By tweaking convergence criteria I can get the model to converge, but when this happens the plotted function appears to be a much worse fit than the unconverged results. Presumably due to very large convergence criteria." CreationDate="2011-02-11T20:08:14.873" UserId="3158" />
  <row Id="11394" PostId="7104" Text="Hmmm. After much fiddling the solution seemed to be a slight tweak of my y parameter from -0.06 to -0.061 to get convergence for all...However, fewer datasets failed to get convergence using your RHS simplification of the first equation, and all but one converged using the c2=0 simplification. Many thanks." CreationDate="2011-02-11T20:43:59.087" UserId="3158" />
  <row Id="11396" PostId="7104" Text="@R_usr Sorry; I was lazy.  I edited the penultimate line to show the new parameters in terms of the original ones." CreationDate="2011-02-11T21:11:37.647" UserId="919" />
  <row Id="11397" PostId="7103" Score="1" Text="That's not a comment @whuber, that's an answer!" CreationDate="2011-02-11T21:35:59.353" UserId="449" />
  <row Id="11398" PostId="7103" Text="@whuber, thanks good remark &quot;Probably 5-16 quantiles will do well&quot; I think that by this remark you enter the problem of compression (even if you deny it afterward).  The remark about gaussian measure in my question is not to illustrate what I would do with wind power but rather to illustrate simple fact about compression of time series of distribution (rather more from a theoretical point of view)." CreationDate="2011-02-11T21:39:01.547" UserId="223" />
  <row Id="11399" PostId="7045" Text="I mean the R function called glm, which is actually the http://en.wikipedia.org/wiki/General_linear_model (general linear_model)" CreationDate="2011-02-11T21:42:38.767" UserId="1808" />
  <row Id="11400" PostId="7103" Score="1" Text="@robin Sorry, I didn't mean to appear to deny this is a compression problem. *Any* method of storing an empirical distribution is implicitly a compression method.  The issue concerns how best to go about it while not wasting computing resources.  I have established that storage is not the bottleneck, implying you should focus on methods that make subsequent analysis sufficiently efficient and accurate." CreationDate="2011-02-11T21:44:08.093" UserId="919" />
  <row Id="11401" PostId="7104" Text="@whuber no need to apologise. If I'm fitting equations I should know how to manipulate them efficiently." CreationDate="2011-02-11T22:09:58.200" UserId="3158" />
  <row Id="11402" PostId="6952" Text="Thanks a lot, this is very usefull to me" CreationDate="2011-02-11T22:59:22.010" UserId="1808" />
  <row Id="11403" PostId="7112" Score="1" Text="If your software is well written it automatically standardizes internally to avoid numerical precision problems.  You shouldn't have to do anything special." CreationDate="2011-02-12T01:14:30.760" UserId="919" />
  <row Id="11404" PostId="7047" Text="Voting to close.  The expertise to answer this question (without the MatLab restriction) resides at http://photo.stackexchange.com/ ." CreationDate="2011-02-12T01:22:59.530" UserId="919" />
  <row Id="11405" PostId="7113" Text="@Henry: Appreciably simple and clear answer. You must be a great teacher. People like you must write a small book on Introduction to Statistics in 200 pages" CreationDate="2011-02-12T02:54:48.633" UserId="2956" />
  <row Id="11407" PostId="7075" Text="wow thanks - that was it - i hadn't even considered the possibility :)" CreationDate="2011-02-12T04:23:03.273" UserId="1463" />
  <row Id="11408" PostId="6746" Score="1" Text="@Chris Taylor, I think part of the problem lies in the fact that the process you've laid out is not fully defined by the characteristics you've listed. For a sample of size $n$, you've given $n\choose 2$ linear constraints for $2^n$ parameters. Many processes could satisfy the constraints and yet lead to different achievable classification rates. Your $R$ code *does* uniquely define a process, but it seemed you intended that as a concrete example instead of as the main object of interest. " CreationDate="2011-02-12T04:43:32.727" UserId="2970" />
  <row Id="11409" PostId="7088" Text="Could you elaborate more on what you mean by format?" CreationDate="2011-02-12T04:44:07.097" UserId="183" />
  <row Id="11410" PostId="7088" Text="By format i mean the expected flow of how you start and end your presentation." CreationDate="2011-02-12T06:53:15.217" UserId="3150" />
  <row Id="11411" PostId="7090" Text="@Whuber: Ok Ok, thanks!&#xA;@rolanda2: Yes, you are right. But if I generate survival times in years I keep all decimal points. Thx" CreationDate="2011-02-12T07:08:19.700" UserId="3019" />
  <row Id="11412" PostId="7067" Text="@Nick : Thanks for the paper!" CreationDate="2011-02-12T08:03:43.213" UserId="3133" />
  <row Id="11413" PostId="7118" Score="1" Text="+1, but why would you not use unstandardised regression coefficients for prediction?" CreationDate="2011-02-12T09:43:56.407" UserId="449" />
  <row Id="11414" PostId="7054" Text="@DrWho, i think it would be more helpful if *you* said a few more words about the design! Are the numbers under 'cases' incident cases or prevalent cases? Are they always different people in each year? If someone became a case in 2006 and died in 2007, how would their data be entered in the above table?" CreationDate="2011-02-12T09:52:34.397" UserId="449" />
  <row Id="11415" PostId="7118" Score="1" Text="@Jeromy (+1) About assessing variable importance, I think the [relaimpo](http://cran.r-project.org/web/packages/relaimpo/index.html) R package does a good job (but see [Getting Started with a Modern Approach to Regression](http://j.mp/dE338Q)). There was also a nice paper by [David V. Budescu](http://j.mp/enzteH) on dominance analysis (freely available on request)." CreationDate="2011-02-12T10:09:30.400" UserId="930" />
  <row Id="11416" PostId="7118" Text="@onestep oops. typo. It's changed now." CreationDate="2011-02-12T11:13:32.843" UserId="183" />
  <row Id="11417" PostId="7054" Text="@onestop: Thank you for your help. Cases are new cases that  occurred every year (incidence). Deaths are within 6 months from onset of diasease. Every year new cases and deaths from that group are only counted. Population is the same but cases are new and different every year. If a person suffered from the disease in 2006 and dies in 2007, he is classified under deaths for the year 2006 if that death occurred within 6 months of onset of disease and attributable to that disease.Kindly advise me which is the best design- historical controls or time-series analysis or  Meta-analysis." CreationDate="2011-02-12T11:54:17.423" UserId="2956" />
  <row Id="11418" PostId="7126" Text="@sesqu &quot;your distribution is parametric&quot; can you explain why you say that? " CreationDate="2011-02-12T12:14:07.177" UserId="223" />
  <row Id="11419" PostId="7123" Text="@Jeromy Anglim: Thank you for the additional information. Can you please educate me why we must use different terms if it is just differing numbers of n and k. Is there a practical significance?" CreationDate="2011-02-12T12:18:59.027" UserId="2956" />
  <row Id="11420" PostId="7118" Text="Hi Jeromy - great answer.  Please have a look at my question here:  http://stats.stackexchange.com/questions/6478/how-to-measure-rank-variable-importance-when-using-cart-specifically-using-r  and see if you might add something there.  Thanks!" CreationDate="2011-02-12T12:45:45.173" UserId="253" />
  <row Id="11421" PostId="7064" Text="Thank you Onestop" CreationDate="2011-02-12T12:46:14.377" UserId="686" />
  <row Id="11422" PostId="7126" Text="@robin You said you have a forecast, which implies a model for a random variable. That model comprises a family of distributions for the variable and a function to identify the distribution you predict from the data. The parameter is the processed data that identifies the predictive distribution - without one, there would be no variability in your predictions." CreationDate="2011-02-12T13:00:23.560" UserId="2456" />
  <row Id="11423" PostId="7126" Text="could be non parametric density forecast ? " CreationDate="2011-02-12T13:02:06.287" UserId="223" />
  <row Id="11424" PostId="7126" Text="Sure, if your model family is, say, an equal mix of multinomials $N(\vec {data}, \vec 1)$, you will have to store all data, since that is your parameter. In this case, you will have to compress the data according to some prior. However, unless your model assumes subsequent time points are independent (and it doesn't, or you wouldn't have specified time series), you can still take advantage of the dependence to refine the prior for $X_t|data_t,X_{t-1}$." CreationDate="2011-02-12T13:35:22.567" UserId="2456" />
  <row Id="11425" PostId="7134" Text="the comment on a reviewer is extremely sad, one would hope that a person in that position would at least not openly display their ignorance, and by doing so, further support the misinterpretation of the statistical method." CreationDate="2011-02-12T17:29:04.407" UserId="656" />
  <row Id="11426" PostId="7134" Score="2" Text="Correct me if I'm wrong, but the randomness of sampling simply affects the degree to which you can generalize findings. In contrast, random assignment is the more critical feature for causal inference." CreationDate="2011-02-12T17:29:25.460" UserId="364" />
  <row Id="11428" PostId="7054" Text="@DrWho: please look here: http://goo.gl/HPAy4" CreationDate="2011-02-12T17:48:05.170" UserId="609" />
  <row Id="11430" PostId="7133" Text="ok, thanks a lot. Is there anyone who can suggest a different analysis?" CreationDate="2011-02-12T18:50:57.583" UserId="3180" />
  <row Id="11432" PostId="7134" Text="Mike, I agree with you.  Do you make this point to extend the discussion or to indicate disagreement with something I've said?" CreationDate="2011-02-12T23:08:05.613" UserId="2669" />
  <row Id="11433" PostId="6953" Text="Actually, I'm not sure I quite follow how these functions work. The examples give results for a normal distribution, but where do I plug in my probability density function?" CreationDate="2011-02-12T23:19:44.133" UserId="52" />
  <row Id="11434" PostId="7136" Text="Wolfgang - interesting and helpful points.  I should have made clear, though, that much of my work is on surveys." CreationDate="2011-02-12T23:28:22.950" UserId="2669" />
  <row Id="11435" PostId="6953" Text="(I've done tests, and my data doesn't seem to be remotely normal.)" CreationDate="2011-02-12T23:40:22.673" UserId="52" />
  <row Id="11436" PostId="7133" Text="Could you please state your research question(s) and how many ratings each subject provided." CreationDate="2011-02-12T23:48:29.860" UserId="2669" />
  <row Id="11437" PostId="7133" Text="Hi, thanks a lot for your reply. Actually there were 36 stimuli, and each stimulus was evaluated only once by each of the 12 participants. So each subject provided 36 ratings on the 9 point Likert scale, one for each stimulus. Any suggestion?" CreationDate="2011-02-13T01:59:36.657" UserId="3180" />
  <row Id="11438" PostId="6953" Text="@Alan H., plug your density function in to `dF`. That is `dF` should return density function value at given argument." CreationDate="2011-02-13T04:39:26.233" UserId="2116" />
  <row Id="11439" PostId="7123" Text="@drwho I've updated my answer with a few thoughts." CreationDate="2011-02-13T05:08:50.003" UserId="183" />
  <row Id="11440" PostId="7141" Score="1" Text="This doesn't qualify as an answer, but I think something along the lines of a maximum entropy distribution for $\mu_{i}$, subject to your constraints $\left| \mu_i - \mu_{i+1}\right| \le \gamma$.  Set this distribution as the prior, and then set jeffreys prior for $\sigma$.  Then update using Bayes theorem to a posterior for $\mu_n$, and calculate the probability $Pr(\mu_{n}&gt;0)$, and use it like a p-value.  Calculating the MaxEnt distribution may be difficult to do though." CreationDate="2011-02-13T05:27:40.750" UserId="2392" />
  <row Id="11441" PostId="7141" Text="@probabilityislogic: We can assume that $\sigma$ is known, which should simplify this computation." CreationDate="2011-02-13T05:50:42.823" UserId="795" />
  <row Id="11442" PostId="7141" Score="2" Text="@shabbychef, if you set up a likelihood ratio test, then under both $H_0$ and $H_1$, you have a convex program to solve for $\hat{\mu}_i$. If you look at the KKT conditions, you might even be able to find an explicit solution. Maybe try &quot;walking backward&quot; from $i = n$, e.g., under $H_0$, it seems that if $|X_{n-1}| &gt; \gamma$, then $\hat{\mu}_{n-1}$ is likely equal to $\mathrm{sgn}(X_{n-1}) \gamma$. You might also look at the *fused lasso*, which has a &quot;similar&quot; (but not identical) form as your problem. I'm not sure that much is known about distributions of likelihood ratios in that setting." CreationDate="2011-02-13T05:59:11.170" UserId="2970" />
  <row Id="11447" PostId="7146" Text="About the picture, post to some image hosting and put a link -- I'll convert it into a pasted-in picture." CreationDate="2011-02-13T08:35:32.897" UserId="88" />
  <row Id="11448" PostId="7146" Text="+1, interesting question, you might find that soon you will have enough reputation :)" CreationDate="2011-02-13T08:46:04.783" UserId="2116" />
  <row Id="11449" PostId="7146" Text="yup i have thanks to all that helped me!!" CreationDate="2011-02-13T08:49:57.633" UserId="3178" />
  <row Id="11450" PostId="7141" Text="@cardinal, we have the same amount of $\mu_i$ as there are data points. Will this not pose a problem for estimation?" CreationDate="2011-02-13T08:54:41.863" UserId="2116" />
  <row Id="11451" PostId="7146" Score="2" Text="judging from the graph, the problem is not the lag. What you have plotted is roughly goodness of fit. So it seems that Weibull distribution is not apropriate for your data. I see that there is a bunch up near zero, do you have zero values in your data? In that case you will need to model zero values separately. So first suggestion would be to try Weibull for non zero values. Also why Weibull, is there particular reason, some reference from similar work perhaps?" CreationDate="2011-02-13T09:02:59.097" UserId="2116" />
  <row Id="11452" PostId="7146" Text="well yes mainly from literature and other works on wind for the purpose of wind farms" CreationDate="2011-02-13T09:08:17.023" UserId="3178" />
  <row Id="11453" PostId="7110" Text="Connected question: http://stats.stackexchange.com/questions/812/what-is-the-difference-between-functional-data-analysis-and-high-dimensional-data" CreationDate="2011-02-13T09:11:20.500" UserId="223" />
  <row Id="11454" PostId="7146" Text="i tried ploting it with out zero's but still there is a lag" CreationDate="2011-02-13T09:12:24.227" UserId="3178" />
  <row Id="11456" PostId="7146" Score="1" Text="note that 'lag' is a term used mainly in analysis of data in time, referring to one thing occurring after another. This isn't a lag - it's perhaps more accurately called a shift - or maybe an offset - but shift is probably more common for distributions, they shift and scale." CreationDate="2011-02-13T10:28:50.960" UserId="1549" />
  <row Id="11457" PostId="7148" Text="thank you very much!!    " CreationDate="2011-02-13T11:04:42.137" UserId="3178" />
  <row Id="11458" PostId="7148" Text="do you tink you can help me with my second question: calculating wind energy based on the density function?" CreationDate="2011-02-13T11:05:45.210" UserId="3178" />
  <row Id="11459" PostId="7128" Text="What is your research question? Also, what are you calling a stimulus - one type of audio, or the pair? " CreationDate="2011-02-13T11:45:36.057" UserId="183" />
  <row Id="11460" PostId="7151" Text="+1 MDS is the main kind of analysis that I've seen for this type of study." CreationDate="2011-02-13T11:48:54.010" UserId="183" />
  <row Id="11462" PostId="7148" Score="2" Text="I'm not a physicist and I don't know the necessary equations, but I imagine it will involve a numerical integral over the density. R's `integrate()` function may be useful for that." CreationDate="2011-02-13T12:48:42.123" UserId="449" />
  <row Id="11463" PostId="7131" Text="@onestop: Thank you for the detailed and very useful advice. If I use only Region 1 and Region 2, and apply Pearson's chi-squared test to the data for each year separately, would it be better to do a Meta-analysis also for these 2 regions? There is heterogeneity in the sample data and a Random model looked more appropriate and gave a significance at a p value of 0.000. Egger’s linear regression method intercept (B0) 10.34631, 95% confidence interval (1.05905, 19.63357), with t=3.54535, df=3. The 1-tailed p-value is 0.01911, and the 2-tailed p-value is 0.03822 (significant funnel asymmetry)." CreationDate="2011-02-13T13:11:28.253" UserId="2956" />
  <row Id="11464" PostId="7148" Text="i know the equation my problam is i want to compute the percent of time the wind is at each speed " CreationDate="2011-02-13T14:18:22.400" UserId="3178" />
  <row Id="11465" PostId="7090" Text="@whuber, @rolando2 Make answers of that." CreationDate="2011-02-13T14:36:10.930" UserId="88" />
  <row Id="11466" PostId="7152" Score="1" Text="More info is needed here, by zero do you mean missing?  Why do you think you have to do something special because the value is zero?  " CreationDate="2011-02-13T14:39:53.670" UserId="2310" />
  <row Id="11467" PostId="7148" Text="what i meAn is can you help me with the integration --&gt; thank you" CreationDate="2011-02-13T14:45:02.663" UserId="3178" />
  <row Id="11468" PostId="7141" Score="1" Text="@mpiktas, not in general since the constraints can force the &quot;perfect fit&quot; to be infeasible. Of course, if the perfect fit is feasible for $H_0$ then the LRT will be 1. But, I think that's the desired answer in that case anyway!" CreationDate="2011-02-13T15:33:02.537" UserId="2970" />
  <row Id="11469" PostId="7155" Text="If you want to know for a specific distribution then ask about your example.  It will be different for different situations." CreationDate="2011-02-13T15:55:47.737" UserId="601" />
  <row Id="11470" PostId="7155" Score="4" Text="Well, I would expect that you will have a `rigorous definition of an outlier` when you'll be able to define `unreasonable amounts of subjectivity` objective manner ;-), Thanks" CreationDate="2011-02-13T16:00:42.393" UserId="3170" />
  <row Id="11471" PostId="7156" Score="2" Text="@Alexander, are you interested in the *change* in the level of corruption or are you interested in detecting a change in the *rate* of increase or decrease of corruption? They're quite different questions. " CreationDate="2011-02-13T16:06:57.503" UserId="2970" />
  <row Id="11472" PostId="7155" Text="But the definition can vary by underlying distribution and situation.  I could say ± 1.5 IQR, or 3 SD, or some such.  But I could take a totally different approach if I have two kinds of measures, say reaction time and accuracy. I can say RT's conditioned on a level of accuracy.  They can all be good and mathematically rigorous and have different applications and meanings." CreationDate="2011-02-13T16:08:15.643" UserId="601" />
  <row Id="11473" PostId="7128" Text="ok, thanks a lot, I will have a look to MDS and try to use the R functions." CreationDate="2011-02-13T16:15:03.657" UserId="3180" />
  <row Id="11474" PostId="7153" Text="@onestop, @Marco, CW?" CreationDate="2011-02-13T16:15:11.750" UserId="2970" />
  <row Id="11475" PostId="7128" Text="@Luca Please register your account -- this way you won't lose your account again. You can do this here: http://stats.stackexchange.com/users/login" CreationDate="2011-02-13T16:17:21.400" UserId="88" />
  <row Id="11476" PostId="7159" Score="1" Text="for bootstrap i would choose more cores, since paralelism is more important. Question though is more apropriate for stackoverflow" CreationDate="2011-02-13T17:09:29.517" UserId="2116" />
  <row Id="11477" PostId="7159" Text="Do you mean you're running two nested bootstraps, i.e. a bootstrap procedure within a bootstrap procedure? That would certainly be CPU intensive. I would have thought it might perhaps be ok to do the model selection w/o bias correction, then use bias-correction on the final model only. I haven't read Harrell's book though." CreationDate="2011-02-13T17:18:14.263" UserId="449" />
  <row Id="11478" PostId="7159" Text="No, I am not nesting bootstraps. The two bootstrap procedures are different. It's a fairly small dataset 500 (subjects) X 500 (variables) for each. Under Windows, the CPU consumption peaks at &gt;70% on a Dual core older Intel processor and the analyses take several hours. In the near future, I would be tackling even bigger datasets (eg. NHANES) with similar analyses strategies." CreationDate="2011-02-13T18:13:07.177" UserId="3183" />
  <row Id="11479" PostId="7142" Text="Thanks very much- I thought I was on the right track but wasn't sure how express why. That makes sense!" CreationDate="2011-02-13T18:55:21.347" UserId="3171" />
  <row Id="11480" PostId="7140" Text="Thanks, that's great. The paper also stated that the first set of post-scan questionnaires (5 mins after) for the 97 particpants who did return the second set of questionnaires were compared to those of the dropouts. No mention was made of any indication of representativeness. The paper then asks WHY they did this- I'm assuming that is to check that scores weren't significantly higher for the dropouts, as particularly high anxiety could be the reason for dropping out? Wow, talk about a trick question!" CreationDate="2011-02-13T19:01:53.850" UserId="3171" />
  <row Id="11481" PostId="7153" Text="@cardinal: CW? What does that mean???" CreationDate="2011-02-13T19:01:55.557" UserId="3019" />
  <row Id="11482" PostId="7152" Score="1" Text="I would close it as duplicate http://stats.stackexchange.com/questions/1444/how-should-i-transform-non-negative-data-including-zeros/1445#1445 " CreationDate="2011-02-13T19:04:24.487" UserId="223" />
  <row Id="11483" PostId="7159" Text="You're aware that R won't &quot;automatically&quot; use all your cores, right? You need to load up a package that will implement parallel computing. My favorite for embarallel computing (as in the case of bootstrapping) is combining the `doMC` package (`doSMP` if using Windows) and `plyr`'s `.parallel=TRUE` arugment." CreationDate="2011-02-13T19:06:07.823" UserId="364" />
  <row Id="11484" PostId="7153" Text="@Marco, this may be more appropriate if marked as Community Wiki. " CreationDate="2011-02-13T19:14:31.267" UserId="2970" />
  <row Id="11485" PostId="7158" Score="2" Text="Not all outliers are generated from an experiment, however. I worked with a large dataset that involved the collection of real-estate information in a region (sale price, number of bedrooms, square footage, etc), and every now and then, there would be data entry mistakes and I'd have a 400,000 bedroom house go for 4 dollars, or something nonsensical like that. I would think that part of the goal of determining an outlier is to see whether it's possible to be generated from the data, or if it was just an entry error." CreationDate="2011-02-13T19:25:01.540" UserId="1118" />
  <row Id="11486" PostId="7158" Score="2" Text="@Christopher Aden: I'd consider that part of the experimental process. Basically, in order to be able to remove outliers, you have to understand how the data were generated, i.e. no removing outliers without a good reason. Otherwise you're just stylizing your data. I've edited my answer to reflect this a bit better. " CreationDate="2011-02-13T19:40:45.703" UserId="198" />
  <row Id="11487" PostId="7152" Text="That question asked about transformations rather than about inferential issues." CreationDate="2011-02-13T19:41:15.313" UserId="2129" />
  <row Id="11489" PostId="7141" Score="1" Text="I fully agree with cardinal's answer, there is not a very big problem here and hence certainly no litterature (however I like the question :) +1 for that). &#xA;&#xA;However this is certainly a simple particular case of what is called [&quot;Adaptive tests of linear hypotheses by model selection by &quot; Y. Baraud, S. Huet, and B. Laurent][1]&#xA;&#xA;&#xA;  [1]: http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.aos/1046294463" CreationDate="2011-02-13T19:49:37.567" UserId="223" />
  <row Id="11490" PostId="2806" Text="Can you tell us why you want to do PCA ? " CreationDate="2011-02-13T19:54:13.940" UserId="223" />
  <row Id="11492" PostId="7141" Text="@robin girard: The case where the $\mu_i$ are affine with slope $\gamma$ does seem to be amenable to the test of Baraud _et. al._ However, I am looking for a test which performs reasonably well under the more general case of the Lipschitz bound. " CreationDate="2011-02-13T20:12:56.580" UserId="795" />
  <row Id="11495" PostId="7141" Text="@shabbychef I was pretty sure you knew Baraud's paper (from past questions and comments...). I though it contained alternatives similar to yours I'll take a look once. " CreationDate="2011-02-13T20:29:53.813" UserId="223" />
  <row Id="11496" PostId="7141" Text="@robin, @shabbychef, maybe I was misunderstood. I think this problem is interesting. I also find disappointing the dearth of good quality work on inference for the many variants of constrained least-squares estimation that have become so popular over the last fifteen years. If @shabbychef can crack that nut, even for this particular example, it might even be publishable." CreationDate="2011-02-13T20:58:55.900" UserId="2970" />
  <row Id="11497" PostId="7164" Score="2" Text="@bayer, divide the domain of your non-bijective function into parts where the function is bijective and then apply change of variables. This might work." CreationDate="2011-02-13T21:05:43.913" UserId="2116" />
  <row Id="11498" PostId="7041" Text="@whuber Sorry I couldn't get back to this sooner...scheduling issues. I edited my original post based on your answer (see above).  I think I now get it.  Just a few questions.  1) shouldn't your code for t be t&lt;-floor(N*(p + z.alpha*s.p/sqrt(N)))?  2) Isn't size where you're back-checking the 99% value?   3) Isn't sureness where you're back-checking the 1% value? " CreationDate="2011-02-13T21:05:45.020" UserId="2775" />
  <row Id="11499" PostId="7156" Text="@cardinal The rate, thanks for asking. I should have been more clear on that." CreationDate="2011-02-13T21:08:14.703" UserId="3182" />
  <row Id="11500" PostId="7156" Text="@Alexander, that seems unintuitive on the surface---at least to me. Is there reason to believe that levels of corruption generally trend rather than stay constant at some level? Just curious, as I have no domain knowledge there at all. :)" CreationDate="2011-02-13T21:15:23.493" UserId="2970" />
  <row Id="11501" PostId="7156" Text="Corruption is an area where there is little data, since it's hard to measure. You may be right, in that there isn't any good reason to think it's trending. However, changes could be caused by things other change a new president, though that isn't the same as trending I guess. Also, I've edited the question a bit, since my first comment. I realized I'm not sure if it's best to look at the rate or the change." CreationDate="2011-02-13T21:22:02.423" UserId="3182" />
  <row Id="11502" PostId="7156" Text="@Alexander, interesting. I can imagine it's not an easy thing to measure. My guess is that the more corrupt, and less transparent, the government, the harder a precise measurement becomes. You should also be careful of trying to conclude causality as well, as, e.g., the current situation in Egypt could end up demonstrating." CreationDate="2011-02-13T21:25:13.607" UserId="2970" />
  <row Id="11503" PostId="7156" Text="Good point! Thanks for helping me to clarify the question. Do you have any thoughts on how to set up the test? Yes, there was an interesting take on that in a recent issue of The Economist, where they introduced 'The Shoe-Thrower's Index'. http://www.economist.com/node/18114401?story_id=18114401" CreationDate="2011-02-13T21:29:38.903" UserId="3182" />
  <row Id="11504" PostId="7166" Text="What exactly are you trying to model/predict here? " CreationDate="2011-02-13T21:52:47.287" UserId="1913" />
  <row Id="11505" PostId="7146" Text="Regarding your question about energy, it's going to be difficult to answer. Do you know anything about your device? Do you know anything of the relationship between wind speed/direction and power? Do you know anything about the layout of the turbines at this particular site since that has a huge impact on the power as you no doubt are aware." CreationDate="2011-02-13T21:58:04.423" UserId="2110" />
  <row Id="11506" PostId="7168" Text="(+1) Would you mind expanding on your opinion on the RMS textbook (of course, it's purely off-topic)?" CreationDate="2011-02-13T22:26:11.500" UserId="930" />
  <row Id="11507" PostId="7168" Text="@chl, I'll see if I can dig out the notes I jotted down on it and will post a couple (if I can find them). As I was reading the book, I recall coming across several remarks and recommendations that struck me as incorrect or very questionable. This colored my opinion of it. But, as I stated above, my overall impression is generally positive." CreationDate="2011-02-13T22:36:28.010" UserId="2970" />
  <row Id="11508" PostId="7141" Text="@cardinal, could you provide a link to some kind of work illustrating inference for constrained least-squares estimation. This seems really interesting." CreationDate="2011-02-13T22:39:03.267" UserId="2116" />
  <row Id="11509" PostId="7141" Text="@mpiktas, sorry, could you clarify? A link to some kind of work illustrating what?" CreationDate="2011-02-13T22:40:54.277" UserId="2970" />
  <row Id="11510" PostId="7141" Text="@cardinal, you mention dearth of good quality work, so I wondered that there is plenty of low quality work :) Any links would be fine, I just want to get an idea." CreationDate="2011-02-13T22:46:20.603" UserId="2116" />
  <row Id="11511" PostId="7141" Text="@mpiktas, I didn't quite mean to suggest that. To me, there just doesn't seem to be much work at all, and certainly very little that is compelling. Knight and Fu (2000) and some of the follow-ons by others are the exception. But, very little (that I am aware of, or can remember being aware of) exists regarding constructing valid confidence intervals and performing hypothesis tests. Chatterjee &amp; Lahiri have a paper on a modified bootstrap for the lasso (standard one is bad). Meinshausen also seems to be doing work in this area. But, the results seem sparse so far." CreationDate="2011-02-13T23:01:26.323" UserId="2970" />
  <row Id="11512" PostId="7141" Text="@cardinal, thanks, I understand that you did not mean to suggest that. It seems interesting that nobody does any work on this, there might some underlying reason for that." CreationDate="2011-02-13T23:10:43.460" UserId="2116" />
  <row Id="11513" PostId="7141" Text="@mpiktas, for one, I think it's generally difficult. And, it's much easier just to dream up some minor iteration on the lasso or $k$-means or PCA, run a few simulations, find one &quot;real&quot; example where your method works better than &quot;competing&quot; ones, and publish without saying (virtually) anything about uncertainty. It's become a veritable cottage industry. You wanted my unvarnished opinion, right? :)" CreationDate="2011-02-13T23:15:41.173" UserId="2970" />
  <row Id="11514" PostId="7141" Text="@cardinal, yep, I treasure those :)" CreationDate="2011-02-13T23:18:24.323" UserId="2116" />
  <row Id="11515" PostId="7174" Text="Thank you. I did sample on the outcome. I edited my question above to be more clear. The database included two different strata: enrolled companies and unenrolled companies. I chose 65 from each stratum. In this case, what would you suggest I use for the model? Thank you! " CreationDate="2011-02-14T01:27:26.560" UserId="834" />
  <row Id="11516" PostId="7176" Text="I assume that *combining errors in quadrature* means calculating the *square root of the sum of the squares*" CreationDate="2011-02-14T02:06:47.983" UserId="2958" />
  <row Id="11517" PostId="7176" Text="re-reading this question the answer feels like &quot;ignore the tiny error in A&quot; so let's go with the more general question: what if the errors in A and/or B are on the order of 10-30%..." CreationDate="2011-02-14T02:07:32.070" UserId="629" />
  <row Id="11518" PostId="7176" Text="@Henry yes. ..." CreationDate="2011-02-14T02:09:58.010" UserId="629" />
  <row Id="11519" PostId="7160" Text="@DWin: Thank you for the appreciably detailed answer with references." CreationDate="2011-02-14T02:18:49.643" UserId="2956" />
  <row Id="11520" PostId="7158" Text="@Jonas:  This is perfectly reasonable, but assumes you already have a decent amount of prior knowledge about what the true distribution is.  I was thinking more in terms of scenarios where you don't and it could be very heavy tailed or bimodal." CreationDate="2011-02-14T03:06:38.110" UserId="1347" />
  <row Id="11521" PostId="7158" Text="@dsimcha: I don't think you can identify outliers in that case (see also my edit). " CreationDate="2011-02-14T04:08:07.380" UserId="198" />
  <row Id="11522" PostId="7082" Text="thanks for the document. i will take a close look at it." CreationDate="2011-02-14T05:12:08.403" UserId="3107" />
  <row Id="11523" PostId="7166" Text="Thank you. I've edited the question. Hope it is now more clear" CreationDate="2011-02-14T05:12:41.430" UserId="3184" />
  <row Id="11524" PostId="7166" Text="Sounds like a Poisson Process problem to me. Let me see what do others think about it." CreationDate="2011-02-14T05:18:25.827" UserId="1307" />
  <row Id="11525" PostId="7181" Text="The response is very much negatively skewed. With the majoirty of projects around 2500 hour mark and a tail stretching out to a couple of 10000-14000 hrs projects. The continous (scale) predictors are a mix of distributions while some predictors are categorical (nominal). What types of distributions are required for LASSO (or what else do you need to know from me)?  -- btw thanks for the response!" CreationDate="2011-02-14T06:21:09.760" UserId="3189" />
  <row Id="11526" PostId="7181" Text="@Shane, the LASSO is a general concept of penalizing with $||$ (modulus or absolute value) function. It is independent of any distribution. If you check the `glmnet` package (see function: `glmnet`), it gives you options to fit the `glm` (linear regression is a special case) with $L_1$ penalty for a variety of distributions. It is pretty fast and amazing at the same time." CreationDate="2011-02-14T06:38:00.487" UserId="1307" />
  <row Id="11527" PostId="7181" Text="Checking SPSS help it talks about a feature called &quot;Categorical Regression Regularization&quot; or CATREG.  It seems to address Lasso and Ridge methods.  For some reason it's not enabled in my version.  If anyone knows why I'd be appreciative." CreationDate="2011-02-14T06:39:27.553" UserId="3189" />
  <row Id="11528" PostId="7181" Text="@Shane If my memory doesn't fail me, I have seen @AndyW post *fancy* SPSS code. It (code) impresses me all the time! " CreationDate="2011-02-14T06:48:33.813" UserId="1307" />
  <row Id="11529" PostId="7184" Text="Thanks for the help. Seeing this many rows at once (which are in this case probesets) can be useful during the hypothesis-generating phase of a microarray analysis. Colleagues sometimes request heatmaps of this size, and this makes it easy to zoom through and see which genes are in a sub-cluster." CreationDate="2011-02-14T07:31:45.120" UserId="1699" />
  <row Id="11530" PostId="7153" Text="@cardinal: Oh, OK! But I still cannot create a new tag..." CreationDate="2011-02-14T07:52:19.440" UserId="3019" />
  <row Id="11531" PostId="7163" Text="Thank you for the proposal! I will have a look at it!" CreationDate="2011-02-14T07:53:39.440" UserId="3019" />
  <row Id="11532" PostId="7172" Score="2" Text="Could you tell us *where* you came by the concept?" CreationDate="2011-02-14T07:53:50.020" UserId="449" />
  <row Id="11533" PostId="7173" Text="Could you clarify how you sample from the four industry strata? Did your sampling scheme specify that you picked a certain number from each stratum? Also do you have data at the employee level or only at the company level?" CreationDate="2011-02-14T07:57:47.893" UserId="449" />
  <row Id="11534" PostId="7190" Text="Try increasing the  `workspace` argument of `fisher.test`. That may avoid the error message, if your computer has enough memory (and you have enough patience). See `?fisher.test`. See my previous answer http://stats.stackexchange.com/questions/4023/chi-square-test-for-equality-of-distributions-how-many-zeroes-does-it-tolerate/4029#4029" CreationDate="2011-02-14T08:27:53.113" UserId="449" />
  <row Id="11535" PostId="7179" Text="In the jargon of web marketing, this probability is called the 'conversion rate'. Not my field, but the Wikipedia articles may give you some leads: http://en.wikipedia.org/wiki/Conversion_rate" CreationDate="2011-02-14T08:39:49.773" UserId="449" />
  <row Id="11536" PostId="7146" Text="well untill 4 m/s there is no wind generation and the max is 15 m/s, wind stronger than 15 m/s will make the same amount of energy     " CreationDate="2011-02-14T08:48:50.363" UserId="3178" />
  <row Id="11537" PostId="7146" Text="i thought of integrating seperetly until 4 m/s, between 4 and 15m/s and over 15m/s, and comparing it to a situation of 100% of the time a turbine working in optimal speed" CreationDate="2011-02-14T08:50:44.123" UserId="3178" />
  <row Id="11538" PostId="7170" Text="Thank you for the response, however, I see no value in it: 1. As far as I see, Quantization is not focused (primarily or solely) on equal-freq-histograms 2. Determining the number of bins per hand or per automatic optimization (via sum-of-squared-errors) is an approach which can be applied anywhere." CreationDate="2011-02-14T09:10:25.587" UserId="264" />
  <row Id="11539" PostId="7086" Text="@cardinal: just added some details, regarding the nature of the data, and analysis, followed with my motivations for the analysis I proposed." CreationDate="2011-02-14T09:23:18.847" UserId="3014" />
  <row Id="11540" PostId="7169" Score="1" Text="(+1) for mutual information. Additional remark: a) I suggest Information Gain as special case of mutual information. b) Automatic feature selection will not only remove the redundant but also all features which have a negative impact on class discrimination." CreationDate="2011-02-14T10:14:15.683" UserId="264" />
  <row Id="11541" PostId="7071" Text="@onestop, @mpiktas, i usually explain the white noise to students through the spectral density concept, at least it gives light to (through the analogy with white color) why the noise is &quot;white&quot;, and why the $AR(1)$ process could be could called &quot;red&quot; and there is no &quot;black noise&quot; :)" CreationDate="2011-02-14T10:34:57.387" UserId="2645" />
  <row Id="11542" PostId="7169" Text="Thanks! This sounds very promising, I will look into it." CreationDate="2011-02-14T10:54:12.057" UserId="977" />
  <row Id="11543" PostId="7170" Text="&quot;No value&quot; was a little bit too harsh, I meant: &quot;no value&quot; for the specific nature of my question which is focused on equal-freq-histograms (and rules of the thumb for it)." CreationDate="2011-02-14T11:19:37.507" UserId="264" />
  <row Id="11544" PostId="7141" Text="I misread the question, so I retract my answer, since I do not see for now how can I rewrite it to be useful." CreationDate="2011-02-14T12:02:52.280" UserId="2116" />
  <row Id="11545" PostId="6746" Text="@Chris Taylor, @cardinal, the problem should have to have known solution, which is probably found in W.Palma Long Memory time series: Theory and Methods. The point is that autocorrelation function may be used to obtain by Yule Walker system of equations the parameters of $AR(\infty)$ representation of the process, the point is when such representation exists (invertability) and what truncation is acceptable by the means of say MSE. For $R$ code in my PhD I used `fracdiff` package." CreationDate="2011-02-14T12:21:05.287" UserId="2645" />
  <row Id="11546" PostId="7094" Text="@whuber Quite right! I stick by the model I set out in the first para, but my equation was screwed up in several ways... Goes to show I haven't actually used log-linear modelling of contingency tables since my MSc, and I haven't got the notes or books to hand. I believe I've fixed it now though. Let me know if you agree! Apols for the delay. Some days my brain just doesn't do algebra." CreationDate="2011-02-14T12:51:09.890" UserId="449" />
  <row Id="11547" PostId="7195" Score="1" Text="Birth-death process attempts to analyze the throughput of the process (here, the parking garage clerk, or similar). I have a feeling that DavidD is looking for a method to predict the amount of cars that will try to check in the garage (not the queue, but the demand). " CreationDate="2011-02-14T12:52:52.837" UserId="1496" />
  <row Id="11548" PostId="7166" Text="@David D. See my comment to Dmitrij Celov's answer and clarify your question." CreationDate="2011-02-14T12:53:31.913" UserId="1496" />
  <row Id="11549" PostId="7181" Text="@Shane, it appears the CATREG command has been around for quite a few versions of SPSS, but you probably need some advanced regression module/licenses to use it. In the current edition you need the &quot;premium&quot; stat suite to get this functionality. I would just suggest checking out the R packages suncoolsu mentions (its free!). " CreationDate="2011-02-14T13:36:43.700" UserId="1036" />
  <row Id="11550" PostId="598" Score="1" Text="As a note GeoDa and SatScan aren't open source, they are freeware (not that it makes much difference to me though!)" CreationDate="2011-02-14T14:06:01.543" UserId="1036" />
  <row Id="11551" PostId="7087" Text="@cardinal I am afraid I can't fit a non-regularized model to my data using lm.. do you suggest using glmnet in this case?" CreationDate="2011-02-14T15:48:33.017" UserId="3139" />
  <row Id="11552" PostId="7173" Text="I first divided each stratum (insured and uninsured) by industry (4 groups). Within these industry groups I further divided the sample into 4 groups that correspond to the size of the company. This was to ensure that I collected similar numbers within each industry and that i didn't up with only small firms. I then numbered all observations and used quota sampling with a random number generator to select firms from each of the 32 groups. I used sampling weights in the analysis to ensure that each observation was weighted by the inverse of the probability of being selected. " CreationDate="2011-02-14T15:58:10.950" UserId="834" />
  <row Id="11553" PostId="7173" Text="No, I don't look at any employee level data within the sample. I realize that my sampling is probably not the best but I was mainly interested in getting some descriptive results. However, I'd also like to be able to use the logistic regression analysis to understand the factors affecting whether or not a company enrolls in health insurance. Thank you! " CreationDate="2011-02-14T15:58:47.360" UserId="834" />
  <row Id="11554" PostId="7087" Score="1" Text="@Marina, can you give us an idea of the size of your model? At the very least the order of magnitude of both the number of observations and the number of coefficients. You might also look at the **biglm** R package. The **lars** package does not use **lm**, though. Be aware that **glmnet** does *not* compute the full regularization path, but instead discretizes it. " CreationDate="2011-02-14T16:05:01.100" UserId="2970" />
  <row Id="11555" PostId="6746" Text="@Dmitrij, @Chris, the OP specifically states he is interested in binary-valued processes (I've got a pretty good guess at what he's likely interested in), for which an AR formulation via Yule-Walker would strike me as ad-hoc at the least. Perhaps you could throw a logistic around it to estimate a conditional probability, but it's still important to recognize the assumptions one is making in that case. Also, for long-memory processes, the choice of truncation can be important and induce nontrivial artifacts. " CreationDate="2011-02-14T16:16:27.097" UserId="2970" />
  <row Id="11556" PostId="7172" Text="Hi onestop, A faculty member mentioned it to me and I didn't find anything by this name online." CreationDate="2011-02-14T16:23:56.153" UserId="253" />
  <row Id="11557" PostId="7182" Text="Thank you for your very comprehensive answer - it's very helpful. " CreationDate="2011-02-14T16:39:12.473" UserId="2635" />
  <row Id="11558" PostId="7188" Text="I assume you mean http://wordnet.princeton.edu/?  I was not aware or this, but it might be useful." CreationDate="2011-02-14T16:40:09.727" UserId="1026" />
  <row Id="11560" PostId="7193" Text="Thanks! Great info, and definitely puts me on the right path. With 2 more rep points, I'd happily give an arrow-up." CreationDate="2011-02-14T17:44:09.860" UserId="3187" />
  <row Id="11561" PostId="7193" Text="Also, I did a search for book recommendations, but I'm not sure what to look for. Books on &quot;binary classification&quot;?" CreationDate="2011-02-14T17:58:04.900" UserId="3187" />
  <row Id="11562" PostId="7210" Text="@dsimcha, that was a fantastic explanation!" CreationDate="2011-02-14T18:18:05.460" UserId="2798" />
  <row Id="11564" PostId="7173" Text="Clarification: I wrote that I used quota sampling but it was in fact stratified sampling. " CreationDate="2011-02-14T19:32:49.493" UserId="834" />
  <row Id="11565" PostId="7194" Text="I like the book too, but the OP didn't really ask about Econometrics." CreationDate="2011-02-14T20:56:33.440" UserId="334" />
  <row Id="11566" PostId="7194" Text="Just going by the Contents, he will find the linear regression model and violations of its assumptions, bayesian approach, Logit, Probit, Tobit models, time series analysis, forecasting and robust estimation. So, even if the title is econometrics, I suppose it covers a large amount of statistical tools that are usefull outside of econometrics. " CreationDate="2011-02-14T21:23:32.950" UserId="1766" />
  <row Id="11567" PostId="7211" Text="For a next time, it would be better to ask two questions." CreationDate="2011-02-14T21:44:22.980" UserId="88" />
  <row Id="11568" PostId="7219" Score="1" Text="Could this be (overly) simplified as something like the difference between generative and discriminative models?" CreationDate="2011-02-14T22:19:53.037" UserId="1764" />
  <row Id="11569" PostId="7219" Text="@Wayne:  Yeah, I think this is actually a pretty fair way of putting it." CreationDate="2011-02-14T22:33:48.593" UserId="1347" />
  <row Id="11570" PostId="7219" Text="&quot;One should solve the [classification] problem directly and never solve a more general problem as an intermediate step...&quot; - Vapnik" CreationDate="2011-02-14T22:42:03.673" UserId="1764" />
  <row Id="11571" PostId="7144" Text="This sounds like the &quot;kernel trick&quot; applied to PCA. http://en.wikipedia.org/wiki/Kernel_PCA&#xA;It's a very good way of handling certain large matrices." CreationDate="2011-02-14T23:11:24.123" UserId="2833" />
  <row Id="11572" PostId="7194" Text="On further reflection, I would not quote the book as the definitve, most thorough source on concepts, as the OP requires." CreationDate="2011-02-14T23:18:20.760" UserId="1766" />
  <row Id="11573" PostId="7219" Text="I disagree that no inference can be done with ML; one can squeeze quite a lot from importance scores some methods produce and/or some feature selection schemes. " CreationDate="2011-02-14T23:20:01.057" UserId="88" />
  <row Id="11574" PostId="7207" Score="2" Text="This paper just must appear in a context: http://www.biostat.wisc.edu/~page/rocpr.pdf" CreationDate="2011-02-14T23:42:18.417" UserId="88" />
  <row Id="11575" PostId="7224" Score="60" Text="+1 for filtering out justin bieber" CreationDate="2011-02-14T22:42:34.127" />
  <row Id="11576" PostId="7224" Score="9" Text="What's your development platform? This can easily be done in .NET because it superior over all other programming environments. Simply call the Page.EradicateBieber() function. Microsoft foresaw this need and graciously provided it for us out-of-the-box in .NET 4.5. (Those of you on older versions will have to wait.) (That is, of course, all tongue-in-cheek.)" CreationDate="2011-02-14T22:46:18.400" />
  <row Id="11577" PostId="7224" Score="22" Text="I think I can safely assert that SO doesn't need a `[justin-bieber]` tag." CreationDate="2011-02-14T22:47:31.833" UserId="2345" />
  <row Id="11578" PostId="7224" Score="2" Text="I can safely assert that people spend more upvotes on comments and *this question* than the close option (the one that deserves the votes)." CreationDate="2011-02-14T22:50:11.093" />
  <row Id="11579" PostId="7224" Score="9" Text="A Justin Bieber audio filter would be good too" CreationDate="2011-02-14T22:58:09.340" />
  <row Id="11580" PostId="7224" Text=":( This question wasn't insincere or a joke." CreationDate="2011-02-14T23:42:14.173" />
  <row Id="11581" PostId="7224" Text="@ʞɔıu: Well, the way it was tagged obviously made it look like one." CreationDate="2011-02-14T23:44:58.853" />
  <row Id="11582" PostId="7224" Text="Heh! I just tried uploading a random Justin Bieber image through celebrity facial recognition at MyHeritage.com - identified him as Catherine Hepburn! http://celebrity.myheritage.com/FP/Company/face-recognition-results.php?temp=4d1qj7d05cab19t7&amp;server=Server76&amp;database=2&amp;startYear=1800&amp;endYear=2005" CreationDate="2011-02-15T00:41:35.730" UserId="3201" />
  <row Id="11583" PostId="7208" Text="For inter-rater reliability with binary outcomes, I think usually one uses the tetrachoric coefficient." CreationDate="2011-02-15T00:48:50.230" UserId="795" />
  <row Id="11584" PostId="7224" Score="4" Text="http://stackoverflow.com/questions/953714/face-recognition-library" CreationDate="2011-02-15T00:54:36.333" UserId="68" />
  <row Id="11585" PostId="7219" Text="@mbq:  I didn't mean to imply that no inference can be done, just that it's not the main goal and that usually p &gt;&gt; n in ML, making it a lot harder." CreationDate="2011-02-15T01:00:27.687" UserId="1347" />
  <row Id="11586" PostId="7224" Text="You could try the tineye API: http://www.tineye.com/commercial_api" CreationDate="2011-02-15T01:06:49.380" UserId="364" />
  <row Id="11588" PostId="7227" Score="2" Text="Yeah, maybe set a threshold of 2-4 possible duplications (to handle the new-baby case) before you reject a photo. Depends on what you're going to do with the photos, I guess. " CreationDate="2011-02-15T01:49:38.233" UserId="3206" />
  <row Id="11589" PostId="7199" Text="Thanks @Wolfgang; your explanation helps a lot. A follow up question that I have then is this. I am indeed analyzing a repeated-measures model with a single treatment factor. Each subject is randomly assigned to either treatment A or B. Then they are measured at 0 mins, 15 mins, 30 mins, 60 mins, 120 mins and 180 mins. From my understanding, time should be a random factor because it is just samples from time 0 to 180 mins. So, should I do: lme(UOP.kg~time*treat, random=~time|id, raw3.42)?" CreationDate="2011-02-15T01:59:12.990" UserId="1663" />
  <row Id="11590" PostId="7181" Text="Ok will do.  What other techniques are 'good'?  Are there any good references I can use for defence of techniques on use of regression on small samples?" CreationDate="2011-02-15T02:52:08.407" UserId="3189" />
  <row Id="11591" PostId="7054" Text="@Andrej: Thank you. The reference is useful." CreationDate="2011-02-15T03:02:31.330" UserId="2956" />
  <row Id="11592" PostId="7225" Text="Of course one way I can quickly get an estimate of R-squared is by fitting a linear model predicting the fitted values from the original data and taking the R-squared from that. But this seems like it would be a massively-overfit and biased estimate of R-squared." CreationDate="2011-02-15T03:13:15.947" UserId="36" />
  <row Id="11593" PostId="7181" Text="@Shane, you can use `Ridge Regression` in your case when $n$ is close to $p$, but as you mention, your aim is to select predictors, $LASSO$ is the recommended way to go!. You can choose not to use _penalized regression_ ie simple linear regression, but that would be dangerous, as your small sample size will result in elevated standard error for your parameters." CreationDate="2011-02-15T03:13:46.790" UserId="1307" />
  <row Id="11594" PostId="7181" Text="Just a note on CATREG - it seems like my version *should* have it as part of the PASW Regression module.  It's PASW Version 18 - student edition.  Do people know if it's an installer thing?" CreationDate="2011-02-15T03:27:39.813" UserId="3189" />
  <row Id="11595" PostId="7227" Score="2" Text="Simple, elegant solution. +1." CreationDate="2011-02-15T03:40:48.733" UserId="2227" />
  <row Id="11596" PostId="7141" Text="@shabbychef, off the top of my head, I don't think the power will necessarily be small. Intuitively, it seems it should be roughly a monotonic function of $|\mu_n|$. I say this because under both the null and alternative, the &quot;left-hand tail&quot; of the sequence is allowed to float, while under the null, you've pinned the &quot;right-hand tail&quot; to zero." CreationDate="2011-02-15T03:43:56.547" UserId="2970" />
  <row Id="11597" PostId="7235" Text="Note: For the bootstrap to be an effective solution, you'd have to bootstrap the entire procedure, starting before any screening occurs, screen the bootstrapped sample, then calculate coefficients. But now you have different sets in predictors in each regression and it's no longer clear how to calculate the distribution for any one of them. Bootstrapping confidence intervals for predicted values of the outcome may be effective, however." CreationDate="2011-02-15T03:44:09.170" UserId="401" />
  <row Id="11598" PostId="7227" Score="5" Text="People could use different pictures of the same person." CreationDate="2011-02-15T03:47:37.447" UserId="1142" />
  <row Id="11599" PostId="7147" Text="@mpiktas: Thank you." CreationDate="2011-02-15T04:19:38.330" UserId="2956" />
  <row Id="11600" PostId="7233" Score="1" Text="@DrWho - it would help us if you can tell us which software you used and some sample commands. At the moment, I think, we can only guess because outputs are _sometimes_ software dependent." CreationDate="2011-02-15T05:14:40.353" UserId="1307" />
  <row Id="11601" PostId="7223" Text="@B_Miner. People sometimes do this - unknowingly (ie misuse Statistics, because they get the desired result) and knowingly (they did bootstrap and it did not affect the result substantially). Your point is valid, and Professor Harrell points this out in the Preface of his book that bootstrap is beneficial." CreationDate="2011-02-15T05:19:44.670" UserId="1307" />
  <row Id="11602" PostId="7193" Text="The task classification (with special case binary classification) is part of every basic book on machine learning and data mining (my favorite: &quot;Data Mining: Practical Machine Learning Tools and Techniques&quot; by Witten &amp; Frank)" CreationDate="2011-02-15T07:10:01.843" UserId="264" />
  <row Id="11603" PostId="7227" Text="(+1) at Rebecca and (-1) @ PPPPPP: This just shifts the problem." CreationDate="2011-02-15T07:15:20.810" UserId="264" />
  <row Id="11604" PostId="7232" Text="Unfortunately, the feature generation step is both the hardest and most important one :(." CreationDate="2011-02-15T07:19:01.197" UserId="264" />
  <row Id="11605" PostId="7147" Text="@DrWho, you are welcome." CreationDate="2011-02-15T08:26:34.373" UserId="2116" />
  <row Id="11606" PostId="6746" Score="1" Text="@cardinal, @Chris. oh, I as usually missed the part of the task ^__^ In the case of binary-valued process it seems to be a very well known (studied) problem of traffic measurement that comes from communication networks or so called ON/OFF process that exhibits long range dependence (long memory) property. As for the particular example, I'm a bit confused, since in &quot;one way to predict&quot; Chris actually takes the previous value, not using the ACF only (or I'm even more confused by the term &quot;classification rate&quot;)." CreationDate="2011-02-15T08:44:28.367" UserId="2645" />
  <row Id="11607" PostId="6835" Text="@ducnguyen, In addition to Rob's comment, try to add a bit more context and what do you put into the term &quot;best&quot;. Best for what and for whom? Since trend is a part that is not observable, and obtained by the decomposition, using either low/high pass filters or model based approaches." CreationDate="2011-02-15T09:01:27.993" UserId="2645" />
  <row Id="11608" PostId="7195" Text="@bgbg, the problem with parking is that cars in the garage are not staying for the whole time, when you drive into the city, you see how many free places are in a particular garage, so you decide either to occupy it or search for free of charge place somewhere else (I assume that it is a kind of the parking lot, but just underground, here I agree that more details would be useful).Since cars in the garage are not staying fro the whole day(s) so you DO need a Birth-death process to describe if a particular place is free or occupied,this time of a day and of a week. Waiting for David's comments." CreationDate="2011-02-15T09:11:19.673" UserId="2645" />
  <row Id="11609" PostId="7240" Score="1" Text="Good question, but I don't have (a reference for) a good answer. There's more than one level of variation in mixed models, so there's more than one component of variance to explain, plus it's debateable whether random effects can really be said to 'explain' variance. I think the whole concept of 'proportion of variance explained' is less useful in mixed models." CreationDate="2011-02-15T09:18:48.010" UserId="449" />
  <row Id="11610" PostId="7222" Text="I will use logistic-regression-related methods like Generalized additive models and Kernel Logistic Regression, so these are due to the large sample size nearly not affected by unequal classes." CreationDate="2011-02-15T09:39:33.807" UserId="2549" />
  <row Id="11611" PostId="7199" Text="Yes, but I would think of it this way: You are essentially allowing the intercept and slope of the regression line (of UOP.kg on time) to differ (randomly) between subjects within the same treatment group. This is what random=~time|id will do. What the model will then tell you is the estimated amount of variability in the intercepts and the slopes. Moreover, the time:treat interaction term indicates whether the *average* slope is different for A and B." CreationDate="2011-02-15T10:13:11.910" UserId="1934" />
  <row Id="11612" PostId="7208" Text="Could you elaborate on that? I am definitely no expert when it comes to statistics and I can't seem to find a straight forward approach to calculating a tetrachoric coefficient." CreationDate="2011-02-15T10:23:43.457" UserId="1205" />
  <row Id="11613" PostId="7136" Text="If the primary goal is to make some kind of inference to the population and the sampling mechanism is of such a nature that the representativeness of the sample is questionable, then indeed, any inference will also be rather questionable. Essentially, you can only make an inference to that part of the population that the sampling mechanism provides a representation of. In principle, the inferences you make will be appropriate for that part of the population. Whether that part of the population is of any interest to you (or the readers) is another issue." CreationDate="2011-02-15T10:24:45.160" UserId="1934" />
  <row Id="11614" PostId="7218" Score="2" Text="onestop provided an answer that will hopefully give you enough to go on. If you *really* want to know about the distribution of the sample correlation coefficient itself, then the definite reference is: Hotelling, H. (1953). New light on the correlation coefficient and its transforms. Journal of the Royal Statistical Society, Series B, 15, 193-232. Note that this isn't light reading." CreationDate="2011-02-15T10:30:54.950" UserId="1934" />
  <row Id="11615" PostId="7172" Text="After asking the faculty member with whom I spoke about it, he said it's supposed to be located in the book &quot;Density estimation for statistics and data analysis&quot; by Silverman. (and maybe it's rabbits ear chasing, not hunting)" CreationDate="2011-02-15T10:37:15.847" UserId="253" />
  <row Id="11616" PostId="7232" Text="@steffen There is some suggestion that the OP is messing with the faces, so have some descriptor generator." CreationDate="2011-02-15T12:15:24.160" UserId="88" />
  <row Id="11617" PostId="7151" Text="Hi Jeromy,&#xA;I am convinced that likely the MDS is not the right analysis, since my table&#xA;does not contain the distances. Indeed if you have a look to the diagonal of the&#xA;table you can see that the values are not 0 (which should express zero distance&#xA;between the same stimuli).&#xA;The values I have in my table are not distances but average scores from a 9 points&#xA;Likert scale, assessing the degree of coherence between two modalities (audio and&#xA;haptic) simulating different pairs of materials.&#xA;&#xA;So my question now is: which analysis should I perform in your opinion?&#xA;&#xA;&#xA;Thanks in advance" CreationDate="2011-02-15T12:40:40.313" UserId="3180" />
  <row Id="11619" PostId="7232" Text="@mpq: I did not doubt that, however, if the OP does not have one feature per pixel, then he has to find a meaningful aggregation level. I did not downvote, I just wanted to point to the complexity which lies behind this answer (which is, of course, correct)." CreationDate="2011-02-15T13:00:05.663" UserId="264" />
  <row Id="11620" PostId="7225" Text="I add this as a comment since I am asking a &quot;similar&quot; question in a nearby post (so I dont know if I qualify as giving an *answer*), but for your question specifically it seems like you can calculate R-squared without requiring any distributional assumptions (they are needed for hypothesis tests in the ordinary way though). Can't you use a hold out set to calculate r-squared or use a k-fold validation if you don't have enough data (at each fold run your full penalized process and average the r-squares from each of the folds not used in the fitting)?" CreationDate="2011-02-15T13:00:13.933" UserId="2040" />
  <row Id="11621" PostId="7232" Text="Right, the feature generation step is the hard part.  I was assuming OP could do this since he has some mechanism for processing the images already.  Even if he does though they might only be useful features for detecting face/not face instead of Bieber/not Bieber...it really depends on the features." CreationDate="2011-02-15T13:17:17.507" UserId="2485" />
  <row Id="11622" PostId="5369" Text="I suspect SPSS does exactly this (which is why I think it's a common procedure)." CreationDate="2011-02-15T13:24:08.073" UserId="144" />
  <row Id="11623" PostId="5423" Score="2" Text="Not true. Factor Analysis differs from PCA in the variance-covariance matrix used. That only the rotation would make the difference, is a common misconception. " CreationDate="2011-02-15T13:29:58.540" UserId="1124" />
  <row Id="11624" PostId="7225" Text="@B_Miner, $k$-fold cross validation tends to give quite biased estimates of $R^2$, as it generally isn't estimating the true quantity of interest. Many (most?) similar procedures have the same problem." CreationDate="2011-02-15T14:20:48.723" UserId="2970" />
  <row Id="11625" PostId="7225" Text="@Stephen, is $R^2$ *really* the quantity you are interested in? Because of the bias induced by penalization, looking at only the variance explained is probably not desirable unless you already have a very good estimate of the bias. The whole idea of using $R^2$ as a basis for inference is predicated on the unbiasedness of the estimates. Even major textbooks on regression seem to &quot;forget&quot; this. (See, for example, Seber and Lee's somewhat faulty treatment of $R^2$ in the multiple regression case.)" CreationDate="2011-02-15T14:26:48.323" UserId="2970" />
  <row Id="11626" PostId="7247" Text="As in Aniko's answer, you need to use `129` for the first parameter to `ppois`, since $F(X\ge 130)=F(X&gt;129)$." CreationDate="2011-02-15T15:15:57.310" UserId="2456" />
  <row Id="11627" PostId="7243" Text="@Bernd: Thank you. " CreationDate="2011-02-15T15:27:41.167" UserId="2956" />
  <row Id="11628" PostId="7247" Text="Good catch. Corrected." CreationDate="2011-02-15T15:35:44.840" UserId="1934" />
  <row Id="11629" PostId="7233" Text="@suncoolsu: Thank you for your help. I have used Comprehensive meta analysis v2. Run analysis was the command used" CreationDate="2011-02-15T15:39:22.227" UserId="2956" />
  <row Id="11630" PostId="7029" Score="1" Text="It's been published: http://dx.doi.org/10.1093/biomet/asq017." CreationDate="2011-02-15T15:39:44.747" UserId="1979" />
  <row Id="11631" PostId="7029" Text="@John D. Cook: Which special case are you approximating? I've read it, but can't really relate your expression to the expressions given in the paper...? " CreationDate="2011-02-15T15:44:16.367" UserId="1979" />
  <row Id="11632" PostId="7243" Text="@Bernd: Thank you for your help. I have included the other results herewith. Q-value-28.056,	df (Q)	4.000 , 	P-value	0.000	, I-squared	85.743&#xA;" CreationDate="2011-02-15T15:47:30.877" UserId="2956" />
  <row Id="11633" PostId="7250" Text="apologies if I've asked too many questions about clustering over the last few days. I'm trying to become more familiar with this field quickly (also I did post this question on SO, http://stackoverflow.com/questions/4997870/using-the-cluster-package-in-r-for-kmeans-clustering but it was suggested to move it here)" CreationDate="2011-02-15T16:03:18.097" UserId="2635" />
  <row Id="11634" PostId="7245" Score="3" Text="+1  Your discussion in point 1 leads to the crux of the question: how is one to test a *post hoc* hypothesis that was developed (essentially on the spur of the moment, it seems) from an examination of the data?  It's unclear what the hypothesis is even supposed to be.  This is the old data-snooping problem looming large.  A good case can be made that the proper answer is that trying to apply the mechanics of hypothesis testing to such one-off observations just makes nonsense of statistical significance." CreationDate="2011-02-15T16:14:42.503" UserId="919" />
  <row Id="11635" PostId="7241" Text="@onestop: Thanks for adding the DOIs/links." CreationDate="2011-02-15T16:22:48.437" UserId="1934" />
  <row Id="11636" PostId="7250" Text="@celenius That's ok, this is not TCS.SE (-;" CreationDate="2011-02-15T16:47:36.483" UserId="88" />
  <row Id="11638" PostId="7254" Text="	&#xA;ok managed to find the solid part but still in search of a way to fit (Gaussian/pdf) curve as in here stats.stackexchange.com/questions/2573/… (this one was done in R)" CreationDate="2011-02-15T17:41:17.043" UserId="3172" />
  <row Id="11639" PostId="7248" Text="I'm not sure if it is a good idea for this site to have questions where you have to read the question in a blog. Could you &quot;contextualize&quot; the link? " CreationDate="2011-02-15T17:45:34.127" UserId="223" />
  <row Id="11640" PostId="6702" Text="@chl that's what I need to do, yes, but I was hoping I wouldn't have to re-invent the wheel." CreationDate="2011-02-15T17:50:21.400" UserId="2817" />
  <row Id="11641" PostId="7255" Text="it is good that you mention it! I can't figure out the intuition, do you? " CreationDate="2011-02-15T17:51:21.010" UserId="223" />
  <row Id="11642" PostId="6999" Text="Thanks for the answer. I had thought that the simple methods had inherent weaknesses, but was also concerned that stochastic methods may suffer from lack of consistency between analyses. Need to look into it further." CreationDate="2011-02-15T17:54:12.467" UserId="229" />
  <row Id="11643" PostId="7255" Text="@robin This is what makes this conjecture so special: a completely elementary statement, some obvious approaches which fail miserably (characteristic functions), and one is left with nothing to grasp... By the way, should one bet on the conjecture being true or false? Even that is not obvious (to me)." CreationDate="2011-02-15T18:13:33.830" UserId="2592" />
  <row Id="11644" PostId="7248" Text="@smms Please update question as robin suggests, otherwise it will be a strong candidate for closure ;)  Thanks." CreationDate="2011-02-15T18:19:57.757" UserId="8" />
  <row Id="11645" PostId="7243" Text="@DrWho I would only report the results based on the random effects model (I mean, I would always report the REM results, regardless of the actual test statistics...). " CreationDate="2011-02-15T18:56:50.270" UserId="307" />
  <row Id="11647" PostId="7259" Text="@Sam - what you're asking for shouldn't be that difficult, but it's difficult to wrap my head around what you're after without some sample data. Can you update your question with some sample data? Maybe try `dput()` on the first 100 or so rows and paste it into your question?" CreationDate="2011-02-15T19:26:44.970" UserId="696" />
  <row Id="11648" PostId="7239" Text="@teucer could you at a minimum provide the input data that you are using?" CreationDate="2011-02-15T20:24:53.497" UserId="1381" />
  <row Id="11649" PostId="7268" Text="I'm guessing you get those errors from `xts()` because the `dates` column is a factor." CreationDate="2011-02-15T21:26:13.440" UserId="1657" />
  <row Id="11650" PostId="7268" Text="I'm really new to R ... I created the dates column from the strptime function. The original data is from read.csv." CreationDate="2011-02-15T21:30:42.920" UserId="2770" />
  <row Id="11651" PostId="7262" Text="Thank you for your reply! I am glad to hear your answer, as that was my original assumption. I was told otherwise by a professor at my institution, so I really had to go digging. I am using SPSS and I now see that the parameter estimates are indeed identical (e.g., B values). I see now that what had confused me originally were the different p values. The ANCOVA statistic is based on a standard F test, while the GZLM is based on the Wald Chi-Square, correct? I have read that the Wald Chi-Square is used when you use parameters of the sample (such as in the GZLM). " CreationDate="2011-02-15T21:31:22.117" UserId="3262" />
  <row Id="11652" PostId="7262" Text="Based on this answer, I added a question to the original post! " CreationDate="2011-02-15T21:37:44.140" UserId="3262" />
  <row Id="11653" PostId="7262" Text="Ok, i've added a corresponding para to my answer in response." CreationDate="2011-02-15T21:58:08.303" UserId="449" />
  <row Id="11654" PostId="7269" Score="3" Text="+1. On Google: &quot;the univariate distribution of&quot; has 25,600 hits.  &quot;the joint distribution of&quot;: 1,080,000. &quot;the multivariate distribution of&quot;: 85,100.  &quot;the bivariate distribution of&quot;: 89,800.  This sounds like the &quot;joint&quot; version is popular with &quot;univariate,&quot; &quot;bivariate,&quot; and &quot;multivariate&quot; occasionally used, each with similar frequencies.  These likely are used in circumstances requiring clarification.  (I have often seen &quot;the univariate distribution of&quot; used in this sense.)" CreationDate="2011-02-15T23:30:04.340" UserId="919" />
  <row Id="11655" PostId="7225" Text="@cardinal Can you please explain why k-fold will not give reliable results? Cross validation and bootstrapping are mainstays in statistical learning - no?? Or are you stating that for the case of biased estimates resulting from penalized fitting?" CreationDate="2011-02-16T01:46:29.510" UserId="2040" />
  <row Id="11656" PostId="7235" Text="@charlie: [Do I read you correctly that you you are only speaking to I.(model selection) not II. (penalized)] Are you saying that for prediction intervals, it is valid to use model selection and then bootstrap the predictions from that model, but for anything else you need to bootstrap the entire process? " CreationDate="2011-02-16T01:53:28.600" UserId="2040" />
  <row Id="11657" PostId="7235" Text="@charlie Regarding the suggested solution of screening on a sample. Would that be along the lines of partitioning the data, (ab)using one set (model selection etc) and then applying that model to the remaining data - and on that data with the model that was fit using traditional theory for hypothesis tests, CIs etc?" CreationDate="2011-02-16T02:03:32.337" UserId="2040" />
  <row Id="11658" PostId="7274" Score="1" Text="I dimly remember the world map of the facebook network, which was done in R. I think the author described his process in some detail in his blog. I suppose using that approach would generate a map that is informative even with 4 million nodes. " CreationDate="2011-02-16T02:53:38.077" UserId="1766" />
  <row Id="11659" PostId="7270" Text="@andresmh, for some of these measures it whether R can handle it or note will depend on how many separate people (nodes) the network has. R may not necessarily be the best tool for the computational aspects. There's a guy with the last name of Leskovec who used to be at Carnegie Mellon---I think as a student---that did lots of stuff with descriptive statistics on large graphs. There are lots of utilities out there to &quot;visualize&quot; graphs, but mostly I've found they're pretty hard to interpret or make much sense out of. Graphing just the degree distributions might be a first start." CreationDate="2011-02-16T03:03:54.343" UserId="2970" />
  <row Id="11660" PostId="7277" Score="2" Text="Typically one uses a fixed scale only with models like logistic regression or Poisson regression, where the response is a count or indicator/frequency variable. In this case there is no analogue to the scale parameter in normal regression." CreationDate="2011-02-16T04:07:18.290" UserId="1569" />
  <row Id="11661" PostId="7278" Text="What's wrong with the formula given by http://en.wikipedia.org/wiki/Chi_squared ? _i.e._ $(1-2t)^{-m/2}$" CreationDate="2011-02-16T04:15:03.350" UserId="795" />
  <row Id="11662" PostId="7278" Text="There's nothing wrong with it, but I'd like to derive it." CreationDate="2011-02-16T04:30:16.463" UserId="2387" />
  <row Id="11664" PostId="7278" Text="ah, gotcha. the question does not say 'derive', however, but 'find'. " CreationDate="2011-02-16T05:37:33.230" UserId="795" />
  <row Id="11665" PostId="7238" Text="`xcorr2` is applied to two matrices (if only one given, it is used in both contexts). How are you using it? That is, how do the x,y,z direction readings become matrices?" CreationDate="2011-02-16T05:46:32.767" UserId="795" />
  <row Id="11666" PostId="7243" Text="@Bernd: Thank you for the appreciably fast and early guidance." CreationDate="2011-02-16T06:17:11.750" UserId="2956" />
  <row Id="11667" PostId="1903" Text="Hmm - my question is not very well defined. The only thing I can do is pick *some* model for q() that permits setting parameters, and maximise the goodness of fit by fiddling with those parameters. That is - no matter what I do I will have to make some assumptions about what q() basically looks like." CreationDate="2011-02-16T06:35:44.297" UserId="997" />
  <row Id="11668" PostId="7256" Text="Applying PCA to reduce dimension in a binary classification problem is nonsence. The directions of highest variablility are not necessarily the direction of highest separation, could be the opposite or anything. Do you need examples to be convinced?" CreationDate="2011-02-16T07:14:39.813" UserId="223" />
  <row Id="11669" PostId="7264" Text="@David pvr and pri are user supplied data: they are the predicted regressors..." CreationDate="2011-02-16T08:10:29.770" UserId="1443" />
  <row Id="11670" PostId="7221" Text="sorry to un-accept.  When I tried to use this answer, I found it did not work for the situation I'm interested in (high correlation coefficients)." CreationDate="2011-02-16T09:06:00.157" UserId="2665" />
  <row Id="11671" PostId="7256" Text="@robin I'm not convinced it's nonsense; it depends on your aims. It's not going to identify the linear combinations of covariates most strongly associated with the binary outcome, but that may not be the aim. E.g. PCA is popular in analysing diet-disease studies as a step in reducing the huge amount of info collected from dietary diaries. One advantage is that as the principal components are chosen *before* their association with the binary outcome is examined, the analysis is free of charges of 'data snooping'." CreationDate="2011-02-16T09:06:59.263" UserId="449" />
  <row Id="11673" PostId="7256" Text="@onestop I said it is nonsence in a binary classification problem. I assume that a binary classification problem is a problem where you need to predict a class that can be zero or one (binary) according to explanatory variables. If the aim is not to identify the relation (not necessarily linear) between covariates and binary outcome, I am wrong about my definition of binary classification. It is not because an algorithm is &quot;popular&quot; that it is a good idea to use it without thinking if it suites to your problem.  " CreationDate="2011-02-16T09:19:35.697" UserId="223" />
  <row Id="11674" PostId="7285" Text="A very interesting problem. Do you have a sample of &quot;typical population of email addresses&quot; at hand ? Additionally it is not guaranteed, that the email-addresses of the visitors do have the same another/different structure, but since you are only looking for an approximation.... Second question: Are you able to set the seed of the RNG ?" CreationDate="2011-02-16T09:26:32.693" UserId="264" />
  <row Id="11675" PostId="7218" Text="@Mark I don't think your graphs are right. I've just drawn some graphs of the distribution derived from the Fisher formula which show it is correctly centered. In fact it's pretty obvious from the formula that it must be asympototically unbiased for $N \rightarrow \infty$. Could you post the mathematical core of your code?" CreationDate="2011-02-16T09:51:14.483" UserId="449" />
  <row Id="11676" PostId="7218" Text="@onestop Sure.  Added Mathematica code." CreationDate="2011-02-16T10:04:23.663" UserId="2665" />
  <row Id="11677" PostId="7256" Text="@robin Fair enough, I assume you wouldn't class the type of diet-disease study i'm thinking of as a binary classification problem then, and you're probably correct." CreationDate="2011-02-16T10:04:32.960" UserId="449" />
  <row Id="11678" PostId="7218" Text="@Mark That's not how pdfs transform -- it's a bit more complicated. See http://en.wikipedia.org/wiki/Probability_density_function#Dependent_variables_and_change_of_variables" CreationDate="2011-02-16T10:11:20.487" UserId="449" />
  <row Id="11679" PostId="7285" Score="4" Text="Sounds like you want a 'hash function': http://en.wikipedia.org/wiki/Hash_function &#xA;This is in the realms of computer science rather than statistics though, so I'm not sure it belongs on CrossValidated." CreationDate="2011-02-16T10:20:45.863" UserId="449" />
  <row Id="11680" PostId="7221" Text="@Mark, I did some simulations with R, everything holds pretty good for correlation 0.75" CreationDate="2011-02-16T10:27:15.340" UserId="2116" />
  <row Id="11681" PostId="7221" Text="@mpiktas Yes, you're right thank you.  I made a mistake in my notebook." CreationDate="2011-02-16T10:28:53.253" UserId="2665" />
  <row Id="11682" PostId="7218" Text="@onestop Of course.  Thank you.  I realized there was a problem after I posted the code, but it would have taken me a while to figure out how to fix it." CreationDate="2011-02-16T10:29:34.203" UserId="2665" />
  <row Id="11683" PostId="7285" Score="1" Text="hmpf ;) ... I intended to write the same. @Jeromy: Especially this part of the site (http://en.wikipedia.org/wiki/Hash_function#Hashing_data_with_other_distributions) could be interesting for you." CreationDate="2011-02-16T10:51:07.410" UserId="264" />
  <row Id="11684" PostId="7286" Text="How come you don't have +100 bonus from your SO record?" CreationDate="2011-02-16T10:57:45.363" UserId="88" />
  <row Id="11685" PostId="7286" Text="@mbq, probably because I created this account long ago when I was 0 on OS as well...pity, 100 on CW looks like a big deal, since you must be a real chap to answer stuff in here :)" CreationDate="2011-02-16T11:11:17.223" UserId="1542" />
  <row Id="11686" PostId="7268" Score="1" Text="Let's see `str()` of the data.frame." CreationDate="2011-02-16T11:30:26.227" UserId="144" />
  <row Id="11687" PostId="7284" Text="Thanks for the answer it confirmed my supposition that this method shouldn't be used." CreationDate="2011-02-16T12:10:42.757" UserId="1643" />
  <row Id="11688" PostId="7158" Score="1" Text="@dsimcha - you **always** have prior knowledge!  for how were the data given to you?  you *always* *always* know that much.  data doesn't magically just show up.  and you can always make tentative assumptions.  &quot;outliers&quot; based on these assumptions basically give you a clue that something in your assumptions is wrong.  by studying the &quot;outlier&quot; (which is always relative) you can improve your model." CreationDate="2011-02-16T12:50:41.443" UserId="2392" />
  <row Id="11689" PostId="6071" Score="2" Text="from this model, $g_i$ and $\delta_i$ are not identifiable.  For $g_i + \delta_i=(g_i+c) + (\delta_i-c)$ and the adjustment leaves the variances unchanged.  This model would be identifiable if $g_i=g$ (i.e. same for all i) or if it was the same across clusters (as in multilevel models)." CreationDate="2011-02-16T13:11:46.017" UserId="2392" />
  <row Id="11690" PostId="2173" Text="(+1)@radek. I am also a rapidminer fan, but in my opinion it is not flexible enough for sophisticated visualizations." CreationDate="2011-02-16T13:39:45.040" UserId="264" />
  <row Id="11691" PostId="7289" Text="@Peter. I have seen this paper - very interesting. Two questions. 1) Lets take logistic regression. It sounds like the only way to conduct CI or hypothesis tests is to build a model in the style of hosmer and lemeshow (precluding any data sets with big p)? So you are left with &quot;using&quot; the model for only point estimates? 2) Your paper discusses the lasso among other alternatives. Are you of the mind that this allows later hypothesis testing or is &quot;simply&quot; given as a better option of model selection?" CreationDate="2011-02-16T13:42:54.473" UserId="2040" />
  <row Id="11692" PostId="7029" Text="@fabians: The case I had in mind was sigma^2 = tau^2 = 1 in Theorem 1. It says the horseshoe density is bounded above and below by multiples of log(1 + c/x^2). So maybe the distribution I mentioned above is more of a simplification of the horseshoe density than an approximation." CreationDate="2011-02-16T13:54:50.223" UserId="319" />
  <row Id="11693" PostId="7285" Text="@onestop Thanks for the tip about hashtags. With regards to whether the question is on topic for the site, I think random allocation of participants to groups is inherently related to study design, which in turn is related to inferences from data." CreationDate="2011-02-16T14:20:18.543" UserId="183" />
  <row Id="11694" PostId="7268" Text="@Roman Thanks for the str() function, I wasn't aware of that. So, getting rid of the Factor column, I can generate an xts object like this, x&lt;-xts(d[,3:5],order.by=d[,1]).  I was then able to apply to.hourly, which shortens the data from 19720 objects down to 480. I'm not sure if this will get me where I want, but I'm closer now, I think." CreationDate="2011-02-16T14:22:35.663" UserId="2770" />
  <row Id="11695" PostId="7285" Text="@Jeremy A hash function is not the same thing at all as a hashtag! I see your point about study design though. I admit to not reading the whole of your question properly." CreationDate="2011-02-16T14:27:06.200" UserId="449" />
  <row Id="11696" PostId="7278" Text="sure, but contextually it's obvious I'm not looking for a link" CreationDate="2011-02-16T14:37:16.053" UserId="2387" />
  <row Id="11697" PostId="7225" Text="@B_Miner, cross validation tends to be biased for the generalization error of the model. However, if the bias remains relatively constant with respect to a parameter over which an optimization is being done for model selection purposes, than the &quot;right&quot; decision will still be made in the end. My experience is that the generalization-error bias tends to be quite large in many situations, but it also tends to be quite constant, e.g., as a function of the regularization parameter in ridge regression. So, for model selection it still works quite well. " CreationDate="2011-02-16T14:50:50.813" UserId="2970" />
  <row Id="11698" PostId="7287" Text="A minor note about the above method is that there are a bunch of valid characters in email addresses - punctuation in particular - that you would want to consider if you were doing this." CreationDate="2011-02-16T14:51:43.760" UserId="527" />
  <row Id="11699" PostId="7287" Text="@dsol: I agree. You could easily be caught out with a &quot;+&quot; in an email address. For a quick and dirty solution, I would probably just skip any punctuation characters that I hadn't specified in my look-up table." CreationDate="2011-02-16T14:59:54.803" UserId="8" />
  <row Id="11700" PostId="7270" Text="Even plotting 4 million points might take a while..." CreationDate="2011-02-16T15:00:57.640" UserId="1351" />
  <row Id="11701" PostId="7290" Score="1" Text="@probabilityislogic, all you've appeared to do above is describe leave-one-out cross validation and $k$-fold cross validation. The former is rarely used these days due to high variance and usually large computational costs (some regression settings being the exception). As for your remarks on influence, if $p &gt; n$ there are no unique least-squares estimates, which is a complication. Also, the signs of the parameter estimates can be different as well. I'm not positive, but even when the OLS estimates exist, there may still be situations where your ratio could be $&gt; 1$ for some parameters. " CreationDate="2011-02-16T15:02:18.663" UserId="2970" />
  <row Id="11702" PostId="7040" Text="@onestop: I am sorry i missed your question. The comparison is on log odds ratio. Thank you for your interest." CreationDate="2011-02-16T15:02:35.950" UserId="2956" />
  <row Id="11703" PostId="7293" Text="@gappy, can I assume you know the eigenvalues of $B$? If so, no approximation is needed. " CreationDate="2011-02-16T15:05:49.300" UserId="2970" />
  <row Id="11704" PostId="7282" Text="Thanks for mentioning SNAP. I am looking into it. Have you used it? The centrality sample that comes with it seems close to what I want. I tried modifying it so it works with my multi directed graph data but it failed to compile. I am not sure if it is appropriate to ask a question about it here, so I might create a new Q." CreationDate="2011-02-16T15:13:15.643" UserId="1762" />
  <row Id="11705" PostId="7282" Score="1" Text="@andresmh, you might try reducing your graph to have a single observation per directed pair first. For the eigenvalue stuff, your data is likely similar or equivalent to a weighted random walk on the graph. I'm not sure if SNAP supports that, but it's likely to. If all else fails, you might send a very specific email to Jure. He's a very nice guy, so I wouldn't be surprised if he provided some quick guidance. " CreationDate="2011-02-16T15:32:19.637" UserId="2970" />
  <row Id="11706" PostId="7296" Text="Of course! Thanks." CreationDate="2011-02-16T15:35:58.927" UserId="30" />
  <row Id="11707" PostId="7270" Text="@wok, nah. Piece of cake on today's computers. Anyway, you could always dump to a PNG first and that's likely to be good enough for the degree distribution. The OP's graph really isn't all that big." CreationDate="2011-02-16T15:36:05.080" UserId="2970" />
  <row Id="11708" PostId="7286" Text="You can get the bonus by clearing all associations and then associating accounts again." CreationDate="2011-02-16T15:37:22.867" UserId="1351" />
  <row Id="11709" PostId="7296" Score="1" Text="@gappy, sure. Regards." CreationDate="2011-02-16T15:37:47.307" UserId="2970" />
  <row Id="11710" PostId="7235" Text="I was thinking only of model selection, but that's largely because I don't know all that much about penalized regression. I would say that you need to bootstrap the entire process in order to get inference on predictions from the model. The whole issue is that, in any one sample, you are likely to find spurious correlations that get magnified when you include some variables and leave others out. The only way to get around this is to look at multiple samples---i.e., bootstrap. Of course, no one actually does this." CreationDate="2011-02-16T15:55:23.943" UserId="401" />
  <row Id="11711" PostId="7235" Text="Right, you use one partition of your sample to come up with your model using model selection procedures, then do your inference on either the other partition or the full sample." CreationDate="2011-02-16T15:59:23.873" UserId="401" />
  <row Id="11712" PostId="7264" Text="@teucer are they predicted from user-supplied values of `ir` and `vr`? if so, how? and if so, why not calculate `pri` and `pvr` yourself inside the JAGS model?" CreationDate="2011-02-16T16:06:29.410" UserId="1381" />
  <row Id="11713" PostId="7294" Text="thanks robertsy, for good reference. indeed the process is not markov. Even if acceptance probability is independent of the past, the transition kernel of the process is a function of the proposal density and thus depends on the whole chain." CreationDate="2011-02-16T16:07:17.533" UserId="1542" />
  <row Id="11714" PostId="7264" Text="@teucer, this is a shot in the dark since I still don't understand your model, and this might be the same as what you already have, but would it make sense to append the `pir` and `pvr` to the `ir` and `vr` columns, get rid of the second `for` loop, and then consider the values of `mu[]` estimated using `pir` and `pvr` to be the posterior predictive estimates of mu?" CreationDate="2011-02-16T16:07:39.610" UserId="1381" />
  <row Id="11715" PostId="7286" Text="thanks @wok, nice to be reach:)" CreationDate="2011-02-16T16:15:54.983" UserId="1542" />
  <row Id="11716" PostId="7297" Text="Could you please define your initialisms, or avoid using initialisms?" CreationDate="2011-02-16T17:10:02.407" UserId="449" />
  <row Id="11717" PostId="7291" Text="Multiplying is not the best idea, I think. Especially if your initial overflow is the regular one - modulo some power of 2. You will get a lot of factors that are even, so most of your lower bits will be 0. Adding the numbers together instead would already be a lot better. If you need even better randomness, use [some sort of hash function](http://en.wikipedia.org/wiki/Hash_function) and use any bits of the result. If you want it to be difficult to guess anything about the outcome for people other than you, use a salted strong cryptographic hash function." CreationDate="2011-02-16T17:17:52.810" UserId="2898" />
  <row Id="11718" PostId="7299" Text="thanks for your reply :)" CreationDate="2011-02-16T17:34:05.230" UserId="2534" />
  <row Id="11719" PostId="7304" Text="thanks for helping me" CreationDate="2011-02-16T17:43:45.520" UserId="2534" />
  <row Id="11720" PostId="7301" Text="I am not very familiar with ggplot2. Thank you for the suggestion." CreationDate="2011-02-16T17:44:01.927" UserId="3263" />
  <row Id="11721" PostId="5369" Text="exactely @carcal, thanks for the acclaration." CreationDate="2011-02-16T17:56:38.483" UserId="1808" />
  <row Id="11722" PostId="7301" Text="Sure - I really recommend trying it out as it has some wonderful visualization features. I've also updated my answer to include Deducer (which is an alternative GUI for ggplot2)" CreationDate="2011-02-16T17:58:31.337" UserId="2635" />
  <row Id="11723" PostId="7235" Text="@charlie - Thanks! In the world of big N, using a split sample for selection and then applying the model to the other split or full data, then using that to do all the normal inference, is certainly attractive." CreationDate="2011-02-16T18:01:33.097" UserId="2040" />
  <row Id="11724" PostId="7297" Text="Can you post an example plot? -- this seems like a computational error, but it is hard to tell from the description only." CreationDate="2011-02-16T18:05:05.283" UserId="88" />
  <row Id="11725" PostId="7297" Text="FT=Fourier transform, FFT = fast Fourier transform, DFT = discrete Fourier transform" CreationDate="2011-02-16T18:05:29.943" UserId="3272" />
  <row Id="11726" PostId="7305" Text="The trick applies to non-integer powers, but the proof is a little different." CreationDate="2011-02-16T18:07:08.927" UserId="795" />
  <row Id="11727" PostId="7301" Text="@celenius thank you, I will try out those things too.  I think I am getting somewhere.  I shall update the question as soon as I find a solution. " CreationDate="2011-02-16T18:26:13.823" UserId="3263" />
  <row Id="11728" PostId="7280" Text="Thanks for the help." CreationDate="2011-02-16T18:28:55.633" UserId="2387" />
  <row Id="11729" PostId="7174" Text="I am still hoping someone can answer my question, based on further details in response to comments from &quot;onestop&quot;, above. Thanks! " CreationDate="2011-02-16T18:41:42.927" UserId="834" />
  <row Id="11730" PostId="7259" Text="@chase, I hope that the example is enough now.  Can you point to a solution.  As mentioned by @celenius, I am looking at ggplot2, but I am still very much looking for a solution." CreationDate="2011-02-16T18:50:45.817" UserId="3263" />
  <row Id="11731" PostId="7297" Text="@Roman, I second @mbq's remarks. A figure would be helpful, especially in terms of understanding what you mean by &quot;noise&quot;. Just because the time domain signal is smooth does not mean the Fourier transform will be. Just think of a sinusoid to convince yourself. in terms of plotting, the first thing that comes to mind is: are you plotting the magnitudes and not, say, accidentally, just the real part. Also, do you know about windowing?" CreationDate="2011-02-16T19:21:43.463" UserId="2970" />
  <row Id="11732" PostId="7301" Text="I put an example of some code up there - this shows a box plot, and faceting, and is somewhat illegible based on the data that I chose, but it should illustrate the basics." CreationDate="2011-02-16T19:51:49.617" UserId="2635" />
  <row Id="11733" PostId="7308" Score="1" Text="@Jyotirmoy, what if the minimum happens at the boundary of your parameter space?" CreationDate="2011-02-16T20:18:44.990" UserId="2970" />
  <row Id="11734" PostId="7282" Text="@cardinal: I found a sample code in SNAP that does exactly what I want but for a undirected graph. I **think** my graph is what the SNAP docs calls &quot;directed multi-graph&quot;. So I changed just one line in `centrality.cpp` from `TUNGraph` to `TNEGraph` (see http://pastebin.com/GHUquJvT line 24). It is not compiling anymore. I suspect it requires a different type of node? The error I get is: `centrality.cpp:24: error: conversion from ‘TUNGraph::TNodeI’ to non-scalar type ‘TNEGraph::TNodeI’ requested`&#xA;(see full error at http://pastebin.com/86mCbByG)&#xA;" CreationDate="2011-02-16T20:20:46.393" UserId="1762" />
  <row Id="11735" PostId="7274" Text="Apologies for the naive question, but how do I convert a table into what you have as `src` and `dst`. This is what I typically do to load the file (now a tab-delimited file):  `el &lt;- read.csv(&quot;comment-net/comments-ouids.tsv&quot;,header=T,sep=&quot;\t&quot;)`" CreationDate="2011-02-16T20:45:59.817" UserId="1762" />
  <row Id="11737" PostId="7312" Text="Well, as an experienced statistician lacking fundamental maths education I still know about Mean and Median differences and applications a lot. What I need here is somebody to DRAW - logically or mathematically - either (a) from (b) or (b) from (a), for me. I feel I can't harmonize (a) with (b) rationally myself. Marco, I find very difficult understanding your notation. If your formulas is the deduction I need please could you &quot;chew over&quot; the idea less technically for me?" CreationDate="2011-02-16T20:59:04.770" UserId="3277" />
  <row Id="11738" PostId="7297" Text="Some idea of the source of the sampled curve could be helpful too." CreationDate="2011-02-16T21:05:32.527" UserId="449" />
  <row Id="11739" PostId="7274" Text="read.csv() should produce a data.frame. as.network() may read that directly or you may need to do as.matrix(el)." CreationDate="2011-02-16T21:10:12.393" UserId="3265" />
  <row Id="11740" PostId="7188" Text="@mariana You might be interested in my next question: http://stats.stackexchange.com/q/7313/1026" CreationDate="2011-02-16T21:19:21.600" UserId="1026" />
  <row Id="11741" PostId="7307" Text="So what you're after for the first pair is a proof that the median, as usually defined as the middle-rank value (for an odd number of values anyway, to start with the simplest case) is also the value that minimizes the sum of absolute deviations? Preferably a proof that also gives some intuitive insight? I don't know of any proof myself, so it seems a good question, and one i'd like to know the answer to as well." CreationDate="2011-02-16T21:24:07.917" UserId="449" />
  <row Id="11742" PostId="7174" Text="I'm no expert on analysis of survey data so I'm not certain, but I tend to agree with DWin that conditional logistic regression is not necessary here. Ordinarly logistic regression with survey weights should be ok. I do have a question though: If you already have all the data available in a database, why sample at all? Why not use all the firms in the database? Or are you then going on to collect additional data on the sampled firms?" CreationDate="2011-02-16T22:18:19.417" UserId="449" />
  <row Id="11743" PostId="7264" Text="@David I want to have some credible interval for the predictions (`pri`), this is why having `mu` is not enough. If I append `pvr` and `pir` I would get `alpha`, `b.vr` and `b.ir` for the whole `ri`, right? Or can I have `ri` with missing values e.g. (1,2,3,NA,NA) and append the regressors?" CreationDate="2011-02-16T22:18:47.860" UserId="1443" />
  <row Id="11744" PostId="7312" Text="P.S. As long as your two inequalities got finally displayed correctly on my screen I see it's merely my (b) statements. You write, &quot;it can be shown that...&quot;. So do show me that. I need a kind of mathematical proof put in terms that are intelligible for data analyst who is not a professional mathematician." CreationDate="2011-02-16T22:36:11.227" UserId="3277" />
  <row Id="11746" PostId="7297" Text="@cardinal: No, these are magnitudes, I've checked it. I've heard of windowing, but I haven't used it before." CreationDate="2011-02-16T22:48:58.647" UserId="3272" />
  <row Id="11747" PostId="7297" Text="@Roman, now you should be able to post images. Think of windowing as applying a low-pass LTI filter to the Fourier coefficients. That essentially takes local weighted averages of the coefficients, which might make peak detection in your case more feasible as long as the peaks are spaced &quot;far enough&quot; apart. " CreationDate="2011-02-16T22:53:31.727" UserId="2970" />
  <row Id="11748" PostId="7307" Text="You feel me correctly. (a) and (b) are currently separate aspects/properties in my mind, for both statistics; but intuition suggests the two aspects are tied. I want to know - _how_ they are tied, to understand it all deeply." CreationDate="2011-02-16T23:08:03.883" UserId="3277" />
  <row Id="11749" PostId="7297" Text="@Roman, was the constant that was subtracted very close in value to the last data point? Otherwise, any zero padding done by the FFT to get the length out to a power of 2 would induce an artifact due to the step down at around time 100. " CreationDate="2011-02-16T23:15:45.187" UserId="2970" />
  <row Id="11750" PostId="7312" Text="@ttnphns:  your request for a mathematical, rather than an intuitive, answer seems incompatible with your request for something less technical than what people have offered." CreationDate="2011-02-16T23:23:19.470" UserId="2669" />
  <row Id="11752" PostId="7315" Score="1" Text="This sweeps some stuff under the rug concerning when you hit the next $x_i$ value, but is close enough to the handwavey proof desired, I think, at least for the median." CreationDate="2011-02-17T00:56:58.787" UserId="795" />
  <row Id="11753" PostId="7312" Text="Can we simplify the situation to 2 or three points and ask whether the median in the double summation non-strict inequality above has a unique value? With two points it would seem to be satisfied by any point between the 2." CreationDate="2011-02-17T01:03:28.183" UserId="2129" />
  <row Id="11754" PostId="7285" Text="@onestop thanks for the terminology correction." CreationDate="2011-02-17T01:06:08.633" UserId="183" />
  <row Id="11755" PostId="7228" Text="Dear Jeromy,&#xA;thanks a lot for your feedback. Anova seems correct to me. Anyways I am just having a look to a &quot;conjoint measurement analysis&quot; and apparently it seems to fit my problem.&#xA;&#xA;All the best" CreationDate="2011-02-17T01:18:50.910" UserId="3180" />
  <row Id="11756" PostId="6795" Text="@mpiktas Apologies...wasn't trying to confuse; clearly didn't ask the question effectively.  Thanks for the help in clarifying it!" CreationDate="2011-02-17T02:40:02.790" UserId="988" />
  <row Id="11757" PostId="7316" Text="@Belmont, are you asking about how to specify the knots in $R$ (i.e., via arguments to **ns**) or are you asking about strategies for deciding on where to place the knots?" CreationDate="2011-02-17T03:47:01.367" UserId="2970" />
  <row Id="11758" PostId="7308" Text="@cardinal. Your are right, my argument won't work in that case. But Wooldridge is considering the case where the minimum is in the interior. Isn't he wrong in that case?" CreationDate="2011-02-17T04:02:14.543" UserId="1393" />
  <row Id="11759" PostId="7308" Text="@Jyotirmoy, it can certainly be only positive semidefinite. Think of linear functions or a function where the set of minimum points forms a convex polytope. For a simpler example, consider any polynomial $f(x)=x^{2n}$ at $x = 0$." CreationDate="2011-02-17T04:13:30.943" UserId="2970" />
  <row Id="11760" PostId="7181" Text="Actually SPSS do charge for this separately in SPSS Categories.  Ugh... suncoolsu you were right.  Anyone know how to get an eval of SPSS version 18 so I can check if this helps me?" CreationDate="2011-02-17T04:49:23.160" UserId="3189" />
  <row Id="11761" PostId="7308" Score="1" Text="@cardinal. True. What is troubling me is the phrase &quot;even positive semidefinite&quot; in the quoted statement." CreationDate="2011-02-17T05:04:19.623" UserId="1393" />
  <row Id="11762" PostId="7322" Score="2" Text="&#xA;You are welcome to ask questions like these, but adding details about what are the variables (not only variable names and tittle details) help. It will help others with similar questions in the future.&#xA;" CreationDate="2011-02-17T07:01:17.170" UserId="1307" />
  <row Id="11764" PostId="7321" Text="I can think of a simple solution for the case of an infinite 'axis', but i assume yours is finite, and I think boundary effects would completely mess up my idea... A bit more info might help: Are the 'fields' all the same length, or similar lengths, or not? How long is the 'axis' compared to a (typical) 'field'?" CreationDate="2011-02-17T07:25:48.943" UserId="449" />
  <row Id="11766" PostId="7323" Text="Well, actually this is the study of “Physical Growth of California Boys and Girls from Birth to Eighteen Years” by Read D. Tuddenham and Margaret M. Snyder back from 1954. The idea was that people in this study misused statistics and hence, made false inferences about what the dataset was supposed to show. Plotting the residuals was one way of showing it. Now, let me define some of the variables: regression soma.WT9 represents the regression of so called somatotype on weight at nine years of age. and regression of HT9.WT9 represents the regression of height at 9 years of age onto weight at 9yr" CreationDate="2011-02-17T07:33:36.467" UserId="3008" />
  <row Id="11767" PostId="7323" Text="further, i then take the residuals from both of those regressions and plot them against each other." CreationDate="2011-02-17T07:35:02.487" UserId="3008" />
  <row Id="11768" PostId="7323" Text="Unfortunately that's not available online... Could you just tell us what 'soma' is?" CreationDate="2011-02-17T07:35:25.860" UserId="449" />
  <row Id="11769" PostId="7323" Text="@onestop: the data is in the R package alr3 as BGSall BGSgirls BGSboys (see http://rgm2.lab.nig.ac.jp/RGM2/func.php?rd_id=alr3:BGSall) and &quot;soma&quot; seems to be &quot;Somatotype, a 1 to 7 scale of body type&quot; developed by William Herbert Sheldon (described in http://en.wikipedia.org/wiki/Somatotype_and_constitutional_psychology)" CreationDate="2011-02-17T08:17:34.727" UserId="2958" />
  <row Id="11770" PostId="7321" Score="3" Text="In a very simple manner, this can be done as a control chart (just in space not time) -- namely by converting it to number of counts over discrete segments and then calculating deviation form Poissson distribution. " CreationDate="2011-02-17T09:19:13.353" UserId="88" />
  <row Id="11771" PostId="7297" Text="@Roman, What do you want to do with this FFT ? Using DWT (discrete wavelet transform) is often more suitable when you have time varying frequency. " CreationDate="2011-02-17T10:20:54.453" UserId="223" />
  <row Id="11772" PostId="7084" Text="Could you elaborate on &quot;using a mathematical model evaluates functional meaning of the measured data&quot; ? How does the mathematical model evaluate the functional meaning? What do you mean by 'evaluate'? What do you mean by 'functional meaning'?" CreationDate="2011-02-17T10:49:48.657" UserId="449" />
  <row Id="11773" PostId="6919" Text="@cardinal, good answer, note that most often $D_{TV}$ is defined another way which makes it half the $L_1$ distance.  " CreationDate="2011-02-17T10:52:33.757" UserId="223" />
  <row Id="11774" PostId="7326" Text="@Mortimer, good question, can you tell us what you want to do with this sample  ? I guess this info will help to get better answer about the size. From a cost loss perspective, the loss of small size is the loss of information which may be different depending on the use of the sample, and the cost of a large size is the computation cost that higly depends on what you do with the sample. " CreationDate="2011-02-17T12:25:22.053" UserId="223" />
  <row Id="11775" PostId="7084" Text="@posdef If I understood you want to build up a test of normality that will generate data from a normal random variable (with suitable mean and variance) and compare it to your orginial sample ? Why do you think this can be a better test than usual test of normality ? " CreationDate="2011-02-17T12:50:25.090" UserId="223" />
  <row Id="11776" PostId="1432" Text="One really easy way to check model fit is a plot of the observed vs the predicted proportions.  But this won't work if you have bernoulli regression (i.e. all of your observations have unique combinations of the independent variables, so that $n_i=1$), because you will just see a line of zeros and ones." CreationDate="2011-02-17T13:17:44.417" UserId="2392" />
  <row Id="11777" PostId="7326" Text="I am not an expert on this stuff, so someone please correct me if i am wrong. I agree with robin that it really depends on what you need to do with the data. In your case a small sample will lead to a loss of information from the original sample (missing tags), which means your sample will not be truly representative. For the more popular tags you can use a simple significance test (like a chi-square test), but for the rarer tags that are missing from your sample, their absence will be significant. It is up to you if this is acceptable. " CreationDate="2011-02-17T13:19:12.707" UserId="2965" />
  <row Id="11778" PostId="7327" Text="Thank you for the response. It will take me at least a few days to process the materials you presented. I would also like to say your dissertation is incredibly nice and after reading your introduction I look forward to reading the rest." CreationDate="2011-02-17T13:50:26.007" UserId="1036" />
  <row Id="11779" PostId="7315" Text="@shabbychef, thank you for the elegant explication. However, it sounds to me so: &quot;That number _y_ which small change doesn't change function _Sum|x_i-y|_ does not depend on each _x_i_ and is called median&quot;. It's an interesting note on median of an even _n_ data. But I was asking to prove this: &quot;That number _y_ which minimizes function _Sum|x_i-y|_ does not depend on each _x_i_ and is called median&quot;. And similarly: &quot;That number _y_ which minimizes function _Sum(x_i-y)^2_ depends equally on each _x_i_ and is called mean&quot;." CreationDate="2011-02-17T13:57:00.557" UserId="3277" />
  <row Id="11780" PostId="7318" Score="2" Text="In what way is the data &quot;non-normal&quot;?  skewed? &quot;fat tails&quot;? outliers? You may be able correct this by using a different distribution (e.g. t or cauchy for fat tails and outliers, and rescaled beta distribution for skewness, off the top of my head), and save some power in your test." CreationDate="2011-02-17T14:09:06.507" UserId="2392" />
  <row Id="11781" PostId="7330" Score="1" Text="I think you'd be better asking this on math.SE, as the question clearly applies equally to anything that can be represented in $n$-dimensional space, not just a probability distribution." CreationDate="2011-02-17T14:10:08.967" UserId="449" />
  <row Id="11782" PostId="7330" Text="@onestop - I am thinking the same thing.  I'll leave this question here anyway, just in case somebody in stats world knows the answer.  One thing I have found with stats (particularly Bayesian stats) is that you tend to come across just about every branch of maths (pardon the pun).  So maybe some stats person may know it." CreationDate="2011-02-17T14:19:35.480" UserId="2392" />
  <row Id="11783" PostId="7084" Text="@onestop: this is essentially a systems biology model, we check how different networks are affected by different conditions. I can't really go into detail on what the model is or how it works, as it'll take a long time and place, and I don't think it's very relevant either.. What is important/relevant is; I have abundance levels for a long list of molecules, which come from an instrument with high technical variation, and an unknown underlying distribution. I want to see that my model can distinguish data from real biology, and made-up/random data, from a similar distribution." CreationDate="2011-02-17T14:28:48.313" UserId="3014" />
  <row Id="11784" PostId="7330" Text="I guess if you can &quot;rearrange&quot; the indexes (apply permutations) in you space the answer is yes otherwise it is no. " CreationDate="2011-02-17T14:29:53.663" UserId="223" />
  <row Id="11785" PostId="7084" Text="@robin girard: not exactly... see the comment above in reply to @onestop. I have a model and I test the significance of the results of my model, by comparing it to data of no biological significance. I intend to sample from normal distribution instead of permuting the same values over and over again. Could this normality approximation be justified somehow, that is my question." CreationDate="2011-02-17T14:31:16.157" UserId="3014" />
  <row Id="11786" PostId="7326" Text="thanks. What we are trying to do is to show how search works on such tagging system and we need to manually annotate the tags. In particular, someone might use &quot;javaisland&quot;, someone else &quot;java_island&quot; etc.. That's why we can't use the whole dataset. We assumed that randomly selecting tag won't make a difference, but it seems that this leads to a very &quot;computer/web oriented&quot; vocabulary. This seems to be the case in the whole delicious dataset apriori, but I would like to have a hard number to check that this distribution of vocabulary isn't just a feature of the sampling." CreationDate="2011-02-17T14:39:29.360" UserId="3291" />
  <row Id="11787" PostId="7326" Text="so, to make it more concrete, the tag &quot;design&quot; appears a lot in the sample, it also appears a lot in the dataset and the tag &quot;crazywebsite&quot; rarelly appears in the dataset and rarelly appears in the sample. Now, I can looks at them all manually, but they don't all have exactly the same distribution and would expect that there is some test to tell me that the general distribution of each class is within some interval of the original distribution." CreationDate="2011-02-17T14:46:54.820" UserId="3291" />
  <row Id="11788" PostId="7308" Text="@Jyotirmoy, is there a specific form of the M-estimator given in the book that you could provide? Also give the parameter space under consideration. Maybe then we can figure out what the author had in mind. In general, I think we've already established that the author's assertion is correct. Placing further constraints on the form of $q$ or the parameter space being considered might alter that. " CreationDate="2011-02-17T14:47:58.667" UserId="2970" />
  <row Id="11789" PostId="7330" Text="@robin- that is what I implicitly meant by rotating about &quot;arbitrary&quot; dimensions, in that it doesn't matter which &quot;dimension&quot; you call 1 and 2." CreationDate="2011-02-17T14:54:02.523" UserId="2392" />
  <row Id="11790" PostId="7332" Text="thanks for your answer, especially the terminology - I've never heard of &quot;givens&quot; rotation before." CreationDate="2011-02-17T14:57:49.993" UserId="2392" />
  <row Id="11791" PostId="7332" Text="Thanks for this, you have saved me a lot of useless algebra!  Because I can go &quot;rotation 2-D&quot; new prior invariant... &quot;rotate 2-D&quot; new new prior invariant ..... etc." CreationDate="2011-02-17T15:13:03.593" UserId="2392" />
  <row Id="11792" PostId="7332" Text="@probabilityislogic, Givens rotations are very popular in numerical linear algebra due to their simplicity and good numerical-stability characteristics. Some QR decomposition methods use them and I think any good SVD implementation will use them as well. " CreationDate="2011-02-17T15:21:38.777" UserId="2970" />
  <row Id="11793" PostId="7264" Text="@teucer yes, you can have ri with missing values; this is a neat feature of bugs that simplifies calculation of a predictive interval." CreationDate="2011-02-17T15:46:27.460" UserId="1381" />
  <row Id="11794" PostId="7208" Score="1" Text="I don't think you are right in the first calculation. If I'm not mistaken, the agreement by chance should be 0.5, giving a kappa of 0." CreationDate="2011-02-17T17:24:12.070" UserId="449" />
  <row Id="11795" PostId="7084" Score="2" Text="&quot;just focus on the question at hand instead of inquiring further and further into the motivations behind why&quot;. I'm sorry; we're not just being curious or trying to be awkward, honest. Statisticians learn, often the hard way, that it's usually a very bad idea to tell clients how to do something without first enquiring why they want to do it." CreationDate="2011-02-17T17:26:00.480" UserId="449" />
  <row Id="11796" PostId="7336" Text="Good question. Have a look at this [question](http://stats.stackexchange.com/questions/3532/random-numbers-and-the-multicore-package) on random numbers and the multicore package" CreationDate="2011-02-17T17:54:43.683" UserId="8" />
  <row Id="11797" PostId="7309" Text="Thank you @chase, this pretty much answers my question. I was wondering if it is possible to do this without the inbetween step of creating the &quot;extra&quot; data-frame.  Also I used to think that `cbind` is more efficient.  Is it not?" CreationDate="2011-02-17T17:56:41.683" UserId="3263" />
  <row Id="11799" PostId="7306" Text="Thanks for this ... I think I might need to reword the question though, or ask a new one.  Looking at this question http://stats.stackexchange.com/questions/980/how-do-i-reduce-the-number-of-data-points-in-a-series, I now think getting the means is not exactly what I'm after." CreationDate="2011-02-17T18:18:41.990" UserId="2770" />
  <row Id="11800" PostId="7337" Text="You also need to coordinate between the RNG streams. Snow does that, multicore may now." CreationDate="2011-02-17T18:23:13.393" UserId="334" />
  <row Id="11801" PostId="7332" Text="(+1) @Cardinal One small thing: the resulting diagonal matrix does not need to be the identity.  It can have an even number of -1's.  But those can be converted to +1's by means of suitable rotations (of 180 degrees).  It's also worth nothing the implicit assumption that the only rotations one can use are with respect to a fixed basis.  If not, then *any* rotation can always be written in the form $R_n(\theta)$ (that is, as a single such matrix) by choosing an appropriate basis adapted to that rotation." CreationDate="2011-02-17T18:27:10.113" UserId="919" />
  <row Id="11802" PostId="7291" Text="Agreed.  Just wanted to suggest idea to illustrate bit shifting in order to generate (roughly) pseudo-random numbers." CreationDate="2011-02-17T18:48:55.320" UserId="3489" />
  <row Id="11803" PostId="7335" Text="@Zubin, @andresmh, Maslov and Sneppen (*Science*, 2002) have a visualization that might be useful in this context. Searching through recent stats/comp-sci--related citations of this work, I found [this](http://statistics.stanford.edu/~ckirby/techreports/GEN/2010/2010-13.pdf) as well. [Here](http://arxiv.org/PS_cache/arxiv/pdf/1012/1012.0201v1.pdf) may be another related work." CreationDate="2011-02-17T19:06:24.497" UserId="2970" />
  <row Id="11804" PostId="7332" Text="@whuber, Good catch! Thank you, and I've updated it accordingly." CreationDate="2011-02-17T19:11:31.573" UserId="2970" />
  <row Id="11805" PostId="5430" Text="@Theodor If you discard the observations for distance&lt;4, is't it more convenient to fit an exponential distribution?&#xA;&#xA;Because if that is the case, take the difference between the exponential fit and the data. I'm sure the peak at 7 will stand out clearly." CreationDate="2011-02-17T19:36:50.473" UserId="2902" />
  <row Id="11806" PostId="1432" Text="Yeah - sadly I usually am using a Bernoulli DV." CreationDate="2011-02-17T20:05:18.503" UserId="196" />
  <row Id="11807" PostId="7315" Text="how am I supposed to prove the 'is called median' part? That's crazy." CreationDate="2011-02-17T20:14:31.360" UserId="795" />
  <row Id="11808" PostId="7340" Text="Ralph, I do not really understand what you mean by: &quot;build a confidence interval of this distribution to see if the population mean is included.&quot;. What we want is to show properties such as the amount of spell variants and synonyms, so we are doing this, but if our sample is not representative of the general population, it's not clear that we can generalize." CreationDate="2011-02-17T20:29:23.400" UserId="3291" />
  <row Id="11810" PostId="7321" Score="1" Text="@tim More information would be helpful (honestly, it's essential).  Either implicitly or explicitly, this problem needs a probability model for the &quot;fields&quot; under a null hypothesis.  For example, we might posit that the fields have fixed length and their right endpoints are uniformly distributed over the possible values (in which case the distribution of the number of overlaps at most predefined points is Binomial).  Bear in mind, too, that you can treat this as a 2D problem (by recording each interval as an ordered pair such as left endpoint and length)." CreationDate="2011-02-17T21:46:37.447" UserId="919" />
  <row Id="11811" PostId="7345" Text="Sometimes I don't the forest for all the trees ... Thanks!" CreationDate="2011-02-17T21:46:41.113" UserId="977" />
  <row Id="11812" PostId="7208" Text="I don't really understand the `information-retrieval` tag here." CreationDate="2011-02-17T22:03:04.737" UserId="930" />
  <row Id="11813" PostId="7208" Text="I don't know, I am working on an information-retrieval task, where people judge whether documents are relevant or not, hence the kappa statistic. But everybody can retag posts here, so feel free to do so! @onestop, following [this standard guide](http://nlp.stanford.edu/IR-book/html/htmledition/assessing-relevance-1.html) my numbers are correct, the pooled marginals are .75 and .25, respectively, and both squared and added to each other equal .625" CreationDate="2011-02-17T22:20:38.387" UserId="1205" />
  <row Id="11814" PostId="7348" Score="4" Text="Please don't [cross post](http://stackoverflow.com/q/5034966/271616).  At the very minimum, you could tell people you're cross-posting." CreationDate="2011-02-17T22:33:07.087" UserId="1657" />
  <row Id="11815" PostId="7348" Text="Fair enough, I confess I wasn't aware of the non-crossposting etiquette. I haven't had a question where I was unsure in my own mind about were to post it before. It fell between data-visualization and r programming. I have deleted the SO question for now." CreationDate="2011-02-17T22:51:14.863" UserId="2726" />
  <row Id="11816" PostId="7208" Text="I hope that's not a standard guide in information retrieval as it's just plain wrong on this! The correct formula can be found by [googling 'Cohen's kappa'](http://www.google.com/search?q=cohen%27s+kappa) and following any of the top three hits." CreationDate="2011-02-17T23:24:39.937" UserId="449" />
  <row Id="11817" PostId="7346" Score="4" Text="I was starting to worry you wouldn't volunteer :-)." CreationDate="2011-02-17T23:24:59.640" UserId="919" />
  <row Id="11819" PostId="7352" Text="I'm just wondering if anyone else noticed that the values at the top of page 2 of the 4th link (whatisbayes.pdf) are a little farther off than simple rounding error?  I'm getting about 0.040, 0.097, 0.138, 0.139." CreationDate="2011-02-18T02:59:06.120" UserId="601" />
  <row Id="11821" PostId="7358" Score="1" Text="(+1) Good question!" CreationDate="2011-02-18T04:53:08.670" UserId="1307" />
  <row Id="11822" PostId="7346" Score="1" Text="+1, I was starting to worry also :)" CreationDate="2011-02-18T07:42:36.017" UserId="2116" />
  <row Id="11823" PostId="7264" Text="@David could please edit your answer accordingly so that I can accept?As I said earlier having only the mean is not enough for my purposes, I would like to have the other statistics..." CreationDate="2011-02-18T08:06:54.953" UserId="1443" />
  <row Id="11824" PostId="7364" Text="@gappy, there are several articles about the limit theory for factor models, but in [econometrics context](http://www.econ.nyu.edu/user/baij/publication.htm). I dabbled myself in factor models but for me your formulation of the factor model is anything but standard. Could you please provide a reference? As it now stands it is more of regression model than the factor model." CreationDate="2011-02-18T08:10:18.663" UserId="2116" />
  <row Id="11825" PostId="7359" Text="+1, great answer. I always wondered why the original formula is used for $R^2$ instead of square of correlation. For linear regression it is the same, but when applied to other contexts it is always confusing." CreationDate="2011-02-18T08:22:28.790" UserId="2116" />
  <row Id="11826" PostId="7362" Text="@DrWho, something is missing in your first sentence. As it stands now it is incomprehensible from linguistic point of view, for me at least." CreationDate="2011-02-18T08:25:08.537" UserId="2116" />
  <row Id="11827" PostId="7344" Score="1" Text="@Zach, have you tried plot on the `lm` or `glm` object? How what you are trying to achieve is different from what `plot.lm` does?" CreationDate="2011-02-18T08:29:12.713" UserId="2116" />
  <row Id="11828" PostId="7356" Score="1" Text="That's a nice one. Just what the doctor ordered." CreationDate="2011-02-18T08:56:00.543" UserId="144" />
  <row Id="11829" PostId="7366" Text="Can't you collect the mean? You're not going to be able to calculate a $z$-score without it." CreationDate="2011-02-18T09:12:49.040" UserId="449" />
  <row Id="11830" PostId="7359" Text="@cardinal (+1) Very elegant response, indeed." CreationDate="2011-02-18T09:14:04.263" UserId="930" />
  <row Id="11831" PostId="7366" Text="@onestop I can collect mean, also editing question accordingly. Thanks" CreationDate="2011-02-18T09:16:29.770" UserId="2170" />
  <row Id="11832" PostId="7356" Score="2" Text="@Dirk (+1) Also of interest, `smoothScatter {RColorBrewer}` and `densCols {grDevices}`. I can confirm it works pretty well with thousand to million of points from genetic data." CreationDate="2011-02-18T09:30:25.490" UserId="930" />
  <row Id="11833" PostId="7367" Score="2" Text="I would like to second this. If you're looking for a fast and easy way to interface compiled code with R, `Rcpp` with `RcppArmadillo` is the way to go. Edit: Using Rcpp, you also have access to all the RNGs implmented in the C-code underlying R." CreationDate="2011-02-18T10:05:29.360" UserId="1979" />
  <row Id="11834" PostId="7336" Text="@CSgillepsie:&gt; thanks for the pointer, but i'm not sure it's the same problem: the way i understand the question you pointed to, all the processes are spawned by mclapply. Here it's a bit different: on each of the machines, all the processes are spawned by mclapply, but this is not the case *across* machines." CreationDate="2011-02-18T10:09:54.083" UserId="603" />
  <row Id="11835" PostId="7372" Text="@onestop, why did you delete your answer? I was just about ready to upvote it." CreationDate="2011-02-18T11:08:11.983" UserId="2116" />
  <row Id="11837" PostId="7373" Text="if I provide z-scores, correlations can be determined by 3rd party tools like WEKA, am I right?" CreationDate="2011-02-18T11:49:16.960" UserId="2170" />
  <row Id="11838" PostId="7372" Text="(+1) Let's say my vote goes to both of you :-)" CreationDate="2011-02-18T12:26:30.280" UserId="930" />
  <row Id="11839" PostId="7364" Score="1" Text="The factor model I provide *is* the standard definition. Check, say http://learning.eng.cam.ac.uk/zoubin/papers/ul.pdf, and references therein; or Econometric Analysis of Cross Section and Panel Data by Wooldridge, or Econometrics by Hayashi. I am well aware of Bai and Ng's work, as well as Reichlin et al., and Stock and Watson; but their limit theory has nothing to do with this-- is proves asymptotic convergence of PCA to the factor model for $N$ and $T$ going to infinity." CreationDate="2011-02-18T12:26:40.110" UserId="30" />
  <row Id="11840" PostId="7346" Score="7" Text="@whuber @mpiktas Let's see you both volunteer too!" CreationDate="2011-02-18T12:38:10.077" UserId="5" />
  <row Id="11841" PostId="7376" Score="1" Text="@Milktrader what do you mean by &quot;valid statistical calculation&quot; you should say valid statistical (estimation) calculation of something. Here the something is very important. Correlation is a valid calculation of the linear relation between two set of data. I don't see why you need stationarity, did you mean auto-correlation ? " CreationDate="2011-02-18T13:13:14.870" UserId="223" />
  <row Id="11843" PostId="7364" Text="@gappy this paper http://www.i-journals.org/ejs/viewarticle.php?id=267&amp;layout=abstract may at least give referencse ? " CreationDate="2011-02-18T13:17:33.070" UserId="223" />
  <row Id="11844" PostId="7365" Text="@mpiktas, That's some interesting notation there (I know it's not yours). A $w$ on the left-hand side and $y$ and $x$ on the right-hand side. I'm guessing $w = (x,y)$ or something like that. Also, I'm assuming the squaring should be happening to $y - m(x,\theta)$ and not just to $m(x,\theta)$. No?" CreationDate="2011-02-18T13:17:34.260" UserId="2970" />
  <row Id="11845" PostId="7368" Text="@onestop: Thank you so much for the unexpectedly fast and simple worded guidance.You must be an excellent teacher. My treatment is very simple and has already saved thousands of lives." CreationDate="2011-02-18T13:18:40.623" UserId="2956" />
  <row Id="11846" PostId="7373" Text="I mean, attribute selection or information gain calculations are enough to keep some z-scores and remove unnecessary ones, aren`t they?" CreationDate="2011-02-18T13:19:20.940" UserId="2170" />
  <row Id="11847" PostId="7365" Text="@mpiktas, I'm not *quite* sure how to interpret your first sentence due to the wording. I can see two ways, one that I'd call correct and the other I wouldn't. Also, strictly speaking, I don't agree with the second sentence in your first paragraph. As I've shown above, it is possible to be at a local minimum in the interior of the parameter space without the Hessian being positive definite." CreationDate="2011-02-18T13:24:03.880" UserId="2970" />
  <row Id="11848" PostId="7373" Text="You've lost me completely there, I'm afraid. Others may understand what you're talking about, but it must be outside my field." CreationDate="2011-02-18T13:25:38.797" UserId="449" />
  <row Id="11849" PostId="7359" Text="@mpiktas, @chl, I'll try to expand on this a little more later today. Basically, there's a close (but, perhaps, slightly hidden) connection to hypothesis testing in the background. Even in a linear regression setting, if the constant vector is not in the column space of the design matrix, then the &quot;correlation&quot; definition will fail." CreationDate="2011-02-18T13:26:11.723" UserId="2970" />
  <row Id="11850" PostId="7365" Text="@cardinal, yes you are right. Wooldridge uses $w$ for consistency reasons, $y$ and $x$ is reserved for response and predictors throughout the book. In this example $w=(x,y)$." CreationDate="2011-02-18T13:26:30.423" UserId="2116" />
  <row Id="11851" PostId="7362" Text="@Bernd: Thank you for your interest to help me. I am sorry I was not clear and wasted your time. I am trying to calculate Orwin's fail-safe N. The treatment is inexpensive but has saved thousands of lives already.I am trying to decide on the best values to use for the criterion for a trivial log odd's ratio and mean log odds ratio in missing studies to calculate the Number of missing studies needed to bring down odds ratio (because Classic fail-safe method yielded 1014 studies). I am using Comprehensive meta-analysis v2.2" CreationDate="2011-02-18T13:30:11.657" UserId="2956" />
  <row Id="11852" PostId="6919" Text="@robin, thanks for your comment. Yes, I realize this. I was just trying to avoid a messy extraneous constant in the exposition. But, strictly speaking, you're correct. I've updated it accordingly." CreationDate="2011-02-18T13:33:29.720" UserId="2970" />
  <row Id="11853" PostId="7326" Text="@Mortimer and @rm999 if you don't use @name to answer then I can know that you are talking to me (the sofware detect @robin and send me a notification somehow when it comes).  " CreationDate="2011-02-18T13:34:23.847" UserId="223" />
  <row Id="11854" PostId="7365" Text="@cardinal, I fixed my wording. Now it should be ok. Thanks for pointing out the problem. " CreationDate="2011-02-18T13:42:18.047" UserId="2116" />
  <row Id="11855" PostId="7376" Score="1" Text="there is a new site which might be more suitable for your question: http://quant.stackexchange.com. Now you are clearly confusing calculation with interpretation." CreationDate="2011-02-18T13:45:44.143" UserId="2116" />
  <row Id="11856" PostId="7344" Text="@mpiktas I'm looking for something to supplement plot.lm or plot.glm.  Plot.lm shows residuals vs Fitted, Scale-Location, Normal Q-Q and Residuals vs. leverage plots.  What I'm looking for is plots of the actual relationship between Solar.R and and Ozone, and the predicted relationship from my model.  Run my example code, and you will see what I mean." CreationDate="2011-02-18T14:33:28.390" UserId="2817" />
  <row Id="11857" PostId="7376" Text="@mpiktas, the quant community is settled on using returns vs prices because of the stationarity of returns and the non-stationarity of prices. I'm asking here for something more than an intuitive explanation of why this should be so. " CreationDate="2011-02-18T14:38:05.210" UserId="3306" />
  <row Id="11858" PostId="7340" Text="Mortimer, I am not sure exactly what you are trying to do. But I am suggesting that if you know that &quot;javaisland&quot; + &quot;java_island&quot; represent 2% of your population, you bootstrap sample from your 500 sample, say 10000 times.  You obtain a confidence interval for your &quot;javaisland&quot; + &quot;java_island&quot; term frequency of 1.5% plus/minus 1%.  Since the confidence interval contains the population mean of 2%, you have no reason to suspect that your sample is not representative for the term &quot;java_island&quot;." CreationDate="2011-02-18T15:07:05.773" UserId="3489" />
  <row Id="11859" PostId="7376" Text="@robin, there are several things that may have you question a statistical analysis. Sample size comes to mind, as does more obvious things such as manipulated data. Does non-stationarity of data call into question a correlation calculation?" CreationDate="2011-02-18T15:10:01.297" UserId="3306" />
  <row Id="11860" PostId="7351" Text="Hi all, Thanks a lot for the pointers. I will be sure to look." CreationDate="2011-02-18T15:13:02.743" UserId="3301" />
  <row Id="11861" PostId="7376" Text="@Milktrader not the calculation, maybe the interpretation if the correlation is not high. If it is high it means high correlation (i.e. high linear relation), two non stationnary time series say $(X_t)$ and $(Y_t)$ can be  potentially highly correlated (for example when $X_t=Y_t$. " CreationDate="2011-02-18T15:25:32.770" UserId="223" />
  <row Id="11862" PostId="7367" Text="Thanks for the vote of confidence. I was about to suggest the same ;-)" CreationDate="2011-02-18T15:32:03.597" UserId="334" />
  <row Id="11863" PostId="7342" Text="Can you help me understand why I'm getting NAs when running this?" CreationDate="2011-02-18T16:01:58.123" UserId="2770" />
  <row Id="11864" PostId="7373" Text="Many apologies, @baris_a, I've deleted my comment that replied &quot;no&quot; to your first comment above as I've just realised the answer is in fact &quot;yes&quot;, i.e. you *can* compute correlations from $z$-scores. Truly sorry, I must have been having a brainstorm earlier. I still don't know what 'attribute selection' or 'information gain calculations' are though, I'm afraid." CreationDate="2011-02-18T16:04:22.980" UserId="449" />
  <row Id="11865" PostId="6811" Text="@Dirk, thanks a lot for offering to help. We have a couple of students in our lab who may look into this.  We'll let you know as soon as we figure things out :)" CreationDate="2011-02-18T16:08:55.100" UserId="2798" />
  <row Id="11866" PostId="6816" Text="@Dikran, that sounds great! Thanks for being willing to help. We have a couple of students in our lab and they might reach out to you if we need help." CreationDate="2011-02-18T16:10:22.167" UserId="2798" />
  <row Id="11867" PostId="7377" Text="Thanks Dirk - I had a feeling you'd answer sooner rather than later :). Given that I want code I can call from other software (Python mainly, but Matlab too) perhaps a good workflow would be to prototype in Rcpp/RcppArmadillo and then move to &quot;straight&quot; Armadillo? The syntax, etc looks very similar." CreationDate="2011-02-18T16:13:52.350" UserId="26" />
  <row Id="11868" PostId="7377" Text="Hope you found it helpful." CreationDate="2011-02-18T16:16:11.220" UserId="334" />
  <row Id="11869" PostId="7361" Text="Thanks for the tip. Boost looks like kind of a big hammer for my little nail, but mature and maintained." CreationDate="2011-02-18T16:25:20.530" UserId="26" />
  <row Id="11870" PostId="7377" Text="Re your 2nd question from the edit: Sure. Armadillo depends on little, or in our case, nothing besides R. Rcpp / RcppArmadillo would help you interface and test prototyped code that can be re-used standalone or with a Python and Matlab wrappers you can add later.  Conrad may have pointers for something; I don't have any for Python or Matlab." CreationDate="2011-02-18T16:40:54.613" UserId="334" />
  <row Id="11871" PostId="7378" Text="The mathematical point of view explanation is what I was looking for. It gives me something to contemplate and explore further. Thanks. " CreationDate="2011-02-18T16:41:13.273" UserId="3306" />
  <row Id="11872" PostId="7379" Score="3" Text="This estimates the standard error of the &quot;accuracy&quot; as measured from &quot;total trials&quot; draws from a Bernoulli variable." CreationDate="2011-02-18T16:54:30.267" UserId="919" />
  <row Id="11873" PostId="7378" Text="This response seems to sidestep the original question: Aren't you just saying that yes, calculating correlation makes sense for stationary processes?" CreationDate="2011-02-18T16:58:20.620" UserId="919" />
  <row Id="11874" PostId="7349" Text="Reducing the overlap with `unique` or by rounding can result in biased (deceptive) plots.  It's important to somehow indicate the amount of overlap through some graphical means such as lightness or with sunflower plots." CreationDate="2011-02-18T17:01:15.313" UserId="919" />
  <row Id="11875" PostId="7379" Text="@whuber - Why not just use `number correct / total trials` like everywhere else?" CreationDate="2011-02-18T17:14:25.840" UserId="2019" />
  <row Id="11876" PostId="7215" Score="4" Text="I almost didn't bother to comment since it goes without saying that @mbq has been a great moderator. In particular, in pushing forward the journal club." CreationDate="2011-02-18T17:14:48.647" UserId="8" />
  <row Id="11877" PostId="7378" Text="@whuber, I was answering the question having in mind the comment, but I reread the question again and as far as I understand the OP asks about calculation of correlation for non-stationary data. Calculation of correlation for stationary processes makes sense, all the macroeconometric analysis (VAR, VECM) relies on that." CreationDate="2011-02-18T17:35:39.340" UserId="2116" />
  <row Id="11878" PostId="7379" Text="Because they're not actually estimating accuracy but the standard error.  If they're actually estimating accuracy this way it's just WRONG. Check the values, are there any over 0.5?  If there are then this isn't the accuracy estimate equation." CreationDate="2011-02-18T17:55:43.787" UserId="601" />
  <row Id="11879" PostId="7215" Score="2" Text="Huh, I haven't comment on myself. Of course, I'll support him! On a related point, I like those &quot;compulsive&quot; edits that make CV so nice looking (on tex.SE, questions/title are rarely reformatted--so strange for a site promoting a correct application of typographical rules...)" CreationDate="2011-02-18T17:55:46.140" UserId="930" />
  <row Id="11880" PostId="7342" Text="Hi Scott, I haven't actually used the `aggregate.zoo` function, though I have used the `zoo` package. Did you make sure that your object was a `zoo` object first? The documentation that I pointed to should help you there." CreationDate="2011-02-18T17:58:30.170" UserId="401" />
  <row Id="11881" PostId="7378" Text="@mpiktas I'll try to clarify my question with a response." CreationDate="2011-02-18T18:57:29.430" UserId="919" />
  <row Id="11882" PostId="7378" Text="@whuber my take away from the answer is that a correlation based on non-stationary data yields a random variable, which may or may not be useful. Correlation based on stationary data converges to a constant. This may explain why traders are attracted to &quot;x-day rolling correlation&quot; because the correlated behavior is fleeting and spurious. Whether &quot;x-day rolling correlation&quot; is valid or useful is for another question." CreationDate="2011-02-18T19:12:42.583" UserId="3306" />
  <row Id="11883" PostId="7306" Text="@JVM Can you explain how the getmeans function works, and why you didn't just use the mean or colMeans functions?" CreationDate="2011-02-18T19:14:49.053" UserId="2770" />
  <row Id="11884" PostId="7379" Text="If the result seems similar to the input, it's only a coincidence.  The equation is just a reworded formula for the standard error of a proportion, as is shown halfway down at http://en.wikipedia.org/wiki/Margin_of_error" CreationDate="2011-02-18T19:37:30.310" UserId="2669" />
  <row Id="11885" PostId="7306" Text="The ddply() function cuts the original dataset into subsets defined by hosts and hour. It then passes these to getmeans() as a data.frame. For your task, using colMeans() would probably work just fine, but you would probably need to first remove the columns you don't need. The nice thing about using ddply() this way is that you can calculate any arbitrary stat for which you might be interested; e.g., sd(), range(), etc. " CreationDate="2011-02-18T19:39:05.803" UserId="3265" />
  <row Id="11886" PostId="7382" Text="@whuber it is good that your answer points that out but I wouldn't say the process are correlated, I would say they are dependent. This is the point. Calculation of correlation is valide and here it will say &quot;no correlation&quot; and we all know this does not mean &quot;no dependence&quot;. " CreationDate="2011-02-18T19:48:16.620" UserId="223" />
  <row Id="11887" PostId="7382" Score="1" Text="@robin That's a good point, but I constructed this example specifically so that for potentially long periods of time these two processes are *perfectly* correlated.  The issue is not one of dependence versus correlation but inherently is related to a subtler phenomenon: that the relationship between the processes changes at random periods.  That, in a nutshell, is exactly what can happen in real markets (or at least we ought to worry that it can happen!)." CreationDate="2011-02-18T19:56:18.037" UserId="919" />
  <row Id="11888" PostId="7359" Text="@cardinal - thanks! this really helps!" CreationDate="2011-02-18T20:10:50.147" UserId="36" />
  <row Id="11889" PostId="7359" Text="If you have a reference other than the Seber/Lee textbook (not accessible to me) I would love to see a good explanation of how variation explained (i.e. 1-SSerr/SStot) differs from the squared correlation coefficient, or variance explained. Thanks again for the tip." CreationDate="2011-02-18T21:25:42.593" UserId="36" />
  <row Id="11890" PostId="7264" Text="@teucer I have updated my answer, I will be interested to know if it works." CreationDate="2011-02-18T22:02:06.650" UserId="1381" />
  <row Id="11891" PostId="7264" Text="@David Thx! Actually the both approaches work as outlined in [link](http://www.stanford.edu/group/mapss/colloquium/papers/hierarchicalJackman.pdf) (page 39, second paragraph). Your solution is obviously more eficient..." CreationDate="2011-02-18T23:07:41.790" UserId="1443" />
  <row Id="11892" PostId="7364" Score="1" Text="@gappy, is that really the convergence statement you're interested in? A variance bound of $O(n^{-2})$ seems pretty strong. Also, what do you mean by &quot;closed form&quot; when you say that for the $\Sigma = I$ case, one gets a &quot;closed form&quot; solution? Thanks." CreationDate="2011-02-18T23:16:10.097" UserId="2970" />
  <row Id="11893" PostId="7379" Text="If the output seems similar to the accuracy, that means the experiment is unreliable, which worrying and the whole point of the exercise. This only happens when accuracy is low and number of trials is small." CreationDate="2011-02-18T23:51:40.883" UserId="2456" />
  <row Id="11894" PostId="7318" Text="The data is not normal in various ways. Some are skewed (Poisson type distributions), some are like the combination multiple normal distributions, several just have a few extreme outliers. The issue is that I have been asked to specifically test for normality (I use the Chi-squared goodness of fit test). The data strongly reject the null hypothesis of normal distribution. Transformation is not an option - and in many cases doesn't work anyway. I agree this should be something to try, but I have been disallowed this option, not my choice. Thanks though!" CreationDate="2011-02-19T01:25:27.757" UserId="3285" />
  <row Id="11895" PostId="7188" Text="I strongly recommend you to check wordnet, and yes is the address you posted. And I am glad you posted that question, It looks like what I recommended worked." CreationDate="2011-02-19T02:54:21.760" UserId="1808" />
  <row Id="11897" PostId="7377" Text="Sorry to pull the rug out :) I want the enter key to give a carriage return, but it submits my comment instead. Anyhow, thanks for your help - I've been enjoying myself tinkering and digging back through the Rcpp mailing list all day today. " CreationDate="2011-02-19T04:50:53.900" UserId="26" />
  <row Id="11898" PostId="7391" Score="1" Text="@David, I'm not sure about the two-sample problem, but as for references for the former, see:&#xA;A. P. Dempster, [Generalized $D_n^+$ Statistics](http://projecteuclid.org/euclid.aoms/1177706275), *Ann. Math. Statist.* Volume 30, Number 2 (1959), 593-597. Also, [here](http://www.springerlink.com/content/ln8hj5273837522p/), [here](http://www.springerlink.com/content/mrx551586853v724/) and [here](http://projecteuclid.org/euclid.bsmsp/1200514106) might have some material of interest. The first and last links should be to publicly available articles. Sadly, the middle two aren't." CreationDate="2011-02-19T05:42:19.097" UserId="2970" />
  <row Id="11900" PostId="7382" Text="@whubert yes, and this is a very good example showing that there are processes that have very high correlation for potentially long periods of time and still are not correlated at all (but highly dependent)  when regarding the larger temporal scale. " CreationDate="2011-02-19T07:47:56.307" UserId="223" />
  <row Id="11901" PostId="7389" Text="@levon9 I have reformulated the title to make it fit with current policy (sentence capitalization, etc.); please check if I didn't alter its original meaning." CreationDate="2011-02-19T08:30:57.187" UserId="930" />
  <row Id="11902" PostId="7315" Text="It's a trope of cause. _This_ part is not to prove, I hoped you understand." CreationDate="2011-02-19T08:31:41.473" UserId="3277" />
  <row Id="11907" PostId="7379" Text="@whuber, @rolando2 Make answers, not comments (-;" CreationDate="2011-02-19T11:36:17.640" UserId="88" />
  <row Id="11909" PostId="7386" Score="2" Text="That's a very good news. You already contributed a lot of particularly good answers and comments, and I'm sure your experience will be really appreciated here on CV." CreationDate="2011-02-19T14:58:04.703" UserId="930" />
  <row Id="11910" PostId="7386" Score="1" Text="Fantastic!  Thanks @whuber!  Now we have three great nominees." CreationDate="2011-02-19T15:03:19.750" UserId="5" />
  <row Id="11911" PostId="7388" Text="@Matthew - I probably won't be able to help you, but knowing what OS you're running, software version, and any other specific details of your system that may be relevant will help those than can help you provide some insight." CreationDate="2011-02-19T15:19:02.010" UserId="696" />
  <row Id="11912" PostId="7395" Text="Thanks. Can you provide a quotable source for this?" CreationDate="2011-02-19T19:10:19.417" UserId="977" />
  <row Id="11913" PostId="7402" Score="1" Text="See [Wikipedia article 'Statistical power'](http://en.wikipedia.org/wiki/Statistical_power)" CreationDate="2011-02-19T21:01:10.990" UserId="449" />
  <row Id="11914" PostId="7395" Text="It seems to be standard in education, e.g. http://books.google.co.uk/books?id=3LhPwUhrVIcC&amp;pg=PA397 or http://www.umich.edu/~exphysio/MVS250/SpearmanRankCorr.doc " CreationDate="2011-02-19T21:54:51.313" UserId="2958" />
  <row Id="11915" PostId="7384" Text="@caracal Nice response, plenty in there that I'm not familiar with, so I need to try it out. Still, looking at my data with your methods, I'm thinking I need to show the high points in my data as well.  Thanks" CreationDate="2011-02-19T22:16:16.667" UserId="2770" />
  <row Id="11916" PostId="7402" Text="I would rephrase this question as &quot;how do I find the power of a general test, such as $H_{0}:\mu=\mu_{0}$ versus $H_{1}:\mu &gt; \mu_{0}$?&quot;  This is often the more frequently performed test.  I don't know how one would calculate the power of such a test." CreationDate="2011-02-20T00:24:24.263" UserId="2392" />
  <row Id="11917" PostId="7409" Text="That shows there is only a little difference when the population is big.  It is worth noting that there is a large difference when it is small: with 600 [phyper(35, 240, 360, 100)] gives about 0.157 while an even more extreme example of 110 [phyper(35, 44, 66, 100)] gives about 0.001" CreationDate="2011-02-20T00:35:09.190" UserId="2958" />
  <row Id="11918" PostId="7409" Text="@Henry Good catch, edited my answer to stress the importance of big $n$." CreationDate="2011-02-20T00:44:55.180" UserId="1909" />
  <row Id="11919" PostId="7389" Text="@chl - couldn't find a way to message directly (still learning my way around here) - could you please point me to the policy? I found the part about it not being &quot;just&quot; luck somewhat relevant given that these are stochastic processes and therefore non-deterministic.  Thanks." CreationDate="2011-02-20T02:01:02.800" UserId="3311" />
  <row Id="11920" PostId="7393" Text="Thanks for the paper reference, I'll check it out. Without having read the paper though, let me restate that I too am going to use benchmark functions (the set I mentioned above) .. I just want to be able to somehow quantify that my (hopefully) better numbers are not just due to &quot;luck&quot; but statistically significant. I.e., I have seen people report just numbers (such as time to execute or number of evaluations), but I am not sure that's sufficient. Be curious about other comments. Thanks." CreationDate="2011-02-20T02:03:20.347" UserId="3311" />
  <row Id="11921" PostId="7405" Text="Hi, its nice paper, thanx for giving pdf link.. I'll surely go through this paper.." CreationDate="2011-02-20T03:52:11.453" UserId="3325" />
  <row Id="11922" PostId="7409" Text="Thanks caracal and Henry. It's really helpful as a beginner to have a few hints about where you're heading." CreationDate="2011-02-20T04:00:58.437" UserId="3317" />
  <row Id="11924" PostId="7392" Text="I made sure to estimate robust standard errors. Also, in the book &quot;Mixed effects models and extensions in ecology with R&quot; by Zuur, et al, 2009, on page 261, they mention, &quot;if the mean of the response variable is relatively large, ignoring the truncation problem, then applying a Poisson or negative binomial (NB) generalised linear model (GLM), is unlikely to cause a problem.&quot; Fortunately, the means of my response variables are large, so I feel a little more comfortable deprioritizing zero-truncation compared to the GEE and negbinomial aspects of my regressions." CreationDate="2011-02-20T07:28:34.280" UserId="3309" />
  <row Id="11925" PostId="7392" Text="Sounds like you already know more about this topic than I do! Or anyone else on this site, judging by the lack of other responses." CreationDate="2011-02-20T08:29:41.247" UserId="449" />
  <row Id="11926" PostId="7392" Text="It is a little unbelievable; who knew that overdispersed longitudinal count data would be so difficult to analyze (without doing a GLMM, which I haven't even looked into doing yet)? If only my data were zero-inflated, that would be another story." CreationDate="2011-02-20T08:46:25.727" UserId="3309" />
  <row Id="11927" PostId="363" Text="There are really three separate questions here! 1) What is the single most influential book in statistics; 2) What book should every statistician read; 3) What book have you read that you most wish you'd read much earlier. (2) and (3) probably have considerable overlap; (1) may be quite distinct." CreationDate="2011-02-20T08:50:11.760" UserId="449" />
  <row Id="11928" PostId="5319" Text="As far as I can tell, this application of align.time after using endpoints is what Adal wanted (apart from regarding his mention of taking &quot;the previous one&quot; in the original question). Anyway, it is what I wanted, so thanks, Joshua." CreationDate="2011-02-20T08:54:35.910" UserId="3329" />
  <row Id="11929" PostId="7389" Score="1" Text="@levon9 Ok, I reverted the title back to the original one. About general policy on title and question wording, please refer to meta, [Is there a style guide that provides guidelines for question title and question content?](http://meta.stats.stackexchange.com/questions/575/is-there-a-style-guide-that-provides-guidelines-for-question-title-and-question-c)." CreationDate="2011-02-20T09:16:17.863" UserId="930" />
  <row Id="11930" PostId="7399" Score="2" Text="@mpiktas That's a good news! I'll second this nomination." CreationDate="2011-02-20T11:13:23.023" UserId="930" />
  <row Id="11931" PostId="487" Text="Thats a wonderful article, written in Cohen's lucid and conversational style." CreationDate="2011-02-20T15:30:17.650" UserId="656" />
  <row Id="11932" PostId="7418" Text="The pscl package, I believe, only fits zero-inflated and hurdle models. Hurdle models incorporate both a left-truncated count component and a right-censored hurdle component. I don't how or even if I am able to run a hurdle model without the hurdle component, but I will look into the sandwick package. As for the geepack package, it seems to have the same problem as gee package; when I specify a &quot;negative.binomial&quot; family (from MASS), without specifying a theta, it will ask for a theta. However, when I specify a theta value, it will spit out an error saying it's an unrecognized family." CreationDate="2011-02-20T15:52:18.797" UserId="3309" />
  <row Id="11933" PostId="7418" Text="@Casey - sorry I misread your requirements re zero truncation. Shame that geepack doesn't work with that family function. If I think of anything else, I'll update here." CreationDate="2011-02-20T16:56:33.127" UserId="1390" />
  <row Id="11934" PostId="7182" Text="@Mariana: Well done thank you!" CreationDate="2011-02-20T17:07:20.457" UserId="2040" />
  <row Id="11935" PostId="7418" Text="@Gavin - Thanks for your thoughts." CreationDate="2011-02-20T17:46:52.237" UserId="3309" />
  <row Id="11936" PostId="7423" Text="Now that I re-read your post, I'm unsure what you were asking. The code I provided computes a volatility and Peak hour to avg. hour ratio score for each hour and each category, collapsing across days (eg. the volatility for email in hour 0 is computed using the hour 0 data for email from all days).  " CreationDate="2011-02-20T18:35:50.127" UserId="364" />
  <row Id="11937" PostId="7175" Text="@celenius Are you looking for further clarifications or are you unhappy with @mariana's response? I guess it concerns your very first question (2nd §). If this is the case, maybe you should update your question so that people understand why you're setting a bounty on this question." CreationDate="2011-02-20T18:58:55.717" UserId="930" />
  <row Id="11938" PostId="7175" Text="@chl I will update it to make it clearer. I'm just looking for some guidance on interpreting the clustering comparisons, as don't understand what the output means. @mariana's response was very helpful explaining some of the terms associated with this method." CreationDate="2011-02-20T19:15:03.977" UserId="2635" />
  <row Id="11939" PostId="7423" Text="well this looks like a good way but what it means is first aggregate the entire 10 day period into one single 24 hour period (for each category by summing the volumes). And then for each category on this aggregated 24 hour period measure the volatility and the Peak to Avg ratio for each &quot;Category&quot;" CreationDate="2011-02-20T19:45:14.873" UserId="2101" />
  <row Id="11940" PostId="7423" Text="@mike I've added a sample output in the question :)." CreationDate="2011-02-20T19:52:22.980" UserId="2101" />
  <row Id="11941" PostId="7422" Text="Does the identical hour on different days count as the same hour or not? If an hour has no use of particular application, do you want that to be counted as 0 for the standard deviation and average calculation?  When you say &quot;Ratio of volume of the maximum hour to the vol. of the average hour&quot;, I presume *vol.* means *volume* rather than *volatility*.   " CreationDate="2011-02-20T21:49:52.210" UserId="2958" />
  <row Id="11942" PostId="7425" Text="@bayer How do I go about examining a cluster fit using a minimax approach? My knowledge level of clustering is very basic, so at the moment I'm just trying to understand how to compare two different clustering approaches." CreationDate="2011-02-20T21:56:29.207" UserId="2635" />
  <row Id="11943" PostId="7422" Text="@Henry yes, what it should do is aggregate all the hours of all the days by adding the volumes per category so all the corresponding hourly volumes and added into one representing all the days in one single 24 hour period. then perform the calculations for each Catgory giving me the result in the form shown." CreationDate="2011-02-20T22:07:23.897" UserId="2101" />
  <row Id="11944" PostId="7422" Text="@Henry and yes vol. is for volume and if the category is not present in any hour it is 0 (though there are no such cases in the actual data :)." CreationDate="2011-02-20T22:08:24.690" UserId="2101" />
  <row Id="11945" PostId="7425" Text="@bayer Could you please share the R code for the attached figure?" CreationDate="2011-02-20T22:14:52.233" UserId="609" />
  <row Id="11946" PostId="7428" Text="@Andrej @Bernd Thank you for your time. I see that Cochrane handbook, Egger's paper(PMID9310563) and MIX2.0 software use different funnelplots, but they all agree that effect size on the horizontal axon and a meassure of sample size on the vertical axon. I don't have means and standard deviations of the groups (patients, controls) of each study. I have only the effect size, and the sample size n1,n2 of patients and controls. So I cannot calculate SE of the effect sizes to use them in the linear regression. Could I use only a funnel plot with samplesize on vertical and effectsize on horizontal?" CreationDate="2011-02-20T23:14:12.287" UserId="3333" />
  <row Id="11947" PostId="7428" Text="@Staty Despair: Please be more specific about what type of effect size you have (odds ratio, risk ratio...). Do you know the [Practical Meta-Analysis Effect Size Calculator](http://www.campbellcollaboration.org/resources/effect_size_input.php)? This tool might help you to get the standard errors. " CreationDate="2011-02-20T23:31:33.777" UserId="307" />
  <row Id="11948" PostId="7389" Score="1" Text="@levon9 General idea is to make title show what the question is about, preferably using proper words. The sentence capitalization policy is just to establish some level of order (people do spit less on clean floors)." CreationDate="2011-02-20T23:59:33.373" UserId="88" />
  <row Id="11949" PostId="7425" Text="@Andrej My guess is a Gaussian cloud (`x&lt;-rnorm(N);rnorm(N)-&gt;y`) split into 3 parts by r (with one of them removed)." CreationDate="2011-02-21T00:03:43.277" UserId="88" />
  <row Id="11950" PostId="7428" Text="@Bernd @Andrej Thanks again. I don't know what type is the effect size (SMD, OR, RR etc). I only know that can take values from -1 to +1, to show the size and the direction of the difference. The variable compared between patient and control groups is a typical scale variable, but no means and standard deviations available from the included studies. The effect size is called SDM value (www.sdmproject.com). I know (automatically calculated) the effect size from each included study, but not the formula used for the calculation. I also now sample sizes n1, n2 of patients,controls in each study." CreationDate="2011-02-21T00:07:34.810" UserId="3333" />
  <row Id="11951" PostId="7428" Text="@Bernd this website includes probably the most complete and well organized collection of online calculators, ...but still not helpful in my case :(" CreationDate="2011-02-21T00:18:04.090" UserId="3333" />
  <row Id="11952" PostId="7428" Text="@Staty Despair: I am sorry but I do not have any experience with this type of analysis. You might want to check this paper: [Voxel-wise meta-analysis of grey matter changes in obsessive–compulsive disorder](http://bjp.rcpsych.org/cgi/reprint/195/5/393). " CreationDate="2011-02-21T00:24:38.563" UserId="307" />
  <row Id="11953" PostId="7424" Text="@onestop, Thanks for your answer. I guess I am looking for stratified sampling. (I was looking for algorithms, which are not computationally very expensive, as not parsing the entire population, to make a representative set, does not even make sense. :-))" CreationDate="2011-02-21T01:03:15.673" UserId="3292" />
  <row Id="11954" PostId="7428" Text="@Bernd Yes, this is one of the 5 papers using this method of meta-analysis. The other 4 are (Pubmed, PMIDs): 19699306, 20603451, 21078227, 21300524. Despite the analytical description in methods section of the papers, it is still difficult to decide the category of the effect size used (SMD, RR, OR, CC), to understand how it is calculated, and calculate the Standard Errors needed to perform Egger test and synthesise a funnel plot! Thank you for your time..." CreationDate="2011-02-21T01:20:54.560" UserId="3333" />
  <row Id="11955" PostId="7412" Text="http://getthedata.org/ bills itself as a question and answer site for finding data" CreationDate="2011-02-21T03:02:11.273" UserId="183" />
  <row Id="11956" PostId="7433" Text="Thanks.  I'm in the need of hand holding case.  As such answers directing me to other tools (read R etc) aren't really what I'm after.  I'm interested in SPSS users feedback." CreationDate="2011-02-21T03:47:49.587" UserId="3189" />
  <row Id="11957" PostId="7434" Score="1" Text="just adding to @JMS answer, a great reference to check out about is **Bayesian Inference in Statistical Analysis** by Box and Tiao. It presents conceptual ideas behind it too." CreationDate="2011-02-21T06:20:27.790" UserId="1307" />
  <row Id="11958" PostId="7429" Text="can you please be more specific about what you are doing exactly? I think you are missing something here as @dsimcha mentions." CreationDate="2011-02-21T06:23:05.913" UserId="1307" />
  <row Id="11959" PostId="7412" Score="1" Text="@Jeromy thanks, posted http://getthedata.org/questions/399/where-can-i-get-real-data-of-big-network-topology" CreationDate="2011-02-21T06:43:05.823" UserId="3328" />
  <row Id="11960" PostId="7" Score="1" Text="You might like http://getthedata.org/ a question and answer site dedicated to finding data sets" CreationDate="2011-02-21T07:02:17.727" UserId="183" />
  <row Id="11961" PostId="7382" Text="@whuber, @robin girard, I think the key here is that for non-stationary processes the theoretical correlation varies with time, when for the stationary processes theoretical correlation stays the same. So with sample correlation which basically is one number, it is impossible to capture the variation of true correlations in case of non-stationary processes." CreationDate="2011-02-21T07:09:28.057" UserId="2116" />
  <row Id="11962" PostId="7433" Score="1" Text="@Shane, in general it is useful to check out how other tools are approaching the same problem. It is the same as reading different books on the same topic, every author has its unique perspective, combining them gives you better understanding. Yes it is a slow process, but if you want to trily understand something this is  in my opinion the only way." CreationDate="2011-02-21T07:15:56.253" UserId="2116" />
  <row Id="11963" PostId="7428" Text="@Staty Despair Consider to use R with meta or rmeta packages. It's much easier than do it from scratch." CreationDate="2011-02-21T07:41:13.827" UserId="609" />
  <row Id="11964" PostId="7425" Text="I don't know of a practical algorithm that fits according to that quality measure. You probably still want to use K-Means et al. But if the above measure breaks down, you know that the data you are looking at is not (yet!) suitable for that algorithm." CreationDate="2011-02-21T07:43:41.793" UserId="2860" />
  <row Id="11965" PostId="7425" Text="@Andrej I don't use R (coming from ML rather than stats :) but what mbq suggests seems fine." CreationDate="2011-02-21T07:45:37.580" UserId="2860" />
  <row Id="11966" PostId="7364" Text="@gappy, I reviewed the Wooldridge book and found nothing about factor models, could you please provide page number?" CreationDate="2011-02-21T08:25:16.307" UserId="2116" />
  <row Id="11967" PostId="7437" Score="1" Text="@Joseph As is, this looks like a quizz! Tell us what you already tried, or if you have some difficulties understanding the statistical concept of [sampling distribution](http://en.wikipedia.org/wiki/Sampling_distribution)." CreationDate="2011-02-21T09:27:30.517" UserId="930" />
  <row Id="11968" PostId="7437" Text="Yes it is a quiz and poorly formatted one. If you want help please put in some work in asking the question." CreationDate="2011-02-21T09:30:38.597" UserId="2116" />
  <row Id="11969" PostId="7437" Text="The first [link](http://www.intmath.com/counting-probability/14-normal-probability-distribution.php) in google search on &quot;probabilities for Standard Normal Distribution&quot; answers all the questions together with explanations. Voting to close." CreationDate="2011-02-21T09:37:08.887" UserId="2116" />
  <row Id="11970" PostId="7437" Score="1" Text="Sorry, but I must at least encourage you to try to understand those questions. " CreationDate="2011-02-21T10:11:23.393" UserId="88" />
  <row Id="11971" PostId="7418" Text="@Casey I've added a note about the `gamlss` package which might fit the bill in R also." CreationDate="2011-02-21T10:27:25.863" UserId="1390" />
  <row Id="11972" PostId="7439" Text="Please provide the example of what you want to achieve. What exactly you do not get?" CreationDate="2011-02-21T10:32:20.313" UserId="2116" />
  <row Id="11973" PostId="7439" Score="3" Text="Here is [my blog post](http://vzemlys.wordpress.com/2010/01/06/time-series-data-aggregation-using-r-and-reshape/) with example of using `melt` and `cast`. There conversion from wide to long format is done at one stage. There really isn't anything more special." CreationDate="2011-02-21T10:35:43.057" UserId="2116" />
  <row Id="11974" PostId="7440" Text="please fix your latex, use $\log$ instead of $log$." CreationDate="2011-02-21T10:51:55.377" UserId="2116" />
  <row Id="11977" PostId="7447" Text="@mbq, Thanks for the edits. They are much appreciated. As you can probably see, I am new to statistics." CreationDate="2011-02-21T12:01:21.157" UserId="3292" />
  <row Id="11978" PostId="7440" Text="@bayer, sorry for posting the incorrect answer in the first place. I just looked at $x-\mu_1$ and immediately thought that the integral is zero. The point that it was squared completely missed my mind :)" CreationDate="2011-02-21T12:02:03.823" UserId="2116" />
  <row Id="11979" PostId="7448" Text="@Henry, Thanks for the reply. It was very helpful! :)" CreationDate="2011-02-21T12:02:39.500" UserId="3292" />
  <row Id="11980" PostId="844" Score="2" Text="@svadali it is is more usual to use ratio here since the distribution of ratio of chi square is tabulated (Fisher's F). However, the problematic part of the question (i.e. the dependency between $X$ and $Y$) is still there whatever you use. It is not straightforward to build up a test with two dependent chi square... I tryed to give an answer with a solution on that point (see below). " CreationDate="2011-02-21T12:19:39.930" UserId="223" />
  <row Id="11981" PostId="7439" Text="Welcome to stats.  You might find it helps to include a small, reproducible dataset in your question to explain what you want.  Read http://www.sigmafield.org/2011/01/18/three-tips-for-posting-good-questions-to-r-help-and-stack-overflow for more." CreationDate="2011-02-21T12:45:34.063" UserId="114" />
  <row Id="11982" PostId="7428" Text="@Andrej No experiance with R, unfortunatelly. But I believe that this analysis cannot be done by using the standerd R rmeta pachages, because the dimention of spatial distribution has to be taken into account. Nevertheless, I already have the pooled effect size, my question is how to calculate the standard errors of the sdm values (=effect size) to use them to explore for publication bias, by doing the funnel plot and the Egger test." CreationDate="2011-02-21T13:28:40.033" UserId="3333" />
  <row Id="11983" PostId="7386" Score="2" Text="I think that behind the score there is also a constant will of @whubert to create and generate constructive connexions and exchanges. impressive !" CreationDate="2011-02-21T13:34:01.960" UserId="223" />
  <row Id="11985" PostId="7346" Text="I guess &quot;contribute&quot; is not strong enough, maybe 24h/24 devotion is close to reality? really of great value to the community." CreationDate="2011-02-21T13:49:52.970" UserId="223" />
  <row Id="11986" PostId="7428" Text="@Staty Despair: I think we reached a point where you need to contact the authors. This looks promising: &quot;For this analysis, we extracted SDMvalues [...] for each study and calculate the standard error of SDM values based on sample size of patient and controls groups&quot; [(Bora et al 2011)](http://www.sciencedirect.com/science?_ob=ArticleURL&amp;_udi=B6TC2-5243NDH-1&amp;_user=2875156&amp;_coverDate=02%2F06%2F2011&amp;_rdoc=1&amp;_fmt=high&amp;_orig=search&amp;_origin=search&amp;_sort=d&amp;_docanchor=&amp;view=c&amp;_acct=C000056617&amp;_version=1&amp;_urlVersion=0&amp;_userid=2875156&amp;md5=5023f799be7dee0cb0e0a7f1d651f930&amp;searchtype=a)." CreationDate="2011-02-21T14:14:47.170" UserId="307" />
  <row Id="11988" PostId="7448" Text="+1 Nice summary.  I like that you point out the role of the loss function first." CreationDate="2011-02-21T14:21:41.357" UserId="919" />
  <row Id="11990" PostId="7400" Text="What does &quot;similar&quot; mean?  The chi-squared test and the KS test, for example, test whether two histograms are close to *identical.*  But &quot;similar&quot; might mean &quot;have the same shape,&quot; *ignoring any differences of location and/or scale.*  Could you clarify your intent?" CreationDate="2011-02-21T14:32:42.663" UserId="919" />
  <row Id="11991" PostId="7410" Text="You are correct that the KS test is not appropriate for histograms when it is understood as a hypothesis test about the distribution of the underlying data, but I see no reason why the KS *statistic* wouldn't work well as a measure of sameness of any two histograms." CreationDate="2011-02-21T14:34:30.337" UserId="919" />
  <row Id="11992" PostId="7401" Text="Subtracting the mean changes the null distribution of the KS statistic.  @David Wright raises a valid objection to the application of the KS test to histograms anyway." CreationDate="2011-02-21T14:36:12.940" UserId="919" />
  <row Id="11994" PostId="7428" Text="@Bernd Unbelievable, but I've already conducted the authors asking for definition on exactly the same line you suggest, from Bora et. al paper... " CreationDate="2011-02-21T14:45:06.740" UserId="3333" />
  <row Id="11995" PostId="7389" Text="@chl - thanks for the link     " CreationDate="2011-02-21T15:58:27.553" UserId="3311" />
  <row Id="11996" PostId="7389" Text="@mbq - thanks for the comments" CreationDate="2011-02-21T15:59:50.030" UserId="3311" />
  <row Id="11998" PostId="7452" Text="Unfortunatly this paper does not say anything about the truncated distribution..I found some packages in R that deal with Zipf estimation parameter in a simply way (zipfR, VGAM) but the truncated distribution need a &quot;special treatment&quot;. With your last sentence did you mean that it is possible to model a power-law dataset with an e.g. exponential distribution and then apply some estimation parameter process for &quot;truncated&quot; exponential distribution? I'm very newbie in this topic!" CreationDate="2011-02-21T16:27:00.653" UserId="3342" />
  <row Id="11999" PostId="7452" Text="In the paper, the authors re-analyses different data sets where a power-law has been fitted. The authors point out that in a number of cases the power-law model isn't that great and an alternative distribution would be better." CreationDate="2011-02-21T16:29:04.590" UserId="8" />
  <row Id="12001" PostId="7450" Text="@Maurizio, to be clear, what precisely are you right-truncating? The distribution of values or the Zipf plot itself? Do you know the truncation point? Is the truncation an artifact of the data or an artifact of the *data processing* (e.g., some decision that you or the experimenter made)? Any additional details would be helpful." CreationDate="2011-02-21T17:00:06.460" UserId="2970" />
  <row Id="12002" PostId="7458" Text="Amazing thanks." CreationDate="2011-02-21T17:17:45.627" UserId="3348" />
  <row Id="12003" PostId="7450" Text="@cardinal. (part 1/2) Thanks cardinal. I will give more details:&#xA;I have a VoIP generator that generates calls following the Zipf (and other distribution) for volume per caller. I have to verify that this generator really follows these distributions.&#xA;For Zipf Distribution i need to define the truncation point (hence it is known and it refers to the distribution of values) which is the maximum number of generated call by the user and the scale parameter. In particular in my case this value is equal to 500, that indicates that one user can generate maximum 500 calls." CreationDate="2011-02-21T17:20:37.527" UserId="3342" />
  <row Id="12004" PostId="7450" Text="(part 2/2) The other parameter to set is the scale parameter for Zipf that defines the spread of the distribution (this value in my case is 0.9). I have all the parameters (size of sample, frequency per user, etc.) but i have to verify that my dataset follows the zipf distribution." CreationDate="2011-02-21T17:21:08.260" UserId="3342" />
  <row Id="12005" PostId="7449" Text="The first line is trivially 0 because the two terms are equal with opposite sign.  The &quot;x&quot; at the bottom of the integral signs has no meaning, either.  Copy-and-paste errors?" CreationDate="2011-02-21T17:39:07.430" UserId="919" />
  <row Id="12006" PostId="7458" Text="Side note: I think it works this way for essentially all modern spreadsheet programs - I know it does for OpenOffice.org Calc and Google Docs' spreadsheet." CreationDate="2011-02-21T17:40:24.353" UserId="2898" />
  <row Id="12007" PostId="7450" Text="@Maurizio, so you're apparently renormalizing the distribution by $\sum_{i=1}^{500} i^{-0.9}$, since for, what I would think of as a &quot;truncated Zipf&quot;, a scaling parameter of 0.9 would be impossible. If you can generate lots of these data and you &quot;only&quot; have 500 possible outcomes, why not just use a chi-square goodness-of-fit test? Since your distribution has a long-tail, you may need a pretty large sample size. But, that would be one way. Another quick-and-dirty method would be to check that you get the right empirical distribution for *small* values of the number of calls." CreationDate="2011-02-21T17:40:51.560" UserId="2970" />
  <row Id="12008" PostId="7449" Text="@whuber, yes copy-paste. The same error is in the OP, will fix it." CreationDate="2011-02-21T17:58:16.557" UserId="2116" />
  <row Id="12009" PostId="7449" Text="It's much clearer now, thanks." CreationDate="2011-02-21T18:05:21.287" UserId="919" />
  <row Id="12010" PostId="7459" Text="+1, great answer as usual. You should nominate yourself as a moderator, there is still 1 hour left :)" CreationDate="2011-02-21T18:20:46.940" UserId="2116" />
  <row Id="12011" PostId="7459" Text="@mpiktas, thanks for the compliments and the encouragement. I'm not sure I could justify nominating myself with the already very strong slate of candidates who have, uniformly, participated more extensively and for longer than I have." CreationDate="2011-02-21T18:49:01.287" UserId="2970" />
  <row Id="12012" PostId="7458" Score="2" Text="Good answer--but it (and the original question) belong on a different site." CreationDate="2011-02-21T19:04:08.493" UserId="919" />
  <row Id="12013" PostId="7458" Text="On a windows machine, you can toggle through the &quot;lock down&quot; methods with the F4 key. " CreationDate="2011-02-21T19:11:46.640" UserId="696" />
  <row Id="12014" PostId="7098" Text="JMP is one of the nicest stats applications out there. Statisticians should definitely learn how to use it, if only to impress their clients. It's expensive, but cheap if you're a student or staff member at a school/college/uni" CreationDate="2011-02-21T19:33:10.370" UserId="74" />
  <row Id="12015" PostId="7439" Text="It was a general question, but I'll edit my question as soon as I can." CreationDate="2011-02-21T19:56:13.277" UserId="3140" />
  <row Id="12016" PostId="7459" Text="@cardinal, here are some links to alternative to Hill's estimator: [original article](http://www.springerlink.com/content/t411403874770047/) by Paulauskas and follow-ups by [Vaiciulis](http://www.springerlink.com/content/4388079545358pn2/) and [Gadeikis and Paulauskas](http://www.springerlink.com/content/f372367226545j25/). This estimator supposedly had better properties than original Hill's." CreationDate="2011-02-21T20:08:46.873" UserId="2116" />
  <row Id="12017" PostId="7466" Text="@Max, this looks like multiple change-point problem. I found this [link](http://genome.jouy.inra.fr/ssb/preprint/SSB-RR-12-cart.pdf) which might be helpful. I hope someone will provide more details." CreationDate="2011-02-21T20:14:44.023" UserId="2116" />
  <row Id="12018" PostId="7466" Text="@mpiktas, thanks, it seems that &quot;change-point&quot; is the term that I was looking for." CreationDate="2011-02-21T20:27:56.343" UserId="255" />
  <row Id="12019" PostId="7182" Text="I am really happy it was helpfull for you." CreationDate="2011-02-21T20:46:48.777" UserId="1808" />
  <row Id="12021" PostId="7216" Score="1" Text="here is a 5-part video series: http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-loading.html" CreationDate="2011-02-21T20:52:35.120" UserId="74" />
  <row Id="12022" PostId="7468" Text="The change-point algorithms seem to be able to determine the optimal number of clusters. Is there any way to determine when to stop before single point is left?" CreationDate="2011-02-21T21:37:00.577" UserId="255" />
  <row Id="12023" PostId="7459" Text="Excellent, detailed answer. Thanks" CreationDate="2011-02-21T22:27:24.553" UserId="8" />
  <row Id="12025" PostId="6542" Score="5" Text="I disagree, Jaynes' book is a terrible recommendation in this circumstance: 1) the notation is sloppy and non-standard, which makes it difficult to cross reference with other sources, 2) he's long winded and gets bogged down in silly and irrelevant arguments (the OP asked for the &quot;shortest route&quot;) 3) there's also the errors (such as the marginalization paradox)" CreationDate="2011-02-21T22:56:01.087" UserId="495" />
  <row Id="12026" PostId="6543" Score="1" Text="+1 All of Statistics: it would be a great place to start." CreationDate="2011-02-21T23:20:17.930" UserId="495" />
  <row Id="12027" PostId="7459" Text="@mpiktas, thanks for the links. There are quite a few &quot;new-and-improved&quot; versions of the Hill estimator. The main drawback of the original approach is that it requires a choice of &quot;cutoff&quot; on where to stop averaging. I think mostly that's been done by &quot;eyeballing&quot; it which opens one up to charges of subjectivity. One of Resnick's books on long-tailed distributions discusses this in some detail, if I recall. I think it's his more recent one." CreationDate="2011-02-21T23:30:42.820" UserId="2970" />
  <row Id="12028" PostId="7465" Text="@Neil McGuigan, this seems like a bit of an odd list. To my knowledge, Carnegie Mellon is the *only* school on that list that (formally) offers an undergraduate degree in statistics. Neither Caltech nor MIT even have graduate programs in statistics." CreationDate="2011-02-22T00:07:37.110" UserId="2970" />
  <row Id="12030" PostId="7465" Text="@cardinal. why must you doubt me? :) I put in links to the undergrad stats courses at those fine institutions. Also, mixing and matching courses from the best schools will beat out following a degree path from a worse school. " CreationDate="2011-02-22T00:28:44.013" UserId="74" />
  <row Id="12031" PostId="6538" Score="1" Text="I think it's almost always important to develop domain-specific expertise as well. A lot of statistics is learning the models relevant to specific fields." CreationDate="2011-02-22T00:30:14.467" UserId="493" />
  <row Id="12032" PostId="7428" Text="@Bernd I don't close the question until receiving author's answer. As I am waiting, I am thinking constructing an alternative funnel plot, as I cannot calculate standard error, and I start a new discussion on this." CreationDate="2011-02-22T00:35:16.453" UserId="3333" />
  <row Id="12033" PostId="7465" Text="@Neil, OCW is certainly a very fine resource and a great initiative. This is no doubting that. As for your assertion that mixing and matching from the &quot;best schools&quot; is a superior solution, I find that highly suspect, particularly for undergraduate studies. While a highly motivated student is bound to get a very good undergraduate education at any of those schools, an undergrad education as good or better can be found at many, many &quot;worse&quot; schools. Schools such as those that you list do tend to &quot;win out&quot; for graduate education, I would say." CreationDate="2011-02-22T01:40:18.837" UserId="2970" />
  <row Id="12034" PostId="7476" Text="@staty what percent of your studies lack an estimate of SE? Have you considered using any of these transformations? http://stats.stackexchange.com/q/2917/1381" CreationDate="2011-02-22T02:53:29.950" UserId="1381" />
  <row Id="12036" PostId="7481" Score="1" Text="A few questions: Is $y$ univariate? Then you have a multiple logistic regression, not a multivariate logistic regression. How are you doing the factor analysis? While there are methods applicable to ordinal variables that are analogous to factor analysis for continuous manifest variables, it isn't clear to me what you're using. Finally, what would you consider to be an inappropriate use of factor analysis? If you look on it as a dimension reduction technique (turning 11 variables into 2) then there's nothing inherently wrong with it. Whether it's a good idea or not is another question." CreationDate="2011-02-22T05:07:38.477" UserId="26" />
  <row Id="12037" PostId="5816" Text="@nico, I think @cespinoza implied programing something relevant to the hot topic re-sequencing application (a.k.a. High-Throughput sequencing, Next Generation Sequencing). Blast would not work for that. You want to look at something like [Mosaik](http://bioinformatics.bc.edu/marthlab/Mosaik), [SOAP Aligner](http://soap.genomics.org.cn/soapaligner.html), [ABySS](http://www.bcgsc.ca/platform/bioinfo/software/abyss), [Bowtie](http://bowtie-bio.sourceforge.net), [BWA](http://bio-bwa.sourceforge.net), or [Maq](http://maq.sourceforge.net)" CreationDate="2011-02-22T05:15:51.240" UserId="1794" />
  <row Id="12038" PostId="7472" Score="1" Text="Well, I'm pretty much in love with this answer.  I would like the emphasize that, in following point 4, you should keep the checks simple - there are often little incongruities in real data that your entry-folks should be able to enter but which you won't be able to foresee.  For example, I make sure that dates are entered as dates; but I've stopped having hard rules about date ranges, and instead check that with reports, and follow up out-of-bounds values with whomever did the data entry." CreationDate="2011-02-22T05:42:56.830" UserId="71" />
  <row Id="12039" PostId="7472" Text="Preventing someone from entering a value that they believe to be right can introduce more error than detecting and investigating it." CreationDate="2011-02-22T05:43:52.553" UserId="71" />
  <row Id="12041" PostId="5816" Text="@Aleksandr: of course I cannot read the mind of cespinoza... but aligning 25bp to an existing genome sounds a lot like checking where some primers bind to the DNA to me... But, again, I may be wrong... I haven't done more than a handful of PCRs in the last few years, and I moved my interests to other fields of biology ;)" CreationDate="2011-02-22T06:21:27.403" UserId="582" />
  <row Id="12042" PostId="7477" Text="@cardinal, I loved the very short story. " CreationDate="2011-02-22T07:04:22.007" UserId="2116" />
  <row Id="12043" PostId="7477" Text="@cardinal, ex-Soviet universities have separate undergraduate statistics studies. In Vilnius University for example you can get bachelor degree in statistics. From what I see with the students I wholeheartedly agree that master or even doctorate-level education is needed for interesting jobs." CreationDate="2011-02-22T07:13:29.853" UserId="2116" />
  <row Id="12044" PostId="7474" Score="2" Text="your definition for harmonic mean does not agree with [wikipedia](http://en.wikipedia.org/wiki/Harmonic_mean)" CreationDate="2011-02-22T07:20:14.997" UserId="2116" />
  <row Id="12045" PostId="7472" Text="@whuber +1, great answer. I agree with Matt, I too love this answer:) " CreationDate="2011-02-22T07:51:46.290" UserId="2116" />
  <row Id="12046" PostId="7259" Text="@Sam, you might want to ask new question, not update the old one. Also consider asking it in stackoverflow.com, since the update asks question about R." CreationDate="2011-02-22T08:00:21.537" UserId="2116" />
  <row Id="12048" PostId="7485" Text="@Marco, it is not clear what do you want to complete. Furthermore as it stands now I do not see problems with identification. Including intercept will identify the mean of $v$. Please provide more details." CreationDate="2011-02-22T08:57:47.393" UserId="2116" />
  <row Id="12049" PostId="7487" Text="Please use proper sentences and explain the problem in more detail. I've no idea what you are trying to do based on this description." CreationDate="2011-02-22T09:19:45.343" UserId="159" />
  <row Id="12050" PostId="7487" Text="@Endris, please read [faq](http://stats.stackexchange.com/faq) on how to ask the questions. As it stands now the question is unreadable and not understandable. " CreationDate="2011-02-22T09:21:28.050" UserId="2116" />
  <row Id="12051" PostId="7485" Text="@Marco, yes your update makes sense, but still it is not clear what is your question." CreationDate="2011-02-22T09:31:38.820" UserId="2116" />
  <row Id="12052" PostId="7485" Text="@mpiktas Thank you! Yes, I know my question is not very rigourous and I hesitated before posting it. But your confirmation makes me glad and I think I do not need any further explanation. Sorry for the lack of rigor!" CreationDate="2011-02-22T09:51:16.767" UserId="3019" />
  <row Id="12053" PostId="7485" Text="@Marco, I advise you to ask rigorous question, since there is a chance that I confirmed the wrong thing." CreationDate="2011-02-22T10:08:23.297" UserId="2116" />
  <row Id="12054" PostId="7485" Text="@mpiktas: promised!" CreationDate="2011-02-22T10:14:34.010" UserId="3019" />
  <row Id="12055" PostId="7477" Text="@cardinal, @mpiktas 4 years in BS + 2 yearts in MS + 4 years in PhD makes ten years to learn something interesting :) I would give $+\infty$ to this great answer if possible. Most books are new to me." CreationDate="2011-02-22T10:41:27.303" UserId="2645" />
  <row Id="12056" PostId="7459" Text="@cardinal, thank you very much, you are very kind and very detailed! Your example in R was very useful for me, but how can I perform a formal chi-square test in this case? (i used chi-square test with other distributions like uniform, exponential, normal, but i have many doubts about the zipf..Sorry but this is my first approach to these topics). &#xA;&#xA;Question to modetators: have i to write another Q&amp;A like &quot;how perform chi-square test for truncated zipf distribution?&quot; or continue in this Q&amp;A maybe updating tags and title?" CreationDate="2011-02-22T10:49:27.847" UserId="3342" />
  <row Id="12057" PostId="7459" Text="@Maurizio: In general, you should only edit your question to help clarify details - not ask new questions. This avoids answers referring to past questions. So ask a new question." CreationDate="2011-02-22T11:45:01.640" UserId="8" />
  <row Id="12058" PostId="7470" Text="@Rick Wicklin: Thank you" CreationDate="2011-02-22T12:46:37.640" UserId="2956" />
  <row Id="12059" PostId="7491" Text="@Jeromy (+1) Good that you mentioned the use of Factor scores directly (and their correspondence with raw scores under certain conditions)." CreationDate="2011-02-22T13:23:26.243" UserId="930" />
  <row Id="12060" PostId="7485" Text="@Marco , I still don't see any question. You may be interested in this Stata FAQ, http://www.stata.com/support/faqs/stat/xtreg2.html " CreationDate="2011-02-22T13:23:50.610" UserId="1036" />
  <row Id="12061" PostId="7459" Text="@Maurizio, in principle, chi-square testing in the truncated Zipf case is even easier than for a continuous distribution since you don't need to make any decisions regarding binning, as the data are already discrete. The observed counts are $O_i = n d_i$ and the expected counts are $E_i = n p_i$, where I've calculated the $p_i$ values for you in the $R$ code. From there all you have to do is plug and chug. Hope that helps. " CreationDate="2011-02-22T13:44:40.977" UserId="2970" />
  <row Id="12062" PostId="7459" Text="@cardinal, If I could I would offer you a pint of beer! In this afternoon I'll try to do some tests and i'll report my results here. Thank you very much!" CreationDate="2011-02-22T14:08:46.673" UserId="3342" />
  <row Id="12063" PostId="7483" Score="1" Text="Playing devil's advocate, one &quot;otherwise&quot; is that academics need to publish papers and get grants to be awarded tenure or to be promoted.  Also methods are often in fasion and get used more than their true value would suggests (until the next bandwagon comes along).  The majority of papers published in machine learning probably do not represent an improvement on standard methods, such as kNN or LDA or logistic regression. " CreationDate="2011-02-22T14:39:02.947" UserId="887" />
  <row Id="12064" PostId="7472" Score="1" Text="@Matt Good points, both of them.  I completely agree.  Concerning the first one, a good approach is to test the data entry procedures on a small representative subset of the data and thoroughly go over all the issues that arise.  This won't address everything that can possibly come up, but it identifies most major issues early on and lets you deal with them effectively." CreationDate="2011-02-22T14:44:16.703" UserId="919" />
  <row Id="12065" PostId="7483" Score="1" Text="@Dikran, yes you are right. But in the book I linked the overview is given in terms of practical use. Not everybody from academia publishes articles solely for purpose of promotion or tenure. " CreationDate="2011-02-22T14:45:53.700" UserId="2116" />
  <row Id="12066" PostId="7485" Score="2" Text="@Marco You seem to be combining two different representations of the same model.  You can have either that $Y = X\beta + v$ where $X$ includes a constant term and you stipulate $\mathbb{E}[v]=0$ or else you omit the constant term in $X$ and allow the expectation of $v$ to be a parameter.  Doing both, as your edit points out, is redundant, but for the trivial reason that you have effectively included the same parameter twice in the model (both as the intercept and the mean of the expectation of the residuals)." CreationDate="2011-02-22T14:49:29.970" UserId="919" />
  <row Id="12067" PostId="7493" Score="2" Text="+1 This answer is a good read due to its clarity, authority, and consistently helpful focus on responding to the questions." CreationDate="2011-02-22T14:53:15.737" UserId="919" />
  <row Id="12068" PostId="7483" Text="@mpiktas, not soley, no (I was just being cynical ;o)  It is important to realise though that the increment in performance introduced by most new methods over traditional techniques tend to be rather modest and often do not translate from evaluation on benchmark problems to real world applications.  ISTR David Hand wrote a good paper on this a few years ago, and it is well in accord with personal experience (e.g. linear SVMs and ridge regression tend to give very similar performance, with neither dominating the other - try writing a grant proposal to investigate ridge regression!)." CreationDate="2011-02-22T14:57:14.593" UserId="887" />
  <row Id="12069" PostId="7484" Text="@mpiktas This is a nice start and provides some guidance when the CV is low.  But even in practical, simple situations it is not clear that the CLT applies.  I would expect reciprocals of many variables not to have finite second or even first moments when there's any appreciable probability their values could be close to zero.  I would also expect the delta method not to apply due to the potentially large derivatives of the reciprocal near zero.  Thus it could help to more precisely characterize the &quot;simple applications&quot; where your method might work.  BTW, what is &quot;D&quot;?" CreationDate="2011-02-22T14:58:46.180" UserId="919" />
  <row Id="12070" PostId="7494" Text="This [article](http://onlinelibrary.wiley.com/doi/10.1002/bimj.200410103/abstract) will certainly clarify matters for you. Unfortunately access is restricted." CreationDate="2011-02-22T14:58:57.770" UserId="2116" />
  <row Id="12071" PostId="7484" Text="@whuber, D is for variance, $DX=E(X-EX)^2$. By simple applications I meant the ones for which variance and mean of reciprocal exists. As you say for random variables with appreciable probability that their values could be close for zero, reciprocal may not even have mean. But then the answer to original question is no. I assumed that the OP asked whether it is possible to calculate standard deviation when it exists. It clearly does not for a lot of random variables." CreationDate="2011-02-22T15:22:11.847" UserId="2116" />
  <row Id="12072" PostId="7484" Text="@whuber, BTW out of curiosity $DX$ is pretty standard notation for me, but one might say that I come from Russian probability school. It is not so common in &quot;capitalistic West&quot;? :) " CreationDate="2011-02-22T15:23:55.900" UserId="2116" />
  <row Id="12073" PostId="7484" Text="@mpiktas I have never seen this notation for variance.  My first reaction was that $D$ is a differential operator!   The standard notations are mnemonic, such as $Var[X]$." CreationDate="2011-02-22T15:31:37.967" UserId="919" />
  <row Id="12074" PostId="7496" Text="While said F-test is relevant for univariate multiple regression (one predicted variable), I could not find information about its use in multivariate multiple regression (many predicted variables analyzed simultaneously)." CreationDate="2011-02-22T15:36:04.433" UserId="1909" />
  <row Id="12075" PostId="7502" Text="@Justin some potentially useful information: what exactly are you going to be measuring, what do you expect the range of values to be, and what size change would be meaningful?" CreationDate="2011-02-22T15:40:37.550" UserId="1381" />
  <row Id="12076" PostId="7503" Score="1" Text="That agrees with my subjective prior belief that there's no such thing as a non-informative prior." CreationDate="2011-02-22T15:58:44.177" UserId="449" />
  <row Id="12077" PostId="7501" Text="Thanks ! your &quot;short answer&quot; perfectly points out an interesting existing literature." CreationDate="2011-02-22T16:07:10.073" UserId="223" />
  <row Id="12078" PostId="7500" Text="@JMS: Your answer is very nice, but you don't actually state what a &quot;non-informative prior&quot; and how it differs from a &quot;vague prior&quot;;)" CreationDate="2011-02-22T16:16:17.160" UserId="8" />
  <row Id="12079" PostId="7498" Text="@Dikran this begs the question of would the uniform be truly non-informative? what would be the max and min of the non-informative uniform? Would the prior be non-informative after transformation to another scale?" CreationDate="2011-02-22T16:16:47.020" UserId="1381" />
  <row Id="12080" PostId="7503" Text="@onestop please consider contributing to my CW answer" CreationDate="2011-02-22T16:17:36.830" UserId="1381" />
  <row Id="12081" PostId="7094" Text="I don't think this works. Assume $p_i=1/n$ and $p_{ij}=0 \forall i \ne j$. This is a valid combination of probabilities, realized when $I$ is a uniform random variable $\in\{1,...,n\}$ and $X_I=1$ and all $X_j=0 \forall j\ne I$. Still the formula above would be 0 for all events. Still thanks for helping!" CreationDate="2011-02-22T16:20:52.273" UserId="3152" />
  <row Id="12082" PostId="7503" Text="This is precisely the issue addressed in the paper I included in my answer ($IG(\epsilon, \epsilon)$ with $\epsilon \rightarrow 0$. The WinBUGS documentation used them long after these priors were discredited." CreationDate="2011-02-22T16:34:12.403" UserId="26" />
  <row Id="12083" PostId="7498" Text="Uniform priors are not necesarily un-informative and vice versa, it depends on the nature of the problem.  I think the idea of a minimally informative prior is often more realistic, using MAXENT and transformation groups to decide on a prior distribution that conveys the minimum amount of information consisten with known constraints (e.g. that the prior should be invariant to scaling).  For many problems, improper priors work well, so there isn't necessarily a need for there to be a well defined minimum and maximum." CreationDate="2011-02-22T16:37:51.487" UserId="887" />
  <row Id="12084" PostId="7506" Text="Gelman himself has moved beyond &quot;noninformative&quot; priors, see eg http://www.stat.columbia.edu/~cook/movabletype/archives/2007/07/informative_and.html and the paper I reference in my answer, and other works of his. This attitude seems increasingly prevalent among applied statisticians." CreationDate="2011-02-22T16:38:17.110" UserId="26" />
  <row Id="12085" PostId="7500" Text="@cgillespie: You're right, edited :)" CreationDate="2011-02-22T16:39:06.420" UserId="26" />
  <row Id="12086" PostId="7506" Text="@JMS Gelman et. al. 2003 made this point, and I have tried to summarize it, they seem to consider the noninformative priors for background / theoretical / heuristic purposes " CreationDate="2011-02-22T16:43:02.327" UserId="1381" />
  <row Id="12087" PostId="7500" Text="@JMS following from your last paragraph and supporting the quixotic nature of the search: if we know that we know *nothing* about the parameter in question, then we know *something* about it." CreationDate="2011-02-22T16:46:10.763" UserId="1381" />
  <row Id="12088" PostId="7500" Text="Quite! The great irony is that many who spend so much energy hand-wringing over their choice of prior don't think twice before throwing down the likelihood. A misspecified likelihood is much more liable to cause troubles, generally. But that's another question entirely..." CreationDate="2011-02-22T16:53:15.853" UserId="26" />
  <row Id="12089" PostId="7484" Text="@whuber, but D is mnemonic also, it stands for dispersion. In Russian and in Lithuanian term for variance is dispersion. I tend to use $Var$, but today old habit probably kicked in, also D is shorter to write:) It is a relief that I did not write $M$ instead  of $E$. If I remember correctly it is still used in schools in Lithuania. $M$ stands for mathematical expectation, which for some strange reason was translated to mathematical hope in Lithuanian." CreationDate="2011-02-22T16:53:55.773" UserId="2116" />
  <row Id="12091" PostId="7506" Text="Yeah, a textbook on Bayesian statistics certainly can't ignore them. I only meant to add that the trend in applied statistics since then has been to move away from these traditional choices, although work continues on &quot;objective Bayes&quot;" CreationDate="2011-02-22T16:57:12.773" UserId="26" />
  <row Id="12092" PostId="7508" Text="@jonjo: I hope you don't mind, but I converted your question into latex. One thing, should $Z_{\alpha}$ not be $Z_{\alpha/2}$ " CreationDate="2011-02-22T17:04:57.850" UserId="8" />
  <row Id="12093" PostId="1311" Text="@svadali I did not replace 365 by $n$; I replaced it by 2, which gives a simple problem we can check by hand.  At any rate, Step 2 has a typo and the reasoning in Step 3 is incorrect.  In fact, the approach is flawed because it does not consider various complications that arise with multiple &quot;collisions,&quot; such as two people sharing one birthday and another two sharing another birthday.  I get the value of 6/16 (four people, two possible birthdays, at most two per birthday) because of the 16 possible configurations, 6 have two people share one birthday and the others sharing the other birthday." CreationDate="2011-02-22T17:06:38.430" UserId="919" />
  <row Id="12095" PostId="7484" Text="@mpiktas I suspected &quot;D&quot; was mnemonic.  In the same manner you often see $\Sigma$ and $\sigma$ (Greek S and s, short for &quot;standard deviation&quot;) frequently used for sd in the statistical literature.  This is a fairly universal convention.  The story about &quot;M&quot; is funny: by that reasoning, *every* symbol used in mathematics ought to begin with an &quot;M&quot;!" CreationDate="2011-02-22T17:10:05.133" UserId="919" />
  <row Id="12096" PostId="7477" Text="I had a look at Boyd &amp; Vandenberghe, Convex Optimization as it appears to be available online for free, and I found it a bit concerning. It had no mention of any of the following phrases: &quot;conjugate gradient&quot; &quot;BFGS&quot; &quot;Broyden&quot;(of BFGS). Those are standard modern methods, so I am confused why they would not be at least mentioned." CreationDate="2011-02-22T17:22:53.117" UserId="1146" />
  <row Id="12097" PostId="7508" Text="If it is clear for folks I don't mind at all. You are right should be a 2-sided alpha." CreationDate="2011-02-22T17:35:38.483" UserId="712" />
  <row Id="12098" PostId="7477" Score="1" Text="@John Salvatier, you're correct that those methods aren't covered in this text. Then again, this strikes me as more a matter of taste, particularly since the text's main focus is not on algorithms. To wit, your concerns are directly addressed by the authors in the introduction (pg. 13). " CreationDate="2011-02-22T17:49:21.950" UserId="2970" />
  <row Id="12099" PostId="7477" Text="@John, My personal interpretation of the intent of this text is to provide the machinery necessary to *recognize* a problem as falling within the class of convex optimization problems rather than develop the algorithmic tools necessary to solve them (though some of this *is* presented) safely. Perhaps an appropriate analogy can be found in the standard treatments of linear regression. " CreationDate="2011-02-22T17:56:31.977" UserId="2970" />
  <row Id="12100" PostId="7273" Score="1" Text="Thank you for your answer. I had one further question about the `clusplot()` function. What is the relationship between the PCA values, and the clustering groups? I've read elsewhere about the link between kmeans and PCA, but I still don't understand how they can be displayed on the same bivariate graph. (Perhaps this should be a new question in itself). " CreationDate="2011-02-22T18:32:45.457" UserId="2635" />
  <row Id="12102" PostId="7511" Score="1" Text="What is a &quot;measurement&quot;?  Is it (a) a determination that the array has detected a signal; (b) a determination *for each pixel in the array* as to whether or not it has detected a signal; (c) an estimate of total number of photons reaching the array; (d) an estimate of photons reaching *each pixel* (*i.e.*, an image); or (e) something else?" CreationDate="2011-02-22T18:40:15.290" UserId="919" />
  <row Id="12103" PostId="7511" Text="@whuber: We are primarily interested to know (a) if we measured a signal, but at a later stage would be interested in (c) as well." CreationDate="2011-02-22T18:43:03.220" UserId="56" />
  <row Id="12104" PostId="7479" Text="Thanks a lot for the answer. It was really helpful. Indeed HIV had an effect on the treatment group as well as some other predictors and so i am running separate models for the two groups which includes these other predictors too.However, in discussing my findings,do i concentrate more on the model with the treatment group as my aim is to evaluate the effect of HIV on birthweight of infants? I am using SPSS 19." CreationDate="2011-02-22T18:47:17.583" UserId="3355" />
  <row Id="12105" PostId="7479" Text="I wrote: &quot;Given that you observe different coefficients for HIV, you will need to run another model which includes an interaction effect...&quot;. I would report the findings of the interaction model. However, reporting the findings of two separate models (control and treatment), can be more instructive, esp. for an audience with less statistical background.  " CreationDate="2011-02-22T18:50:52.403" UserId="307" />
  <row Id="12106" PostId="7508" Text="What are $\theta R$ and $\psi R$ ? Should they be $\theta_R$ and $\psi_R$? " CreationDate="2011-02-22T19:02:59.400" UserId="449" />
  <row Id="12107" PostId="6542" Text="@Dikran Marsupial, do you own the Schervish text on statistical inference? I've been on the fence regarding whether to purchase it or not, so was curious, since you appear to align yourself pretty strongly with the Bayesian approach. " CreationDate="2011-02-22T19:07:45.720" UserId="2970" />
  <row Id="12108" PostId="6542" Text="I wouldn't say I was strongly aligned to the Bayesian approach.  It is the approach I understand the best, which is not the same thing.  Essentially I am an engineer at heart, and I want both tools in my toolbox, maintained in good order!  A proper understanding of the benefits and disadvantages of each approach is what we should aim for.  I haven't got Shervishes book, but I did read a paper of his on Bayes factors that seemed quite flawed to me (I'll see if I can find it and post a question for someone to explain it to me!)." CreationDate="2011-02-22T19:24:13.370" UserId="887" />
  <row Id="12110" PostId="7479" Text="You generate a new variable which is the product of HIV and treatment ($x_1 \times x_2$). Then, include your new variable as well as HIV ($x_1$) and treatment ($x_2$) in your model: $... = ... + b_1x_1 + b_2x_2 + b_3x_1x_2$. However, you seem to have less experience with this kind of analysis. Don't get me wrong but may I suggest that you contact a local statistician or other person who can help you." CreationDate="2011-02-22T19:39:10.930" UserId="307" />
  <row Id="12111" PostId="7511" Text="@honk Is this a series of discrete measurements or is the array constantly being monitored?  In other words, can we treat this as a hypothesis test or is it a control chart/changepoint/signal detection problem?" CreationDate="2011-02-22T20:11:21.147" UserId="919" />
  <row Id="12113" PostId="7511" Text="@whuber: We take measurements at singular points in time." CreationDate="2011-02-22T20:16:53.020" UserId="56" />
  <row Id="12114" PostId="7479" Text="Thanks, i think i will just do that." CreationDate="2011-02-22T20:19:12.800" UserId="3355" />
  <row Id="12115" PostId="7273" Text="That new question wouldn't be clear without its context :)" CreationDate="2011-02-22T20:53:28.923" UserId="2902" />
  <row Id="12116" PostId="7515" Score="2" Text="Linking 1-D CDFs makes me think of [copula](http://en.wikipedia.org/wiki/Copula_%28statistics%29)s. Not sure if they'd be any use to you." CreationDate="2011-02-22T20:59:29.337" UserId="449" />
  <row Id="12117" PostId="7273" Score="1" Text="Actually, the PCA values and the clustering groups are independent. PCA creates &quot;new&quot; coordinates for each observation in `mydata`, that is what you actually see on the plot. The shape of the points is plotted using `fit$cluster`, the second parameter of `clusplot()`. Maybe you should take a deeper look into PCA. Let me know if this helped you, or if you further references." CreationDate="2011-02-22T21:01:23.390" UserId="2902" />
  <row Id="12118" PostId="6963" Text="Are you writing your program in C, MATLAB, R, ...?" CreationDate="2011-02-22T21:10:06.260" UserId="2902" />
  <row Id="12119" PostId="7518" Text="Tip: next time use the button with binary numbers (or indent it with 4 spaces) to make the markdown engine interpret the code as code." CreationDate="2011-02-22T21:24:13.580" UserId="88" />
  <row Id="12120" PostId="7516" Text="+1 Nelsen is quite readable.  I bought a copy a few years ago even after going through a lot of the online materials." CreationDate="2011-02-22T21:36:45.360" UserId="919" />
  <row Id="12121" PostId="7273" Score="1" Text="It helps (in the sense that I am honing in on my problem!). How is `fit$cluster` related to the PCA &quot;coordinates&quot;? I think I understand how PCA works, but as I understand it, each Component cannot be explained using variables from the original data (rather it is a linear combination of the raw data), which is why I don't understand how it can be related to the clusters." CreationDate="2011-02-22T22:15:35.453" UserId="2635" />
  <row Id="12122" PostId="7519" Text="Have you read the Wikipedia article?  http://en.wikipedia.org/wiki/Jeffreys_prior" CreationDate="2011-02-22T23:12:13.597" UserId="919" />
  <row Id="12123" PostId="7519" Text="Yes, I had looked there. perhaps I am missing something, but I do not feel that the Wikipedia article gives an adequate answer to my questions." CreationDate="2011-02-22T23:23:07.033" UserId="3347" />
  <row Id="12124" PostId="7484" Text="The paper &quot;Inverted Distributions&quot; by E. L. Lehmann and Juliet Popper Shaffer is an interesting read regarding distributions of inverted random variables. " CreationDate="2011-02-22T23:27:32.710" UserId="530" />
  <row Id="12125" PostId="7456" Text="I found a link in MathWorld, about margin of error: http://mathworld.wolfram.com/MarginofError.html    " CreationDate="2011-02-23T00:22:14.793" UserId="3341" />
  <row Id="12126" PostId="7456" Text="@K-1 Good find.  Note the &quot;additional qualification&quot; that is needed: there is no consensus on what a &quot;margin of error&quot; really is.  Notice, too, that the Wikipedia article offers *three* definitions: http://en.wikipedia.org/wiki/Margin_of_error#Definition .  Regardless, in simple situations (like your least squares fit) the &quot;margin of error&quot; ultimately is computed from an estimate of the sampling variance, as shown in my sample code.  Its square root, the sampling standard deviation, is often multiplied by some constant." CreationDate="2011-02-23T00:25:40.560" UserId="919" />
  <row Id="12127" PostId="7428" Text="@Berndt @Andrej Thank you both for the contribution. The answer is that SDM value could be treated as a SMD. Accordingly, SE could be calculated from the (also suggested by Cochrane) formula SE(SMD)=sqr[N/(n1*n2)+(SMD^2)/(2*(N-3.94))], where SDM=SDMvalue and N=n1+n2. Then, of course, precision can be calculated as precision=1/SE. Nevertheless, I am still not convinced that SDM value could be treated as SMD. Why is a SMD?  " CreationDate="2011-02-23T01:14:11.593" UserId="3333" />
  <row Id="12128" PostId="7456" Text="1- By the way, it looks that the calculated error margin has no relation with &quot;Confidence Level&quot;. For example if you add the option ConfidenceInterval-&gt;0.9 to the LinearModelFit there would be no change in the root mean square error." CreationDate="2011-02-23T01:43:50.380" UserId="3341" />
  <row Id="12129" PostId="7456" Text="2- Instead of using LinearModelFit to calculate the constant in the fit, I suggest to use NonlinearModelFit. It enables you to define the pattern of fit." CreationDate="2011-02-23T01:49:20.313" UserId="3341" />
  <row Id="12130" PostId="7476" Text="@David All included studies lack an estimate of SE, SD, means, CI due to the nature of the analysis. See http://stats.stackexchange.com/questions/7426/eggers-test-in-spss for the description of the problem. Thanks for the suggested transformations." CreationDate="2011-02-23T02:08:14.580" UserId="3333" />
  <row Id="12131" PostId="7427" Text="@Andrej I see... Egger's test of the null hypothesis that intercept b=0 (or tests the null hypothesis that there is no funnel plot asymmetry). In this case the regression line will run through the origin. If the intercept b deviates from zero, the intercept b provides a meassure of asymmetry. The larger the interceptor's deviation from zero point, the larger the asymmetry. The two-sided p-value should be reported. (Synopsis from the book Publication bias in meta-analysis: prevention, assessment and adjustments. Po avtorjih A. J. Sutton). Thanks." CreationDate="2011-02-23T02:34:56.353" UserId="3333" />
  <row Id="12132" PostId="7493" Text="@Bernd Thanks for the clear answer. I am going to start a new thread on Peters et al 2006, JAMA paper." CreationDate="2011-02-23T03:02:40.173" UserId="3333" />
  <row Id="12133" PostId="7523" Score="2" Text="Could you provide more information on the context and what you would do with the weighted composite?" CreationDate="2011-02-23T03:38:26.893" UserId="183" />
  <row Id="12134" PostId="7484" Text="Thank you all for your prompt replies. I appreciate it. But if I have a finite sample set, is it possible to calculate variance or SD for harmonic mean of that set. " CreationDate="2011-02-23T04:46:31.827" UserId="3353" />
  <row Id="12135" PostId="7529" Text="[This post](http://www.biostat.wustl.edu/archives/html/s-news/1999-07/msg00212.html) by Bill Venables explains well the issues involved in this kind of analysis." CreationDate="2011-02-23T07:07:30.883" UserId="2975" />
  <row Id="12136" PostId="7521" Score="3" Text="@fmark, using logs sometimes helps. Log-log models are popular in econometrics, because they are easy to interpret, that is another bonus of using them." CreationDate="2011-02-23T08:23:10.523" UserId="2116" />
  <row Id="12137" PostId="7533" Text="this returns the time of processes there operation was used not the estimated time of operation. Thank you for trying.&#xA;&#xA;component timeMax timeMin timeAverage   timeSD samples&#xA;1       op1     100      30    62.33333 27.98333       6&#xA;2       op2     100       0    52.00000 42.61455       4&#xA;3       op3     100       0    16.66667 40.82483       1&#xA;4       op4     100       0    42.66667 47.42433       3&#xA;" CreationDate="2011-02-23T10:21:40.560" UserId="3376" />
  <row Id="12138" PostId="7533" Text="How do you define the &quot;time of operation&quot; if it's not the time of the processes where the operation is used? Is it the time of the process divided by the number of operations used in that process or...?" CreationDate="2011-02-23T10:32:14.237" UserId="3377" />
  <row Id="12139" PostId="7531" Text="The first step is grouping of simmilar cases and calculating min,max,count,average and SD for all of them.&#xA;The second is to make subtraction of this (grouped) records." CreationDate="2011-02-23T10:44:13.153" UserId="3376" />
  <row Id="12140" PostId="7512" Score="1" Text="You may need to provide more information. What do you mean by correlate? What are n and N?" CreationDate="2011-02-23T12:38:44.877" UserId="495" />
  <row Id="12141" PostId="7338" Text="I waited a bit for other answers, but yours was very elaborate. Thank you for the tips." CreationDate="2011-02-23T12:58:04.023" UserId="1205" />
  <row Id="12143" PostId="7510" Text="@Andrej (+1) I have (re)read your paper, which looks pretty interesting, although I found amazing the use of a Rasch model in clusters of genes. Did you compare your results with a sparse PLS-DA approach?" CreationDate="2011-02-23T14:03:32.713" UserId="930" />
  <row Id="12144" PostId="7428" Text="@Staty Despair: As far as I can see the SDM papers have been published in good journals. So, *I* would trust the authors; of course, you should not trust me because I have absolutely no idea about this SDM stuff. Do you understand how the SDM is actually computed?  " CreationDate="2011-02-23T14:08:51.277" UserId="307" />
  <row Id="12145" PostId="7485" Text="@whuber, Thank you for your comment. Now, this becomes clear to me!" CreationDate="2011-02-23T14:19:10.553" UserId="3019" />
  <row Id="12146" PostId="7456" Text="@K-1 (1) It all depends on what you are looking for confidence in.  This is yet another reason why &quot;margin of error&quot; is so vague.  You are better off computing an actual confidence interval (for a parameter or a prediction) and calling it that.  (2) NonlinearModelFit is not what you specified in your question: you explicitly wrote down a linear model." CreationDate="2011-02-23T14:48:47.253" UserId="919" />
  <row Id="12147" PostId="7428" Text="@Bernd No, I ve never seen the SDM value formula. Actually SDM software is something like a black box for me. I enter the data, I take the results and trying to do the interpretation according to the tutorials, manual and published papers. But no much info available about the actual calculations taking place..." CreationDate="2011-02-23T15:26:07.650" UserId="3333" />
  <row Id="12148" PostId="7517" Text="@whuber (+1) I think there's a typo with the closing `]` around function args list (should be before `:=`). " CreationDate="2011-02-23T15:39:31.390" UserId="930" />
  <row Id="12149" PostId="7517" Text="@chl You have great eyes!  However, this was cut and pasted from working code.  *Mathematica* is flexible about where the criteria (following &quot;/;&quot;) need to appear.  In my experience they are most reliably interpreted when they all appear outside the arguments list, as shown here, even though in some cases that can slow down execution slightly." CreationDate="2011-02-23T15:46:56.330" UserId="919" />
  <row Id="12150" PostId="7428" Text="@Staty Despair So, you have no idea about &quot;voxel-based neuroimaging studies&quot;? I mean I have no idea about &quot;voxel&quot; or &quot;neuroimages&quot;. But since you are working with this kind of data, I assume that you have at least a basic understanding. Are you preparing a publication/thesis/whatever?  " CreationDate="2011-02-23T15:51:06.947" UserId="307" />
  <row Id="12151" PostId="7273" Score="1" Text="You almost got it :) `fit$cluster` is unrelated to PCA. What `clusplot()` does is to plot the points using the &quot;new&quot; coordinates and label them using `fit$cluster`. I got '+' for cluster 3, 'o' for cluster 1 and a triangle for cluster 2. The function `clusplot()` is useful to visualize the clustering. " CreationDate="2011-02-23T15:52:02.390" UserId="2902" />
  <row Id="12152" PostId="7517" Text="@whuber Indeed, it works if there's no carriage return before `/;` -- my fault!" CreationDate="2011-02-23T15:55:27.137" UserId="930" />
  <row Id="12153" PostId="7517" Text="@chl No, you're right.  I had changed the carriage returns to fit the pasted version within the space allotted.  It turns out *Mathematica* objects when a carriage return precedes `/;` but not if it follows `/;`.  (I call that bad parsing, but that's another matter.)  I have changed it so that anyone who pastes this code wholesale back into *Mathematica* will encounter no error messages." CreationDate="2011-02-23T16:01:58.100" UserId="919" />
  <row Id="12154" PostId="7273" Text="What do you mean by the &quot;new&quot; coordinates and how can they be related to `fit$cluster`? I had thought that the coordinates of PCA1 and PCA2 are completely unrelated to `fit$cluster` This is the part that confuses me, I think. Thank you for your patience in explaining this!" CreationDate="2011-02-23T16:13:20.530" UserId="2635" />
  <row Id="12155" PostId="7542" Score="3" Text="You have demonstrated the calls are *not* uniformly distributed.  The $\chi^2$ calculation is not &quot;spoiled&quot;: it worked!" CreationDate="2011-02-23T16:23:53.523" UserId="919" />
  <row Id="12156" PostId="7474" Text="(-1) Although using exponentials is a good approach to understanding the problem, I have reluctantly downvoted because the overt error pointed out by @mpiktas has not been corrected." CreationDate="2011-02-23T16:28:16.903" UserId="919" />
  <row Id="12157" PostId="7542" Text="ok, mmm..maybe is there a &quot;conceptual&quot; error during the implementation of the generator? because for my thesis, my supervisor asked me to demonstrate formally that the generator (which is the result of months&amp;months of study and they use for a long time) really follows the uniform distribution. Maybe can I use a sort of &quot;approximation&quot;?" CreationDate="2011-02-23T16:35:38.747" UserId="3342" />
  <row Id="12158" PostId="7542" Score="3" Text="@Maurizio What would you like to approximate?  Obviously you cannot validly demonstrate this generator produces uniform results: the data flatly contradict that.  The next step depends on what actions you contemplate.  Could the data be wrong?  Could they inaccurately reflect what the generator is doing?  Could they indicate an unexpected phenomenon?  Could the design of the generator be wrong?  You need to raise (and eventually address) questions like these.  The one question that is definitively settled, though, is the one you originally asked: these data are not uniform!" CreationDate="2011-02-23T16:45:21.180" UserId="919" />
  <row Id="12159" PostId="7510" Text="@chl Not yet; working on that." CreationDate="2011-02-23T16:48:02.813" UserId="609" />
  <row Id="12160" PostId="7542" Text="thanks, i think it is better to speak with who has implemented the generator before reaching hasty conclusions." CreationDate="2011-02-23T16:52:52.510" UserId="3342" />
  <row Id="12161" PostId="7273" Score="1" Text="By the &quot;new&quot; coordinates I mean PCA1 and PCA2. You're right, they are completely unrelated to `fit$cluster` :)" CreationDate="2011-02-23T18:18:57.080" UserId="2902" />
  <row Id="12162" PostId="7540" Text="Thank you for your answer and driving my attention to regressors. It would definetely help me to explain diferrent results. Anyway I decided to go with Lag=12, as it is used in Analysis_of_Financial_Time_Series by TSAY and Modeling_financial_time_series_with_s-plus by Zivot, Wang where stationarity was made by using logarithmic differention.  Thank you once more for finding time and answer my question in great detail. Good luck" CreationDate="2011-02-23T18:19:09.660" UserId="3378" />
  <row Id="12163" PostId="2701" Text="As @ars points out there is the notion of the &quot;truncated mean&quot; which is the area under the survival curve. Michael Berry discusses it and also mentioned as &quot;Mean survival restricted to time L&quot; at the following: http://support.sas.com/resources/papers/proceedings10/252-2010.pdf  " CreationDate="2011-02-23T18:22:07.893" UserId="2040" />
  <row Id="12165" PostId="7273" Score="1" Text="But if they are unrelated, how can they be plotted on the same axes?" CreationDate="2011-02-23T18:32:43.533" UserId="2635" />
  <row Id="12166" PostId="7545" Text="Yes, sort of. Was rewriting question while you answered. But I have ~1000 records and ~50 components in real task." CreationDate="2011-02-23T18:36:15.953" UserId="3376" />
  <row Id="12167" PostId="7545" Text="That doesn't seem like a problem, you can do simple linear regression with many more variables. Even non-linear regression is feasible I think." CreationDate="2011-02-23T18:39:23.540" UserId="3369" />
  <row Id="12168" PostId="7545" Text="What is the best way to do this. I know how to solve simple quations on paper.. but do not know how to achieve this on computer for the scale I need. Could you tell me there should I search?" CreationDate="2011-02-23T18:49:11.033" UserId="3376" />
  <row Id="12169" PostId="7545" Text="I think almost any decent software for numerical computations can do regression. You can even do it in Excel (also in Matlab, R, probably octave, python, etc.). I'd search for &quot;linear regression in X&quot;, where X is your favorite environment. If you don't have a favorite, I'd recommend R, because it's free and powerful. Here's a [link](http://www.jeremymiles.co.uk/regressionbook/extras/appendix2/R/) to a tutorial on doing regressions in R. The specific R command you probably want is glm." CreationDate="2011-02-23T19:08:12.210" UserId="3369" />
  <row Id="12170" PostId="2917" Text="@Henrik for small $n$, the t-statistic is appropriate, and as you said, as $n$ increases, t approximates Z. See also http://math.stackexchange.com/q/23246/3733" CreationDate="2011-02-23T19:14:14.967" UserId="1381" />
  <row Id="12171" PostId="7545" Text="+1 For making some sense of the question!" CreationDate="2011-02-23T19:37:08.623" UserId="919" />
  <row Id="12172" PostId="7326" Text="@robin I was talking to Mortimer :)" CreationDate="2011-02-23T19:39:22.897" UserId="2965" />
  <row Id="12173" PostId="7273" Score="1" Text="Remember that `fit$cluster` is a categorical variable indicating the cluster assigned to each observation. It is used to define the labels of the points, that's how they are plotted." CreationDate="2011-02-23T19:39:41.613" UserId="2902" />
  <row Id="12174" PostId="7273" Text="So, am I correct in thinking that a point, with a cluster category is essential a linear combination, or vector, of the variables? Is the PCA just a rescaling of this vector? (Apologies if my terminology is incorrect). I have read a few papers which talk about the link between kmeans and PCA, which is perhaps why I'm wondering about this. (If I could vote your answer, up more than once I would!)." CreationDate="2011-02-23T20:02:08.457" UserId="2635" />
  <row Id="12175" PostId="7547" Score="1" Text="@Ralph The whole point of the comments following the question is that although the distribution of the data might &quot;resemble&quot; a uniform distribution, it is extremely unlikely that the data were truly obtained from this distribution.  That's why we use quantitative methods in statistics: they keep us from fooling ourselves (and others) with suggestive patterns or resemblances that are not justifiable." CreationDate="2011-02-23T20:46:41.037" UserId="919" />
  <row Id="12176" PostId="7544" Text="@user3386 This is a nice idea.  But the proposed analysis will draw many values from a normal distribution.  When the quantile of such a value exceeds (n-1/2)/n or is less than (1/2)/n, what data value would correspond to it?  There is no good answer to this with the information we have.  In short, extrapolation--which appears to be an essential need--is impossible with your proposal." CreationDate="2011-02-23T20:51:09.920" UserId="919" />
  <row Id="12178" PostId="5714" Text="In the meanwhile, I've added a bit more documentation on adaboost: http://packages.python.org/milk/adaboost.html so the above comment might be less valid than it was earlier." CreationDate="2011-02-23T21:41:26.623" UserId="2067" />
  <row Id="12180" PostId="7495" Text="Well, in many practical situations you more-less know the environment, so optimizing selection of a classifier makes sense and do help." CreationDate="2011-02-23T21:57:03.543" UserId="88" />
  <row Id="12181" PostId="7482" Text="Why do you call kNN nonparametric? " CreationDate="2011-02-23T22:00:06.303" UserId="88" />
  <row Id="12182" PostId="7523" Text="Hi Jeromy, I'm attempting to improve several models for horse racing ratings and want to introduce how successful a trainer is at a particular course. Eg if a trainer performance at Catterick was 6 wins from 48 = 13%  (6/48) then this is a fair reflection on on the percentage win rate. However, if the trainer won a single race at Exeter (1/1) this would be 100% and not a good measure. I could either a) Set as 0% for races under a certain denominator, b) create 2 variables percentage &amp; denominator or c) what's advised here" CreationDate="2011-02-23T22:12:37.310" UserId="3373" />
  <row Id="12183" PostId="7551" Text="Using the Hmisc:redun() function I can confirm that the variables are not redundant so long as I only include 10 of the 11" CreationDate="2011-02-23T22:27:14.873" UserId="3388" />
  <row Id="12184" PostId="7551" Text="I found another suggestion to change 'tol' to a value smaller than its default 1e-7. The results are quite similar to changing 'penalty' above. Changing it to smaller values reduces the 'Offending variables' list to only 'admityear2000'" CreationDate="2011-02-23T22:27:53.297" UserId="3388" />
  <row Id="12185" PostId="7273" Text=":D Indeed, a point in the `clusplot()` is a linear combination of the original variables. Maybe the papers that you mention are about clustering after PCA, but that's definitely another topic." CreationDate="2011-02-23T22:31:28.133" UserId="2902" />
  <row Id="12187" PostId="7533" Text="I had updated question and tried to explain that I mean." CreationDate="2011-02-23T23:08:18.547" UserId="3376" />
  <row Id="12188" PostId="7428" Text="@Bernd it is the first time I am involved in the organization of a neuroimaging study. I dont feel familiar with these theories yet." CreationDate="2011-02-23T23:12:04.497" UserId="3333" />
  <row Id="12189" PostId="7545" Text="Maybe I am doing something wrong, but Statistica 8 complains about type of independent variables while trying to perform multiple linear regression analysis. Is this type of analysis is suitable for boolean (1/0) independent/predictor variables?" CreationDate="2011-02-23T23:43:59.763" UserId="3376" />
  <row Id="12190" PostId="7428" Text="@ Staty Despair I see! Good luck with your further analyzes!" CreationDate="2011-02-23T23:50:55.837" UserId="307" />
  <row Id="12191" PostId="7545" Text="The standard linear regression assumes the independent variables are continuous. It should work without a problem for the specific case when the only values that you observe are 0 and 1. The problem you have with the software might just be a question of casting to a different data type." CreationDate="2011-02-24T00:16:46.623" UserId="3369" />
  <row Id="12192" PostId="7545" Text="A deeper issue is that a more elaborate model might take into account the binary nature of the variables and do it more efficiently or more accurately. But I don't know of any such standard models. So I'd try the standard regression first and look for alternatives only if that doesn't work." CreationDate="2011-02-24T00:18:35.217" UserId="3369" />
  <row Id="12193" PostId="7525" Text="&quot;Weighting by the denominator itself may be reasonable&quot; - Please could you explain how to weight by denominator properly? I understand  your method of combining percentages but I don't think I can apply it in my data as the &quot;experiments&quot;/or courses are different. Eg if trainer at Catterick won 6/48 = 13% races and same trainer won 1/1 = 100% at Exeter how should I adjust the percentage or introduce new variables for NN/Decision Tree/SVM predictors for a predictor named eg &quot;% success at todays course&quot;?" CreationDate="2011-02-24T00:26:04.890" UserId="3373" />
  <row Id="12194" PostId="7555" Score="1" Text="Your topic is interesting, but you ask so many questions about different aspects of modeling and meta-analysis, and the questions are so much of a judgment-oriented rather than a how-to nature, that I'm afraid you're not going to get them answered satisfactorily in this venue.  If you could narrow down...." CreationDate="2011-02-24T00:41:15.193" UserId="2669" />
  <row Id="12195" PostId="7459" Text="@Maurizio, sorry for the delay in responding. My pleasure. How did things turn out?" CreationDate="2011-02-24T00:53:55.157" UserId="2970" />
  <row Id="12196" PostId="7555" Text="@rolando2 I agree, but the topic is hot. Also there is no meta-regression tag available. I hope that someone would spend time to add a paragraph for reference! " CreationDate="2011-02-24T00:55:07.433" UserId="3333" />
  <row Id="12197" PostId="6542" Text="@Dikran, Your (potential) question sounds interesting. I look forward to a post on it." CreationDate="2011-02-24T00:56:21.503" UserId="2970" />
  <row Id="12198" PostId="7555" Score="1" Text="@Staty Despair: I modified your list of tags, hope you agree. Before I start answering your questions, I'd like to know how many data points you have. " CreationDate="2011-02-24T01:18:48.143" UserId="307" />
  <row Id="12199" PostId="7557" Text="Thanks. I think is enough to test for normal distribution with a Kolmogorov-Smirnov test on the effect size. What do you mean &quot;the errors of the linear regression model have to be N(0,1)? Weighted regression? This means that I have to weight for SE my dependend variable? I've recently heard about this guy, Peters et al. 2006 JAMA wrote that they modified his test, Macascill's test, to produce the alternative Egger's regression. Unfortunatelly I don't have access to full text of Macascill et al. paper, but Peters et al modification does not include weighting as far as I understand their paper." CreationDate="2011-02-24T01:25:57.570" UserId="3333" />
  <row Id="12200" PostId="7428" Text="@Bernd Thanks a lot, best wishes!" CreationDate="2011-02-24T01:30:58.403" UserId="3333" />
  <row Id="12202" PostId="7557" Score="1" Text="Peters et al (678) write: &quot;In preference to Egger’s regression test, we recommend a simple weighted linear regression with lnOR as the dependent variable and the inverse of the total sample size as the independent variable.&quot; No, this does not mean that you have to weight your dependent variable. Just estimate a WLS regression model. " CreationDate="2011-02-24T01:34:49.287" UserId="307" />
  <row Id="12203" PostId="7555" Text="@Bernd Twenty included studies, three independent variables, let's say 60 data points more or less, for the meta-regression." CreationDate="2011-02-24T01:47:35.943" UserId="3333" />
  <row Id="12204" PostId="7541" Text="Thank you for the reference. However i currently do not have access to jstor. Thank you for the idea as well about partitioning the chi-square." CreationDate="2011-02-24T02:32:12.413" UserId="3379" />
  <row Id="12205" PostId="7553" Text="Assuming Outcome is an ordinal factor, I believe lmr() will treat it as such. At least the documentation for lmr() and lmr.fit() seem to indicate that it will be. &lt;br /&gt;Assuming your categorical variable is set up as a factor, as suggested by DWin, and your response variable is set up as an ordered factor, the inability to estimate the model may be occurring because of separation. Basically, some categories of admityear may perfectly `explain' a particular outcome. " CreationDate="2011-02-24T04:35:23.990" UserId="3265" />
  <row Id="12206" PostId="7543" Text="@whuber: THANK YOU! Not because you answer is the solution to all my problems, but because you have put effort to communicate in a level on which the supplied information can become useful. A lot of times I find myself frustrated and at times borderline angry, when two parties clearly miss one another during exchange of information, and this thread has proven to be one for me. Regarding your suggestions, I'll try to look into the article, but I feel like this has gone on for too long, and in order to finish the software I'm gonna have to implement the analysis as I initially thought... " CreationDate="2011-02-24T09:18:19.027" UserId="3014" />
  <row Id="12207" PostId="7543" Text="and leave validation for further development. This way I can hopefully implement something that's acceptable both biologically and statistically (it's surprising how often they don't coincide) " CreationDate="2011-02-24T09:19:55.203" UserId="3014" />
  <row Id="12208" PostId="7545" Text="It does not work. At least with the raw data. It returns Beta,beta std err, B, B std err, t(2),p last two are about significance, Nor B, nor Beta nor their combination in $y=\beta*x+B$ maner give me the desired results. The most close results I got with &quot;breakdown table&quot; method (Statistica 8) - it calculated initial ranges for all variants." CreationDate="2011-02-24T09:35:00.997" UserId="3376" />
  <row Id="12209" PostId="7552" Text="Thanks for the link." CreationDate="2011-02-24T09:48:41.537" UserId="3383" />
  <row Id="12210" PostId="7365" Text="@mptikas. Neither Wooldridge nor I are claiming that the Hessian has to be positive definite everywhere. My claim is that for an interior maximum the empirical Hessian has to be positive semidefinite as a necessary condition of a smooth function reaching its maximum. Wooldridge seems to be saying something different. " CreationDate="2011-02-24T10:07:16.423" UserId="1393" />
  <row Id="12211" PostId="7531" Text="It looks like the desired method should be a **multiple regression** with one continuous dependent variable and multiple independent/predictor **binary** variables. Like logistic regression but vice-versa and multiple predictors." CreationDate="2011-02-24T10:11:50.010" UserId="3376" />
  <row Id="12212" PostId="7544" Text="@user3386 - this is not a very good idea if you want to know what's going on with your data.  It is for the data to tell you whether it is or isn't &quot;normal&quot;.  The only time this would be a good idea is when it is useless - because the normal approximation already holds.  It's like you're &quot;telling nature how its gona be&quot;." CreationDate="2011-02-24T10:17:34.113" UserId="2392" />
  <row Id="12213" PostId="7531" Text="Found article about [Multiple Regression with Categorical Variables](http://www.psychstat.missouristate.edu/multibook/mlt08m.html), but this technique requires coding of dummy variables, wich is not suitable in my case (&gt;50 binary predictors and the number may extend in future)." CreationDate="2011-02-24T10:44:08.857" UserId="3376" />
  <row Id="12214" PostId="6542" Text="I expect I am just missing something, but it is the paper by Lavine and Schervish &quot;Bayes factors: what they are and what they are not&quot;.  The example 2 seems a bit odd as they use a Bayes factor in a situation where they have a well specified (and highly informative) prior on the hypotheses.  As a result it seems unsurprising that it behaves a little oddly given that the information (mu) is used in computing the Bayes factor, but then treated as unknown in the hypothesis test itself.  I like reading papers on what things actually mean though and being wrong is a good way of learning!" CreationDate="2011-02-24T11:25:58.867" UserId="887" />
  <row Id="12215" PostId="7477" Text="I would also recommend Boyd and Vandenberghe, especially if it is available for free." CreationDate="2011-02-24T11:27:58.210" UserId="887" />
  <row Id="12216" PostId="7546" Score="3" Text="Check out this answer: http://stats.stackexchange.com/questions/5344/how-to-choose-nlme-or-lme4-r-library-for-mixed-effects-models/5350#5350" CreationDate="2011-02-24T12:43:40.987" UserId="1979" />
  <row Id="12217" PostId="7459" Text="@cardinal, I'll post a new answer with the results of my test!" CreationDate="2011-02-24T12:47:58.033" UserId="3342" />
  <row Id="12218" PostId="7477" Text="@cardinal: Scandinavian universities usually do offer bachelors level degrees as well. That being said, I think statisticians take themselves a bit too seriously. I disagree that you'd need a doctorate degree in order to get an &quot;interesting&quot; job. I believe that as science and research becomes more and more cross-disciplinary statistics have been imposed on studies from many different areas. Half of the articles on high impact journals have some questionable statistical analysis, just to meet the demands, even though it might not make any sense, given in original context/domain of the problem. " CreationDate="2011-02-24T12:52:40.130" UserId="3014" />
  <row Id="12219" PostId="7477" Text="[continuation from previous comment] ... I dont know if this is the case with the OP, but I can definitely relate to the dilemma of trying to fast forward, especially if you are done with your education. It's simply unrealistic to go through a long list of books, exercises and spend months and years to be able to do your job, which might not have anything to do with theoretical statistics... (I'm sorry if I sound too frustrated)" CreationDate="2011-02-24T12:53:02.780" UserId="3014" />
  <row Id="12220" PostId="7477" Text="@posdef, I certainly empathize with your feelings of frustration. I take a little issue with some remarks that you've made that I suspect are mostly a product of this frustration. To respond to your comments completely would require an essay. Modern applied statistics is a mathematically and computationally sophisticated field which requires a fairly high degree of specialized knowledge. It strikes me as a bit presumptuous to want to become good/competent in the field without putting in the effort. Click on the link I provided for a more eloquent version of this. " CreationDate="2011-02-24T13:57:50.807" UserId="2970" />
  <row Id="12221" PostId="7477" Text="As someone who originally came from another discipline, I understand the desire to learn quickly enough to be productive. Unfortunately, my experience is that that is difficult and potentially dangerous. What is your background? If someone with a B.S. In statistics came to you and said &quot;I need to know your field. Make me an expert as quickly as possible,&quot; what would you realistically suggest to them? When would *you* feel safe calling them an expert and handing them their own project that *your* business or livelihood depended on?" CreationDate="2011-02-24T14:03:17.123" UserId="2970" />
  <row Id="12222" PostId="7477" Text="Like I said, I think a well-thought-out essay is needed and this is hardly a good format for it, unfortunately. The points you raise regarding poor statistical analysis in the literature are valid. My argument would be that this is often due to precisely the problem we are trying to address in this thread. There are many, *many* people doing statistics without fully grasping what they are doing. There are also too few statisticians that have sufficient domain expertise in the area of application they are working in. " CreationDate="2011-02-24T14:07:18.853" UserId="2970" />
  <row Id="12223" PostId="7522" Score="1" Text="Hong's response is correct (and I tested that the code works). You almost surely want to use the ExportMatrixToR module (2nd method), because they the 'a' object in R is a real R object, and not just a literal." CreationDate="2011-02-24T14:12:12.067" UserId="2773" />
  <row Id="12224" PostId="7553" Text="Thanks so much to both of you. I learned about 5 new things already.  I'll add responses and lessons to my question&#xA;" CreationDate="2011-02-24T14:46:04.810" UserId="3388" />
  <row Id="12225" PostId="7365" Text="@Jyotirmoy Bhattacharya, I've updated my answer with more details, hope it explains my intuition which motivated the initial answer." CreationDate="2011-02-24T14:57:44.337" UserId="2116" />
  <row Id="12226" PostId="7543" Score="1" Text="@posdef Best of luck.  I'm sorry to hear about the conflicts of biological and statistical acceptability you have encountered.  Any such conflict bothers me, because &quot;statistical unacceptability&quot; is tantamount to *unscientific.*  There has to be a resolution to this.  Maybe it needs better communication and understanding on both sides." CreationDate="2011-02-24T15:11:15.610" UserId="919" />
  <row Id="12227" PostId="7571" Score="1" Text="I'm a bit confused, perhaps by the title. Are you wanting to interpolate between two time points or are you asking about something like smoothing (as opposed to filtering)?" CreationDate="2011-02-24T15:12:57.993" UserId="1764" />
  <row Id="12228" PostId="2627" Text="+1 (Sorry about the belated vote): nice illustrations." CreationDate="2011-02-24T15:13:56.730" UserId="919" />
  <row Id="12229" PostId="7559" Text="The title of your question is asking a totally different question from what you really ask­. I, for one, was curious about the real differences between the Linear SVM (let's take the hard margin SVM) and other linear discriminants." CreationDate="2011-02-24T15:19:07.937" UserId="1320" />
  <row Id="12230" PostId="7477" Text="@cardinal: While I mostly agree with you, what I find highly troubling is the assumption of &quot;interest to master&quot; the field of statistics. When I ask, or answer, a question on StackOverflow, I don't make assumptions on intentions to master anything, I only intend to learn to solve my problems. Likewise I don't assume people want to master the field of programming when they ask something there. Mastery, as you have mentioned, comes over time and effort of course. My experience SO vs CV and statistics vs programming have been bitterly different in terms of help/utility :(" CreationDate="2011-02-24T15:22:56.153" UserId="3014" />
  <row Id="12231" PostId="7543" Text="@whuber: Thnx. Regarding the quote: &quot;Any such conflict bothers me, because &quot;statistical unacceptability&quot; is tantamount to unscientific. There has to be a resolution to this. Maybe it needs better communication and understanding on both sides.&quot; - I couldn't agree more..." CreationDate="2011-02-24T15:26:59.953" UserId="3014" />
  <row Id="12232" PostId="7477" Text="@posdef, thanks for your comments. My response was tailored to the question of the OP who &quot;wanted the equivalent knowledge of a quality stats degree.&quot; I read that as a desire for a certain degree of mastery. While prigramming requires a very significant degree of both logic and creativity, I would speculate that most of the questions on SO are about more direct implementation issues. CV is more geared towards helping others understand and apply statistics, which generally requires a deeper dialogue with the OP to do well (correctly) and involves addressing more subtleties. My $0.02." CreationDate="2011-02-24T15:32:25.727" UserId="2970" />
  <row Id="12233" PostId="7477" Text="@cardinal: ah, i guess you're right... I'm sorry for creating an off-topic discussion like this, and for being very bitter on the subject. I gotta work on that, somehow." CreationDate="2011-02-24T15:48:50.743" UserId="3014" />
  <row Id="12234" PostId="7557" Text="@Bernd Do you mean I have to use &quot;SE&quot; as the Weight Variable in SPSS WLS regression model?" CreationDate="2011-02-24T16:34:47.180" UserId="3333" />
  <row Id="12235" PostId="7557" Score="1" Text="@Staty Despair: No, its in my answer: &quot;...are weighted by the inverse variance of the estimate...&quot;, i.e. $w_i = \frac{1}{SE_i^2}$." CreationDate="2011-02-24T16:40:11.830" UserId="307" />
  <row Id="12236" PostId="3405" Text="+infinity.  I do bioinformatics and this one caused me a world of pain.  Happens in openoffice.org, too." CreationDate="2011-02-24T17:09:44.557" UserId="1347" />
  <row Id="12238" PostId="7365" Text="@mpiktas +1 for the idea, even though it seems there are problems with it.  I don't follow your first paragraph: it seems to be self-contradictory.  Maybe there's a &quot;not&quot; missing somewhere?  I am unable to reproduce the results of your example.  I do find that in many cases it is difficult to find the global minimum and, occasionally, that failure can lead to a Hessian with a small negative eigenvalue.  In thousands of simulations, using a non-derivative based minimizer in *Mathematica*, I always obtain positive definite Hessians." CreationDate="2011-02-24T17:41:46.097" UserId="919" />
  <row Id="12239" PostId="7308" Text="@Cardinal Good question.  I wonder whether there could be problems with $q$ that are not continuously second differentiable.  In short, we need to find violations of the hypotheses of the theorem (implying non-negative definiteness of $H$) because the validity of the theorem is not in doubt!" CreationDate="2011-02-24T17:47:27.070" UserId="919" />
  <row Id="12240" PostId="7581" Score="1" Text="&quot;In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule and its result (the estimate) are distinguished.&quot;  (First line of the Wikipedia article http://en.wikipedia.org/wiki/Estimator )." CreationDate="2011-02-24T19:15:43.340" UserId="919" />
  <row Id="12242" PostId="7571" Text="@Wayne I'd like to interpolate between the two measurement times, but include information about the future measurement -- not just the previous measurements." CreationDate="2011-02-24T19:34:12.233" UserId="3405" />
  <row Id="12243" PostId="7545" Text="I ran regression in Matlab with the set of 7 procedures and four operations in your opening post. The results it gave are: [25 31 34 22] for the operation times, with confidence interval for the first operation time of [4, 47], and similar values for the others. Keep in mind that this is on a very small set. So my question is, do you get similar results and just want something different? E.g. it gives you a (probabilistic) confidence interval and maybe you want hard constraints instead. Or do you get something completely different?" CreationDate="2011-02-24T20:12:14.367" UserId="3369" />
  <row Id="12245" PostId="7583" Score="2" Text="This sounds like you are asking for a computer program to implement the criteria stipulated in the question.  If so, we need to migrate this question to SO.  Otherwise, you need to clarify what you mean by &quot;determine the trend.&quot;" CreationDate="2011-02-24T20:51:45.150" UserId="919" />
  <row Id="12246" PostId="7559" Score="1" Text="I guess the main difference is the objective function they optimize. SVM tries to maximize the margin. Other linear classifiers do other things (e.g. perceptron optimizes the reconstruction error). These different objectives have different properties (e.g. maximizing the margin improves the generalization error). They also give you different problems to solve (some of which may be easier or more efficient, others can be solved online, etc.)." CreationDate="2011-02-24T20:54:53.110" UserId="3369" />
  <row Id="12247" PostId="7465" Text="Actually, this was the first thing I tried. I tried this before posting the question. Finding a list of courses wasn't hard, but finding information about which books actually got used for those courses and what sections of those books were covered was much more difficult. " CreationDate="2011-02-24T21:02:10.263" UserId="2912" />
  <row Id="12248" PostId="7585" Score="5" Text="How does this distinguish an estimator from any garden-variety random variable?  How does an estimator then differ from an observation, for instance?" CreationDate="2011-02-24T21:34:43.207" UserId="919" />
  <row Id="12249" PostId="7586" Text="You have nicely defined a *predictor.*  It is subtly (but importantly) different from an estimator.  The estimator in this context is the least squares formula used to compute the parameters 1 and 6 from the data." CreationDate="2011-02-24T21:36:24.827" UserId="919" />
  <row Id="12250" PostId="7581" Text="+1 I am upvoting this question (despite the presence of a well formulated answer on an obvious Wikipedia page) because initial attempts at answering it here have pointed to some subtleties." CreationDate="2011-02-24T21:38:36.773" UserId="919" />
  <row Id="12252" PostId="7586" Text="Hmm, I didn't mean it that way, @whuber, but I think your comment illustrates an important ambiguity in my language that I didn't notice before. The main point here is that you can think of the generic form of the equation Y = mX + B (as used above) as an estimator, whereas the particular predicted values generated by specific examples of that formula (e.g., 1 + 6X) are estimates. Let me try to edit the paragraph above to capture that distinction..." CreationDate="2011-02-24T21:53:33.630" UserId="3396" />
  <row Id="12253" PostId="7586" Text="btw, I'm trying to explain this without introducing the &quot;hat&quot; notation that I've encountered in most textbook discussions of this concept. Perhaps that's the better route after all? " CreationDate="2011-02-24T21:58:57.667" UserId="3396" />
  <row Id="12254" PostId="7582" Text="+1 In addition, look at the sandwich package which works hand-in-hand with functions in the pscl package to provide appropriate estimation of model covariance matrices that account fot the lack of independence." CreationDate="2011-02-24T22:12:50.663" UserId="1390" />
  <row Id="12255" PostId="7586" Score="2" Text="@ashaw I think you have hit a nice medium between accuracy and technicality in your original answer: keep it up!  You don't need hats, but if you can manage to show how an estimator is distinguished from other, similar-looking things, that would be most helpful.  But please notice the distinction between *predicting* a value Y and *estimating* a parameter such as *m* or *b*.  Y could be interpreted as a random variable; m and b are not (except in a Bayesian setting)." CreationDate="2011-02-24T22:15:30.347" UserId="919" />
  <row Id="12256" PostId="7586" Text="indeed, a very good point in terms of parameters versus values there. Editing again..." CreationDate="2011-02-24T22:24:22.110" UserId="3396" />
  <row Id="12257" PostId="7597" Text="Oh no. You beat me to it. I will delete my answer!" CreationDate="2011-02-24T23:50:24.067" UserId="1307" />
  <row Id="12258" PostId="7592" Text="+1: That's a really useful SO thread there. Pretty much reduces my question to a repost :P." CreationDate="2011-02-24T23:53:13.757" UserId="1118" />
  <row Id="12259" PostId="7594" Text="This sounds an awful lot like a homework question to me. Could you provide some context?" CreationDate="2011-02-24T23:55:12.110" UserId="3396" />
  <row Id="12260" PostId="7545" Text="Looks like I do something wrong. I get completely different results in Statistica. I had tried R for this: &#xA;`d &lt;- read.table(&quot;data.csv&quot;,header=T,sep=&quot;,&quot;)&#xA;r &lt;- lm(time ~ p1+p2+p3+p4, data=d)&#xA;summary(r)&#xA;p1             2.000  &#xA;p2            25.000 &#xA;p3            22.000&#xA;p4            22.000 &#xA;`&#xA;Results are closer, but still strange. I think I did something wrong with lm function." CreationDate="2011-02-25T00:17:32.300" UserId="3376" />
  <row Id="12261" PostId="7418" Text="Accepting your answer because of the multiple suggestions for resources and functions that have improved my understanding. It seems like 'gamlss' would be a possible way to solve my issue, but because I'm actually a non-statistician, I don't currently have the background in math nor the time to open that can of worms right now (but maybe eventually I will). As mentioned in another comment, for my data at least, it seems that ignoring zero-truncation won't change my estimates and std errors much. For my intended audience, I believe a nbinomial GEE will do just fine. Thanks!" CreationDate="2011-02-25T00:32:43.493" UserId="3309" />
  <row Id="12262" PostId="7594" Text="It isn't a homework question, but the original problem domain isn't hotels -- it is cloud computing. I am merely translating the problem. I am looking for pointers to techniques and approaches that can solve this question. Even -- can it be solved with the given data or is more data required (for example the initial occupancy rate)." CreationDate="2011-02-25T00:57:17.097" UserId="3418" />
  <row Id="12263" PostId="7600" Text="What kind of additional information? If we had the average occupancy rate, could we estimate the average length of stay? Or, if we had the average length of stay, we could estimate the occupancy rate?" CreationDate="2011-02-25T01:03:18.600" UserId="3418" />
  <row Id="12264" PostId="7600" Text="@Chiradeep: With three out of four pieces of information you can estimate  the fourth. Broadly speaking: the product numbers of rooms available times occupancy rate equals the number of rooms occupied each night, which in turn equals the product numbers of arrivals per day times length of stay.    " CreationDate="2011-02-25T01:17:12.960" UserId="2958" />
  <row Id="12265" PostId="7545" Text="It looks like what you did was &quot;regression with offset&quot;. I.e. the function implicitly adds a column of all ones and tries to build a linear function with offset. I expect it wouldn't matter on your large dataset, but for a small dataset it skews the result. What you should try is &quot;regression through origin&quot; (i.e. without the offset). I believe the command is time ~ 0+p1+p2+p3+p4." CreationDate="2011-02-25T01:17:50.370" UserId="3369" />
  <row Id="12266" PostId="7545" Text="Looks like the solution! Thank you very much!!! May I edit your answer to insert the R code?" CreationDate="2011-02-25T01:26:19.647" UserId="3376" />
  <row Id="12267" PostId="7545" Text="Sure. Glad it helped." CreationDate="2011-02-25T01:31:46.157" UserId="3369" />
  <row Id="12268" PostId="6556" Text="This hasn't got anything to do with Bayes; it's just the factorization lemma and is a direct consequence of the definition of sufficiency. Bringing Bayes into the mix just complicates things." CreationDate="2011-02-25T01:53:09.267" UserId="26" />
  <row Id="12269" PostId="7594" Score="1" Text="Imagine two jails with the same number of cells and same arrival rate (we can even say it's equal to the departure rate, so occupancy stays constant). One jail has all prisoners stay for a week. Another jail has 90% of the prisoners released same day (zero nights) and 10% stay for 10 weeks. Same arrival rate, same occupancy rate, different distribution of stay duration." CreationDate="2011-02-25T02:03:07.943" UserId="2993" />
  <row Id="12270" PostId="7318" Text="Are any of your DVs actually Poisson-distributed? I.e., are they counts of the frequencies of events over successive periods of time, and does the mean roughly equal the variance? If so, you may be able to use Poisson regression to test the interactions you're interested in." CreationDate="2011-02-25T03:23:57.090" UserId="2669" />
  <row Id="12271" PostId="7553" Text="There may be separation when cross-classifying by two or more factors that doesn't show up on a simple predictor by Outcome tabulation. You would need to look at multi-way tables to find it." CreationDate="2011-02-25T04:08:52.213" UserId="2129" />
  <row Id="12272" PostId="7566" Text="Thanks Dmitrij, your advice is much appreciated." CreationDate="2011-02-25T04:29:56.853" UserId="179" />
  <row Id="12274" PostId="7603" Score="1" Text="In that last paragraph aren't you effectively arguing against any use of p-values?  Why, then, use a hypothesis test at all?  Why not just estimate the effect sizes and be done with it?" CreationDate="2011-02-25T06:18:55.400" UserId="919" />
  <row Id="12275" PostId="7594" Text="@jonw Good example.  The difficulty is even more basic than that.  An extreme case is a city with so many hotels that nobody ever has to leave!  This makes it obvious that we need to know the average rate at which people depart hotels, or equivalent information." CreationDate="2011-02-25T06:22:57.237" UserId="919" />
  <row Id="12276" PostId="7606" Text="Actually generalized Pareto (GP) is a limiting distribution for extreme data (extreme are characterize by all data which exceed a high threshold). A suitable threshold (model gives the good fit to data above this threshold) indicated by the stability of parameter estimates which is not so easy to determined by look at a graphical plot. I want to know whether there is any other way/test to find the best threshold (i.e the best data set for GP) " CreationDate="2011-02-25T07:19:41.590" UserId="3423" />
  <row Id="12277" PostId="6556" Text="@JMS - Yes I do understand your &quot;factorisation&quot; theorem argument.  What I am saying is that using Bayes Theorem will 1) tell you if a sufficient statistic of reduced dimensionality exists, and 2) what the sufficient statistic is for your problem.  The use of &quot;sufficient statistics&quot; is basically a way to bring &quot;frequentist&quot; statistics&quot; closer to &quot;Bayesian statistics&quot; without admitting that one is doing so.  They also reduce the calculations one has to perform in an analysis.  Sufficiency is also closely related to the maximum entropy method (aka &quot;ultimate inference&quot;)." CreationDate="2011-02-25T08:37:16.587" UserId="2392" />
  <row Id="12278" PostId="7574" Text="@vqv - could you adapt your answer so that it matches the notation of the question?  To what is $x^2-y^2$ referring?  Where does this get inserted into the equations given in the question?" CreationDate="2011-02-25T08:44:57.643" UserId="2392" />
  <row Id="12279" PostId="7605" Text="Not a &quot;proper&quot; answer, but if two variables are correlated, but you put them into different factors, then it does make intuitive sense that the two factors should also be correlated." CreationDate="2011-02-25T08:48:11.260" UserId="2392" />
  <row Id="12280" PostId="6978" Text="Are you able to calculate statistics for the data set? say for example the sum, or averages of data items?" CreationDate="2011-02-25T08:53:28.723" UserId="2392" />
  <row Id="12281" PostId="7608" Text="@whuber (+1) A very precise and detailed response." CreationDate="2011-02-25T09:14:05.867" UserId="930" />
  <row Id="12282" PostId="7605" Text="@Scott How do you measure the increase in goodness-of-fit?" CreationDate="2011-02-25T09:15:48.220" UserId="930" />
  <row Id="12283" PostId="7612" Text="Thanks very much - very helpful. I will investigate `anova.coxph` too." CreationDate="2011-02-25T10:10:41.713" UserId="3429" />
  <row Id="12284" PostId="7610" Text="I think this is a duplicate of (http://stats.stackexchange.com/questions/3458/alternatives-to-classification-trees-with-better-predictive-e-g-cv-performanc). The only additional value of the question here could be &quot;criterions for selecting classifiers&quot; (which would make the question a very generic one). If it's a duplicate, vote for close, else vote for cw ;) " CreationDate="2011-02-25T11:02:47.133" UserId="264" />
  <row Id="12285" PostId="7365" Text="@whuber, I rewrote the answer. Also I included the data, so it will be easier to reproduce. I also fixed the bug which was present in the previous version. " CreationDate="2011-02-25T11:25:07.093" UserId="2116" />
  <row Id="12286" PostId="7610" Text="@steffen: Your referenced question is helpful, though, I think it's not a duplicate. Indeed my question is rather generic. I'm not looking for a solution to a particular problem but for general reasons why to use which learners - I'll update my question accordingly." CreationDate="2011-02-25T11:36:22.170" UserId="2230" />
  <row Id="12287" PostId="7571" Text="just to clarify, by `*` you mean the usual product or [convolution](http://en.wikipedia.org/wiki/Convolution)?" CreationDate="2011-02-25T12:49:48.750" UserId="2116" />
  <row Id="12288" PostId="6028" Score="2" Text="Another excellent resource for learning about and understanding survival analysis is _Applied Longitudinal Data Analysis_ by Singer and Willett. The also give [example code/output for all their models using every stats package under the sun](http://www.ats.ucla.edu/stat/examples/alda.htm)." CreationDate="2011-02-25T13:06:32.423" UserId="1916" />
  <row Id="12289" PostId="7613" Score="5" Text="These are two separate questions. The answer to the second is [yes](http://en.wikipedia.org/wiki/Principal_component_analysis#Relation_between_PCA_and_K-means_clustering). The first one is too vague. First what is the magnitude of the vector? Also what kind of relationship you are looking for? Principal eigenvectors in PCA are the eigenvectors of either covariance or correlation matrix of original variables, so they are clearly related to correlations. " CreationDate="2011-02-25T13:13:41.247" UserId="2116" />
  <row Id="12290" PostId="7613" Score="2" Text="The wikipedia page is pretty verbose and should be sufficient for question #1, http://en.wikipedia.org/wiki/Principal_component_analysis ." CreationDate="2011-02-25T13:50:33.413" UserId="1036" />
  <row Id="12291" PostId="7537" Text="Thanks for the advice. I will definitely try to examine the predictions before deciding on the model" CreationDate="2011-02-25T14:41:57.447" UserId="1871" />
  <row Id="12292" PostId="7539" Text="The intention is that i will eventually try to model this using Bayesian, but i was trying to understand how i can make a decision before fitting the models. If there is a possibility that ignoring the nested nature of the data messes things up, them i will try them GLMMs first. The only package for R that i am aware of that can do multilevel ZINB is glmmADMB. Would you recommend any other packages? " CreationDate="2011-02-25T14:48:21.983" UserId="1871" />
  <row Id="12293" PostId="7616" Score="3" Text="-1 Absolutely incorrect workflow for large p small n, FS overfitting is guaranteed. " CreationDate="2011-02-25T14:51:07.370" UserId="88" />
  <row Id="12294" PostId="7576" Text="Thanks for your answer. Indeed, i started thinking about whether different items could produce the 0s versus any other count and I actually think that there are a couple of my variables that would only explain 0s vs any other count.  So probably i should at least try ZINB first to see if my these variables work the way i would expect them to work. " CreationDate="2011-02-25T14:59:52.710" UserId="1871" />
  <row Id="12295" PostId="7616" Text="isn't kNN a lazy learner rather than an eager one (as you don't actually do anything until you really have to when a pattern to classify comes along)?  Any pre-processing you do before applying the classifier is likely to have a larger effect on performance than the difference between classifiers; feature selection is especially difficult (easily leads to over-fitting), and methods like the SVM with regularisation usually perform better without feature selection.  I certainly wouldn't recommend neural networks, far too many potential pitfalls." CreationDate="2011-02-25T15:28:04.487" UserId="887" />
  <row Id="12296" PostId="7603" Text="I stop short of calling *p*-values completely useless.  I do advocate for the use of other methods, esp. when I hear people struggling with hypothesis testing and when they seem to have been taught such testing in the flawed way that I and many of us seem to have been taught.  I've commented on this elsewhere on the site so I won't repeat too much of my views here." CreationDate="2011-02-25T16:04:53.950" UserId="2669" />
  <row Id="12297" PostId="7620" Text="Though not entirely uncommon for this type of thing to occur some people might like to see this as a teaching or learning case.  If you could share the data it would be great.  You also might get good feedback on your specific problem." CreationDate="2011-02-25T16:23:48.110" UserId="601" />
  <row Id="12298" PostId="7613" Score="2" Text="I think I provide an answer to question 2 in an earlier thread, [Classification after factor analysis](http://stats.stackexchange.com/questions/2467/classification-after-factor-analysis/3475#3475)." CreationDate="2011-02-25T16:28:35.140" UserId="930" />
  <row Id="12299" PostId="7622" Text="@mbq (+1) About class imbalance, we can still rely on stratified sampling during bagging." CreationDate="2011-02-25T16:31:50.097" UserId="930" />
  <row Id="12300" PostId="6028" Text="@M Adams Thanks for adding this link. Yes, the UCLA server is really full of useful resources." CreationDate="2011-02-25T16:38:12.443" UserId="930" />
  <row Id="12301" PostId="7608" Text="@chl Thank you.  It could not have been this way without the help of the previous replies and commentary, which suggested which points may be worth emphasizing." CreationDate="2011-02-25T17:37:31.000" UserId="919" />
  <row Id="12302" PostId="7629" Text="+1, nice analysis. I think that is why Wooldridge included the remark. I still think it is possible to think of some example where the hessian will be indefinite. Artificially restricting the parameter space for example. In this example the parameter space is whole plane, that is why the local minimum will give semi-positive hessian. I think the time has come to write a nice email to Wooldridge to get his take on the question:)" CreationDate="2011-02-25T18:00:24.267" UserId="2116" />
  <row Id="12303" PostId="7622" Text="@mbq, doesn't make coffee? That's a deal-breaker right there. " CreationDate="2011-02-25T18:06:40.920" UserId="2970" />
  <row Id="12304" PostId="7626" Score="1" Text="@Mike:  Path analysis is an interesting idea, but I don't see your point about violation.  Usually in nonexperimental studies predictors are correlated to some degree.  Without that there's little point in using statistical control:  zero-order correlations of each X with Y will be the same as partial correlations." CreationDate="2011-02-25T18:26:22.440" UserId="2669" />
  <row Id="12305" PostId="7629" Text="@mpiktas Yes, I'm sure there exist problems where an interior *global* minimum has an indefinite Hessian, *yet where all parameters are identifiable.*  But it simply is not possible for the Hessian at a sufficiently smooth interior global minimum to be indefinite.  This sort of thing has been proven again and again, such as in Milnor's *Topology from a Differentiable Viewpoint*.  I suspect Wooldridge may have been misled by errant numerical &quot;solutions.&quot;  (The typos on the quoted page suggest it was written hastily, by the way.)" CreationDate="2011-02-25T18:31:48.823" UserId="919" />
  <row Id="12306" PostId="7605" Text="Hey chl - looking at RMSEA, SRMS, and BIC." CreationDate="2011-02-25T18:46:40.623" UserId="3424" />
  <row Id="12307" PostId="7614" Text="Thanks! So if I want to validate a orthogonal-rotated factor-loading with CFA (as I have done), it makes sense to validate that with a CFA model where the factors are uncorrelated. Since it's not a great fit, you are saying I should re-run my EFA using an oblique rotation, and then build a CFA model where the factors may correlate." CreationDate="2011-02-25T18:48:56.940" UserId="3424" />
  <row Id="12309" PostId="7619" Text="Thanks M- Adams. In response to 1: I've noticed that the structures are nearly consistent. For 2) Thanks! I am validating against a sample (split my big set in half) 3) Are you suggesting a model where they may be a factor driving other factors? Thanks!" CreationDate="2011-02-25T18:52:41.067" UserId="3424" />
  <row Id="12310" PostId="7311" Text="@zyxo:  I visited the link and read your praise of the decision tree.  Have you considered the way this method treats all main effects (save the first) as if they were interaction effects?  I have found that highly misleading and counterproductive on many occasions." CreationDate="2011-02-25T19:06:24.353" UserId="2669" />
  <row Id="12311" PostId="7629" Text="@whuber, even at the boundary, hessian will be positive? I'll check out the book, I see that I really lack extensive knowledge in this area. Classical theorems are very simple, so I assumed that there should not be something else very complicated. That maybe one of the reasons why I had so much difficulty answering the question. " CreationDate="2011-02-25T19:19:19.713" UserId="2116" />
  <row Id="12312" PostId="7631" Text="Thanks for editing mpiktas!" CreationDate="2011-02-25T19:42:06.227" UserId="1102" />
  <row Id="12313" PostId="7631" Text="Do you by any chance mean that X2 is distributed as U[0, X1]?" CreationDate="2011-02-25T19:53:29.990" UserId="3369" />
  <row Id="12314" PostId="7626" Text="@rolando not sure if this is appropriate, but PCA would provide orthogonal predictors" CreationDate="2011-02-25T19:56:12.297" UserId="1381" />
  <row Id="12315" PostId="7629" Text="@mpiktas At the boundary the Hessian won't necessarily even be *defined*.  The idea is this: if the Jacobian/Hessian/second derivative matrix is defined at a critical point, then in a neighborhood the function acts like the quadratic form determined by this matrix.  If the matrix has positive *and* negative eigenvalues, the function *must* increase in some directions and decrease in others: it cannot be a local extremum.  This is what concerned @Jyotirmoy about the quotation, which seems to contradict this basic property." CreationDate="2011-02-25T20:51:31.260" UserId="919" />
  <row Id="12316" PostId="7633" Text="Henry: log(X1) is after integrating (but before substituting limits) for marginal of X2. Your P(X2) is wrong. I believe you are integrating log(X1) that I said which we get after integration itself. " CreationDate="2011-02-25T21:23:53.250" UserId="1102" />
  <row Id="12317" PostId="7631" Text="SheldonCopper: That's correct. I'll change it." CreationDate="2011-02-25T21:24:27.090" UserId="1102" />
  <row Id="12318" PostId="7614" Text="yup, thats it. If you're going to be doing EFA and CFA on one dataset, you might want to look at splitting the data, running EFA on one half and CFA on the other, to increase replicability of the factor structure." CreationDate="2011-02-25T22:04:59.190" UserId="656" />
  <row Id="12319" PostId="7637" Text="possible duplicate of [How to easily determine the results distribution for multiple dice?](http://stats.stackexchange.com/questions/3614/how-to-easily-determine-the-results-distribution-for-multiple-dice)" CreationDate="2011-02-25T22:55:50.273" UserId="919" />
  <row Id="12320" PostId="7631" Score="1" Text="The limits for the marginal of $X_2$ are not from 0 to 1 except when $X_2 = 0$." CreationDate="2011-02-25T23:04:41.860" UserId="919" />
  <row Id="12321" PostId="7633" Text="@Harpreet: testing using R, it is clear to me that $P(X_2 \le x_2) = x_2 (1-\log(x_2))$ is correct for $0 &lt; x_2 &lt; 1$.  I have also expanded the integrals to show how this is obtained.  So which $P(X_2)$ do you think is wrong?  " CreationDate="2011-02-25T23:16:30.270" UserId="2958" />
  <row Id="12322" PostId="7633" Text="P(X2)=int(1/X1)." CreationDate="2011-02-25T23:21:56.227" UserId="1102" />
  <row Id="12323" PostId="7631" Text="Thanks whuber. You are correct. So, we have to substitute limits for marginal density of X2 as X1=X2 to X1=1." CreationDate="2011-02-25T23:24:04.167" UserId="1102" />
  <row Id="12324" PostId="7633" Text="Thanks Henry. But I think what you are doing is correct, however marginal of X2 will be $ln(X_1)$ without limits." CreationDate="2011-02-25T23:26:21.130" UserId="1102" />
  <row Id="12325" PostId="7640" Text="Okay.  I have a table of data 4300 (yeah, it's actually 4300) lines long.  It has 3 independent variables and one dependent variable.  I want to do multivariable linear regression to find how those three IVs affect the DV.  I also want to automatically generate 3-dimensional graphs of each slice of the three IVs (as in leaving one IV constant while using the other two as x and y, and the DV as z)." CreationDate="2011-02-26T01:46:04.533" UserId="3439" />
  <row Id="12326" PostId="7633" Text="$X_1 \le 1$ so $\ln(x_1) \le 0$, which means it cannot be a density or distribution function.  And I still think $X_1$ should not appear in the marginal distribution of $X_2$. http://en.wikipedia.org/wiki/Marginal_distribution says the same thing in &quot;The distribution of the marginal variables (the marginal distribution) is obtained by marginalizing over the distribution of the variables being discarded, and the discarded variables are said to have been marginalized out.&quot;" CreationDate="2011-02-26T01:54:12.567" UserId="2958" />
  <row Id="12327" PostId="7639" Text="Check out the &quot;loess&quot; and &quot;glm&quot; commands in R.  What do you mean by &quot;nonlinear&quot; regression?" CreationDate="2011-02-26T02:08:24.187" UserId="2817" />
  <row Id="12328" PostId="7571" Text="@mpiktas product.  One of the hazards of copying from code.  :)  I've corrected the post for clarity." CreationDate="2011-02-26T02:39:28.043" UserId="3405" />
  <row Id="12329" PostId="7626" Text="Hi David - it would, but I don't see any great need for orthogonality in this case, and with just 4 predictors, not much need for the data reduction that PCA provides.  There's also the risk that Y's relationship with the derived components would be less interpretable than relationships with the 4 original predictors would be." CreationDate="2011-02-26T03:35:15.477" UserId="2669" />
  <row Id="12330" PostId="6556" Text="&quot;What I am saying is that using Bayes Theorem will 1) tell you if a sufficient statistic of reduced dimensionality exists, and 2) what the sufficient statistic is for your problem.&quot; That has nothing to do with Bayes theorem, which is my point. Sufficiency is a property of the likelihood function and exists independently of whatever mode of inference you prefer." CreationDate="2011-02-26T03:39:19.063" UserId="26" />
  <row Id="12333" PostId="5926" Score="3" Text="Taking a wild guess:  If zero correlation, you may as well conduct separate anovas and thus simplify your task. If very high correlation, you may as well conduct anova on just one of the Y variables since results will be largely the same for all the others." CreationDate="2011-02-26T03:53:29.897" UserId="2669" />
  <row Id="12334" PostId="6556" Text="Incidentally, the &quot;&quot;factorisation&quot; theorem argument&quot; isn't mine but a result which goes back to Fisher &amp; Neyman and is in any intro stat inference text." CreationDate="2011-02-26T03:55:12.253" UserId="26" />
  <row Id="12335" PostId="7578" Text="@Erik. Thanks for your answer. I am &quot;accepting&quot; the other answer as I'm allowed to pick only one and that seemed to fit my current situation slightly better. But your answer was very helpful too." CreationDate="2011-02-26T04:06:10.050" UserId="1393" />
  <row Id="12336" PostId="6556" Text="@JMS - I'm not disputing that you can't find sufficient statistics in other ways from Bayes theorem.  Fisher was, I believe, the one who actually coined the term &quot;sufficiency&quot;.  But what I'm saying is that Bayes theorem is one of the easiest ways to *find* a sufficient statistic.  I would have a small bet that most &quot;intro stat text&quot; don't make much mention of Bayesian analysis, which is an essential tool for any good statistician." CreationDate="2011-02-26T04:07:57.960" UserId="2392" />
  <row Id="12337" PostId="7629" Text="@whuber. Thanks you both you and @mpiktas for the very nice analysis. I would tend to agree with you that Wooldridge is confusing numerical difficulties with theoretical properties of the estimator. Let's see if there are any other answers." CreationDate="2011-02-26T04:22:02.543" UserId="1393" />
  <row Id="12338" PostId="7639" Score="3" Text="4000 lines of data is not big.  Try doing analysis on a few million rows!" CreationDate="2011-02-26T04:29:42.720" UserId="2392" />
  <row Id="12339" PostId="7645" Text="Yeah, I figured it out before you gave the answer :) ... Thanks." CreationDate="2011-02-26T04:33:08.800" UserId="1102" />
  <row Id="12340" PostId="6597" Text="@onestop: Am I too naive thinking that, _maybe_, discriminant analysis could be employed here?" CreationDate="2011-02-26T07:53:35.710" UserId="144" />
  <row Id="12341" PostId="6028" Text="Thanks for the great link to UCLA! I'll dig into it... ;)" CreationDate="2011-02-26T08:48:50.513" UserId="2652" />
  <row Id="12342" PostId="7645" Text="$\log (1/x_2) = - \log (x_2)$ which is what I found" CreationDate="2011-02-26T09:29:43.077" UserId="2958" />
  <row Id="12343" PostId="7617" Text="This looks like exactly what I need. Can you explain how you got this expansion? I tried in a lot of ways and was unable to do that ..." CreationDate="2011-02-26T10:16:21.780" UserId="217" />
  <row Id="12344" PostId="7574" Text="+1 Good points in the update, especially the last paragraph.  When the Hessian is available--as is implicitly assumed throughout this discussion--one would automatically use its positive-definiteness as one of the criteria for testing any critical point and therefore this issue simply could not arise.  This leads me to believe the Wooldridge quotation must concern the Hessian at a putative global minimum, not at a mere critical point." CreationDate="2011-02-26T13:52:13.947" UserId="919" />
  <row Id="12345" PostId="7653" Text="Sounds like a term that is specific to that author.  If you have found otherwise please share what you have found elsewhere and show why other uses of the term do not clear things up." CreationDate="2011-02-26T14:39:55.320" UserId="2669" />
  <row Id="12347" PostId="7650" Text="+1 small multiples definitely seem like the way to go here. Also, maybe somebody could add the `data-visualization` tag to this?" CreationDate="2011-02-26T16:18:39.360" UserId="3396" />
  <row Id="12348" PostId="7655" Text="Tank you for this answer" CreationDate="2011-02-26T16:39:49.987" UserId="3019" />
  <row Id="12349" PostId="7644" Text="@probabilityislogic, the Wikipedia article itself provides you the answer to this question. If you assume normality of the &quot;true&quot; regressor, then you need further conditions on the distributions of the errors. If the true regressor is not Gaussian, then you have some hope. See [Reiersol (1950)](http://www.jstor.org/pss/1907835)." CreationDate="2011-02-26T17:07:43.710" UserId="2970" />
  <row Id="12350" PostId="7644" Text="@probabilityislogic, also, what do you mean by &quot;Solutions for just the intercept and slope are fine&quot;. Those are your only two parameters! Or were you hoping to try to back out the &quot;true&quot; regressor as well?" CreationDate="2011-02-26T17:10:15.813" UserId="2970" />
  <row Id="12351" PostId="7477" Text="@posdef, no worries. The discussion is interesting (to me) nonetheless. The best way to fight the feelings of frustration is to continue to learn when the opportunity arises. I, for one, am glad to have your participation here. Applied statistics as a field will only continue to grow through positive interactions with those interested in its applications. I would argue that in many cases the reverse is true as well." CreationDate="2011-02-26T17:15:48.380" UserId="2970" />
  <row Id="12352" PostId="7655" Text="@Marco: just bear in mind that I have not read the book, so I may be wrong! :)" CreationDate="2011-02-26T17:36:54.440" UserId="582" />
  <row Id="12354" PostId="7647" Text="Just out of curiosity... why on Earth are they blocking Google Docs??" CreationDate="2011-02-26T18:17:23.640" UserId="582" />
  <row Id="12355" PostId="7642" Text="Hey I have `/path/to/results.csv` and `/path/to/command.r`, and it gives me `source(&quot;/path/to/command.r&quot;)&#xA;Error in file(file, &quot;rt&quot;) : cannot open the connection&#xA;In addition: Warning message:&#xA;In file(file, &quot;rt&quot;) :&#xA;  cannot open file 'results.csv': No such file or directory`" CreationDate="2011-02-26T18:18:27.490" UserId="3439" />
  <row Id="12356" PostId="7655" Text="@nico I give you a +1 because this response really makes sense, even if you haven't read the book." CreationDate="2011-02-26T18:30:21.127" UserId="930" />
  <row Id="12357" PostId="7647" Text="@Gaetan Given the number of votes to close, you should probably try to reorient your question toward some kind of a statistical aspect rather than ask for technicalities that indeed are of the resort of the Help Center or another SE site." CreationDate="2011-02-26T18:33:48.767" UserId="930" />
  <row Id="12358" PostId="7642" Text="How can I fix the above?" CreationDate="2011-02-26T18:46:16.307" UserId="3439" />
  <row Id="12359" PostId="7658" Text="+1 The last paragraph is very well put." CreationDate="2011-02-26T19:43:03.153" UserId="919" />
  <row Id="12360" PostId="7659" Text="@user3447 What is the intended use of this scale? Do you aim at comparing two or more groups (wrt. mean scores) or so?" CreationDate="2011-02-26T20:09:18.617" UserId="930" />
  <row Id="12361" PostId="7658" Text="@chl: Thank you very much for such a nice answer! Can I ask a couple a words to explain further the sentence &quot;this is a way to reduce the chance of drawing incorrect conclusions from either good or bad data.&quot; In that setting, what are good or bad data? Thx again!" CreationDate="2011-02-26T20:33:34.207" UserId="3019" />
  <row Id="12365" PostId="7649" Text="Awesome comments, very helpful.&#xA;&#xA;No, I agree with you, I wouldn't claim a connection between a given&#xA;seed and performance. I would have varied the starting seed though for&#xA;each *set* of runs.&#xA;&#xA;I didn't know about the powercurve, sounds very useful, I'll check out&#xA;your link and also read up on it some more. I would not generalize the performance of my algorithms beyond the set&#xA;of problems used which I hope to be varied, i.e., unimodal,&#xA;multimodal, etc. Re running the algorithms to &quot;completion&quot; should I worry about&#xA;the starting seed? If so, why couldn't I just use the paired t-test in&#xA;that case?" CreationDate="2011-02-26T21:57:52.410" UserId="3311" />
  <row Id="12366" PostId="7649" Text=".. continued .. I.e., it took so many generations &quot;before&quot; and now so many&#xA;&quot;after&quot; the changes made to the algorithm. &#xA;&#xA;I will have to think about the unpaired t-test approach - my stats&#xA;knowledge is not very deep (or complete in breadth, so I have found&#xA;this site to be quite useful).&#xA;&#xA;Oh, I also plan to use this approach with PSOs as well down the line." CreationDate="2011-02-26T21:59:05.780" UserId="3311" />
  <row Id="12367" PostId="7654" Score="1" Text="You could combine these into `=if(countif(a1:a20,&quot;NO&quot;)=0,&quot;YES&quot;,&quot;NO&quot;)`" CreationDate="2011-02-26T22:00:44.967" UserId="2958" />
  <row Id="12368" PostId="7664" Text="@bayesian, all probability measures assign measure one to the entire sample space (by definition). Is that what you're looking for?" CreationDate="2011-02-26T22:07:40.387" UserId="2970" />
  <row Id="12370" PostId="7665" Text="thank you for repling, The questionnaire is measuring empathy, it has 33items that originally has 9 point likert scale range of which i reduced to a 4 point range. all the items and negative questions were kept the same.  i am looking into empathy score within 4 birth orders catogories. As i have made a change to the scoring of the scale i dont know if i need to run specific tests. because i am looking at birth order, the means for the questionnaire responses will be in the 4 catorgories, first born, middle, last and only.thank you for any help " CreationDate="2011-02-26T21:08:16.197" UserId="3447" />
  <row Id="12371" PostId="7665" Text="@user3447 Please use comments instead of answers. " CreationDate="2011-02-26T23:22:14.927" UserId="88" />
  <row Id="12372" PostId="7666" Text="@Henry, your definition of $\hat{\beta}$ doesn't make any sense to me. Are some &quot;hats&quot; missing?" CreationDate="2011-02-26T23:34:36.327" UserId="2970" />
  <row Id="12373" PostId="7644" Text="@cardinal - I meant that I didn't particularly care about the two scale parameters, and as you say, the &quot;true&quot; regressor $X_{i}$." CreationDate="2011-02-26T23:40:38.907" UserId="2392" />
  <row Id="12374" PostId="7644" Text="@probabilityislogic, I see. That makes sense." CreationDate="2011-02-26T23:45:25.610" UserId="2970" />
  <row Id="12375" PostId="6543" Text="@Christopher, the UC-Davis program looks good and I think you'll get a great education there. I wouldn't consider it &quot;less rigorous&quot; than other places. I thought the comment on their &quot;integrated B.S./M.S. degree&quot; **[page](http://anson.ucdavis.edu/undergrad/bs-ms)** was interesting and relevant to the thread: &quot;There is a high demand for statisticians, but the knowledge and skill achieved by those with a BS degree in Statistics are often not sufficient for the needs in the [government or industrial] workplace.&quot;" CreationDate="2011-02-26T23:57:24.227" UserId="2970" />
  <row Id="12376" PostId="7666" Text="It is mean to be the observed standard deviation of $\{y_i\}$ divided by the observed standard deviation of $\{x_i\}$. I'll change $\sigma$ to $s$ " CreationDate="2011-02-26T23:58:03.273" UserId="2958" />
  <row Id="12377" PostId="7664" Text="well, that is the reason this trick holds. I thought it might have a name" CreationDate="2011-02-26T23:58:19.263" UserId="3347" />
  <row Id="12378" PostId="7667" Text="I guess it does not have a name. thanks." CreationDate="2011-02-26T23:59:10.027" UserId="3347" />
  <row Id="12379" PostId="7666" Text="@Henry, can you clarify some of your comments? Something strikes me as being off based on your current description. Let $\hat{\beta}_{xy}$ be the slope assuming $y$ is the response and $x$ is the predictor. Let $\hat{\beta}_{yx}$ be the slope assuming $x$ is the response and $y$ the predictor. Then $\hat{\beta}_{xy} = \hat{\rho}s_y / s_x$ and $\hat{\beta}_{yx} = \hat{\rho} s_x / s_y$, where $\hat{\rho}$ is the sample *correlation* between $x$ and $y$. Hence the geometric mean of these two slope estimates is just $\hat{\rho}$." CreationDate="2011-02-27T00:15:18.860" UserId="2970" />
  <row Id="12380" PostId="7666" Text="@cardinal: No - when I see $x = by+c$ I mean the slope is $1/b$ since it can be rewritten as $y=x/b-c/b$. When you try to draw the two OLS lines on the same graph together with the observed points (e.g. with $y$ on the vertical axis and $x$ on the horizontal axis) you have to invert one of the slopes.  So I meant that you take the geometric mean of $\hat{\rho}s_y/s_x$ and $s_y/\hat{\rho}s_x$, which is simply $s_y/s_x$. Or, if you are unconventional enough to plot $y$ and $x$ the other way round for both lines and the observed points, then you get the inverse of that as the slope." CreationDate="2011-02-27T00:39:50.350" UserId="2958" />
  <row Id="12381" PostId="7649" Text="If running seed X on the same algorithm, but for different durations, you can use the paired t-test. But running seed X on two different algorithms, you cannot. The reason is that seed X might not interact with Alg.1 in the same way that it interacts with Alg.2. Paired t-tests are for when a result from one sample is related to a result from another sample. You cannot assume that seedX(Alg.1) and seedX(Alg.2) are related. But, you can test if they are, using a correlation coefficient, which is really easy to do." CreationDate="2011-02-27T00:40:45.187" UserId="3443" />
  <row Id="12382" PostId="7666" Text="@Henry - that's quite an interesting answer.  I don't necessarily doubt its validity, but one thing which does surprise me is that the correlation/covariance between $Y$ and $X$ is completely absent from the answer.  Surely this *should* be relevant to the answer?" CreationDate="2011-02-27T01:04:39.517" UserId="2392" />
  <row Id="12383" PostId="7666" Text="@probabilityislogic: Not *completely* as I do use the sign of the correlation/covariance ;)  But the magnitude may just be noise.  In standard OLS, the noise is all assumed to be on the &quot;dependent variable&quot; so the predicted change in the dependent variable due to the change in the independent variable is reduced by the magnitude of the correlation.  But if you don't know where the noise is, or indeed which is dependent and which independent, then how can you decide whether to increase or decrease? " CreationDate="2011-02-27T01:22:33.933" UserId="2958" />
  <row Id="12384" PostId="7557" Text="@Bernd Thank you very much" CreationDate="2011-02-27T02:16:08.003" UserId="3333" />
  <row Id="12385" PostId="7632" Text="Thanks.  I was hoping to see a worked example, but the right key words to search on was helpful.  I think I've found a good description.  Now to apply it." CreationDate="2011-02-27T02:19:49.500" UserId="3405" />
  <row Id="12386" PostId="7573" Text="@Bernd Thank you for your excellent answer and references. I installed R to try metafor. More difficult than I was expected. There are many tutorials available but none for non-statisticians :-)" CreationDate="2011-02-27T02:41:53.327" UserId="3333" />
  <row Id="12387" PostId="7642" Text="Your question here is unclear. Are you having a problem reading in the results.csv file when you run the script? If so, you might double check that you point the read.csv command at the proper directory(e.g. `read.csv(&quot;path/to/results.csv&quot;)`). If it's something else with the script, maybe you could tell us what command(s) you're using to run it?" CreationDate="2011-02-27T03:16:43.450" UserId="3396" />
  <row Id="12388" PostId="7594" Text="Good points on the distribution. In this case the stay duration is probably a &quot;long tail&quot; distribution. " CreationDate="2011-02-27T03:33:24.723" UserId="3418" />
  <row Id="12389" PostId="7666" Text="@Henry, thanks, that clarifies things a bit. Still, the methodology seems a little odd. I may think about this some more and ask another question or two." CreationDate="2011-02-27T03:45:52.333" UserId="2970" />
  <row Id="12390" PostId="7318" Text="I will look into that. Any tips on where to read about Poisson regression - with examples? I'd nevertheless still like to hear about any more general non-parametric interaction tests." CreationDate="2011-02-27T03:59:01.603" UserId="3285" />
  <row Id="12391" PostId="7666" Text="@cardinal: To stress my original point, this is not the best way to approach the problem if you have any idea what might actually be going on.  It does dreadful things to residuals if the correlation is low.  But it is simple if you think you have absolutely no information. Plato is supposed to have said something like *ignorance is the root of all evil*." CreationDate="2011-02-27T04:40:28.607" UserId="2958" />
  <row Id="12392" PostId="6556" Score="1" Text="You're missing my point entirely. Whatever mode of inference you prefer, sufficiency is simply a property of the likelihood function. Bayes theorem doesn't add anything at all. You *aren't* using Bayes theorem when you factor a likelihood! Nor does it add to the results that follow about minimal/complete/ancillary statistics. Deriving sufficient statistics doesn't require you to accept the principles of Bayesian inference; in fact it's intimately related to the ideas behind MLE. " CreationDate="2011-02-27T05:02:04.617" UserId="26" />
  <row Id="12394" PostId="6556" Text="Incidentally, having read many of those textbooks on statistical inference I'd take your bet. Casella &amp; Berger, Bickel &amp; Doksum, Lehman &amp; Casella, etc all at least discuss Bayesian parameter estimation if not some decision theory. Certainly it doesn't get the treatment it deserves but it's unreasonable to expect an introductory text to go too deep - and you can't become a statistician just on the back of them anyway. I'm an avowed Bayesian myself, but a solid understanding of classical statistics is important." CreationDate="2011-02-27T05:12:01.930" UserId="26" />
  <row Id="12396" PostId="7574" Text="+1, good point about $\Psi$ estimators." CreationDate="2011-02-27T05:15:15.763" UserId="2116" />
  <row Id="12397" PostId="7666" Text="@Henry, it seems to me that in the setting of this problem, the solution given is equivalent to assuming that $e_{y,i} = \beta e_{x,i}$ for each $i$. In an OLS setting, this method completely ignores the uncertainty in the data. It *pretends* that $y_i$ and $x_i$ are perfectly correlated and so assumes the data lie ***perfectly*** on a line, despite the fact that the data themselves contradict that. In either case, this strikes me as a very ***strong*** (and false) assumption." CreationDate="2011-02-27T05:20:37.257" UserId="2970" />
  <row Id="12398" PostId="7671" Text="@cardinal: Cool, I will read &quot;a new metric for probability distribution&quot; as soon as possible. Txh" CreationDate="2011-02-27T07:04:51.467" UserId="3019" />
  <row Id="12399" PostId="2467" Text="What software did you use to do FA?" CreationDate="2011-02-27T08:39:42.887" UserId="144" />
  <row Id="12400" PostId="7658" Text="@Marco I guess by &quot;bad data&quot; the author means data that either departs from the initial design (missing value on covariates, lost to follow-up, etc.) or that present unexpected errors discovered afterwards (e.g., randomization to treatment errors, site-specific confounding effect) -- in this case, the statistician still has to cope with the available data. Remember that there is no substitute for good data (even machine learning cannot save us :-), but we must acknowledge the limitations of a particular statistical technique when faced with data of poor quality." CreationDate="2011-02-27T09:04:10.323" UserId="930" />
  <row Id="12401" PostId="7658" Text="@chl: Ok, noted! Thx!" CreationDate="2011-02-27T09:46:09.613" UserId="3019" />
  <row Id="12402" PostId="7671" Text="Thanks! I didn't realize that JSD itself is already analogous to dist**2" CreationDate="2011-02-27T10:53:33.940" UserId="2759" />
  <row Id="12403" PostId="7573" Score="1" Text="@ Staty Despair: You might be interested in [Yet another short introduction to R with some emphasis on meta-analysis](https://github.com/berndweiss/r_meta-analysis_intro/raw/master/report/p_introToR.pdf). " CreationDate="2011-02-27T11:44:20.983" UserId="307" />
  <row Id="12405" PostId="7643" Text="@Peter (+1) Please note that `nls()` is in the base `stats` package. There exist, however, the [nls2](http://cran.r-project.org/web/packages/nls2/index.html) and [nlstools](http://cran.r-project.org/web/packages/nlstools/index.html) packages on CRAN." CreationDate="2011-02-27T12:05:21.310" UserId="930" />
  <row Id="12406" PostId="7666" Text="@cardinal: No - the simple solution cannot make any requirement that the $e_{y,i}$ and $e_{x,i}$ are correlated.  It is discussed in section 2.4 of Gillard's historical overview linked in my original answer, and has the names &quot;Geometric mean regression&quot;, &quot;Reduced major axis regression&quot;, &quot;Standardized principal component regression&quot; or &quot;Ordinary least products regression&quot; in the literature, having been invented several times. I think the strongest criticism is that it can appear to provide a result where vagueness would be better, particularly when the observed correlation is weak." CreationDate="2011-02-27T12:11:13.420" UserId="2958" />
  <row Id="12407" PostId="7672" Text="Thanks for the answer!" CreationDate="2011-02-27T15:21:51.437" UserId="3442" />
  <row Id="12408" PostId="2959" Text="@suncoolsu, that's not the typical definition of admissibility. The one you've given is (much) stronger. An admissible estimator is one that is *not* uniformly dominated, i.e., for every other rule against which it is compared, there is a value of the parameter for which the the risk of the present rule is (strictly) less than that of rule against which it's being compared. Conversely, an *inadmissible* estimator is one that is (weakly) dominated by *some* other estimator for *every* value of the parameter and is strictly dominated for *at least* one value by that same estimator." CreationDate="2011-02-27T15:24:09.203" UserId="2970" />
  <row Id="12409" PostId="7666" Text="@Henry, perhaps I'm having a slow last couple of days and you can find the flaw in my argument. If we fix $\hat{\beta} = s_y / s_x$, then we implicitly assume that the sample correlation between $x$ and $y$ is one. Now, this can **only** happen if $x$ and $y$ are (perfectly) linearly related. If we consider the model of this question, a simple substitution yields $y_i = \alpha + \beta x_i - \beta e_{x,i} + e_{y,i}$. The only way to get a perfect linear relationship between $x$ and $y$ is to take $e_{y,i} = \beta e_{x,i}$ for all $i$. Perhaps I'm being dense. Do you see a flaw?" CreationDate="2011-02-27T16:20:44.490" UserId="2970" />
  <row Id="12410" PostId="7666" Text="@cardinal: You are thinking in OLS terms and seem to have $\hat{\beta}_{xy} = \hat{\rho}s_y / s_x$ fixed in your mind;  in OLS this minimises the sum of squares of vertical residuals, which is not appropriate here as the $x_i$ have errors.   I will put some R code in my answer to illustrate the impact." CreationDate="2011-02-27T17:14:09.880" UserId="2958" />
  <row Id="12411" PostId="7647" Text="To answer nico and chl, they blocked Google Docs because it has extended into becoming a file archiving/sharing domain.  Thus, it could be a source of computer virus...  chl, I understand what you are saying.  I have asked that question else where and have not received any info on the subject whatsoever.  Given the knowledge of this specific community, it is my one last shot.  The question is not all that different from asking how to use any type of statistically oriented software.  I also ask how to use charts, which is technical in nature." CreationDate="2011-02-27T17:30:35.550" UserId="1329" />
  <row Id="12412" PostId="7666" Text="@Henry, my argument is that they are equivalent. Independently of how one arrives at it, by making the choice $\hat{\beta} = s_y / s_x$, this is ***equivalent*** to assuming that your two observable variables $x$ and $y$ are perfectly correlated in an OLS framework. I'm, of course, basing this on the (somewhat limited) information in your post. I'll try to have a glance at the link and await your $R$ code." CreationDate="2011-02-27T17:32:04.303" UserId="2970" />
  <row Id="12413" PostId="7666" Text="@Henry, I took a quick look at your link. Notice that their (2.1) is a weak form of the relation that I noted above. That is, for unbiasedness (though I think they really meant consistency), $\sigma_y^2 = \beta^2 \sigma_x^2$ is required, which is equivalent to a second-moment version of my (stronger) requirement." CreationDate="2011-02-27T18:10:04.317" UserId="2970" />
  <row Id="12414" PostId="7666" Text="@cardinal: Gillard does mean *unbiased*. This is why I said in my original answer &quot;It could also be seen as equivalent to making an implicit assumption that the variances of the two sets of errors are proportional to the variances of the two sets of observations&quot;" CreationDate="2011-02-27T19:27:03.123" UserId="2958" />
  <row Id="12415" PostId="7681" Score="10" Text="Sounds like you're after some sort of [diversity index](http://en.wikipedia.org/wiki/Diversity_index)" CreationDate="2011-02-27T19:29:50.627" UserId="449" />
  <row Id="12416" PostId="7666" Text="@Henry, I'm not so sure about that. First, no model is specified for the latent variable in the surrounding text. Is it fixed (i.e., *functional* form) or random (i.e., *structural* form)? If the former, some hope remains (though I actually doubt this---but haven't done the calculation). If it's the latter, there is **no** way that I can see that $\tilde{\beta}_{\mathrm{GM}}$ would be *unbiased* for all possible distributions of the latent variable. Furthermore, the surrounding text and following math expression strongly hint that they intended to say &quot;consistent&quot; instead of &quot;unbiased&quot;." CreationDate="2011-02-27T20:51:24.690" UserId="2970" />
  <row Id="12417" PostId="7666" Text="@Henry, your example is interesting. The constants are chosen &quot;perfectly&quot; to satisfy the only case in which the estimate could be consistent." CreationDate="2011-02-27T20:58:27.760" UserId="2970" />
  <row Id="12418" PostId="7308" Text="I wonder if this issue of &quot;definiteness&quot; is similar to the &quot;cholesky decomposition&quot; issue of taking negative square roots.  While the observed hessian is always PSD, this is only in *exact arithmetic* with infinite precision.  Perhaps rounding errors can make it negative definite in *floating point arithmetic* when it is &quot;close&quot; to being negative definite." CreationDate="2011-02-27T21:10:00.887" UserId="2392" />
  <row Id="12419" PostId="7609" Text="Thanks. Great support from a fellow adventurer in this world of statistics! ;) I'm currently reading Discovering Statistics using SPSS by Andy Field, which I'm surprised to enjoy (since it's a statistics textbook). I altered my COX analysis to measure survival on days instead of months, which luckily pushed my the significance of my 'risk' covariate below 0,05... :)" CreationDate="2011-02-27T21:18:43.003" UserId="2652" />
  <row Id="12420" PostId="7666" Text="@cardinal: If you mean $2000/400=5$, then it was indeed deliberate.  Apart from that ratio, the constants were arbitrary. At the start of this overlong thread, all I was offering was a simple method with some basic properties. " CreationDate="2011-02-27T21:50:35.723" UserId="2958" />
  <row Id="12421" PostId="7681" Score="2" Text="+1 for diversity indices -- especially if you can figure out what kind of index would be meaningful to your particular audience/topic. You might also check out [GINI Coefficients](http://en.wikipedia.org/wiki/Gini_coefficient), which are used to measure economic inequality on a scale from 0-1. In your case, &quot;more diverse&quot; would be closer to the 0 end of the scale." CreationDate="2011-02-27T21:57:35.000" UserId="3396" />
  <row Id="12422" PostId="7676" Text="Assuming you've got a valid rationale for comparing the same people in the two conditions without a true control group, I'm not sure the correlation across bootstrap iterations is really something to be worried about in itself. It seems like you are, in effect, calculating CIs for the Average Treatment Effect on the Treated (no Wikipedia entry yet, sorry) and interpreting that as a meaningful outcome of interest. That said, I'm having a hard time following exactly what steps you took to estimate the differences between conditions. Is it a simple difference of means? Something else? " CreationDate="2011-02-27T22:17:43.193" UserId="3396" />
  <row Id="12423" PostId="7686" Score="2" Text="Can you describe your problem a bit deeper?" CreationDate="2011-02-27T22:18:52.640" UserId="88" />
  <row Id="12424" PostId="7686" Text="Are you saying that Bayes' theorem $P(A=a|B=b) = P(B=b | A=a)\, P(A=a) / P(B=b)$ requires $P(B=b)$ to be positive, but when $B$ is a continuous random variable, for example Gaussian, then the probability that it takes any particular value is $0$?" CreationDate="2011-02-27T22:33:44.877" UserId="2958" />
  <row Id="12425" PostId="2959" Text="@cardinal Yup. You are right. I will correct it." CreationDate="2011-02-28T00:11:54.190" UserId="1307" />
  <row Id="12426" PostId="2959" Text="@cardinal. Using math is much easier than simplifying it in plain English. But that is just me. Thanks for the correction @cardinal" CreationDate="2011-02-28T00:17:22.083" UserId="1307" />
  <row Id="12427" PostId="2959" Text="@suncoolsu what do you mean by “never unbiased”?  Do you mean with respect to the prior?" CreationDate="2011-02-28T00:46:42.577" UserId="1670" />
  <row Id="12428" PostId="7633" Text="Henry: I am sorry about that. I didn't look at LHS to see that its a CDF. Thanks for your answer and I regret that I chose the answer before I realized it (I realized through the other guy's note)." CreationDate="2011-02-28T01:19:22.360" UserId="1102" />
  <row Id="12429" PostId="7688" Score="4" Text="What is multivariate statistics?  Seriously, this is such a vast subject and covers so much ground that it's impossible to know what you're looking for and certainly it would be impossible for any textbook to claim a &quot;detailed&quot; coverage of the subject!" CreationDate="2011-02-28T03:06:07.397" UserId="919" />
  <row Id="12430" PostId="2959" Text="@vqv There is a theorem by Blackwell which says that &quot;Bayes estimates are almost never unbiased&quot;. Please notice I am talking about asymptotic unbiasedness here." CreationDate="2011-02-28T03:31:33.830" UserId="1307" />
  <row Id="12431" PostId="7688" Text="@whuber. Good point. " CreationDate="2011-02-28T03:32:32.610" UserId="1307" />
  <row Id="12432" PostId="7308" Text="**[EDIT: Wooldridge has removed the statement in the second edition of his book and replaced it with the above argument. See page 414 of http://www.amazon.com/Econometric-Analysis-Cross-Section-Panel/dp/0262232588/ ]** So I guess the original statement was wrong. Closing the question. Apologies for bothering the good people here." CreationDate="2011-02-28T04:08:23.603" UserId="1393" />
  <row Id="12433" PostId="2959" Score="1" Text="@suncoolsu asymptotic unbiasedness is very different from the usual sense of &quot;unbiased&quot;. Any reasonable estimate should be asymptotically unbiased.  One more note: shouldn't the statement about admissible estimators be the other way around? ie every admissible estimator is generalized Bayes." CreationDate="2011-02-28T04:15:13.523" UserId="1670" />
  <row Id="12434" PostId="2959" Text="@suncoolsu, this is definitely one of those instances where a few symbols say something much more clearly than &quot;plain English&quot;. I fear my description is too clumsy, but it's hard to get it precise in few words. Another attempt would be: &quot;An admissible estimator dominates every other estimator at at least one point in the parameter space. The point of domination will typically vary according to the estimator being compared against.&quot; " CreationDate="2011-02-28T04:15:17.703" UserId="2970" />
  <row Id="12435" PostId="7308" Text="As if Wooldridge has read this thread :) The book was released before, but still quite amusing." CreationDate="2011-02-28T04:52:41.247" UserId="2116" />
  <row Id="12436" PostId="7575" Text="Thanks, Alex. Let me address a couple of your questions. In terms of problem's scale, I have ~17,000 events/transport decisions over about 4,000 people. There are around a dozen event level predictors, and a half-dozen at people level. " CreationDate="2011-02-28T05:57:29.797" UserId="3387" />
  <row Id="12437" PostId="7575" Text="I'm very interested in your SEM suggestion. I'll admit that my vague impression of SEM has been that it's like, say, game theory: very general, obstensibly can solve anything, but results in simplistic diagrams and not very satisfying answers. But your description, as simultaneously solving mixed models, makes sense. Can you point me to any papers/examples that seem particularly relevant to my situation? Thanks so much.  " CreationDate="2011-02-28T06:01:57.037" UserId="3387" />
  <row Id="12438" PostId="6157" Text="Thanks for your answer. graphviz is not easy to install with R, but it seams to be a great library" CreationDate="2011-02-28T06:38:04.303" UserId="1709" />
  <row Id="12439" PostId="1239" Score="2" Text="I think this question should be retained. The question would be suitable for stackoverflow. However, I think it's also suitable here on Stats.SE. Stats.SE has many existing questions regarding statistical software and implementation. Also, the question specifically concerns statistics." CreationDate="2011-02-28T06:55:48.927" UserId="183" />
  <row Id="12440" PostId="7477" Text="@cardinal: thanks! I'm trying to get rid of the prejudice and the frustration by trying to ignore the negative sides, and try to figure out the right &quot;mindset&quot;. I believe science is somewhat like language, you'll never be fluent unless you learn to think in that language instead of thinking in some other language and translating in your head. We'll see how well that works out for me... :) " CreationDate="2011-02-28T08:55:43.067" UserId="3014" />
  <row Id="12441" PostId="7695" Text="There is a problem with your first sentence, which probably is the cause of the further problems.  Is this your sentence, or is it citation? What is the pdf of wealth for one person? Pdf combines information about many values random variable takes. Information about one person's many values wealth is usually private and generally only available to tax offices. Usually given the income distribution of whole population and the wealth of particular person you can say what is the probability of finding wealthier or poorer persons. Is this what you are saying?" CreationDate="2011-02-28T09:04:59.757" UserId="2116" />
  <row Id="12442" PostId="7695" Text="The conjecture in the second paragraph is false. The portion of wealth is a **number**, distribution times the constant is **function**." CreationDate="2011-02-28T09:07:00.287" UserId="2116" />
  <row Id="12443" PostId="7695" Text="Thanks, mpiktas - I believe I fixed them.  My basic question is: &quot;Help me understand the relationship between P(X has this much wealth) distribution and distribution of wealth in the society as a whole." CreationDate="2011-02-28T09:28:12.817" UserId="3463" />
  <row Id="12444" PostId="7695" Text="@Marcus, I find your usage of term distribution a bit confusing. P(X has this much wealth)  is a number, $P(X&lt;x)$ for given $x$ is  a distribution of $X$. Do you per chance need something like [Lorenz curve](http://en.wikipedia.org/wiki/Lorenz_curve)?" CreationDate="2011-02-28T09:58:14.810" UserId="2116" />
  <row Id="12445" PostId="7694" Score="1" Text="Please provide the example. Now it is unclear what is your actual problem." CreationDate="2011-02-28T11:08:21.333" UserId="2116" />
  <row Id="12446" PostId="6919" Text="@cardinal Your addendum is the most useful piece of information I ran into on stats.SE, so far. All my warmest thanks for this. I simply reproduce here the reference you gave: http://research-repository.st-andrews.ac.uk/bitstream/10023/1591/1/Endres2003-IEEETransInfTheory49-NewMetric.pdf Endres and Schindelin, A new metric for probability distributions, *IEEE Trans. on Info. Thy.*, vol. 49, no. 3, Jul. 2003, pp. 1858-1860." CreationDate="2011-02-28T12:07:46.137" UserId="2592" />
  <row Id="12447" PostId="7699" Text="Does your regression has an intercept? Omitting intercept sometimes can cause problems. Also could you post reproducible example? It is clear that you made an error, as @onestop pointed out, but with example available it may be possible to point out where the error is exactly." CreationDate="2011-02-28T12:29:50.460" UserId="2116" />
  <row Id="12448" PostId="7689" Text="It escapes me how that paper is relevant to the question. Do you mind elaborating?" CreationDate="2011-02-28T13:10:13.720" UserId="1036" />
  <row Id="12449" PostId="7701" Text="Is your binary categorical variable _O_ your dependent variable? Could you tell us what your model looks like now?" CreationDate="2011-02-28T13:48:38.960" UserId="3396" />
  <row Id="12450" PostId="7705" Text="That will not prevent the predictions of the network from going outside the range 0-1.  If it is necessary to constrain the outputs of the network to lie within a particular range, a non-linear activation function is unavoidable." CreationDate="2011-02-28T13:53:28.990" UserId="887" />
  <row Id="12451" PostId="7654" Text="Sorry for late reply, but THANKS. I ended up using `=IF(COUNTIF(B5:B17, &quot;&lt;&gt;Yes&quot;), &quot;No&quot;, &quot;Yes&quot;)`" CreationDate="2011-02-28T13:55:54.043" UserId="3446" />
  <row Id="12452" PostId="7701" Text="@ashaw: Variable $O$ is the outcome of a number of different laboratory tests. I'm not modeling $O$ as it is categorical. Males who are $True$ for $O$ are older than females who are $True$ for $O$, but this is expected as all males in the cohort are older than females. I want to know if $O$ is only occurring in older males for example. Are older males more likely to be $True$ for $O$?" CreationDate="2011-02-28T14:01:03.633" UserId="2824" />
  <row Id="12453" PostId="7703" Text="I replaced the linear function by a siglog-function. Strangely, the neural net predicts now 0.5 for whatever data inputs. May this be a consequence of still using the sum-of-squares metric?" CreationDate="2011-02-28T14:02:58.257" UserId="3465" />
  <row Id="12455" PostId="7703" Text="The use of the sum-of-squares error metric shouldn't be a problem, assymptotically it will give a network that predicts the conditional mean of the data regardless of the distribution and output layer activation function.  Can you give some more information about the data, size of the netwok, training algorithm etc? What is the distribution of target values like?  Are you using regularisation?" CreationDate="2011-02-28T14:10:06.543" UserId="887" />
  <row Id="12456" PostId="7701" Text="So if I understand you correctly, the first formula I posted below is inaccurate. $O$ is a covariate in your model and you're using $O$, $Age$, and $Gender$ to model some other outcome $Y$. If that's true, I can try to adjust the answer below." CreationDate="2011-02-28T14:10:25.703" UserId="3396" />
  <row Id="12457" PostId="7707" Text="I thought linear regression required that all variables be continuous (I didn't add the linear regression tag). $O$ represents whether the infection in the participant is a particular variant or not (in the subset of the data in question, all participants are &quot;infected&quot;; some are infected with the variant and are thus $True$ for $O$). So I wanted to determine if all the men with this variant are older because the variant only infects older men or because men in the study are, on average, older than women. $O$ is binary." CreationDate="2011-02-28T14:12:43.857" UserId="2824" />
  <row Id="12458" PostId="7707" Text="Aha, that makes things a little different. The difference between linear and logistic regression stems from nature of your **outcome**. If the outcome is continuous and distributed reasonably normally, OLS may apply. If the outcome is binary, logistic regression may apply. (I'm not coming down definitely in either case because there can be many other factors worth taking into account when picking a model). Both logistic and OLS models can accommodate binary, categorical, ordinal and continuous covariates." CreationDate="2011-02-28T14:23:12.907" UserId="3396" />
  <row Id="12459" PostId="7701" Text="Thanks for the answer and discussion. I have to think more about this." CreationDate="2011-02-28T14:25:11.850" UserId="2824" />
  <row Id="12461" PostId="7703" Text="Training algorithm is back propagation. 500 training cycles, learning rate 0.3 and 0.2 momentum. 1 hidden layer with 3 neurons. The data consists of 3 attributes, and one target attribute. 20000 samples. The target attribute is very heavily(!!!) skewed within the range of 0 and 1, the mean is 0.077, SD 0.185." CreationDate="2011-02-28T14:29:38.187" UserId="3465" />
  <row Id="12462" PostId="7701" Text="Feel free to post more information too - you don't necessarily need to provide the specifics of the study, but if you can provide reasonably clear examples there's a good chance somebody can help!" CreationDate="2011-02-28T14:29:49.470" UserId="3396" />
  <row Id="12463" PostId="7706" Text="@Misha, your question body does not mention anything about survival analysis, so I've changed the title. I still left the tag though, since it might generate different answers. Please consider adding more details if the survival analysis is really important in your question." CreationDate="2011-02-28T14:43:10.627" UserId="2116" />
  <row Id="12464" PostId="7707" Text="@ashaw: What does it mean to fit a linear model between a continuous variable ($age$) and a categorical factor ($gender$)? I was using Wilcoxon to investigate continuous versus categorical variables." CreationDate="2011-02-28T14:47:08.693" UserId="2824" />
  <row Id="12465" PostId="7695" Text="The Lorenz curve is very appropriate.  But it's cumulative.  I think I'm looking for the derivative of the Lorenz curve (of a Pareto distribution or in general).  What is that?" CreationDate="2011-02-28T14:57:50.503" UserId="3463" />
  <row Id="12466" PostId="7695" Text="mpiktas, isn't P(X&lt;x) also a number, given a particular x (&quot;this much wealth).  And, when x is a parameter and not a fixed number, so is P(X has x much wealth)." CreationDate="2011-02-28T14:58:43.433" UserId="3463" />
  <row Id="12467" PostId="7707" Text="@SabreWolfy: Regression models can produce coefficients for continuous, categorical or binary variables. In the case of a binary variable, the results usually provide a coefficient for only one of the values (e.g. $Female = True$) with the other value ($Female = FALSE$) implicitly used as the baseline of comparison. " CreationDate="2011-02-28T15:04:35.570" UserId="3396" />
  <row Id="12468" PostId="7707" Text="What sort of outcome variable are you using in this case?" CreationDate="2011-02-28T15:06:03.400" UserId="3396" />
  <row Id="12469" PostId="7542" Text="new udpate! (and new detailed title)" CreationDate="2011-02-28T15:22:29.617" UserId="3342" />
  <row Id="12470" PostId="7642" Text="Open up R and just copy/paste the commands into the console.  Use `MyData &lt;- read.csv('path/to/MyData.csv')`, and then run the rest of the script." CreationDate="2011-02-28T15:29:52.353" UserId="2817" />
  <row Id="12471" PostId="7703" Text="In that case, it might be worth transforming the target to have a less skewed distribution before training.  Is it a problem with many values exactly at zero (like rainfall for example) or are there no special values?  " CreationDate="2011-02-28T15:48:56.380" UserId="887" />
  <row Id="12472" PostId="7365" Text="@mpiktas It's a nice exposition; I wish I could vote it up again!  One thing, though: when you find a Hessian to be indefinite, it's easy to check whether the solution is on the boundary.  If not, don't give up and conclude the &quot;model may be inappropriate&quot;!  It's more likely the minimizing algorithm is not a good one for the problem.  Furthermore, in your case the model is obviously appropriate: it is identical to the model that produced the data.  The difficulty is that (due to the high error variance) it is essentially a one-parameter problem; another parameter is almost unidentifiable." CreationDate="2011-02-28T16:20:08.763" UserId="919" />
  <row Id="12473" PostId="2959" Text="@vqv In re: &quot;Any reasonable estimate should be asymptotically unbiased.&quot; I don't necessarily disagree, but this is a strong statement, no? I haven't got an example offhand but it seems plausible that there exist estimators with nice finite sample properties that happen to be inconsistent. But perhaps not. And I think you're correct that every admissible estimator corresponds to a Bayes rule." CreationDate="2011-02-28T16:22:15.500" UserId="26" />
  <row Id="12474" PostId="2959" Text="@vqv - You are correct as well. Sorry for misinterpreting your question." CreationDate="2011-02-28T16:23:05.033" UserId="1307" />
  <row Id="12475" PostId="2959" Text="@JMS. I think @vqv is correct, this is another way of thinking about consistency." CreationDate="2011-02-28T16:25:41.413" UserId="1307" />
  <row Id="12476" PostId="7468" Text="I ended up implementing a variation of this algorithm with a metric called  Dunn’s clustering validity index" CreationDate="2011-02-28T16:26:47.700" UserId="255" />
  <row Id="12477" PostId="2959" Text="@suncoolsu I wasn't so clear; I should have stuck with asymptotically biased (consistent under L1). Still, I'm not so sure that there do not exist reasonable estimators that are asymptotically biased but have good finite sample properties. But like I said, I can't think of an example offhand. " CreationDate="2011-02-28T16:31:31.690" UserId="26" />
  <row Id="12478" PostId="7712" Text="This isn't really about vectorizing calls to `dpois`. It's more a question about creating vectors." CreationDate="2011-02-28T16:49:28.980" UserId="3396" />
  <row Id="12479" PostId="7713" Text="@ashaw. I should have looked at this carefully." CreationDate="2011-02-28T16:59:22.737" UserId="1307" />
  <row Id="12480" PostId="7714" Text="SORRY THE DATA IS " CreationDate="2011-02-28T17:01:36.300" UserId="3472" />
  <row Id="12481" PostId="7713" Text="no worries, glad I could answer the question." CreationDate="2011-02-28T17:03:15.007" UserId="3396" />
  <row Id="12483" PostId="7713" Text="The only problem is with the 0.1:0.3 part. `1:3` is already a vector and doesn't require concatenation." CreationDate="2011-02-28T17:08:15.763" UserId="697" />
  <row Id="12484" PostId="7713" Text="indeed. edited accordingly." CreationDate="2011-02-28T17:15:27.930" UserId="3396" />
  <row Id="12485" PostId="7714" Score="1" Text="Also, there really is no single &quot;best&quot; test for reliability. Most inter-rater reliability statistics I've encountered are for nominal, ordinal, interval or ratio level data. It's possible that with continuous data, something more like correlation or rank correlation statistics may be more useful. I'd encourage you to look into what sorts of techniques are common in your particular field and/or to say more about the scale involved if you'd like a more informative answer." CreationDate="2011-02-28T17:25:12.913" UserId="3396" />
  <row Id="12486" PostId="2959" Text="@suncoolsu, consistency and asymptotic unbiasedness are different things. To get some notion of equivalence requires more structure on the problem, like uniform integrability. There are some estimators that are consistent yet asymptotically biased and, of course, estimators that are asymptotically unbiased but inconsistent. " CreationDate="2011-02-28T17:38:25.367" UserId="2970" />
  <row Id="12487" PostId="2959" Text="@JMS Asymptotic unbiasedness does not imply consistency.  Asymptotic unbiasedness: $|E \hat{\theta} - \theta| \to 0$; L1 consistency: $E|\hat{\theta} - \theta| \to 0$.  L1 consistency is much stronger.  Anyway, this is probably far off the topic of the original question." CreationDate="2011-02-28T17:49:20.100" UserId="1670" />
  <row Id="12488" PostId="7676" Text="@ashaw: Comparison of conditions experienced by the same individuals is a standard methodological approach that increases power by permitting removal of between-Ss variance that usually obscures effect variance in completely between-Ss designs. Regarding the computation of the difference score CI, I have edited the post above to clarify that on each iteration the condition means were computed and collapsed to a difference, yielding a distribution of differences from which the 95% interval." CreationDate="2011-02-28T18:19:53.510" UserId="364" />
  <row Id="12489" PostId="7713" Score="4" Text="A more compact version of the same answer: `dpois(1:3, 1:3/10)`." CreationDate="2011-02-28T18:33:31.510" UserId="1657" />
  <row Id="12490" PostId="7690" Text="my question arose from calculating the probability of datapoints generated by a gaussian and using bayes theorem in maximizing the likelihood. though the probability of a point is zero it seems the handwavy way of thinking of it as an infinitesimal probability is ok. " CreationDate="2011-02-28T18:38:27.620" UserId="3456" />
  <row Id="12491" PostId="7676" Text="@mike, that helps for sure. Apologies if you read the first phrase of my comment to suggest that the approach was invalid in any way. Rather, I meant to point out that the validity of my response was contingent on the validity of the study design. In any event, now I can mull over a real response a little more..." CreationDate="2011-02-28T18:52:13.960" UserId="3396" />
  <row Id="12492" PostId="7717" Text="Thank you so much for that!! Do you mean the Bald-Altman plot. So the Kappa score is not applicable in the continuous data?? am yet collecting more data( my sample size is 146, but i intend to test 15 for inter-rater reliability)" CreationDate="2011-02-28T19:19:05.113" UserId="3472" />
  <row Id="12493" PostId="7714" Text="@ashaw No, you can't use simple correlation-based measures since they are invariant by shifting the mean of one of the two measures, or adding a constant amount to one of them, which obviously lead to decreased reliability (overall, the two raters disagree to a larger extent) but that would not be reflected in such an association measure." CreationDate="2011-02-28T19:27:47.540" UserId="930" />
  <row Id="12494" PostId="7717" Text="@Edwin @onestop's response is fine, and you can complete this graphical/numerical approach by computing an Intraclass correlation, for example. Several earlier posts dealt with those topics (search around the two tags you associate to your question). I also gave a +1 to your question, so that you can vote up @onestop's response as you seem to consider it being helpful." CreationDate="2011-02-28T19:31:30.220" UserId="930" />
  <row Id="12495" PostId="7715" Text="@sask It is worth rewording a little bit your question so that people understand (a) what kind of data you have, (b) what model you used, and most importantly (c) what you are interested in. As it stands, we don't know if you are after a purely algebraic explanation or have real results at your disposal." CreationDate="2011-02-28T19:37:52.773" UserId="930" />
  <row Id="12496" PostId="7674" Score="1" Text="Great answer. If I had another vote, I would vote for this answer again." CreationDate="2011-02-28T19:38:27.750" UserId="847" />
  <row Id="12497" PostId="7716" Text="@vqv +1 and thanks for the links." CreationDate="2011-02-28T19:39:34.130" UserId="930" />
  <row Id="12498" PostId="7717" Text="@Chl..Thanks a lot for that. I just want to get it clear, so the Bland-Altman plot will be the solution..." CreationDate="2011-02-28T19:40:29.060" UserId="3472" />
  <row Id="12500" PostId="7714" Text="@chl - Definitely, I wouldn't expect simple correlation to actually capture reliability at all. In my head, the emphasis was on the &quot;something more like&quot; part of that sentence. Should be clearer from these comments at least...  " CreationDate="2011-02-28T20:12:24.297" UserId="3396" />
  <row Id="12501" PostId="2959" Text="Of course, where is my head. Good catch." CreationDate="2011-02-28T20:23:44.600" UserId="26" />
  <row Id="12503" PostId="7595" Text="@williamyarberry Register here and on maths with the same OpenID to recover the ownership of your question (if you think it is worth it, of course ;-) ). You can do it by clicking &quot;log in&quot; at the top bar." CreationDate="2011-02-28T20:31:42.720" UserId="88" />
  <row Id="12504" PostId="2959" Text="@everyone. Sorry. I meant - &quot;a step towards consistency&quot;. Because consistency is a sum of squared bias and variance in $L_2$ specifically. " CreationDate="2011-02-28T20:33:04.050" UserId="1307" />
  <row Id="12505" PostId="7512" Text="By correlate, I mean a measure of association - something like pearson's r. N and n were typos; I just meant I have two sample sizes. So, for one variable, I have 50 males and 50 females and I want to correlate their scores together. For another variable, I have 99 males and 99 females and I want to correlate their scores. My variables are measures of delinquency counts, which are relatively rare events. Thanks so much for your response!!" CreationDate="2011-02-28T20:37:51.210" UserId="3368" />
  <row Id="12506" PostId="7717" Text="@Edwin Happy to know that you find B-A plot a good compromise. In this case, you might just upvote @onestop's response and wait few days to see if no other response come up before accepting it." CreationDate="2011-02-28T21:04:03.480" UserId="930" />
  <row Id="12507" PostId="2959" Text="@suncoolsu, you must be using a definition of consistency that is different. The usual definition is a statement about *convergence in probability*, not in any $L^p$ space. Of course, if you have the latter, you get the former for free. " CreationDate="2011-02-28T21:08:58.990" UserId="2970" />
  <row Id="12508" PostId="7720" Score="3" Text="@dfrankow Somewhat crude and certainly very partial help for your first two questions, but `methods(&quot;profile&quot;)` will give you the (S3 in this case) methods associated to an R `profile` object, then you will see than there is a dedicated method for `polr` results, that you can browse on-line by typing `getAnywhere(&quot;profile.polr&quot;)` at the R prompt." CreationDate="2011-02-28T21:13:38.527" UserId="930" />
  <row Id="12509" PostId="7706" Text="@mpiktas, The data are indeed survival data in that the dependent var is the time from diagnosis until death. Out of 1500 patients I have 15 that are still alive and hence right censored. " CreationDate="2011-02-28T21:20:53.590" UserId="1291" />
  <row Id="12510" PostId="7720" Text="Thanks! Source code is good.  Explanation would be even better. :)" CreationDate="2011-02-28T21:21:27.397" UserId="2849" />
  <row Id="12511" PostId="7719" Score="1" Text="@Kieran It's hard to imagine a stacked barplot with SE bars (at least for me!). Why not simply use a [dotplot](http://www.statmethods.net/graphs/dot.html)?" CreationDate="2011-02-28T21:21:28.503" UserId="930" />
  <row Id="12513" PostId="7723" Score="4" Text="There are good questions in here but you're asking too much at once.  Focus on one issue at a time and formulate a specific question for it.  Specify what you do know and what efforts you have made towards answering the question.  &quot;How do I even start&quot; doesn't give us anything to go on--we would just refer you to a textbook or Wikipedia (which is not a bad idea, by the way)." CreationDate="2011-02-28T22:32:36.770" UserId="919" />
  <row Id="12514" PostId="7556" Text="hey i edited the question for a more specific use case for the data" CreationDate="2011-02-28T22:36:42.153" UserId="3392" />
  <row Id="12515" PostId="7587" Text="Hey i edited the question to provide more details, I don't know anything about the distribution of the data, though most likely normal if i have to guess. " CreationDate="2011-02-28T22:37:46.037" UserId="3392" />
  <row Id="12516" PostId="7677" Text="Thank you very much -- your suggestion worked very well indeed. Also, thank you for your pointer to ezMixed, which is also a huge help -- I wasn't aware that ez offered support for mixed models.&#xA;(Actually, while I'm at it: Thank you for your work on ez, which is a great pleasure to use :-) )" CreationDate="2011-02-28T23:04:58.623" UserId="3451" />
  <row Id="12517" PostId="7556" Text="What if the mean is 0? " CreationDate="2011-02-28T23:13:16.810" UserId="2965" />
  <row Id="12518" PostId="7556" Text="If your data is such that the mean can be zero, then this is not a good solution. I.e. it is not meaningful to divide by the mean. This is even if the actual mean is not zero. E.g. you average something like the &quot;x-position&quot; of points. The mean here can be zero, and I'd argue dividing by the mean is not meaningful, even if in your particular sample the mean is not 0." CreationDate="2011-02-28T23:44:35.260" UserId="3369" />
  <row Id="12519" PostId="7715" Text="Thanks for the comment - I am not actually using a particular data set. It is simply an exercise limited to those few pieces of information. So the 'purely algebraic explanation' is what I am after - I am quite sure there is supposed to be a link between the residual variability from which the R-squared is calculated and the formula for the ESS, but I just can't figure it out... Thanks!!" CreationDate="2011-02-28T23:58:08.877" UserId="3473" />
  <row Id="12520" PostId="7715" Text="Isnt the formula Explained sums of squares/Total sum of squares? Does that answer your question?" CreationDate="2011-03-01T00:07:36.683" UserId="656" />
  <row Id="12521" PostId="7642" Text="alternatively, on windows/mac just go source or read csv file.choose() and a window will open up which you can navigate to open the required files." CreationDate="2011-03-01T00:11:18.303" UserId="656" />
  <row Id="12522" PostId="7702" Text="+1 for this nice, readable and freely available resource!" CreationDate="2011-03-01T00:55:29.790" UserId="1050" />
  <row Id="12523" PostId="7721" Score="1" Text="@Andy, can you clarify your question a little bit? Do you have hundreds of thousands of predictors for each response? Or hundreds of thousands of responses? Or perhaps, a data matrix with hundreds of thousands of entries? Or maybe you meant &quot;hundreds ***or*** thousands&quot;? The answer to this will provide some guidance. Also, how many class categories do you have? What sort of computing resources are available? Are your features sparse or dense?" CreationDate="2011-03-01T01:52:46.353" UserId="2970" />
  <row Id="12524" PostId="7721" Text="@Andy, for ***binary*** logistic regression, Paul Komarek built a package as his dissertation research at Carnegie Mellon. It has some optimizations for sparse features. I don't believe it does multinomial logistic regression, but I could be mistaken. He actually uses a ridge version of logistic regression, but with a fixed ridge parameter. His claim is that that is good enough for (most) all problems. The fitting method uses conjugate gradients if I recall. I'm not sure it's amenable to parallelization, though." CreationDate="2011-03-01T01:55:34.820" UserId="2970" />
  <row Id="12525" PostId="7727" Text="@BobC, maybe one of the moderators can have your post migrated to this site. That would be the most reasonable. As for your technical questions, first of all, are you using the FFT to do the correlation? That should be feasible for 2 million data points on a half-decent computer. Your signal-to-noise ratio looks reasonably high, so you should be in business. A quick-and-dirty cut would be to fill in the missing data with either the last available sample or with zeros. The creep from sampling-interval differences may be the most challenging &quot;feature&quot; of your data to deal with." CreationDate="2011-03-01T02:16:36.270" UserId="2970" />
  <row Id="12526" PostId="7727" Text="@cardinal: I did indeed try an FFT, only to get garbage as a result.  The 'interesting' features readily visible in the data are indistinguishable from noise in the FFT.  However, I have done FFTs only on the entire data set: Perhaps a moving window FFT would provide better results, but I haven't yet been able to find a computationally efficient way to implement it.  I suspect a Wavelet transform could help, but I'm unfamiliar with it (but am slowly learning about it)." CreationDate="2011-03-01T02:24:34.397" UserId="3479" />
  <row Id="12527" PostId="7727" Score="1" Text="@BobC, what I meant was, did you consider an FFT-based implementation for calculating the correlation? Direct convolution is $O(n^2)$, but an FFT-based implementation would reduce this $O(n \log n)$, making it feasible. As for looking at the FFT itself, with 2 million data points, your frequency resolution will be very high. Any sampling creep and other stuff is bound to wash the signal out on a per-frequency basis. But, you should be able to aggregate over many bins to bring the signal out of the noise. Something like a Welch approach or maybe a custom windowing technique." CreationDate="2011-03-01T02:29:22.983" UserId="2970" />
  <row Id="12528" PostId="7727" Text="@BobC, off the top of my head, it seems like some variant of an *overlap-and-add* or *overlap-and-save* algorithm could be used to do a sliding-window FFT. Sliding the samples within a window just amounts to a phase shift, so all you have to do is compensate for those samples that &quot;fall off&quot; the left end and those that &quot;come in&quot; on the right end." CreationDate="2011-03-01T02:31:12.903" UserId="2970" />
  <row Id="12529" PostId="2959" Text="@cardinal. I can't resist my temptation to use math anymore. $E[\hat \theta - \theta]^2 = E[\hat \theta - E[\hat \theta] + E[\hat \theta] - \theta]^2 = E[\hat \theta - E[\hat \theta]]^2 + ( E[\hat \theta] - \theta)^2 = Var[\hat \theta] + Bias^2$. So basically @cardinal and me are talking about the same thing but from a different perspective. If Variance **and** Bias go to 0 in **probability**, consistency holds, otherwise not. Or am I still incorrect? Thats what I have learnt in my courses here." CreationDate="2011-03-01T02:37:23.207" UserId="1307" />
  <row Id="12530" PostId="7721" Text="@cardinal : Thanks for your answer. Good questions ! I have 12 classes, and the number of predictors can go from 80k to 200k depending on the source of the data. And yes this is a highly sparse sparse data set. Regarding computing resources, I think I might be able to get access to a cluster (~10nodes or so), but I think if push comes to shove, I can try and get more computing resources also." CreationDate="2011-03-01T02:39:47.077" UserId="3301" />
  <row Id="12531" PostId="2959" Text="@suncoolsu, Let $\hat{\beta}_n$ be an estimator for a parameter, $\beta$, derived from a sample of size $n$. Then $\hat{\beta}_n$ is *consistent* for $\beta$ if $\hat{\beta}_n \to \beta$ ***in probability***. This is a weaker requirement than requiring that $\mathbb{E} |\hat{\beta}_n - \beta|^p \to 0$ for some $p &gt; 0$." CreationDate="2011-03-01T02:44:30.053" UserId="2970" />
  <row Id="12532" PostId="2959" Text="@suncoolsu, I see you edited your comment while I was writing mine. Convergence in *any* $L^p$ space guarantees convergence in probability. This can be seen by a straightforward application of Markov's inequality. But, convergence in probability is weaker than this requirement. Hence, consistency of a parameter is (substantially) weaker than requiring the parameter to converge to its true value in, say, $L_2$, the latter being more or less what it looks like you're stating." CreationDate="2011-03-01T02:48:50.873" UserId="2970" />
  <row Id="12533" PostId="7721" Text="@Andy, how much information are in those features? Can't be much. Are the features continuous-valued or binary? There might be a quick-and-dirty approach where, e.g., you can train 12 binary-logistic regressions separately and then use there predictions in a final step to get a proper distribution for the twelve-class case. (For example, train a 12-feature multinomial logistic regression when you're done training each of the individual ones. Obviously, that's suboptimal, but depending on the application, it might be both computationally feasible and good enough.)" CreationDate="2011-03-01T02:56:00.277" UserId="2970" />
  <row Id="12534" PostId="2959" Text="@cardinal. Yeah I was talking about MSE and not the convergence." CreationDate="2011-03-01T03:02:43.007" UserId="1307" />
  <row Id="12535" PostId="7721" Text="@Andy, I just checked and **[Komarek](http://komarix.org/ac/lr)** does provide some very simple suggestions for multi-class extensions. They're simpler than what I was suggesting. One reason mine might be better is that if you fit 12 separate binary logistic regressions, the associated parameters aren't necessarily on similar scales. By having a final step that is a full multinomial logistic regression, the parameters in the final fit can compensate for those scale differences that a simple renormalization wouldn't. Besides, simple renorming wouldn't change your predicted class anyway." CreationDate="2011-03-01T03:04:23.263" UserId="2970" />
  <row Id="12536" PostId="7587" Text="Thanks for clarifying. If you're dealing with people's behavior on the Internet, the distribution is probably far from normal. In fact, you're likely to find very skewed distributions on almost any measure (that's the fun of the long tail!). I think you're instinct (mean + standard errors in some intuitive fashion) is great. The visualization suggestions others have made should be useful too. You might consider also presenting information about just how unequal the distribution is as well as data about &quot;average&quot; users at the bottom and top of the distribution too." CreationDate="2011-03-01T03:08:55.390" UserId="3396" />
  <row Id="12538" PostId="2959" Text="@suncoolsu, Some people refer to convergence of a parameter in $L_2$ as *mean-square consistency*. But, in general, a consistent estimator need not have asymptotically negligible bias *nor* variance. In fact, it need not even have finite variance. If you're really sadistic, you can undoubtedly cook up an example where the variance *isn't even defined*." CreationDate="2011-03-01T03:13:28.700" UserId="2970" />
  <row Id="12539" PostId="7728" Text="Great point, @rolando - since the original post refers to training and validation datasets I suspect your response may actually be more usable for @Figaro. " CreationDate="2011-03-01T03:15:28.883" UserId="3396" />
  <row Id="12540" PostId="2959" Text="@cardinal. Thanks for the inputs :-) Helps improve my understanding!" CreationDate="2011-03-01T03:17:16.370" UserId="1307" />
  <row Id="12542" PostId="2959" Text="@suncoolsu, mine, too. That's one of the reasons I enjoy this site. Cheers." CreationDate="2011-03-01T03:23:08.257" UserId="2970" />
  <row Id="12543" PostId="7599" Text="In response to your edit, @williamyarberry, I apologize for not realizing what exactly you meant by the original question. I've been poking around in the documentation for MASS and I'm also uncertain why the output doesn't provide any summary information about model fit. Presumably, you could use the `fitted.values` and `residuals` stored in the `M10` object to calculate $R^2$ independently, but I do not know if that's a sensible approach with this method." CreationDate="2011-03-01T03:26:48.567" UserId="3396" />
  <row Id="12545" PostId="7732" Text="@mbq, @imelza, this is pretty much a duplicate of **[this question](http://stats.stackexchange.com/questions/7727/how-to-correlate-two-time-series-with-gaps-and-different-time-bases)**. Usually if there is a question that would provide the answer you need, it's a little frowned upon to submit another that is very similar. You'd be better off waiting to see if you can glean an adequate answer from the original one. Regards." CreationDate="2011-03-01T03:31:36.377" UserId="2970" />
  <row Id="12547" PostId="7732" Text="Hi, I understand that the questions are very similar, but I cannot comment on that question and I really need this answer. " CreationDate="2011-03-01T03:36:49.887" UserId="3233" />
  <row Id="12548" PostId="6919" Text="@Didier, well, it was more a happy accident than anything else. No one was responding to the other question, so I decided to try to figure out what the Jensen-Shannon Divergence was in the first place. Once I found the definition, it seemed reasonable to connect the two questions via my addendum. I'm glad you found it useful. Regards." CreationDate="2011-03-01T03:38:06.200" UserId="2970" />
  <row Id="12549" PostId="7599" Text="I should note that a general point common to many Robust/Resistant techniques (I've mostly worked with Huber estimators) is that they require you to bootstrap standard errors &amp; $R^2$ values. Two discussions I found useful in my searches on this topic (and which contain code for bootstraping standard errors for an lqs model) can be found [here](http://www.stat.psu.edu/~dhunter/R/testboot.html) and [here](http://www.biostat.jhsph.edu/~iruczins/teaching/jf/ch13.pdf) (PDF). I also recommend exploring the sources cited in the MASS documentation." CreationDate="2011-03-01T03:43:13.457" UserId="3396" />
  <row Id="12550" PostId="7732" Text="Is the time scale identical for both of the variables? Can you post an example of your data set?" CreationDate="2011-03-01T04:28:25.127" UserId="2116" />
  <row Id="12551" PostId="7721" Text="@cardinal : Actually I do not have a sense of how much information is likely to be there in the features, since I am yet to get access to the data :-) The features are all nominal and between 0 and 1. But yes, I think I will do some feature selection. I looked at Komarek's software, and it looks promising. Two things are not clear to me though: (1) How do I get the regression weights? It's not clear from his webpage or the software documentation (2)It seems like he is using Ridge regression. I need to use Lasso as well." CreationDate="2011-03-01T04:38:59.030" UserId="3301" />
  <row Id="12552" PostId="7733" Score="3" Text="A question for the quant.SE probably. The whole problem is defining what is valuable. The result will depend on this definition." CreationDate="2011-03-01T06:50:02.193" UserId="2116" />
  <row Id="12554" PostId="7727" Text="@cardinal, did you take a look at the plots I posted as an update to the original question?  The sensors are not measuring identical environments, and the environment does lots of filtering, the nature of which will be quantified by future analysis.  But the changes appear sufficient to preclude use of phase shift." CreationDate="2011-03-01T06:55:32.547" UserId="3479" />
  <row Id="12555" PostId="7727" Text="Hi, I have a similar question. I have 2 time series, each represented by a matrix with its first column corresponding to the values and second column corresponding to the time difference (since the previous value) &#xD;&#xA;&#xD;&#xA;How do I find the correlation between these 2 matrices? I tried to do xcorr2() but it doesn't seem right and doing xcorr would probably calculate correlation with only the values into considering, but I also want to account for the time. &#xD;&#xA;&#xD;&#xA;I'm really confused here, will an FFT help? How would you suggest I go about it?" CreationDate="2011-03-01T03:09:18.637" UserId="3233" />
  <row Id="12557" PostId="7727" Text="@lmelza Converted; yet if you have a question, just ask it as a new one (ASK QUESTION button at the top-right side). " CreationDate="2011-03-01T07:12:53.017" UserId="88" />
  <row Id="12558" PostId="7732" Text="I'm not familiar with Matlab functions... in R I would use `ccf`, that calculates the cross correlation function of the two signals. The FFT won't help here, as you don't want to do anything in the frequency domain, as far as I understand." CreationDate="2011-03-01T07:57:14.820" UserId="582" />
  <row Id="12559" PostId="7728" Text="Thank you both for your valuable input. @rolando2 you where right about my ambiguous wording, algorithmic was the direction I aiming for. " CreationDate="2011-03-01T08:15:23.600" UserId="3401" />
  <row Id="12560" PostId="7690" Text="Thinking of $f(t)dt$ as an &quot;infinitesimal&quot; probability is a good way to go I reckon.  I think of a PDF as a &quot;infinite&quot; histogram with bins of &quot;tiny&quot; width $dt$ and height $f(t)$.  The integral is how you &quot;add them up&quot;,  Which is why its $\int f(t)dt$  &quot;sum up the area&quot;" CreationDate="2011-03-01T08:38:43.973" UserId="2392" />
  <row Id="12561" PostId="3500" Text="You can go one step further and take logs.  Then you have *log-posterior ratio* = *log-prior ratio* + *log-likelihood ratio*" CreationDate="2011-03-01T08:43:36.637" UserId="2392" />
  <row Id="12562" PostId="7723" Text="@Whuber - this is a bit of a cop-out, @LSEactuary is not asking for the answer to everything.  You could easily just provide an answer to one piece of his question." CreationDate="2011-03-01T08:55:27.390" UserId="2392" />
  <row Id="12563" PostId="7738" Score="3" Text="p-value is always associated with statistic. What is your statistic? What type of problem are you solving? " CreationDate="2011-03-01T10:09:07.637" UserId="2116" />
  <row Id="12565" PostId="7738" Text="There are a lot of p-value questions here (for example [here](http://stats.stackexchange.com/questions/6477/interpretation-of-the-p-values-produced-by-levenes-or-bartletts-test-for-homoge/6486#6486), you may find searching these to be useful" CreationDate="2011-03-01T10:41:15.837" UserId="2392" />
  <row Id="12566" PostId="7731" Text="**leaps** is great, I like the plots from it, +1. In real applications some averaging techniques work faster (and better) than pretested estimators even found from all regression models. So I would suggest to go for Bayesian model averaging (BMA package) or the algorithm I like the most - Weighted average least squares (WALS[1]) developed by J.R.Magnus et al. The Matlab code is easily transformable to R code. The good thing for WALS is $N$ computational difficulty instead of $2^N$. &#xA;&#xA; &#xA;  [1]: http://www.tilburguniversity.edu/research/institutes-and-research-groups/center/staff/magnus/wals/" CreationDate="2011-03-01T11:14:23.143" UserId="2645" />
  <row Id="12567" PostId="7716" Text="@vqv, @chl seconded. Great overview." CreationDate="2011-03-01T11:34:23.400" UserId="2116" />
  <row Id="12568" PostId="7732" Text="@nico, in terms of *implementation*, the FFT will most definitely help if the matrices have a large number of rows. Both MATLAB and R use a frequency-domain approach to calculation of crosscorrelation. R's is incredibly naive, unfortunately, unless you understand the underlying approach and resize your matrix to take advantage of it. MATLAB's is a bit smarter, if I recall correctly. " CreationDate="2011-03-01T12:19:40.393" UserId="2970" />
  <row Id="12569" PostId="7739" Score="2" Text="@probabilityislogic, while your response is interesting, maybe it could be refocused to better address the OP's question. Also, ***no*** software for computing least-squares solutions performs matrix inversion, much less a determinant. Ever. Unless it's inverting a $1 \times 1$ matrix. " CreationDate="2011-03-01T12:28:19.110" UserId="2970" />
  <row Id="12570" PostId="7739" Score="1" Text="@probabilityislogic, handling the $2^n$ cases efficiently quickly far outstrips the $O(n^3)$ issues of an efficient least-squares solution. That's where the *leaps-and-bounds* algorithm comes in. " CreationDate="2011-03-01T12:30:39.547" UserId="2970" />
  <row Id="12571" PostId="7731" Score="1" Text="@Dmitrij, thanks for your comments. I've tried to remain fairly agnostic in my response regarding the utility of all-subsets regression. It seems to me that there is nearly always a better solution, but I felt that might seem like too trite of a response to the OP's question. " CreationDate="2011-03-01T12:34:21.230" UserId="2970" />
  <row Id="12572" PostId="7731" Text="@Dmitrij, BMA over main-effects models would still have the same computational complexity as all-subsets regression. No? The main advantage in BMA seems to me to be in trying to figure out which covariates are likely to be influencing the response. BMA does this by essentially averaging the log likelihoods over $2^{n-1}$ submodels. " CreationDate="2011-03-01T12:38:38.623" UserId="2970" />
  <row Id="12573" PostId="7731" Text="Thanks for the pointer to the leaps R package! I didn't know about it and it might come in handy in the future. If I could get some information on specific limitations for N for other popular packages it would be very helpful." CreationDate="2011-03-01T12:44:37.323" UserId="3311" />
  <row Id="12574" PostId="7721" Text="@Andy, how many observations do you have? For a lasso version of logistic regression, there is **glmnet**, but I've found it flaky even on small problems and I doubt it will scale to the size you are interested in. But, it's probably about the only package that even has a chance. The others tend to use iterative versions of the LARS algorithm. I think your first step should be to try some methods of feature filtering to get down to a more manageable size and see how bad that does. Are you at liberty to discuss more details of the application?" CreationDate="2011-03-01T12:45:19.413" UserId="2970" />
  <row Id="12575" PostId="7739" Text="Thanks for the post. &quot;Do the maths or do the time!&quot; :-)  .. I'm actually not even trying to figure out the underlying algorithms used by the packages (thought that is interesting to know), at this point really looking for specific information regarding the limitations of N by the major packages." CreationDate="2011-03-01T12:47:51.650" UserId="3311" />
  <row Id="12576" PostId="7732" Text="@cardinal: thanks, I didn't know about that :)" CreationDate="2011-03-01T12:53:33.727" UserId="582" />
  <row Id="12577" PostId="7727" Text="@BobC, I did take a quick look at the three plot snippets you posted. I'll try to take a closer look later today. It's not entirely germane to your question, but I'm a little puzzled by the data dropouts. I used to work with multichannel accelerometers sampling at 1kHz per channel or more for hours at a time and never experienced any such issues. Assuming you're running each channel into, say, a 12 bit ADC, you can easily pipe that over even a standard serial cable and still have more than enough left over for some additional error correction coding." CreationDate="2011-03-01T12:54:53.830" UserId="2970" />
  <row Id="12578" PostId="7731" Score="1" Text="@levon9, I doubt it will vary much at all by package. The algorithm that **leaps** uses has been state of the art for at least 20 years or so. Even if you found an implementation that was *twice* as fast, that would mean you got to increment the number of variables you can handle by one. For every doubling of the speed, you get one more variable. Hardware, not algorithmic, limitations are your bottleneck in this case. " CreationDate="2011-03-01T13:00:17.937" UserId="2970" />
  <row Id="12579" PostId="7739" Text="@cardinal The updating and downdating algorithms also exist for various matrix decomposition procedures, I suspect that is what was meant by &quot;matrix inverse&quot; etc." CreationDate="2011-03-01T13:04:31.250" UserId="887" />
  <row Id="12580" PostId="7732" Text="@nico, sure. A naive correlation takes $O(n^2)$ operation where it's easy to calculate the leading constant. An FFT-based implementation is $O(n \log n)$, where the leading constant can be taken somewhere between 8 and 16 or so. The value of $n$ doesn't have to be too large to make that worth it. " CreationDate="2011-03-01T13:15:03.987" UserId="2970" />
  <row Id="12581" PostId="7743" Score="1" Text="@Dikran, (+1) good comments. I was trying to avoid going there since it didn't directly address the OP's question. But, I agree. All-subsets is rarely the way to go. And, if you do go that way, you need to understand all the implications. " CreationDate="2011-03-01T13:19:14.920" UserId="2970" />
  <row Id="12582" PostId="7739" Text="@Dikran, several efficient and numerically stable approaches to least-squares exist, including methods for augmenting or reducing a design matrix by one column at a time. Sometimes it's good to understand what is happening under the surface, even if on most days you don't need to care. " CreationDate="2011-03-01T13:29:47.473" UserId="2970" />
  <row Id="12583" PostId="7705" Text="If you mean, it won't go to even 1.01, then you're right.  However, if you train to values that are all between 0 and 1, then eventually the network will train to always give values close to that range.  You can clip the output to 0 to 1 if you want, while leaving the region in between linear, but in most cases if it gives an output of 1.01 or 1.02 occasionally this is not an issue." CreationDate="2011-03-01T13:34:39.373" UserId="2917" />
  <row Id="12584" PostId="7743" Text="@Dirkan thank you for the comments, I am a real stats newbie. I realize the danger of overfitting the model when too many variables are in play, so I am just looking at various automated ways (ie without much benefit of insight) such as the stepwise approach (which might get caught in a local maxima) and the exhaustive all subsets model - and the computational limits it faces (and the external limitations imposed by packages)" CreationDate="2011-03-01T13:48:27.083" UserId="3311" />
  <row Id="12585" PostId="7744" Text="I presume this is homework? What have you tried so far?" CreationDate="2011-03-01T13:50:48.623" UserId="8" />
  <row Id="12586" PostId="7744" Text="@motdepasse, can you give a more explicit statement of the model you're considering? I ask because, in general, the first thing you're asked to prove is false!" CreationDate="2011-03-01T13:50:59.273" UserId="2970" />
  <row Id="12587" PostId="7695" Text="@mpiktas, others, what do you say? It looks to me the Lorenz is simply the CDF, reflected over y=x and scaled.  Is that correct?" CreationDate="2011-03-01T13:52:16.400" UserId="3463" />
  <row Id="12588" PostId="7737" Score="2" Text="@probabilityislogic, you might be cautious with your use of terminology, especially when addressing someone new to stats. For example, pivotal quantities are **not** statistics, so using the phrase &quot;pivot statistics&quot; can be (very) misleading. The OP uses &quot;pivot functions&quot;, which strikes me as a little unconventional, but certainly more correct. Sorry if this seems pedantic of me. Cheers. " CreationDate="2011-03-01T14:01:54.633" UserId="2970" />
  <row Id="12589" PostId="7695" Text="@Marcus, this is concerning your second last comment. Yes $P(X&lt;x)$ is a number, but you talk about distribution of P(X has x much wealth). That would be distribution of a number, which is confusing. Concerning the last comment, no, Lorenz curve is not simply CDF reflected and scaled. It might be for some distributions, but not in general. Your question is still not clear, sorry. Can you provide example with numbers? For simplicity assume that we have population of 2 people with different wealth. What will your question would look like?" CreationDate="2011-03-01T14:03:28.007" UserId="2116" />
  <row Id="12591" PostId="7721" Text="@cardinal: The number of observations can go from 200-20k depending on the type of data" CreationDate="2011-03-01T14:34:57.480" UserId="3301" />
  <row Id="12592" PostId="7703" Score="1" Text="Try a lesser learning rate (say 0.001) and a higher momentum (say 0.99). This will make training take longer but it will also be more robust." CreationDate="2011-03-01T14:34:59.147" UserId="2860" />
  <row Id="12593" PostId="7745" Text="Would be great if you cited whole passage where this occurs. It should be something simple, but I do not have access to the article, so I cannot answer. For example it might mean regression where dependent variable is vector of ones and independent variables are the columns of $W_t$. Usually further formulas in article give hints what authors had in mind." CreationDate="2011-03-01T14:38:53.187" UserId="2116" />
  <row Id="12594" PostId="7743" Score="2" Text="@levon9, you can get over-fitting that is just as serious when you choose the features, so feature selection does not guard against over-fitting.  Consider a logistic regression model used to predict the outcome of flipping a fair coin.  The potential inputs are the outcome of flipping a large number of other fair coins.  Some of these inputs will be positively correlated with the target, so the best all-subsets model will select inputs (even though they are useless) and you will get a model that appears to have skill, but in reality is no better than guessing." CreationDate="2011-03-01T14:39:05.803" UserId="887" />
  <row Id="12595" PostId="7721" Text="@Andy, so even in the best case you have four times as many predictors as observations? And, in the worst case you have 1000 times as many predictors as observations?? In linear regression, the lasso will only give you at most the minimum of the number of observations and predictors in terms of nonzero coefficients. I'd expect this to carry over to the logistic regression case as well. What is the ultimate goal of this study? Are you analyzing SNPs or something?" CreationDate="2011-03-01T14:47:58.230" UserId="2970" />
  <row Id="12596" PostId="5311" Text="It is important to stress to new users that they should almost always use a script window and `ctrl-R` rather the direct command line, both for when things do not work and need to be changed, and for when they do work and are worth saving for later use. " CreationDate="2011-03-01T14:54:34.497" UserId="2958" />
  <row Id="12597" PostId="7721" Text="@cardinal:I have data from different experiments and I need to study the relative importance of values/weights of the predictors obtained from various experiments - different predictors are used across different experiments. One example would be, if I have data about a gene's function (response variable) from experiments EX1 and EX2 (both experiments are likely to predict identical gene function), can I say anything about the relative importance of the predictors? I am sorry if this seems fuzzy. I don't have much background in Statistics, and am trying to learn things as I go along.    " CreationDate="2011-03-01T14:55:58.680" UserId="3301" />
  <row Id="12598" PostId="7752" Text="+1, it is basically the same as my comment, but much nicer written." CreationDate="2011-03-01T15:15:59.880" UserId="2116" />
  <row Id="12599" PostId="7752" Text="I spent too long LaTeXing that so i didn't see your comment before I submitted this answer. It appears that when you're writing an answer SE alerts you to other answers being posted, but not comments." CreationDate="2011-03-01T15:23:05.937" UserId="449" />
  <row Id="12600" PostId="7575" Text="My (also) vague impression of SEM is that it's technically sound but often misapplied by people who by a large margin lack the background necessary to properly formulate/diagnose/test/explain the models they can create using today's software. I'm going to point you to a set of slides with good references and videos (you can skip the videos if you're pressed for time) -- Topic 1 + the ref lists should suffice -- by Muthen &amp; Muthen, who are the authors of Mplus. The demo version of Mplus allows 6 DVs. http://www.statmodel.com/course_materials.shtml" CreationDate="2011-03-01T15:51:07.260" UserId="3408" />
  <row Id="12601" PostId="7575" Text="Oh, and the &quot;Path Analysis&quot; section will be most relevant -- and also the SEM sections -- but none of the examples are the *exact* situation you're looking for. There's another Path Analysis section in Topic 2, which comes close to what you need. Lastly, Mplus has a bit of a learning curve, so be prepared if you go that route." CreationDate="2011-03-01T15:56:40.547" UserId="3408" />
  <row Id="12602" PostId="7731" Text="@cardinal, exactly, BMA has the same computational complexity disadvantage as all-subsets regression (in Eviews it is named as combinatorial approach ^_^). It is why I value WALS more, since it both weights the covariates, is faster and is useful if we do have *focus* parameters (weighted estimator has smaller *pre-test* bias) and parameters that go to auxiliary variables we are not certain about and, yes, it solves the problems @Dikran mentioned in his post. Focus variables are theory based (no room to go spurious or over-fitting), large information set fights pre-test bias problem well. " CreationDate="2011-03-01T16:02:48.727" UserId="2645" />
  <row Id="12603" PostId="7743" Text="@Dikran (+1) the same as @cardinal, I first wrote a similar text, but then decided it is not what @levon9 asked, because he simply was curious about the complexity :)" CreationDate="2011-03-01T16:06:08.583" UserId="2645" />
  <row Id="12604" PostId="7751" Text="For what size of dataset?" CreationDate="2011-03-01T16:17:26.937" UserId="1499" />
  <row Id="12605" PostId="7723" Score="2" Text="@Probability You don't need to attack me.  Let's be constructive.  The original post shares several characteristics of questions the FAQ specifies should *not* be asked, including &quot;there is no actual problem to be solved&quot; and &quot;open-ended, hypothetical question[s].&quot;  In its original formulation at least four distinct questions were posed, including the hugely general &quot;how to calculate a test statistic&quot;.  This needed focusing to achieve appropriate, effective answers." CreationDate="2011-03-01T16:46:48.097" UserId="919" />
  <row Id="12606" PostId="7754" Text="Could you clarify what you mean by &quot;increased by 3?&quot;  Some issues: (1) &quot;increased by 3&quot; could mean &quot;+3&quot; or &quot;*3&quot;; (2) are you referring to the underlying variance $\sigma^2$ or its estimate $\hat{\sigma}^2$; (3) what could this increase possibly mean in either case?  Specifically, if $\sigma^2$ changes, how do you suppose the data would change (and via them, the estimated variance)?  If the estimate of $\sigma^2$ changes, how would that occur without changing the data altogether?" CreationDate="2011-03-01T16:52:11.573" UserId="919" />
  <row Id="12607" PostId="7750" Text="It would be helpful to distinguish variables from the actual data.  Even if the expected bias is zero, the actual bias is likely to be nonzero.  In other words, the problem (as stated) seems to be about *a dataset* and not about an underlying model for the data by means of random variables, so the mention of expectation is a bit of a surprise." CreationDate="2011-03-01T16:54:56.237" UserId="919" />
  <row Id="12608" PostId="5332" Text="I tried TinnR and didn't like it much, it seemed unpolished and clumsy." CreationDate="2011-03-01T17:00:56.123" UserId="1146" />
  <row Id="12609" PostId="7754" Text="@whuber: I think he simply meant that his estimates of the variances went from $69$ for the initial sample to $72$ for the large sample and $72-69=3$." CreationDate="2011-03-01T17:03:08.910" UserId="2958" />
  <row Id="12611" PostId="7750" Text="@whuber, I had the formula in wikipedia page in mind. There the difference between estimate of the regression coefficient and its true value is expressed as sum of two terms. The bias is expectation of this sum. The expectation of one term is always zero, since it involves the regression error. The expectation of another term is zero if there is no correlation between the omitted variables and the variables in the regression. I modified the answer to reflect your comment. " CreationDate="2011-03-01T17:08:46.090" UserId="2116" />
  <row Id="12612" PostId="7750" Text="@whuber, you raised an interesting point. I've updated my question, but I'll need to check it. Unfortunately I am posting now via my phone, which rather limits my editing capabilities, so if I'll have something substantial to add it will be later." CreationDate="2011-03-01T17:18:26.210" UserId="2116" />
  <row Id="12613" PostId="7745" Text="Thanks for comment. I put up the first three pages here, hoping not to violate any rights too much, but I fear the paragraph is not understandable without the rest (http://img815.imageshack.us/i/paper0.jpg/, http://img339.imageshack.us/i/paper1d.jpg/, http://img683.imageshack.us/i/paper2a.jpg/). The interesting part is on the third page and I don't really see any further information :( Unfortunately I don't have access to the original article by Godfrey and Wickens mentioned there. " CreationDate="2011-03-01T17:21:01.457" UserId="3104" />
  <row Id="12614" PostId="7755" Text="Would it not be necessary to weight the SDs given the disparity in group sizes?" CreationDate="2011-03-01T17:26:46.020" UserId="561" />
  <row Id="12615" PostId="7517" Text="Thank you for this detailed answer whuber. You didn't however not address my question (last sentence) at all. Could you make a small comment (in your answer) on that, please?" CreationDate="2011-03-01T17:30:50.353" UserId="56" />
  <row Id="12616" PostId="7736" Text="That's indeed a great reference and will be helpful when extending the cheat sheet with R examples. " CreationDate="2011-03-01T17:32:55.500" UserId="1537" />
  <row Id="12617" PostId="7755" Score="3" Text="@propofol: I think you would weight them if you concluded that the two samples came from the same distribution and wanted to derive a new estimate of the standard deviation for the combined sample; more importantly, you would also need to take account of the difference in the estimates.  But I was assuming that at this stage David was interested in the question of whether the estimates from the two samples were significantly different - on these numbers there is insufficient evidence to conclude there has been a change. " CreationDate="2011-03-01T17:34:04.080" UserId="2958" />
  <row Id="12618" PostId="7735" Text="This is a nice read. Thus far I have not yet considered Dante but perhaps renaming the cheat sheet to &quot;The Statistics Purgatorio&quot; comes to mind ;-)." CreationDate="2011-03-01T17:40:47.723" UserId="1537" />
  <row Id="12619" PostId="7517" Text="@honk I did not address it because, unfortunately, I don't understand it: I don't know what your notation refers to and I don't know what application is implied by the word &quot;use&quot;.  There may be a subtler difficulty here, too: we cannot find the &quot;probability that this measurement was noise&quot; from the information available.  This requires the use of Bayes' theorem and the assumption of a prior probability that a light beam was present." CreationDate="2011-03-01T17:42:37.447" UserId="919" />
  <row Id="12620" PostId="7750" Text="@mpiktas Mighty nice work from a telephone interface!" CreationDate="2011-03-01T17:43:53.780" UserId="919" />
  <row Id="12621" PostId="7754" Text="@Henry that is correct" CreationDate="2011-03-01T17:44:50.637" UserId="1381" />
  <row Id="12623" PostId="7754" Text="@Henry Thanks; this should have been obvious but I missed it." CreationDate="2011-03-01T17:49:42.903" UserId="919" />
  <row Id="12624" PostId="7755" Text="@Henry thanks for your answer; I am familiar with this for the case of adding two IRV's but did not realize that it was the case for calculating the difference as well. I guess it makes sense that it does." CreationDate="2011-03-01T17:52:19.680" UserId="1381" />
  <row Id="12625" PostId="4785" Text="When you estimate the frequencies from the data (as with Fourier analysis) and then include them as sin/cos terms in the regression, their p-values will be meaningless." CreationDate="2011-03-01T17:54:30.687" UserId="919" />
  <row Id="12626" PostId="7754" Text="@whuber if I understand your question: the estimate of $\sigma^2$ should change because the number of samples increases (thus the data are actually different), and I am assuming that the reason that the variance would increase is that there is a greater chance of sampling values in the tails of the distribution." CreationDate="2011-03-01T17:54:40.410" UserId="1381" />
  <row Id="12627" PostId="7754" Text="@David If you are using an unbiased estimate of variance (or, at the least, an estimate whose bias is independent of sample size) then the expected difference in the variances is zero, regardless of the sample size.  Although the larger sample indeed increases chances of obtaining extreme values, it also is likely to have many values close to the mean.  The two effects balance out.  Often, in fact, the bias in biased estimators (such as the MLE) is positive and decreases with sample size, so we would expect a small *decrease* in estimated variance in the large sample!" CreationDate="2011-03-01T18:00:05.817" UserId="919" />
  <row Id="12628" PostId="7755" Score="2" Text="@David The difference of two RVs is the sum of one and the negative of the other.  The variance of the negative of an RV equals the variance of the RV itself." CreationDate="2011-03-01T18:01:58.200" UserId="919" />
  <row Id="12629" PostId="7742" Text="Is Q a distribution over E?" CreationDate="2011-03-01T19:01:55.670" UserId="495" />
  <row Id="12630" PostId="7517" Text="@whuber: By &quot;use&quot; I mean application of a Likelihood-ratio test -- from independent sources we have priors for &quot;beam present&quot;. Or am I totally missing your point as a layman here?" CreationDate="2011-03-01T19:11:28.907" UserId="56" />
  <row Id="12631" PostId="7746" Text="Well I am supposed to have an algebraic proof that the mean of the residuals is zero. I assumed that by showing that the sum of the residuals is zero, then one could state that the mean is similarly zero, but I am not sure if that is accurate. I feel that something is missing.." CreationDate="2011-03-01T19:21:44.233" UserId="3487" />
  <row Id="12632" PostId="7747" Text="I've been playing with R-Studio today, as I had some data to analyze. It's very nice and easy to use. I only wish it had autocomplete for variable names." CreationDate="2011-03-01T19:24:03.717" UserId="582" />
  <row Id="12633" PostId="7746" Text="@cassetete, did you try to differentiate $g$? You are right mean will be zero when sum will be zero." CreationDate="2011-03-01T19:33:24.367" UserId="2116" />
  <row Id="12634" PostId="7746" Text="It finally became clear to me. I just worked it out, many thanks to all!" CreationDate="2011-03-01T19:51:17.373" UserId="3487" />
  <row Id="12635" PostId="7720" Text="Someone pointed me to &quot;Modern Applied Statistics with S&quot; by Venables and Ripley.  Section 7.3 has &quot;A four-way frequency table example&quot; that covers this house model extensively.  Reading.." CreationDate="2011-03-01T19:56:48.360" UserId="2849" />
  <row Id="12636" PostId="7746" Text="@cassetete, congratulations! It would be good if you posted your solution as an answer. So that somebody can find it by googling it. Or you can accept my answer, wink, wink, hint, hint, if it was the proverbial push over the cliff." CreationDate="2011-03-01T20:01:13.620" UserId="2116" />
  <row Id="12637" PostId="7715" Text="Yes, that's correct, thanks! I figured it out as follows: Rsquared = (ESS/n)/(TSS/n) = (ESS/n)/(varianceY). Then the incremental sum of squares is simply the lesser of the ESS's subtracted from the greater ESS (to account for the influence of the third variable added)... Quite simple when you think of it actually.. thanks!" CreationDate="2011-03-01T20:02:38.300" UserId="3473" />
  <row Id="12638" PostId="7720" Text="Actually the section is &quot;a proportional odds model&quot;" CreationDate="2011-03-01T20:03:11.973" UserId="2849" />
  <row Id="12639" PostId="7517" Text="@Honk I am beginning to follow.  However, your last expression, because it completely ignores the strong spatial correlation of any signal, cannot be correct (as you hint at the very end).  And I don't really believe you have any (informative) prior for &quot;beam present;&quot; what you do have is a characterization of the noise distribution." CreationDate="2011-03-01T20:07:14.117" UserId="919" />
  <row Id="12640" PostId="7758" Text="Interesting. How do you initialize the weights? Uniformly, Normal, ... ?" CreationDate="2011-03-01T20:24:51.167" UserId="2860" />
  <row Id="12641" PostId="7622" Text="Thanks for the hint to *Random Forests*. But would you try only them? What if you're not happy with the results? Which classifier you would try else? Or, what would you answer if someone asks: &quot;Why didn't you try other methods?&quot;" CreationDate="2011-03-01T20:49:54.453" UserId="2230" />
  <row Id="12642" PostId="7622" Text="@Oben Well, I understood you are making a kind of one-classifier-per-answer pool." CreationDate="2011-03-01T20:55:48.237" UserId="88" />
  <row Id="12643" PostId="7760" Text="You can make your equations readable by using TeX: enclose expressions in &quot;$\$$&quot;, as in &quot;$\$$ \hat{b} Y_{1.2} $\$$&quot; to display $\hat{b} Y_{1.2}$." CreationDate="2011-03-01T21:08:14.073" UserId="919" />
  <row Id="12644" PostId="7751" Text="Thanks very useful information. Just curious, did this this take long?" CreationDate="2011-03-01T21:34:26.997" UserId="3311" />
  <row Id="12645" PostId="7764" Score="2" Text="Thanks! I was sure it was a well-known result, but I wasn't able to find it myself. I appreciate the reference too; academic writing was so much more colorful back then..." CreationDate="2011-03-01T21:45:28.500" UserId="2111" />
  <row Id="12646" PostId="7720" Text="@dfrankow Yes, my remark about source code was a &quot;pis-aller&quot;... I will try to provide a more consistent response soon." CreationDate="2011-03-01T22:11:12.967" UserId="930" />
  <row Id="12648" PostId="7730" Score="1" Text="@levon9 This question generated a lot of sound answers and comments so that I've +1. But, please, forget about Excel for doing serious job in model selection..." CreationDate="2011-03-01T22:21:55.407" UserId="930" />
  <row Id="12650" PostId="7743" Text="@Dikran +1 because I like such advice." CreationDate="2011-03-01T22:22:57.047" UserId="930" />
  <row Id="12652" PostId="7757" Text="Z-scores normalisation is sometimes used but I have a funny feeling it may the another name for bayer's answer??&#xA;" CreationDate="2011-03-01T20:50:09.943" UserId="3484" />
  <row Id="12653" PostId="7730" Score="1" Text="@levon9 - I was able to generate all possible subsets using 50 variables in SAS.  I do not believe there is any hard limitation other than memory and CPU speed.&#xD;&#xA;&#xD;&#xA;-Ralph Winters&#xD;&#xA;" CreationDate="2011-03-01T14:56:19.553" UserId="3489" />
  <row Id="12654" PostId="7730" Text="For what size of dataset?" CreationDate="2011-03-01T16:17:26.937" UserId="1499" />
  <row Id="12655" PostId="7730" Text="Thanks very useful information. Just curious, did this this take long?" CreationDate="2011-03-01T21:34:26.997" UserId="3311" />
  <row Id="12656" PostId="7721" Text="@cardinal : OK, so I got my first set of data, and with feature reduction (by a factor of 3) I was able to obtain pretty much same cross validation values as those without. It leads me to (carefully) conjecture that I might be able to do significant feature reduction after all..." CreationDate="2011-03-01T23:06:13.310" UserId="3301" />
  <row Id="12657" PostId="7551" Text="a week later, I have figured out more precisely what the problem was.  I was using two factors, admitting school and admit-year.  For some years we only had data from a single admitting school, so the levels for that year and that school were redundant.  Understanding that makes me comfortable having them drop out of the model." CreationDate="2011-03-01T23:06:31.843" UserId="3388" />
  <row Id="12658" PostId="7650" Text="thanks very much for your answer.  It was totally helpful.  That said, the revision of my question made it bit more focused on the graphical representation than I had intended.  While your suggestion to make a lattice was eye-opening and cool, I was already able to make plots for each school." CreationDate="2011-03-01T23:09:50.483" UserId="3388" />
  <row Id="12659" PostId="7650" Text="(oops, editing timed out) ...  What I really needed was a way to calculate predicted values for the entire model so that I could create a single representative plot of the overall effect.  I don't want to communicate between-school differences, I want to average them out and present the sample-wide trends." CreationDate="2011-03-01T23:16:19.793" UserId="3388" />
  <row Id="12660" PostId="7768" Text="I should add I've been discarding new runners from both training and prediction data because of inherent uncertainty but would appreciate any better methods than 'Ignore'" CreationDate="2011-03-01T23:25:49.590" UserId="3484" />
  <row Id="12661" PostId="7730" Text="@chl .. is that because Excel is slow, or just incapable (ie would give inaccurate results?). " CreationDate="2011-03-01T23:56:02.580" UserId="3311" />
  <row Id="12662" PostId="7743" Text="@Dikran thanks for the additional clarifications/comments - and sorry about the typo earlier with your name." CreationDate="2011-03-01T23:59:42.127" UserId="3311" />
  <row Id="12663" PostId="7770" Text="Thanks for your answer. I have heard about it also. I downloaded them but could not find any documentation at all. For ex. what should the input file format be? Am I just not seeing it? " CreationDate="2011-03-02T00:05:34.587" UserId="3301" />
  <row Id="12664" PostId="7567" Text="@Maurizio, for some reason, I missed this post until now. Is there anyway you can edit it and add a plot similar to the last one in my post, but using your observed data? That might help diagnose the problem. I think I saw another question of yours where you were having trouble producing a uniform distribution, so maybe that is carrying over into these analyses as well. (?) Regards." CreationDate="2011-03-02T00:45:19.117" UserId="2970" />
  <row Id="12665" PostId="7668" Text="@raegtin, is there any additional info regarding your question that I could help you address, but haven't already? Just curious." CreationDate="2011-03-02T00:46:27.603" UserId="2970" />
  <row Id="12666" PostId="7573" Text="@Bernd great job, undoubtendly! I don't know why I didn't find this text last days I searched hundrends of R related pages. Nevertheless, i still can't manage a simple meta-analysis using R, ...despite the fact that I already know the results from Cochrane RevMan. The missing piece of the puzzle, is a simple example of a real-life meta-analysis problem, solved by keeping the R syntax as simple/realistic as possible. &quot;One word of truth outweighs the whole world&quot; (Alexander Solzhenitsyn, 1970, Nobel lecture)." CreationDate="2011-03-02T01:16:22.950" UserId="3333" />
  <row Id="12667" PostId="7758" Text="Are these actually normal (using some kind of normality test), or do you just refer to the fact that it's bell-shaped? I would actually have expected some kind of heavy-tailed distribution. Additional information which would be helpful is network structure -- how many layers, units, etc." CreationDate="2011-03-02T01:35:45.287" UserId="3369" />
  <row Id="12668" PostId="7742" Text="Do you mean to ask, is $Q\in E$? Need not be. " CreationDate="2011-03-02T02:24:49.007" UserId="3485" />
  <row Id="12669" PostId="7733" Text="Well, I'm *trying* to define the problem.  I started with a plain average, but I want to weigh more recent data higher.  So thus weighted average." CreationDate="2011-03-02T03:34:49.303" UserId="2664" />
  <row Id="12670" PostId="7772" Score="2" Text="@Andy Amos. Can you please provide some more details. At this point, I can only say -- probably -- hierarchical Bayesian models would work. cf. Gelman and Hill 2007, Cambridge Univ. Press" CreationDate="2011-03-02T04:16:07.797" UserId="1307" />
  <row Id="12671" PostId="7720" Text="(+1) great question." CreationDate="2011-03-02T04:31:13.627" UserId="1307" />
  <row Id="12672" PostId="7716" Text="@vqv. Thanks for your very helpful answer." CreationDate="2011-03-02T05:11:25.427" UserId="1393" />
  <row Id="12673" PostId="7751" Text="@Ralph I have undeleted this post (and merged another of your comments in an edit) because the OP found it useful and others might too.  Thank you for your contribution; please keep it up!  (If you really think it should be deleted, go ahead and do so; I won't contravene your wishes again.)" CreationDate="2011-03-02T05:53:38.873" UserId="919" />
  <row Id="12674" PostId="7777" Text="I will add more." CreationDate="2011-03-02T05:54:24.460" UserId="1307" />
  <row Id="12675" PostId="7730" Text="@levon9, @chl Excel is (in principle) capable of implementing model selection algorithms correctly.  It doesn't do so out of the box.  Does anyone have a particular add-in in mind?" CreationDate="2011-03-02T05:55:49.293" UserId="919" />
  <row Id="12676" PostId="7755" Text="@whuber uggh. I should have recognized that. Thanks for the help." CreationDate="2011-03-02T06:32:48.713" UserId="1381" />
  <row Id="12677" PostId="7778" Score="1" Text="Is this a troll?" CreationDate="2011-03-02T07:02:30.167" UserId="1307" />
  <row Id="12678" PostId="7778" Text="What do you think, mate?" CreationDate="2011-03-02T07:03:18.857" UserId="3503" />
  <row Id="12679" PostId="7622" Text="@mbq: Not really, but it turns out to be such a pool. Probably I did not make myself clear enough in the question. Actually I wanted to know which set of classifiers one should try first, to cover different general classification methods (with different strengths and weaknesses). I always ask myself if I shouldn't try more classifiers. Knowing that the ones I tried already represent the most typical/promising approaches would help here. But for that I need to know for which set of classifiers this is true. (I'm far from being a stats expert, so let me know if my mind is a bit twisted here)" CreationDate="2011-03-02T07:07:41.210" UserId="2230" />
  <row Id="12680" PostId="7779" Text="Exactly. I want to know more about how they read statistics? " CreationDate="2011-03-02T07:16:01.000" UserId="3503" />
  <row Id="12681" PostId="7779" Text="hmmm... I'm sorry, I just woke up and maybe I'm still a bit hazy... but what do you mean? Aside from a simple counter I don't see any statistics in that page. Even complex services like Google Analytics don't really do any proper statistics on the visitor data, also because it's easy to fake those counter (the one you linked is especially bad from this point of view, it will count a visit every time I reload the page, which is hardly an interesting thing to measure)." CreationDate="2011-03-02T07:23:34.783" UserId="582" />
  <row Id="12682" PostId="7778" Score="1" Text="@user3503 I think @suncoolsu has a point because the question links to a commercial web page.  If you're on the level--as a comment of yours suggests--, you won't mind that I edited this out :-).  (Web counters are straightforward to find all over the internet, so linking to a specific example seems unnecessary.  There's even one on the page you're reading right now: look to the right!)" CreationDate="2011-03-02T07:55:03.737" UserId="919" />
  <row Id="12683" PostId="7779" Text="@nico I changed the tag to a more relevant descriptor." CreationDate="2011-03-02T07:55:57.167" UserId="919" />
  <row Id="12684" PostId="7730" Text="@levon9 @whuber My point about Excel was not related to its performance (which I don't know in this particular case) but it was merely to point to &quot;better&quot; software that provide integrated tools for model building, selection, diagnostics (and yes I must admit I'm a little bit biased toward R or Stata for that purpose)." CreationDate="2011-03-02T08:02:18.987" UserId="930" />
  <row Id="12685" PostId="5332" Text="True, I actually changed to Eclipse with Statet. Much better." CreationDate="2011-03-02T08:03:32.660" UserId="1709" />
  <row Id="12686" PostId="7781" Text="@Whuber: Understood, thank you! Is the correlation cor(z1, z2) difficult to obtain? " CreationDate="2011-03-02T08:27:27.427" UserId="3019" />
  <row Id="12687" PostId="7781" Text="@Marco,The correlation is straightforward to calculate because the test statistic is so simple: it's a linear combination of normal variables.  (This is because we assume the variance is known.)  Alternatively, you can think of the second statistic as being a sum of two independent random variables: the first one, $z_1$, plus the change created by the additional data, $z_1 - z_2$. In more complicated cases the correlation might be quite difficult to calculate: that's one reason this somewhat idealized situation is used to motivate the sequential tests!" CreationDate="2011-03-02T08:40:39.080" UserId="919" />
  <row Id="12688" PostId="7781" Text="@Whuber: Thank you very much. Yes, the correlation looks pretty easy to compute. Actually, it was not clear to me that the context was a comparison of the means of two normal distributions. Now, it is clear and you make everything else very clear as well! Thank you!" CreationDate="2011-03-02T08:43:52.203" UserId="3019" />
  <row Id="12689" PostId="7774" Text="An obvious lower bound is $\Pr(T&lt;n) = 0$, but I guess you're aware of that..." CreationDate="2011-03-02T08:56:17.590" UserId="449" />
  <row Id="12690" PostId="7757" Text="It's the same except for the whitening part." CreationDate="2011-03-02T09:37:08.447" UserId="2860" />
  <row Id="12691" PostId="7773" Score="1" Text="@Bernd: Thank you for providing such a clear explanation." CreationDate="2011-03-02T09:38:39.200" UserId="1934" />
  <row Id="12692" PostId="7772" Text="@Andy Latent Rasch Regression and PLS1 Regression are also models such as the one you describe. As @suncoolsu said, it would help giving us some further information, esp. about data collection, working hypothesis." CreationDate="2011-03-02T11:05:33.720" UserId="930" />
  <row Id="12693" PostId="7742" Text="No, you said that Q was a prior. But a prior for what? If it is a prior for P, then it must be a distribution over E, in which case D(P||Q) doesn't really make sense." CreationDate="2011-03-02T11:14:51.533" UserId="495" />
  <row Id="12694" PostId="7772" Text="This sounds like some sort of factor analysis (such as principal component analysis) or maybe a mixture model. I agree that more information would be useful." CreationDate="2011-03-02T11:22:13.747" UserId="495" />
  <row Id="12695" PostId="7786" Text="@user3136 What kind of variables are you considering, and what kind of relationship are you interested in?" CreationDate="2011-03-02T11:42:20.943" UserId="930" />
  <row Id="12696" PostId="7784" Text="Can you elaborate on how exactly one would use it or give a reference?" CreationDate="2011-03-02T11:55:37.350" UserId="2860" />
  <row Id="12697" PostId="7786" Text="The two variables are both continuous, a simple correlation or perhaps a regression per block would be enough to summarize the relationship. I have calculated these already, but my problem is that the sampling is not strictly random so I am not convinced how much I can trust any p-values, confidence intervals etc." CreationDate="2011-03-02T12:10:26.860" UserId="3136" />
  <row Id="12698" PostId="7791" Score="1" Text="@Chris, well you need to start by defining your model. When you say you visit twice a week at random, that can potentially mean many things. For example, you could go every week *exactly* two times, chosen as a random combination of two elements of the set $\{\text{Mon},\ldots,\text{Fri}\}$, or you could go *on average* twice a week where, say, you flip a biased coin with probability of heads of 2/5 and you go every day you see a head. These aren't the only options. " CreationDate="2011-03-02T12:59:39.783" UserId="2970" />
  <row Id="12699" PostId="7790" Text="Check [wikipedia](http://en.wikipedia.org/wiki/Validity_(statistics))  and then come back and clarify your question.  It might be that you mean reliability." CreationDate="2011-03-02T13:00:32.070" UserId="601" />
  <row Id="12700" PostId="7789" Text="Thanks Simon, that seems easy" CreationDate="2011-03-02T13:02:42.503" UserId="686" />
  <row Id="12701" PostId="7723" Text="@LSEactuary There are too much questions here, and I would suggest to split your initial text in (at least) two parts, as suggested by my edit. The 2nd part deserves to be formulated in a new question (IMHO, but see also @whuber's comment), since only pivotal quantities have been addressed in this thread." CreationDate="2011-03-02T13:10:34.537" UserId="930" />
  <row Id="12702" PostId="7791" Text="Also are you assuming that you'll always see him if you visit the dojo on the same day as him? If not, i think we'd need to know something about the length of your sessions and the length of his sessions compared to the length of time each day the dojo is open." CreationDate="2011-03-02T13:11:11.447" UserId="449" />
  <row Id="12703" PostId="7790" Text="@Nir, how were the groups chosen? What sort of events are we talking about? Are they &quot;intrinsic&quot; characteristics of the people sampled or choices they made on the particular day they were sampled or something else? Is each group homogeneous or heterogeneous? The answers to questions like these will provide a starting point to addressing your question. " CreationDate="2011-03-02T13:12:24.053" UserId="2970" />
  <row Id="12705" PostId="7790" Text="Sounds like you're after a [chi-squared test of independence](http://en.wikipedia.org/wiki/Pearson%27s_chi-square_test#Test_of_independence). It would be much easier to do this in a statistics package than in PHP, especially as the PHP function isn't currently documented." CreationDate="2011-03-02T13:20:07.263" UserId="449" />
  <row Id="12706" PostId="7791" Score="1" Text="@Chris, @onestop, this question reminds me of, and is related to, a technique used to sample people that might be reluctant to answer a question truthfully, often due to the social stigma of responding affirmatively. You introduce a random element to the sampling such that with fairly high probability the respondent responds affirmatively (more embarrassing answer) even if in truth they would have responded negatively. If the probability of a randomly determined &quot;yes&quot; is high enough, the &quot;embarrassment bias&quot; is reduced. Of course, one has to sample more people, too. " CreationDate="2011-03-02T13:22:37.120" UserId="2970" />
  <row Id="12707" PostId="7790" Text="Please see additional info above" CreationDate="2011-03-02T13:33:09.937" UserId="3506" />
  <row Id="12708" PostId="7791" Score="1" Text="@Chris You would have to make some more assumptions.  As it is now there are a multiplicity of valid explanations.  Here’s a silly one: Are the individual’s visits independent of yours?  If not, maybe he will only visit when you visit (he looks for your car outside everyday), but tosses a coin (with probability 0.9) before deciding whether to go inside." CreationDate="2011-03-02T13:38:58.340" UserId="1670" />
  <row Id="12709" PostId="7795" Text="@hskin96 - I'm not familiar with `gam` models, but have you examined the different attributes of that object? You can look at the names of the objects with `names(b)`. I'm guessing whatever details you are after will be retained within that object somewhere." CreationDate="2011-03-02T13:56:18.757" UserId="696" />
  <row Id="12710" PostId="7751" Text="@Ralph It seems you are using two different unregistered accounts. I have merged them but you will still need to register." CreationDate="2011-03-02T14:08:25.250" UserId="930" />
  <row Id="12711" PostId="7799" Text="Are you interested in univariate multiple regression (one predicted variable) or multivariate multiple regression (many predicted variables analyzed simultaneously)?" CreationDate="2011-03-02T14:27:32.800" UserId="1909" />
  <row Id="12712" PostId="7788" Text="Some points to clarify: 1. Is your dependent variable the same for both countries?  2. How did you pool both country data? 3. What type of the conclusions you are trying to make? My general observation is that if you pool both countries, you must assume that the same association applies in both of them. This usually is quite strong assumption. If it holds, then simply adding country specific variables to your pooled analysis might be enough." CreationDate="2011-03-02T14:53:13.237" UserId="2116" />
  <row Id="12713" PostId="7775" Score="1" Text="I suggest trying the other models and see whether the code works or not. R is usually nice, so anova should return similar things for different models. The problem is with your initial formula. Does it hold for other models? If it does not, then there is no point in getting code to work, furthermore the code should issue a warning that it is used for models where formula does not hold." CreationDate="2011-03-02T15:05:55.020" UserId="2116" />
  <row Id="12714" PostId="7772" Score="1" Text="@Andy, latent variable means that it is not directly observable, so without model relating the latent variable to the observable variables it is impossible to estimate anything." CreationDate="2011-03-02T15:09:44.720" UserId="2116" />
  <row Id="12715" PostId="7801" Text="oops, I didn't see yours when I posted my answer. Well, it's a little more detailed anyway...." CreationDate="2011-03-02T15:16:45.817" UserId="1979" />
  <row Id="12716" PostId="7801" Text="@fabians, no worries, I would not have posted mine if I saw yours. I outlined general idea, you have provided the code. Since the question asks for code, your answer is better." CreationDate="2011-03-02T15:20:21.497" UserId="2116" />
  <row Id="12717" PostId="7805" Text="What do you mean by &quot;expectation between two variables&quot;?" CreationDate="2011-03-02T15:20:23.057" UserId="2860" />
  <row Id="12718" PostId="7806" Text="+1 for example with trace." CreationDate="2011-03-02T15:20:45.567" UserId="2116" />
  <row Id="12719" PostId="7786" Score="2" Text="If the sample isn't random, it's going to be difficult to make any firm conclusions unless you create a model for the non-randomness (e.g., certain demographic groups are underrepresented)." CreationDate="2011-03-02T15:30:29.803" UserId="401" />
  <row Id="12720" PostId="7803" Text="@M.-Tibbits nice link and simple tutorial, with some good links under &quot;Going Further&quot; at the end." CreationDate="2011-03-02T15:49:22.230" UserId="1381" />
  <row Id="12721" PostId="7805" Score="1" Text="@user3509, you can get an easy *lower bound* by noting that $x^{-1}$ is convex and if $X$ is marginally Gaussian, then $e^{-X}$ is lognormal and so $\mathbb{E}(Y_i Y_j) &gt; (1 + \mathbb{E} e^{-X_i} + \mathbb{E} e^{-X_j} + \mathbb{E} e^{-X_i-X_j})^{-1}$. The expectations on the right hand side are easy to compute. " CreationDate="2011-03-02T15:53:07.110" UserId="2970" />
  <row Id="12722" PostId="7772" Text="@suncoolsu @chl @Simon Byrne The dependent variable Y is a NxM matrix denoting N observations each from M variables. The latent variable X is a NxP matrix about which we know nothing except the dimensions. In addition to these, we have an initial estimate of the coefficient matrix beta based on prior knowledge. My goal is to find the estimates of latent X and coefficient matrix beta by both using the data matrix Y and the initial coefficient matrix. I thought of constructing an EM algorithm but because of the complexity of multivariate data and latent variable concept, I am totally confused." CreationDate="2011-03-02T16:15:26.100" UserId="3499" />
  <row Id="12723" PostId="7805" Text="**Stationary phase** (using a linear approximation to find the critical point) looks like it will work, but the resulting expressions--although they will be algebraic combinations of $\mathbf{M}$ and $\mathbf{S}$--will be *extremely* messy.  How far are you prepared to go with this and what's the purpose of a &quot;closed form&quot; solution (when perhaps a numerical algorithm will in practice be more efficient and accurate)?" CreationDate="2011-03-02T16:40:44.763" UserId="919" />
  <row Id="12724" PostId="7791" Score="1" Text="Simple solution: ask him :-)." CreationDate="2011-03-02T16:49:52.497" UserId="919" />
  <row Id="12725" PostId="7805" Text="@bayer, I mean &#xA;&#xA;E{y_1,y_2}=\int\int s(x_1)s(x_2) p(x_1,x_2) dx_1 dx_2&#xA;&#xA;where y=s(x) is the sigmoid function, p(x,y) is the joint-PDF (gaussian). The integral is over RxR." CreationDate="2011-03-02T17:05:23.170" UserId="3509" />
  <row Id="12726" PostId="7805" Score="1" Text="@user3509 TeX is enabled in comments as well as questions and replies: surround it with $\$$, as in &quot;$\$$E\{y_1,y_2\}=\int\int s(x_1)s(x_2) p(x_1,x_2) dx_1 dx_2$\$$&quot; gives  $E\{y_1,y_2\}=\int\int s(x_1)s(x_2) p(x_1,x_2) dx_1 dx_2$" CreationDate="2011-03-02T17:12:13.063" UserId="919" />
  <row Id="12727" PostId="7805" Text="@whuber, the purpose of this closed form solution is to obtain speed and scalability at the expense of accuracy and implementation simplicity. Ideally if I know which approximations I did I can also ensure a good behavior over a large range of input values. &#xA;&#xA;I am now looking stationary phase methods (wikipedia page) " CreationDate="2011-03-02T17:21:41.037" UserId="3509" />
  <row Id="12728" PostId="7805" Text="@whuber, I don't see how the integral can be formulated as an oscillatory integral. You need some kind of periodic behaviour depending on the integration variables right?.&#xA;&#xA;Sorry for the tex mess, my browser does not diplay it correctly. I'am also having problems with CVs interface." CreationDate="2011-03-02T17:37:59.710" UserId="3509" />
  <row Id="12729" PostId="7811" Text="Thank you so much? However, my question is, in this situation, the more similar two vectors are, the smaller the norm will be. What is an easy way to invert this whereby more similar vectors output higher similarity values." CreationDate="2011-03-02T18:13:09.790" UserId="1514" />
  <row Id="12731" PostId="7811" Text="@Spencer The norm in your case is capped; the vectors (4,4,4,4...) and (1,1,1,1...) are as far apart as possible. You could calculate the highest norm possible and subtract the actual norm from that if you want a similarity measure. " CreationDate="2011-03-02T18:24:59.913" UserId="2965" />
  <row Id="12732" PostId="7770" Text="Just here: http://www.bayesianregression.com/bmr.html" CreationDate="2011-03-02T18:53:33.577" UserId="1739" />
  <row Id="12733" PostId="7786" Score="2" Text="Following on from Charlie, you *might* be in the happy situation where the non-random selection process is the same for each block.  Then your original question 'is the X Y relationship different between blocks?' is actually addressable by the usual methods despite the fact that each method will be arbitrarily wrong about what that relationship actually is..." CreationDate="2011-03-02T18:58:53.747" UserId="1739" />
  <row Id="12734" PostId="7812" Text="I am a bit concerned over the linearization but it is definitely worth trying. Do you know of some available literature on stationary phase aproximation of integrals?. " CreationDate="2011-03-02T19:20:50.200" UserId="3509" />
  <row Id="12735" PostId="7772" Text="@Andy. May be a silly question: How did you control the levels of other variables when you were collecting $Y$. Each row in $Y_i$ matrix contains entries $Y_{ij}$ which denote the observed variable corresponding to $j^{th}$ latent variable. One more request: can you please modify your question instead of comments? " CreationDate="2011-03-02T19:24:24.970" UserId="1307" />
  <row Id="12736" PostId="7812" Text="I was concerned about the linearization, too, but (a) the series actually is linear through second order and (b) extensive plotting of the pdf for means between -3 and 3, sds from 0.1 to 3, and correlations from -1 to 1, indicate it should generally work well (the linearized critical point appears quite close to the global maximum).  I learned this technique from an article in The American Statistician perhaps some 10-15 years ago, but I can't locate it at this moment." CreationDate="2011-03-02T19:27:36.620" UserId="919" />
  <row Id="12737" PostId="7812" Text="@user3509 I found the TAS reference and linked to it in the text: it goes by the name &quot;saddlepoint approximation.&quot;  I apologize for confusing it with stationary phase (they are essentially the same thing: one with a real variable, the other with an imaginary variable)." CreationDate="2011-03-02T19:42:47.673" UserId="919" />
  <row Id="12739" PostId="7817" Text="The variates cannot possibly be iid if there is any effect at all.  Presumably you mean there is a function $f$ such that $y_i - f(x_i)$ are iid conditional on the $x_i$ (or some such thing) and where $f$ has a discontinuity." CreationDate="2011-03-02T19:49:21.383" UserId="919" />
  <row Id="12740" PostId="7817" Text="yes. Corrected. " CreationDate="2011-03-02T19:53:46.123" UserId="603" />
  <row Id="12741" PostId="7814" Score="2" Text="@whuber +1, awesome. How did you come up with functional form for $g$?" CreationDate="2011-03-02T19:59:52.877" UserId="2116" />
  <row Id="12742" PostId="7803" Score="1" Text="Beware of the recommendation for `cv.lm()` from package `DAAG` for k-fold cross-validation though. It silently uses only the first predictor in the validation, i.e., is only suitable for simple linear regression. As is mentioned in its help file, use `CVlm()`instead." CreationDate="2011-03-02T20:13:03.383" UserId="1909" />
  <row Id="12743" PostId="7812" Text="Yes, that was exactly what I could not understand I was looking for the complex exponential somewhere. Thank you very much." CreationDate="2011-03-02T20:13:46.643" UserId="3509" />
  <row Id="12744" PostId="7814" Text="@mpiktas A combination of looking at the formulas for the CDF and plotting the medians versus $\nu$.  The exponential, with two parameters, actually fits about as well as a quadratic polynomial (with three parameters)." CreationDate="2011-03-02T20:36:41.797" UserId="919" />
  <row Id="12745" PostId="7818" Text="I __love__ the UCLA site.  I tell my students taking SAS to go there if they've seen/used a different piece of software, but I didn't know they had compiled as extensive of a section on R.  Splendid!" CreationDate="2011-03-02T20:43:47.687" UserId="1499" />
  <row Id="12746" PostId="7817" Score="5" Text="See the genetic sequences at http://anson.ucdavis.edu/~mueller/changetrees.pdf (at the end).  A dataset of 56 cases where a slope exhibits a discontinuous jump is given at http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mcmc_sect048.htm .  Googling &quot;multivariate changepoint problem dataset -time&quot; yields promising hits." CreationDate="2011-03-02T21:51:52.557" UserId="919" />
  <row Id="12747" PostId="7727" Text="It's a mobile acquisition where minimal size and weight are critical, and no external power is available.  I link to the devices used in the OP.  They lack internal memory sufficient to buffer data during slow writes to a microSD card.  The devices have a variable sample rate, and I'm using them at 512 Hz, their fastest rate, thus obtaining the worst dropouts.  Only sample rates below 60 Hz are gap-free, and such rates completely undersample the events I'm interested in.  The device does have a burst-mode acquisition, but the implementation has limited use." CreationDate="2011-03-02T22:28:34.693" UserId="3479" />
  <row Id="12748" PostId="7806" Text="@fabians Many thanks for your help. When I reproduce your code up to  `plotData &lt;&lt;- c(plotData, pd[[i]])}))&#xA;`, the following message occurs `Error in fBody[[i]] : no such index at level 3`. Any ideas why it does not work?&#xA;" CreationDate="2011-03-02T23:15:54.130" UserId="2264" />
  <row Id="12749" PostId="7819" Text="@Conjugate Prior Your reformulation of the question is exactly correct. I think the multivariate calibration is what I'm looking for. However the initial coefficient matrix is not completely correct so I have to find a way to optimize it by an iterative procedure. But before that, thank you very much." CreationDate="2011-03-02T23:20:35.907" UserId="3499" />
  <row Id="12750" PostId="7742" Text="May be, this is the difference between the two methods. I do not really know Bayesian method completely. As for MaxEnt $Q$ can be any distribution which is mostly a prior knowledge about the system (not the prior you are talking about). Pardon me if I am wrong." CreationDate="2011-03-03T02:23:14.497" UserId="3485" />
  <row Id="12755" PostId="7796" Text="Yes, it holds for every fixed $n$. A (very simple) proof can be found, for instance in Levin, Peres and Wilmer's book Markov Chains and Mixing Times, Proposition 2.4. The proof doesn't work for the lower bound, though." CreationDate="2011-03-03T04:41:40.563" UserId="3500" />
  <row Id="12756" PostId="7817" Score="1" Text="Thank you @WHuber, the google keyword you suggested lead me to examples from a problem i did not know of. Can you post your comment as an answer (Snijders and al 2001)." CreationDate="2011-03-03T04:53:07.420" UserId="603" />
  <row Id="12757" PostId="7747" Score="2" Text="After a couple of revs, it will be the best thing going. Its first beta release is already eminently usable and beats most other options IMO." CreationDate="2011-03-03T04:53:19.400" UserId="26" />
  <row Id="12759" PostId="7796" Text="@cardinal In fact, I might as well transcribe the proof here: &#xA;&#xA;&quot;Let $A_i$ be the event that the $i$-th [coupon] type does not appear among the first $n \log n + cn$ coupons drawn. Observe first that&#xA;$P(\tau &gt;n\log n+cn )=P (\cup_{i} A_i ) \leq \sum_i P(A_i)$. Since each trial has probability $1 − n^{−1}$ of not drawing coupon $i$ and the trials are independent, the right-hand side above is bounded above by $\sum_i (1 - 1/n)^{n \log n + cn} \leq n \exp(\frac{n \log n + cn}{n} ) = e^{-c}$,&#xA;proving (2.7).&quot;" CreationDate="2011-03-03T05:17:13.497" UserId="3500" />
  <row Id="12760" PostId="7806" Text="@hskin96: Dunno. Maybe you're using a differnt version of `mgcv`? I tested my code with `mgcv_1.7-2`, you may need to change the `at` argument of `trace` for earlier versions..?" CreationDate="2011-03-03T08:31:51.013" UserId="1979" />
  <row Id="12762" PostId="7791" Text="@whuber- that made be lol!" CreationDate="2011-03-03T09:45:24.403" UserId="199" />
  <row Id="12763" PostId="7812" Text="@whuber, +1 for the nice answer and the link to the article. Did not know that it is feasible to use such approximations. I would have resorted to numerical integration without even trying to get the closed form solution." CreationDate="2011-03-03T11:29:39.787" UserId="2116" />
  <row Id="12764" PostId="7742" Text="My point was that to calculate D(P||Q), Q must be absolutely continuous with respect to P (so cannot be an arbitrary distributions). " CreationDate="2011-03-03T11:33:04.677" UserId="495" />
  <row Id="12765" PostId="7742" Score="1" Text="You may find this question useful: http://stats.stackexchange.com/q/4978/495" CreationDate="2011-03-03T11:33:50.293" UserId="495" />
  <row Id="12766" PostId="7829" Score="1" Text="@Manisha Do you mean you want to highlight cluster structure (i.e., its characteristics features wrt. external subject-specific variables, for example) or to provide insight into how clusters differ between the two methods?" CreationDate="2011-03-03T12:36:54.457" UserId="930" />
  <row Id="12767" PostId="7483" Text="@Dikran, &quot;papers and grants&quot; would also explain the scarcity of synthetic benchmarks, gensyntheticdata( N, dim, nfeature ... ) to evaluate methods -- or do you know of any at all ?" CreationDate="2011-03-03T13:44:56.613" UserId="557" />
  <row Id="12768" PostId="7799" Text="Good question; I am interested in both but could make do with univariate multiple regression (and use multiple models for different output variables of interest)." CreationDate="2011-03-03T13:45:28.887" UserId="2917" />
  <row Id="12769" PostId="7818" Text="I marked the Quick-R answer from M. Tibbits as &quot;accepted answer&quot; but also upvoted this one as this also looks like a great resource which Google had not led me to.  Thanks!" CreationDate="2011-03-03T14:05:35.460" UserId="2917" />
  <row Id="12770" PostId="7836" Text="Data-generation process is never known, we choose the model in hope that we approximate the data-generation process sufficiently. That is one of the possible answers, it would help if you provided more context, so it is clearer what kind of answer you are looking for. Check out the chat, currently ongoing journal club discusses article where this issue is raised." CreationDate="2011-03-03T15:36:12.007" UserId="2116" />
  <row Id="12771" PostId="7836" Score="2" Text="The answers to this question will vary, as they should, because both &quot;data-generation process&quot; and &quot;model&quot; are used in varied ways by various authors.  @Weijie, do you have a particular reference in mind?" CreationDate="2011-03-03T15:46:10.483" UserId="919" />
  <row Id="12775" PostId="7845" Score="4" Text="Q-Q plots are far better tools for incisive comparison of empirical distributions.  Using them avoids the binning problem altogether." CreationDate="2011-03-03T16:32:04.783" UserId="919" />
  <row Id="12776" PostId="7845" Score="3" Text="@whuber:  Agreed, if you just want a sensitive visualization of whether two distributions are different, but the histogram approach is IMHO better if you want detailed insight into **how** they're different." CreationDate="2011-03-03T16:34:32.083" UserId="1347" />
  <row Id="12777" PostId="7845" Score="2" Text="@dsimcha My experience has been the opposite.  The Q-Q plot clearly shows (in a quantitative way) differences of scale, location, and shape, especially in the thickness of the tails.  (Try comparing two SDs directly from the histograms, for instance: it's impossible when they are close in value.  On a Q-Q plot you need only compare slopes, which is fast and relatively accurate.)  a Q-Q plot is inferior to a histogram in terms of picking out modes, but no histogram is good at that until a decent amount of data have been collected and a good choice of bins has been made." CreationDate="2011-03-03T16:52:11.950" UserId="919" />
  <row Id="12778" PostId="7747" Text="I liked it too. It is for more advanced users that basically just need a better shell that RGui. Emacs+ESS yet has much more features but not everybody likes Emacs. RStudio seems to be simple but very effective at what it wants to do. I look forward to see how it evolves over time." CreationDate="2011-03-03T16:58:22.913" UserId="2020" />
  <row Id="12779" PostId="7844" Text="+1 for most of your post.  I do think 3D plots can be very effective, as I described in http://stats.stackexchange.com/questions/6763/statistically-group-chemical-batches/6768#6768  Clustered boxplots are an alternative worth looking at, too, as are Ralph's bubble plots." CreationDate="2011-03-03T17:01:31.057" UserId="2669" />
  <row Id="12780" PostId="7835" Text="It sounds as if Simpson's paradox will not be in play, since you are going to analyze within populations only....Do you care about collinearity, or about control of other predictors while checking a given predictor's relationship with A-or-B class?  If so, you'll want to look into either discriminant analysis or (probably) the easier method of logistic regression.  But it's not trivial either, and the conceptual issues involving control and variable selection often trip people up just as the technical, computational ones do." CreationDate="2011-03-03T17:07:09.837" UserId="2669" />
  <row Id="12781" PostId="143" Text="+1 for Random Forests. Definitely a good classifier to try 1st, as they don't overfit." CreationDate="2011-03-03T17:23:44.880" UserId="2817" />
  <row Id="12782" PostId="7622" Text="&quot;worse accuracy than SVM for problems desperately requiring kernel trick&quot; what is &quot;kernel trick&quot;?" CreationDate="2011-03-03T17:24:53.943" UserId="2817" />
  <row Id="12783" PostId="7842" Text="Thank you for your contribution.  A short summary of your approach would be appropriate (and welcome), especially because the link does not go to a generally recognized, authoritative source about statistics on the Web." CreationDate="2011-03-03T17:46:26.353" UserId="919" />
  <row Id="12784" PostId="7848" Text="+1 For bringing in new ideas (with references)." CreationDate="2011-03-03T17:51:08.363" UserId="919" />
  <row Id="12785" PostId="7844" Score="1" Text="Thanks. Yes, 3D can be effective, but often it's not obvious what the 3D representation is when it's been projected onto a static 2D computer screen, unless you can grab it and rotate it yourself. I have a 3D version of the above plot, which I'll add to the answer." CreationDate="2011-03-03T17:57:15.113" UserId="1191" />
  <row Id="12786" PostId="7786" Text="I like Charlie's and Conjugate Prior's remarks.  More prosaically, the fraction of the population that was sampled within each block might have some bearing on the reliability of estimates." CreationDate="2011-03-03T17:59:57.627" UserId="2669" />
  <row Id="12787" PostId="7771" Text="First of all, I wouldn't place much trust in the K-S test with samples &lt;25 or &gt;300.  But otherwise, interesting:  how to quantify on a unidimensional, 0-to-1 scale?  On the one hand, it's not hard to imagine that when a distribution shows a perfect fit to the hypothesized one, the effect size would be 1.  But what about radically poor fits--how would one distinguish between the poor fits to the normal of, say, a U-shaped distribution, a uniform, or some multi-humped polynomial one?  Also, there's a question of scale:  it seems *Z* would have to be infinite for the effect size to be 0." CreationDate="2011-03-03T18:10:22.613" UserId="2669" />
  <row Id="12788" PostId="7813" Text="Any thoughts on this? Or if anyone could explain AdaBoost in simple English, that will be helpful also. Thanks." CreationDate="2011-03-03T18:18:34.240" UserId="3301" />
  <row Id="12789" PostId="7483" Text="@Denis, no I've not used many synthetic benchmarks (apart from Ripley's SYNTH benchmark, but that is mostly useful for illustration rather than benchmarking).  I've often wondered about generating &quot;realistic&quot; synthetic benchmark datasets by training a classifier and a kernel density estimator on an real dataset and then using them to generate data in whatever quantities I want.  Haven't had time to investigate how well it works yet." CreationDate="2011-03-03T18:32:22.777" UserId="887" />
  <row Id="12790" PostId="7845" Score="1" Text="I agree that QQ plots are the best solution, although they don't avoid the bin problem, they just force you to place the bins in particular places (the quantiles :-)  On the other hand this does imply that the bins don't, indeed shouldn't be shared by the two distributions." CreationDate="2011-03-03T18:40:54.567" UserId="1739" />
  <row Id="12791" PostId="7796" Text="@David, that's nice and simple enough. I quickly played with expanding the inclusion-exclusion formula out by another term, but didn't get anywhere quickly and haven't had time to look at it further. The event $\{T &lt; t_n\}$ is equivalent to the event that no coupons are left after $t_n$ trials. There should be a martingale associated with that. Did you try Hoeffding's inequality on the (presumed) associated martingale? The asymptotic result suggest strong measure concentration. " CreationDate="2011-03-03T18:48:17.880" UserId="2970" />
  <row Id="12792" PostId="7796" Text="@David, there's a sign flip in your proof above, but I'm sure that's obvious to other readers too. " CreationDate="2011-03-03T18:50:15.163" UserId="2970" />
  <row Id="12794" PostId="7850" Score="1" Text="Could you post a snippet of your data, so we have a better idea of what you want to manipulate?" CreationDate="2011-03-03T19:17:24.187" UserId="2817" />
  <row Id="12796" PostId="7835" Text="If you're only classifying within a population, don't worry about it.  Just make sure you present your results in separate tables (or charts or whatever), one for each class.&#xA;&#xA;If you're trying to classify observations who's class you do not know, then you need to be more careful." CreationDate="2011-03-03T19:31:20.330" UserId="2817" />
  <row Id="12797" PostId="7849" Text="Your answer to #1 converges to zero as the sample size grows, so what justifies interpreting it as an effect size?" CreationDate="2011-03-03T19:33:42.660" UserId="919" />
  <row Id="12798" PostId="7851" Text="x1 is an experimental measured valued. x2 would be some aggregate function (most likely mean) of a set of experimental measured values to which x1 belongs. Does that change anything?" CreationDate="2011-03-03T19:36:13.330" UserId="3527" />
  <row Id="12799" PostId="7622" Text="@Zach http://en.wikipedia.org/wiki/Kernel_trick ; many artificial geometric problems fall in this class." CreationDate="2011-03-03T19:38:29.770" UserId="88" />
  <row Id="12800" PostId="7816" Text="I would also combine #4 and #1: its important to know how to cross validate without overwhelming your computing resources." CreationDate="2011-03-03T19:39:18.593" UserId="2817" />
  <row Id="12801" PostId="7826" Text="One question is, what exactly is it you are trying to estimate. An estimate usually implies that you take a uniform subsample of the population, measure the frequency in that subsample and try to guess how different the result is from the whole population. This doesn't seem to be the case here. If you want to know the prevalence among hospital patients, then you don't need to sample. Just take the entire population of the hospital and compute the exact result. If you want to know prevalence among general population, then hospital patients are certainly not a uniform sample of it." CreationDate="2011-03-03T19:46:32.120" UserId="3369" />
  <row Id="12802" PostId="7847" Text="Could you provide some theoretical justification for this opinion?" CreationDate="2011-03-03T19:46:56.317" UserId="919" />
  <row Id="12803" PostId="7826" Text="And the second question is, can you assume the infections are independent. I.e. for someone who has pneumonia, are they just as likely to get urinary infection as anyone else, or are they more likely to get it (maybe because they are already sick and it's easier for them to succumb to another infection)?" CreationDate="2011-03-03T19:48:07.050" UserId="3369" />
  <row Id="12804" PostId="7839" Text="It's not clear what you mean by &quot;percent error.&quot;  For general guidance, take a look at the Wikipedia article on &quot;level of measurement&quot; (http://en.wikipedia.org/wiki/Level_of_measurement ).  As a rough guide, data considered to be on an &quot;interval scale&quot; are compared additively (e.g., by subtracting a mean from an individual value) whereas data on a &quot;ratio scale&quot; can validly be compared multiplicatively (e.g., by dividing a value by the mean or by a percent difference).  In a regression setting where the slope can be positive or negative, this rules out using percent difference." CreationDate="2011-03-03T20:13:02.480" UserId="919" />
  <row Id="12806" PostId="7849" Text="$D$ is defined as the supremum of absolute differences in the two ECDFs; $Z$ is defined as $\sqrt{n_1 n_2 / (n_1+n_2)} D $. I was just explaining how to compute $D$ from SPSS output. Hope that clarifies." CreationDate="2011-03-03T20:24:34.700" UserId="3408" />
  <row Id="12807" PostId="7857" Text="That would work when the two sample sizes are similar.  But when they are dissimilar, the common bin size (even in standardized units) could be appropriate for one or the other histogram, but not for both.  How would you deal with that case?" CreationDate="2011-03-03T20:26:21.380" UserId="919" />
  <row Id="12808" PostId="7835" Text="@Zach, @rolando2 Make answers!" CreationDate="2011-03-03T20:42:53.473" UserId="88" />
  <row Id="12809" PostId="7843" Score="1" Text="It'd be helpful if you'd summarize the ones you think are best." CreationDate="2011-03-03T20:51:41.307" UserId="2669" />
  <row Id="12810" PostId="7851" Text="No. I would not consider an aggregate statistic an experimental value." CreationDate="2011-03-03T20:52:03.507" UserId="3489" />
  <row Id="12811" PostId="7847" Text="No, just an opinion. But if I had time, I would start my research from the retail packaging world (thin body perception), and incorporate some of the work of Tufte." CreationDate="2011-03-03T20:59:18.633" UserId="3489" />
  <row Id="12812" PostId="7857" Text="Maybe we're thinking about different meanings of standardize. I meant the one I linked to where, for instance, if one population has a stdev of 5 and the other has a stdev of 10, after standardization they would both have a stdev of 1. They could then be more fairly compared with the same bin size since each bin has a comparable amount of pixels and data. Or maybe you were getting at the larger issue that &quot;appropriate bin size&quot; is a bit of a black art and unique to every data set..." CreationDate="2011-03-03T22:21:47.360" UserId="1191" />
  <row Id="12813" PostId="7860" Score="2" Text="@Gappy: A few questions: How many components do you have? Besides the sample size, is there anything that makes the display of this PCA output need to be different from the display of other continuous variables that one might be dealing with? Are you trying to contrast scores of different groups, and if so how many?  Generally, what are you hoping to achieve with your displays?" CreationDate="2011-03-03T23:26:37.130" UserId="2669" />
  <row Id="12814" PostId="7864" Text="That's not a math problem, more a definition problem. Plus it deals with data analysis. You should move this to [Cross Validated](http://stats.stackexchange.com/)." CreationDate="2011-03-03T20:45:03.997" UserId="2036" />
  <row Id="12815" PostId="7864" Text="I am struggling to find a coherent question here: could you please clarify what you're looking for?" CreationDate="2011-03-04T04:08:01.613" UserId="919" />
  <row Id="12816" PostId="7869" Score="2" Text="To be honest, this may be something more appropriate for StackOverflow. " CreationDate="2011-03-04T07:29:11.130" UserId="1118" />
  <row Id="12817" PostId="7869" Text="Christopher, thanks for your comments. Now, I have posted it there too. I posted here because there may be some statistics technique to achieve this." CreationDate="2011-03-04T07:31:02.830" UserId="3535" />
  <row Id="12818" PostId="7869" Text="@Imran, I edited your question, to make this question more apropriate for this site. My question is what is your optimisation criteria? What do you expect from this optimisation? Are there any  long-term criteria for this optimisation? " CreationDate="2011-03-04T07:52:56.960" UserId="2116" />
  <row Id="12819" PostId="7869" Text="@mpiktas, thanks for your kindness. **1. optimisation criteria?** After x views (or conversions), check which item has the highest conversion rate and increase its weight. It will consequently decreases the low converting headings to keep the weight total to 100 . How much would be increased against what, I am not sure about it yet. I need help on this too. **2. Expectation** The more converting headings should be served more and vise versa. **long-term criteria** sorry not sure about it too. I'd really appreciate if you could help with the info provided. I may explain it more as we move along" CreationDate="2011-03-04T09:22:58.330" UserId="3535" />
  <row Id="12820" PostId="7845" Score="1" Text="@dsimcha, I think something like age/gender plots could be useful pictures. Anyway why to use histograms for this? Just plot distribution functions directly. However, if you are playing with empirical things, then QQ plot suggestion is the best choice." CreationDate="2011-03-04T09:24:51.940" UserId="2645" />
  <row Id="12821" PostId="7868" Score="3" Text="@curious Maybe http://gis.stackexchange.com/ would be more appropriate in your case. Anyway, you might think of updating the title of your question because as it stands it doesn't really reflect your aims (the use of &quot;classifier&quot; is misleading here)." CreationDate="2011-03-04T09:37:29.903" UserId="930" />
  <row Id="12822" PostId="7869" Text="I need some point to start with this. We can fine tune it after we have something in working. I hope you understand. Thanks again." CreationDate="2011-03-04T09:37:41.350" UserId="3535" />
  <row Id="12823" PostId="7417" Text="what did you test ? Average number of generations as suggested by the OP ? Difference from (if known) optimal solution ? A combination of both ? How do you test whether one algorithm is more likely to get stuck in local optimas ?" CreationDate="2011-03-04T12:18:22.927" UserId="264" />
  <row Id="12824" PostId="7879" Score="1" Text="@steffen +1, for the links. " CreationDate="2011-03-04T13:30:21.350" UserId="2116" />
  <row Id="12825" PostId="7364" Text="@gappy Is $n$ also the sample size?  Is $\Sigma$ assumed to be diagonal?  I assume that you are asking for a fixed sample size, what happens to the MLFA solution as the uniquenesses (entries of $\Sigma$, assumed to be diagonal) tend to $0$.  It also sounds like you’ve already seen Tipping and Bishop (1999)." CreationDate="2011-03-04T14:32:47.477" UserId="1670" />
  <row Id="12826" PostId="7847" Text="@whuber: it is mostly related to the way our brain processes information. When there are smaller bins, our mind also &quot;shrinks&quot; the boundaries of the curve. Try inversing the size of the bins in fig. #2 to see what I mean." CreationDate="2011-03-04T15:07:06.447" UserId="582" />
  <row Id="12827" PostId="7847" Text="@nico Yes, there is a perceptual element to the question.  But in the forefront is the statistical issue because it has a much larger influence: smaller bins ==&gt; more sample variability in the bins ==&gt; more &quot;ragged&quot; histograms ==&gt; greater difficulty in comparison.  Thus, IMO, any worthwhile answer should adduce support from *statistical* theory (at a minimum)." CreationDate="2011-03-04T15:50:43.840" UserId="919" />
  <row Id="12828" PostId="7857" Text="@xan We share the same meaning of &quot;standardize.&quot;  Choosing a bin size requires judgment and knowledge of context, but it's a stretch to characterize it as a &quot;black art&quot;: see, for example, http://stats.stackexchange.com/q/798/919 ." CreationDate="2011-03-04T15:52:46.633" UserId="919" />
  <row Id="12829" PostId="7881" Text="Doesn't this just push the question down to the issue of selecting appropriate kernel widths and whether (and how) one can compare two smooths using different kernel widths?" CreationDate="2011-03-04T15:53:58.503" UserId="919" />
  <row Id="12830" PostId="7845" Text="@Dimitrij Using the EDFs (empirical [cumulative] distribution functions) is a nice idea.  Moreover, there are reasons to choose between histograms (which essentially are empirical PDFs) and EDFs.  See, for instance, http://stats.stackexchange.com/q/4810/919 .  Thus your suggestion would solve the problem well in some cases but others really do need a histogram to be displayed.  Yet isn't the *display* of a data distribution really separate from the question of *graphical comparison* of two distributions?" CreationDate="2011-03-04T15:58:39.233" UserId="919" />
  <row Id="12831" PostId="7873" Text="Are they strictly interested in survival from one season to the next, or would they prefer to model survival over time?" CreationDate="2011-03-04T15:59:48.733" UserId="71" />
  <row Id="12832" PostId="7880" Text="Indeed it does. Thx Gavin, this are just one great explanation and nice examples. I just did not get that the bar in the data panel is the unit bar. Plus I find it a little irritating that you don't have scales for trend. Anyway great help! thx!" CreationDate="2011-03-04T16:00:53.377" UserId="704" />
  <row Id="12833" PostId="7880" Text="why the **** can't I accept this answer? I'll try to accept it later." CreationDate="2011-03-04T16:01:52.470" UserId="704" />
  <row Id="12834" PostId="7880" Score="1" Text="D'oh. I got it. The color of acceptance is really bad for be, because I am colorblind :) It's much better on SO, maybe CV should change it because there a quite a few colorblind people around... ;)" CreationDate="2011-03-04T16:04:36.337" UserId="704" />
  <row Id="12835" PostId="7847" Text="@whuber: I was referring to the fact that the distribution **look** differently dispersed in the two images. Of course how they look has nothing to do with how much they are really dispersed." CreationDate="2011-03-04T16:07:26.113" UserId="582" />
  <row Id="12836" PostId="7881" Score="1" Text="@whuber - valid point. I wasn't trying to suggest density curves were the be all end all method to use, simply offering other alternatives. It is clear from this post that there are pros and cons to any approach, so was offering this up as another viable alternative to throw into the mix." CreationDate="2011-03-04T16:07:52.583" UserId="696" />
  <row Id="12837" PostId="7483" Text="@Dikran, could you ask around who might use such ? There seem to me (amateur) to be quite different communities -- image processing, text ... with not much commonality. Plus, as you say above, that would kill papers with unknown data." CreationDate="2011-03-04T16:09:43.347" UserId="557" />
  <row Id="12838" PostId="7868" Score="1" Text="@curious I'm not understanding something: if a prefix of the numbers &quot;points to&quot; a geographic region, isn't that your solution?  If a new number has a unique new prefix and you are hoping to determine (or guess) which region it might indicate, then you need to supply more information about the nature of the prefix-region relation in order to begin addressing this problem." CreationDate="2011-03-04T16:14:25.950" UserId="919" />
  <row Id="12839" PostId="7880" Text="@ran2 what do you mean by no scales for trend? Do you mean on the axis or no grey bar for trend? If the latter, I suspect that the trend is such a strong/large part of the variation in the data that the grey bar is so small as to be less than 1px in size. Try plotting on a huge `pdf()` device and see if it shows up? As for the colour of the tick, I'll post something on meta to see if we can change it and point to this accessibility issue." CreationDate="2011-03-04T17:04:14.550" UserId="1390" />
  <row Id="12840" PostId="7888" Text="Oh, right, thanks! Forgot about that syntax for aggregate." CreationDate="2011-03-04T17:53:11.003" UserId="3432" />
  <row Id="12841" PostId="7770" Text="@Conjugate Prior : Thank you very much ! I will certainly give it a try. " CreationDate="2011-03-04T18:02:10.243" UserId="3301" />
  <row Id="12842" PostId="7884" Text="Thanks for all your answers. The data.table solution of @Steve was the fastest by a factor of ~5 on my data set over the aggregate() solution of @Gavin (which in turn was faster than my aggregate() code), and a factor of ~7.5 over the by() solution of @Matt. I didn't time the reshape idea because I couldn't get it working quickly. I'm guessing the solution that @Chase gave will be the fastest and it was actually what I was looking for, but when I started writing this comment, the code wasn't working (I see it's fixed now!)." CreationDate="2011-03-04T18:17:40.457" UserId="3432" />
  <row Id="12843" PostId="7884" Text="Actually @Chase was faster by a factor of ~9 over data.table, so I changed my accepted answer. Thanks again everyone - learned a bunch of new tools." CreationDate="2011-03-04T18:29:55.627" UserId="3432" />
  <row Id="12844" PostId="7884" Text="@lockedoff - sorry, I fixed my code. The one caveat or trick here is to concatenate a value that isn't one of your IDs in the `diff()` so that you can pick up the first ID in `dx`. " CreationDate="2011-03-04T18:30:44.693" UserId="696" />
  <row Id="12845" PostId="7889" Score="1" Text="Clever!  You could also do `dx[c(TRUE, dx$ID[-1] != dx$ID[-length(dx$ID)], ]` for non-numeric data - I get 0.03 for character, 0.05 for factors.  PS: there's an extra `)` in your first `system.time()` function, after the second zero." CreationDate="2011-03-04T18:32:00.230" UserId="71" />
  <row Id="12846" PostId="7889" Text="@Matt - good call and nice catch. I don't appear to be able to copy/paste code worth a flip today." CreationDate="2011-03-04T18:33:57.817" UserId="696" />
  <row Id="12847" PostId="7888" Text="If you'd like to add Chase's solution, here's what I got: `dx$ID &lt;- sample(as.numeric(dx$ID)) #assuming IDs arent presorted&#xA;system.time(replicate(1000, {&#xA;	dy &lt;- dx[order(dx$ID),]&#xA;	dy[ diff(c(0,dy$ID)) != 0, ]&#xA;	}))&#xA;   user  system elapsed &#xA;   0.58    0.00    0.58`" CreationDate="2011-03-04T18:34:06.187" UserId="3432" />
  <row Id="12848" PostId="7881" Text="@Chase In light of that I'm voting up your answer, +1." CreationDate="2011-03-04T20:03:53.667" UserId="919" />
  <row Id="12850" PostId="7888" Text="@lockedoff - done, thanks, but I didn't randomly sample the `ID`s so the result were comparable to other solutions." CreationDate="2011-03-04T20:17:29.627" UserId="1390" />
  <row Id="12851" PostId="7888" Text="And time @Matt Parker's version in the comments to @Chase's answer" CreationDate="2011-03-04T20:21:39.853" UserId="1390" />
  <row Id="12852" PostId="7895" Score="2" Text="Isn't the point of homework that *you* do the homework so *you* actually might learn something!?" CreationDate="2011-03-04T20:25:52.233" UserId="1390" />
  <row Id="12853" PostId="7895" Text="Let's not downvote this. The Q is OT and should be closed, but I don't have the rep here to do that." CreationDate="2011-03-04T20:34:34.577" UserId="1390" />
  <row Id="12854" PostId="7895" Text="@sefa What are you looking for here?  You indicated &quot;chi-squared&quot; as a tag and that's one possible answer.  What have you tried?  How far have you gotten on your own in solving the problem?" CreationDate="2011-03-04T20:37:41.117" UserId="919" />
  <row Id="12855" PostId="7895" Text="@Gavin Thank you for the edits.  I agree that even in its improved form it should be closed, but let's give the OP a chance to clean up the question first." CreationDate="2011-03-04T20:39:22.053" UserId="919" />
  <row Id="12856" PostId="7888" Score="2" Text="Thanks for doing the timings, Gavin - that's really helpful for questions like these." CreationDate="2011-03-04T20:53:32.800" UserId="71" />
  <row Id="12857" PostId="7894" Score="2" Text="Why not just plot the three ROC curves for -1 vs. 0/1, 0 vs. -1/1, and 1 vs. -1/0?" CreationDate="2011-03-04T20:57:17.333" UserId="919" />
  <row Id="12858" PostId="7888" Text="Of course, it just now occurs to me that Chase's solution and my variant both rely on pre-sorted data.  Gavin, would you mind testing to see how sorting the data affects the timings?  It took almost three times as long on my (considerably slower) computer when I used Chase's version of `dx`." CreationDate="2011-03-04T21:08:45.503" UserId="71" />
  <row Id="12859" PostId="7893" Text="@whuber, you are absolutely right. In my answer, I assumed both that $x_{(n)} \neq 0$ and $\alpha &gt; 0$. If you want to post your comment as an answer, I'll delete my answer." CreationDate="2011-03-04T21:21:01.160" UserId="401" />
  <row Id="12860" PostId="7894" Score="1" Text="I noticed you are predicting the train set -- this is wrong, since you'll get a too optimistic approximation. You should use OOB votes instead, which are in `model$votes`." CreationDate="2011-03-04T21:30:36.250" UserId="88" />
  <row Id="12861" PostId="7893" Score="1" Text="@Charlie Why not just edit your answer?  As I said, your idea is correct, so you just need to fix the formula.  Then I can vote it up :-)." CreationDate="2011-03-04T21:32:59.430" UserId="919" />
  <row Id="12862" PostId="7827" Text="FWIW, any Box-Cox transformation of power greater than 1 will reduce negative skew.  In light of Rob Hyndman's response that's not the first thing to try, though." CreationDate="2011-03-04T21:46:20.040" UserId="919" />
  <row Id="12863" PostId="7893" Score="1" Text="@whuber, I didn't want to take any credit from you. Thanks for the assist!" CreationDate="2011-03-04T21:46:40.453" UserId="401" />
  <row Id="12864" PostId="7828" Score="1" Text="+1.  Of note is that there are fairly simple formal procedures to investigate transformations, such as spread-vs.-level plots (described in Tukey's *EDA*)." CreationDate="2011-03-04T21:48:14.470" UserId="919" />
  <row Id="12865" PostId="7893" Text="@Charlie I fixed it up for you :-)." CreationDate="2011-03-04T21:53:49.380" UserId="919" />
  <row Id="12866" PostId="4959" Score="2" Text="The two copies of this question reference each other and, appropriately, the stats site (here) has a statistical answer and the math site has a mathematical answer.  It seems like a good division: let it stand!" CreationDate="2011-03-04T21:59:15.003" UserId="919" />
  <row Id="12867" PostId="7880" Text="no, i just mean the scale on the axis. in my case the trend is not as strong as in your second example, but still strong. But as you example shows: the trend never has a scale no matter if the bar is small or big. It does not really matter here because the relation is important here, but nevertheless i would be interested in the meaning of the scale on the right (repsectively its dimension – if there is any. Or is it just a dimensionless factor?) " CreationDate="2011-03-04T22:14:15.327" UserId="704" />
  <row Id="12868" PostId="7894" Text="And well, this ROC surface is 5D in 6D space... not too feasible for `rgl` =]" CreationDate="2011-03-04T22:35:00.967" UserId="88" />
  <row Id="12869" PostId="7862" Text="A really dynamite addition." CreationDate="2011-03-04T22:35:55.947" UserId="2669" />
  <row Id="12870" PostId="7900" Score="3" Text="I like the Xeger solution linked to in the first post: it obtains the deterministic FSA created by the system and then makes random transitions within it.  (The snarky negative comments in the second post seem altogether ignorant of that simple, valid approach).  AFAIK, nothing in R provides the equivalent capability.  Consider using something like Xeger to generate a text file of strings and then read it into R for the intended statistical processing." CreationDate="2011-03-04T22:59:05.190" UserId="919" />
  <row Id="12871" PostId="7902" Score="1" Text="You want to read up on *non-response bias.*  That may help you elaborate on this question, because to get a suitable answer you need to disclose the nature of the &quot;invalidity&quot; as well what you're really trying to estimate." CreationDate="2011-03-04T23:03:36.183" UserId="919" />
  <row Id="12872" PostId="7854" Text="Thank you very much. That's exactly what I was trying to do." CreationDate="2011-03-04T23:12:11.037" UserId="3008" />
  <row Id="12873" PostId="7902" Text="Thanks, whuber. So by &quot;invalid&quot;, I meant the cases that are not interesting to our study. For example, assume we study the returned items in a supermarket. We are interested in only the cases where the items are damaged or incomplete. These cases are labeled with this reason, so it is straightforward to extract them from the database. But some of them are mislabeled, i.e. they are actually not because of this reason. So we have read through all of them and exclude those &quot;invalid&quot; cases." CreationDate="2011-03-04T23:18:55.953" UserId="3545" />
  <row Id="12874" PostId="7900" Score="1" Text="Or make a &quot;RXeger&quot; wrapper package with rJava and than post it on CRAN ;-)" CreationDate="2011-03-04T23:45:03.880" UserId="88" />
  <row Id="12875" PostId="7903" Score="3" Text="This is a tough problem because you have very little information about the nature of the cheating. How do you differentiate a cheater from a student who studied extra hard? Without more information, you can't. One possibility is if students can cheat by copying off each other, or if subsets of students had access to the same answers. If this is the case, you could create a distance function between students (lower distance means they did well on the same questions) and look for patterns here. This would be more conclusive IMO. " CreationDate="2011-03-05T00:28:57.823" UserId="2965" />
  <row Id="12876" PostId="7906" Text="Thanks, @Matthias Vallentin. I've just used this for generation of a random length word based on a simple regular expression. I had been unable to do the same in Excel earlier. However, it would be nice to have a more robust approach in the future!" CreationDate="2011-03-05T01:28:18.020" UserId="3551" />
  <row Id="12877" PostId="7903" Score="2" Text="Levitt and Dubner describe their approach in *Freakonomics* (http://www.freakonomicsmedia.com/ )." CreationDate="2011-03-05T01:56:01.810" UserId="919" />
  <row Id="12878" PostId="7739" Text="@cardinal - I'm curious about your comment about &quot;least squares&quot; not ever performing matrix inversion.  The main equation for the estimates is $\beta=(X^{T}X)^{-1}X^{T}Y$.  Further the variance of these estimates is given by $\sigma^{2}(X^{T}X)^{-1}$.  The matrix inverse is fundamental to the typical least squares regression, at least in the mathematics.  Although I my be showing my ignorance in the actual computational procedures followed." CreationDate="2011-03-05T05:16:46.580" UserId="2392" />
  <row Id="12879" PostId="7907" Text="@whuber +1, great answer! " CreationDate="2011-03-05T07:24:29.737" UserId="2116" />
  <row Id="12880" PostId="7903" Text="@rm999 To clarify, cheaters had access to the *same* subset of questions (e.g., a partial answer key was leaked prior to the exam administration). I am not interested in cheating that might have happened from copying. I will revise my question over the weekend if this is unclear. " CreationDate="2011-03-05T07:56:26.253" UserId="3432" />
  <row Id="12881" PostId="7903" Text="@whuber Thanks, I will look up the paper (assuming it's published). I listened to the audiobook, but can't remember the details of how they identified cheaters (who were teachers who were fudging students' answers, I believe)." CreationDate="2011-03-05T07:58:46.767" UserId="3432" />
  <row Id="12882" PostId="7880" Text="But there is a scale on the trend panels in both my examples?!" CreationDate="2011-03-05T08:08:05.607" UserId="1390" />
  <row Id="12883" PostId="7908" Text="@SheldonCooper I read the first part of your answer and think it's promising. Two quick notes -- this was multiple choice so probabilities of guessing correctly are 25% or 20%. You are correct in that we may assume a subset of questions was leaked before the exam. Will return to this on Sunday or Monday." CreationDate="2011-03-05T08:12:24.103" UserId="3432" />
  <row Id="12886" PostId="7888" Text="@Matt can I just check what you mean? You version works fine with a random ordering of rows in the original `dx`. The only difference is the result is not in the order of the `ID`s." CreationDate="2011-03-05T09:51:43.697" UserId="1390" />
  <row Id="12888" PostId="7901" Score="1" Text="@nico +1 for the tip about `cex`, but I think the OP wants all stuff on the same plotting region, not on separate ones." CreationDate="2011-03-05T10:28:43.020" UserId="930" />
  <row Id="12889" PostId="7911" Score="1" Text="@chl +1, nice graph, and short code. " CreationDate="2011-03-05T10:44:08.753" UserId="2116" />
  <row Id="12890" PostId="7911" Text="@mpiktas Thanks. This led me to realize I didn't work with the right sample, in fact :-)" CreationDate="2011-03-05T10:50:57.697" UserId="930" />
  <row Id="12891" PostId="7901" Text="Ahh... now I understand the question. Well, then he can just use `curve` or `points` to overplot the three graphs ;)" CreationDate="2011-03-05T10:54:12.657" UserId="582" />
  <row Id="12893" PostId="7911" Text="Almost looks like the one I need, but with real numbers I faced the following problems:&#xA;1) `df.new &lt;- data.frame(x = seq(min(x), max(x), 0.1))` is better.&#xA;2) Ellipse is drawn at 0;0 position, which is not correct and it`s size is also strange (too small). Also tryed `library(car) dataEllipse(df$x, df$y, levels=0.95:1, lty=2)` but it drops all.&#xA;3) The curve (like on correlogramm) is missing. I almost reproduced it by calling `library(car) cr.plots(m0)` but the range of data is incorrect.&#xA;&#xA;Use first 2 lines from my code instead of yours to reproduce." CreationDate="2011-03-05T14:36:40.463" UserId="3376" />
  <row Id="12894" PostId="7796" Text="@David, please see my other posted answer to your question. The method is different than the upper bound you give, but the tools employed are nearly as elementary, in contrast to the answer I gave here." CreationDate="2011-03-05T14:40:39.880" UserId="2970" />
  <row Id="12895" PostId="7911" Text="@Yuriy Ok, I will update my code (no need to make any edit in the meantime), but I cannot see how we could get overlap with real-valued random variates with your $(x,y)$ setting; this is the reason why I use boostrap with replacement (this ensures that ~ 2/3 of the original units are present). `car::dataEllipse` does provide the same facilities than in the `ellipse` package, but it is probably less easy to customize. I guess the superimposed curve is just a *loess*, so it is not difficult to add." CreationDate="2011-03-05T14:56:25.637" UserId="930" />
  <row Id="12896" PostId="7739" Text="@probabilityislogic, one common approach is to use (some variant of) a $QR$ decomposition. So, we write $X = Q R$, where $Q$ is a matrix with orthogonal columns and $R$ is a square triangular matrix. It's an easy exercise to see that the residuals can be written as $\hat{y} = Q Q^T y$ and the parameter estimates are the solution so the triangular system of equations $R \hat{\beta} = Q^T y$. Triangular systems are very efficient to solve. The $Q R$ decomposition using Householder reflections or Givens rotations is very numerically stable. No matrix inversion necessary." CreationDate="2011-03-05T15:11:53.877" UserId="2970" />
  <row Id="12897" PostId="7916" Text="I have made an approach to maximize F-Score using grid search but it's very expensive, almost infeasible. " CreationDate="2011-03-05T15:25:28.810" UserId="1371" />
  <row Id="12898" PostId="7916" Score="1" Text="@matcheek This is the SVM reality -- you can try randomly reducing the dataset and optimizing on it, to at least get an approx. where to look. Or just change the classifier ;-)" CreationDate="2011-03-05T16:03:12.257" UserId="88" />
  <row Id="12899" PostId="7911" Text="@Yuriy Done. There were two errors in my previous code: (1) as there are replicates in the data.frame, length of `x` and `table(x)` will differ, hence the `cex` param was recycled; (2) I forgot to update the location and scale of the ellipse (not clearly apparent when using standard gaussian variates). I hope there are no other ones." CreationDate="2011-03-05T16:06:22.913" UserId="930" />
  <row Id="12900" PostId="7911" Text="Got `Error in ellipse(cor(df$x, df$y), scale = c(sd(df$x), sd(df$y)), centre = c(mean(df$x),  : &#xA;  center must be a vector of length 2` and no ellipse :( Everything else is excellent. Thank you very much." CreationDate="2011-03-05T16:28:25.823" UserId="3376" />
  <row Id="12901" PostId="7911" Text="@Yuriy It works for me; maybe there's a conflict with another function having the same name (in the car package)? Try to source all the code in a new R session. The picture was generated this way." CreationDate="2011-03-05T16:35:13.217" UserId="930" />
  <row Id="12902" PostId="7911" Text="@chl Sorry, my fault. Thank you again!" CreationDate="2011-03-05T16:56:00.800" UserId="3376" />
  <row Id="12903" PostId="7862" Text="Nice. How did you generate the graph above? Is there an R package, or bare hands?" CreationDate="2011-03-05T16:56:57.567" UserId="30" />
  <row Id="12904" PostId="7916" Score="1" Text="The approach the WEKA uses to optimize the hyperparamters, is to:&#xA;first: use a 2-fold cross validation to &quot;roughly&quot; estimate the location of the hyperparamters on the grid, and&#xA;second: do a 10-folds cross validation on the adjacent points (found in the previous steps) to find a more &quot;accurate&quot; estimate of the hyperparameters.. Hope this help speed your code up.. :) &#xA;" CreationDate="2011-03-05T17:34:13.917" UserId="3544" />
  <row Id="12905" PostId="7862" Text="I’ve written my own function for drawing a biplot with ggplot2.  R has the builtin biplot() function, but it’s not that great.  http://www.multivariatestatistics.org/biplots.html is a good place to learn more about the biplot" CreationDate="2011-03-05T18:14:14.137" UserId="1670" />
  <row Id="12906" PostId="7919" Score="1" Text="@analyticsPierce The zero is an observable value, that is your predictor can take such a value? " CreationDate="2011-03-05T19:09:24.463" UserId="930" />
  <row Id="12907" PostId="7912" Score="1" Text="The Mahalanobis distance makes no sense when the distributions differ.  (It's like saying &quot;Peter lives on a sphere and Paul lives on a Euclidean plane; how do we compute the distance between them?&quot;)  Perhaps you could back up a step and help us understand the motivation for the question: what exactly do you want to accomplish here?  What is the statistical context?" CreationDate="2011-03-05T19:16:48.920" UserId="919" />
  <row Id="12908" PostId="7919" Text="If zero is special, that suggests the response is non-negative or bounded at zero. Is that the case?" CreationDate="2011-03-05T19:18:40.307" UserId="1390" />
  <row Id="12909" PostId="7919" Score="1" Text="To echo @chl: in some contexts the zero might be a code for a missing value or an undetectable amount, whereas in other contexts it might be a &quot;true&quot; zero (e.g., a count).  The proper way to deal with these situations varies radically." CreationDate="2011-03-05T19:19:51.070" UserId="919" />
  <row Id="12910" PostId="7912" Text="Alright, I suspected so much. The reason why I ask is that I've seen the following equation being used to compute a 'Mahalanobis' distance, or so it claimed: $$\Delta^T \(\Sigma_1 \Sigma_2\)^{-1} \Delta$$ I'm not too sure that's a Mahalanobis distance; I'm just mirroring what was claimed. Would a Bhattacharya distance work better in its place?" CreationDate="2011-03-05T19:44:48.300" UserId="3586" />
  <row Id="12911" PostId="7915" Score="1" Text="How many datapoints do you have? What kind of SVM are you uisng -- linear or Gaussian kernel (or some other kernel)? Kernel methods in libsvm are much slower." CreationDate="2011-03-05T20:08:11.963" UserId="3369" />
  <row Id="12912" PostId="7915" Text="C-SVC, 26k 9-dimensional vectors, polynomial kernel.&#xA;Just edited grid.py to maximize F-scores instead, but still looking for some clever options to run SVM Classfier." CreationDate="2011-03-05T20:24:31.613" UserId="1371" />
  <row Id="12913" PostId="7912" Text="@k-damato Mahalanobis distance measures distance between points, not distributions." CreationDate="2011-03-05T20:42:42.917" UserId="1670" />
  <row Id="12914" PostId="7919" Text="someone remind me - &quot;predictor variables&quot; are what another third of the world calls &quot;covariates&quot; and another third calls &quot;explanatory variables&quot; yup? Actually there's another third that either calls them dependent or independent variables, I forget which..." CreationDate="2011-03-05T20:54:29.600" UserId="1549" />
  <row Id="12915" PostId="7921" Score="2" Text="@vqv, a couple of recent discussions here have focused on modifications to the KL divergence that result in a proper metric. In case you're interested, see **[here](http://stats.stackexchange.com/questions/7630/clustering-should-i-use-the-jensen-shannon-divergence-or-its-square)** and **[here](http://stats.stackexchange.com/questions/6907/an-adaptation-of-the-kullback-leibler-distance)**." CreationDate="2011-03-05T22:02:07.153" UserId="2970" />
  <row Id="12916" PostId="7912" Text="Alright, so does anyone recognize the above equation as something meaningful? The deltas are displacement vectors." CreationDate="2011-03-05T22:11:56.397" UserId="3586" />
  <row Id="12917" PostId="7566" Text="@Dmitrij Celov Given that a small minority of my sales data is in the range `0.0 to 1.0`, should I add `1` to all my sales figures before using a log transformation?" CreationDate="2011-03-06T01:02:18.327" UserId="179" />
  <row Id="12918" PostId="7739" Text="@cardinal - thanks for that.  So I suppose my &quot;thought bubble&quot; reduces to comparing the QR decomposition speed with numerical integration" CreationDate="2011-03-06T01:21:11.820" UserId="2392" />
  <row Id="12919" PostId="7739" Text="@probabilityislogic, $QR$ decomposition should never be worse than $O(n p^2)$, and that provides an exact (to within numerical precision) answer. To compare that against Monte Carlo integration would require you specifying at least a couple notions of desired precision." CreationDate="2011-03-06T02:23:15.443" UserId="2970" />
  <row Id="12920" PostId="7914" Text="Nice job!  I'd like to ask what may be a naive question.  I've seen and used the formula for the standard error of a proportion umpteen times, but I've never encountered the sqrt{1-f} portion of it.  Under what conditions is this needed?" CreationDate="2011-03-06T02:36:24.510" UserId="2669" />
  <row Id="12922" PostId="7917" Text="@cardinal That's very nice and just what the doctor ordered. Thank you." CreationDate="2011-03-06T05:00:20.980" UserId="3500" />
  <row Id="12923" PostId="7917" Text="@David, just curious: What is the intended application?" CreationDate="2011-03-06T05:01:43.043" UserId="2970" />
  <row Id="12924" PostId="7917" Text="@cardinal Long story. I'm trying to prove a lower bound for the mixing time of a Markov chain that I've cooked up in order to analyze the running time of an algorithm I'm interested in - which turns out to reduce to lower-bounding the c.collector problem.&#xA;&#xA;BTW, I had been playing around with trying to find exactly this kind of Chernoff-style bound, but hadn't figured out how to get rid of that product in $i$. Good call choosing $s = 1/n$ :-)." CreationDate="2011-03-06T05:23:11.837" UserId="3500" />
  <row Id="12925" PostId="7919" Text="@chi yes, the zero is a valid value. these are all spending variables and the 0 indicates that there was no spend. The goal is to understand how spending impacts response. Are the zero's included in the calculation of lm()? If I want to omit them should I convert them to NA and use na.omit to ignore them?" CreationDate="2011-03-06T07:32:45.523" UserId="1422" />
  <row Id="12926" PostId="7566" Text="@fmark, if you don't have exactly zeroes than there is no problem to take log transformation (the only case for zero is if the store sells nothing during a quarter!, but it is a strange case). You have this small range probably due to units you measure your sales. In log-log regression this will be absorbed into intercept term. However, if you do have some true zeroes then we will think about other useful tricks (by the way no need for logs in dummies)." CreationDate="2011-03-06T09:05:53.583" UserId="2645" />
  <row Id="12927" PostId="7417" Text="I tested difference from known optimal solution using approximately equal number of scoring function calculations. " CreationDate="2011-03-06T09:06:01.730" UserId="1496" />
  <row Id="12928" PostId="7923" Text="@Zia Possible related question: [Group differences on a five point Likert item](http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item)." CreationDate="2011-03-06T09:29:09.980" UserId="930" />
  <row Id="12929" PostId="7868" Text="@chl Well, I think it is a general stats question. Instead of GIS, it could have been mapping to anything else. Can you suggest me a better title? I am sorry I am very new to http://stats.stackexchange.com and not very good with stats." CreationDate="2011-03-06T09:29:24.563" UserId="1261" />
  <row Id="12930" PostId="7739" Text="@cardinal - I would suggest that the MC method has the ability to be &quot;scaled-up&quot; (more simulations) or &quot;scaled-down&quot; (less) according to the precision required.  With the QR method, although exact, you are stuck to a degree with the same computing time.  For something like all subsets regression, exactness of the answer may not be the number one priority.  Once again, if you have two methods - one of them will be quicker.  Expanding on my thought bubble would consist of what conditions are required for one method to be quicker than the other - and at what cost." CreationDate="2011-03-06T09:30:25.920" UserId="2392" />
  <row Id="12931" PostId="7868" Text="@curious &quot;Building a classification rule for geographical mapping of cell number&quot;? Maybe provide additional details, per @whuber's request, and this might be clarified further." CreationDate="2011-03-06T09:32:28.520" UserId="930" />
  <row Id="12932" PostId="7913" Text="@Peter Usually, cheating or guessing might be incorporated directly into an IRM. This is in essence what a 3-PL model intend to do, as it includes a parameter for *difficulty*, *discrimination*, and *guessing* which acts as a lower asymptote for the probability of endorsing an item. However, it has been proven to be unrealistic in most situations, and other dedicated *person-fit statistics* have been developed alongside (either in educational testing or psychological assessment). Meijer, Person-Fit research: An introduction. APM (1996), 9: 3-8 has a nice review on aberrant response patterns." CreationDate="2011-03-06T09:45:51.497" UserId="930" />
  <row Id="12933" PostId="7914" Text="Roughly speaking it indicates that the inference will be based on randomisation (over a finite but large population) rather than on a model (of a theoretically infinite population).  You'll see it a lot in survey research.  Notice that as the population converges on the sample the SE shrinks to 0, which is SheldonCooper's first case." CreationDate="2011-03-06T10:45:10.800" UserId="1739" />
  <row Id="12934" PostId="7868" Text="@chl: Done. Thanx for the title suggestion. And, I have provided more information." CreationDate="2011-03-06T11:21:42.327" UserId="1261" />
  <row Id="12935" PostId="7723" Score="1" Text="@whuber - I did not mean my comment as an &quot;attack&quot; but I could have phrased it a bit better, in hindsight.  I apologise for my poor choice of words in that respect.  What I meant was that the comment you have written would have been much better placed as the start of an answer, in which you focus your response on one particular aspect of the question.  The &quot;how do I even start?&quot; kind of question is a very good one I think, because this is often the kind of situation many statisticians actually face (I know I have)." CreationDate="2011-03-06T12:09:38.107" UserId="2392" />
  <row Id="12936" PostId="7737" Text="@cardinal - thanks for the comment.  I have updated my answer accordingly." CreationDate="2011-03-06T12:10:57.903" UserId="2392" />
  <row Id="12937" PostId="7913" Text="@chl  Thanks!  I studied this stuff in grad school, but that was long ago - my last class was in 1996 or so." CreationDate="2011-03-06T12:27:33.430" UserId="686" />
  <row Id="12938" PostId="7666" Text="@Henry - is there a measure of error for the slope parameter?  from my intuition it would seem to have something to do with &quot;how far&quot; apart the two lines are in the geometric average.  For instance, how should I test the hypothesis that the slope is zero?" CreationDate="2011-03-06T13:10:52.280" UserId="2392" />
  <row Id="12939" PostId="7911" Text="Hi chl, great piece of code.  Might I as what does the ellipse actually mean? (p.s: I think I'll repost your answer on my blog, it is a cool piece of code...)" CreationDate="2011-03-06T14:10:35.720" UserId="253" />
  <row Id="12941" PostId="7739" Text="@probabilityislogic, the issue is still that the $O(2^n)$ work for all-subsets is much larger than the $O(n p^2)$ work of a $QR$ decomposition. In the $QR$ case, once the initial full-regression estimates are obtained, conceivably, only simple one-rank updates can be made to find the parameter estimates for smaller models. These rank-one updates are fast. But, for all-subsets, *every* combination of such updates would have to be made. Your method would require re-estimating the integral every time $X$ changes and is not exact. I'd hazard a guess that that's far less efficient." CreationDate="2011-03-06T15:06:58.940" UserId="2970" />
  <row Id="12942" PostId="7666" Text="@probabilityislogic: I would have thought that looking at the   [correlation_coefficient](http://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Inference), taking account of the sample size, ought to tell you this.  It is a measure of how far apart the $y$-on-$x$ and $x$-on-$y$ slopes are. " CreationDate="2011-03-06T15:57:46.303" UserId="2958" />
  <row Id="12943" PostId="7911" Score="1" Text="@Tal The interpretation of the ellipse is the same as in the `corrgram` package: it shows 95% pairwise confidence region assuming a bivariate normal distribution centered on the mean and scaled by SD(x) and SD(y). I'm not a big fan of this when used in a scatterplot, though. But see Murdoch &amp; Chow, [A graphical display of large correlation matrices](http://www.jstor.org/pss/2684435), Am Stat (1996) 50:178, or Friendly, [Corrgrams: Exploratory displays for correlation matrices](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.1268&amp;rep=rep1&amp;type=pdf), Am Stat (2002) 56:316." CreationDate="2011-03-06T16:01:53.013" UserId="930" />
  <row Id="12944" PostId="7723" Score="1" Text="@probability Thank you for clarifying.  You make a good point about the &quot;how do I start&quot; question.  It seems to me, though, that there is a fundamental difference between the student and the statistician: the student is supposed to have the the resources at hand to tackle the problem, which is supposed to have a definite answer.  &quot;Resources&quot; = teacher, textbook, etc.  At a minimum, then, the student should be able to describe their initial efforts and articulate the obstacles they are encountering.  If not, what are we supposed to do in response?  Write a textbook?" CreationDate="2011-03-06T16:10:44.297" UserId="919" />
  <row Id="12945" PostId="7206" Score="2" Text="@cardinal +1, great overview. " CreationDate="2011-03-06T17:05:35.133" UserId="2116" />
  <row Id="12946" PostId="7917" Text="@David, $s = 1/n$, while almost certainly suboptimal, seemed like the obvious thing to try since that gave $e^{s t} = n e^{-c}$, which is the same term as the one obtained in the derivation for the upper bound." CreationDate="2011-03-06T17:48:44.687" UserId="2970" />
  <row Id="12947" PostId="7206" Text="+1 Impressive, useful overview." CreationDate="2011-03-06T18:12:06.497" UserId="919" />
  <row Id="12948" PostId="7888" Text="Hmm, it shouldn't as it's dependent on change in ID from one record to the next to identify first records.  Try: `dx.mix &lt;- dx[order(sample(dx$ID, 400000)), ]`; `dx.mix[c(TRUE, dx.mix$ID[-1] != dx.mix$ID[-length(dx.mix$ID)]), ]`.  I end up with over 390,000 records, instead of 7,000.  What am I missing?" CreationDate="2011-03-06T18:19:26.917" UserId="71" />
  <row Id="12950" PostId="7911" Text="@chl ggplot version is even nicer. It will be better to change colors and line styles: dashed red for ellipse and bolder (dark?)green for loess line... just a suggestion. Nice work. – Yuriy Petr" CreationDate="2011-03-06T21:27:26.393" UserId="3376" />
  <row Id="12951" PostId="7929" Score="1" Text="@ran2, [this book](http://books.google.com/books?id=ca5MkRbF3fYC&amp;printsec=frontcover&amp;dq=analysis+of+integrated+and+cointegrated+time+series+with+R&amp;hl=fr&amp;ei=QxB0TbPxMIGs4AaTxMmnDQ&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CCkQ6AEwAA#v=onepage&amp;q&amp;f=false) should answer all your questions." CreationDate="2011-03-06T22:53:54.283" UserId="2116" />
  <row Id="12952" PostId="7929" Text="thanks! I love the Springer useR series, but I did not know this one..." CreationDate="2011-03-06T23:03:16.300" UserId="704" />
  <row Id="12953" PostId="7880" Text="hell, I am not only colorblind. Now that you say it, I see the scale on the left. Sorry, Gavin. Sometimes it really takes extra patience to teach me things. thx man." CreationDate="2011-03-06T23:06:33.970" UserId="704" />
  <row Id="12954" PostId="6916" Text="@csgillespie Wonderful example!" CreationDate="2011-03-06T23:45:02.210" UserId="919" />
  <row Id="12955" PostId="7933" Text="@whuber +1 for this awesome response!" CreationDate="2011-03-07T00:03:48.440" UserId="930" />
  <row Id="12956" PostId="7927" Text="Thank you so much for this. I had some follow-up questions if you don't mind. Since the comment box doesn't allow me to list them clearly, I included them in the post.&#xA;&#xA;" CreationDate="2011-03-07T00:10:10.943" UserId="3572" />
  <row Id="12957" PostId="7903" Text="If I recall the Freakonomics case, it involved spotting children in the same school/class who had (a) large jumps in attainment compared with a year earlier, (b) different answers for the earlier easier questions, and (c) identical sequences of answers for later harder questions, so being suggestive of a teacher filling in answers which the children had left blank. " CreationDate="2011-03-07T00:24:12.533" UserId="2958" />
  <row Id="12958" PostId="7937" Text="I'd say that's an advantage, not a disadvantage.  It stops you from doing something you probably shouldn't be doing." CreationDate="2011-03-07T00:59:50.150" UserId="686" />
  <row Id="12959" PostId="7937" Text="@Peter:  Why?  I'm assuming that you would properly correct for multiple testing, etc. such that the P-values obtained would be valid." CreationDate="2011-03-07T01:51:40.110" UserId="1347" />
  <row Id="12960" PostId="7941" Text="There's a better place for this than SO: http://stats.stackexchange.com/" CreationDate="2011-03-07T02:55:45.807" />
  <row Id="12961" PostId="7944" Text="+1 Nice answer." CreationDate="2011-03-07T03:48:53.213" UserId="919" />
  <row Id="12962" PostId="7917" Text="**Request**: The proof I've given above is my own. I worked on it out of enjoyment, since the problem intrigued me. However, I make no claim to novelty. Indeed, I cannot imagine that a similar proof using a similar technique doesn't already exist in the literature. If anyone knows of a reference, **please** post it as a comment here. I would be **very** interested to know of one." CreationDate="2011-03-07T04:26:37.830" UserId="2970" />
  <row Id="12963" PostId="7925" Text="@Berk U, what do you mean by two $(t,m,s)$ nets being uncorrelated? In particular, what does this mean when you say &quot;us[ing] a deterministic scrambling algorithm&quot;? Many scrambling algorithms can be applied to arbitrary $(t,m,s)$ nets; I honestly don't know if *all* schemes do. I would imagine the answer might be &quot;no&quot;. (That is, one could construct a scrambling that was specialized enough that it maintained the closure property for a *particular* sequence, but not in general. I don't know about that off-hand.)" CreationDate="2011-03-07T04:38:27.393" UserId="2970" />
  <row Id="12964" PostId="7607" Text="Does nobody knows or is the question not interesting? I think the problem arises each time you want to use the wavelet tool for online analysis. Does nobody do online wavelet analysis?" CreationDate="2011-03-07T06:03:03.123" UserId="1709" />
  <row Id="12965" PostId="7925" Text="@cardinal Sorry that was unclear, so let me try to rephase it. Say I have two $(t,m,s)$ nets $P$ and $Q$ that I use to generate two sequences of 100 points, $\{p_i\}^{100}_{1}$ and $\{q_i\}^{100}_{1}$. If I use a random scrambling algorithm then  $\{p_i\}^{100}_{1}$ and $\{q_i\}^{100}_{1}$ should be uncorrelated, right? Clearly this is not true if I had used a deterministic scrambling algorithm. But what if generated 200 points and only kept even entries of $\{p_i\}^{200}_{1}$ and odd entries from $\{q_i\}^{200}_{1}$. Would these be correlated? And would they still be nicely &quot;spread out&quot;?" CreationDate="2011-03-07T06:44:54.337" UserId="3572" />
  <row Id="12966" PostId="7905" Text="+1 for the attempt" CreationDate="2011-03-07T08:06:15.363" UserId="2116" />
  <row Id="12967" PostId="7891" Text="@etarion, can you please cite these references? " CreationDate="2011-03-07T08:07:41.210" UserId="2116" />
  <row Id="12968" PostId="7880" Text="@ran2 no worries, it did cross my mind earlier that part of your confusion was perhaps missing the alternating y-axes. Glad the discussion/answer has helped." CreationDate="2011-03-07T08:19:48.247" UserId="1390" />
  <row Id="12969" PostId="7946" Score="2" Text="@BB01, the answer depends heavily on what do your values represent, i.e. what does -1 mean for example. Since your data is discrete you should use [Spearman rank correlation coefficient](http://en.wikipedia.org/wiki/Spearman's_rank_correlation_coefficient) or its alternatives. For the graph, again the answer depends on details of your study." CreationDate="2011-03-07T08:21:03.410" UserId="2116" />
  <row Id="12970" PostId="7929" Text="hmm, iiuc these test only whether phi = 1, not if phi &gt; 1. But still the R output says the alternative is: stationarity. So do they check phi &gt; 1, too? Obviously that would be non-stationary as well." CreationDate="2011-03-07T09:25:42.783" UserId="704" />
  <row Id="12971" PostId="7891" Text="@mpiktas - well, it was just the result of a google search on max-margin clustering, the &quot;original&quot; paper on it seems to be http://webdocs.cs.ualberta.ca/~dale/papers/nips04.ps.gz (Xu et al, NIPS 2004)" CreationDate="2011-03-07T09:33:34.240" UserId="3336" />
  <row Id="12972" PostId="7905" Text="@mbq: I'm not quite sure I understand your answer. Did you mean that &quot;the lack of a size constraint&quot; is the central idea? (Then the rest of the answer makes sense to me, kind of)" CreationDate="2011-03-07T09:36:25.233" UserId="3336" />
  <row Id="12973" PostId="7912" Text="@Kristian I have merged your two duplicated accounts. Please use your registered account, from now on." CreationDate="2011-03-07T10:19:42.277" UserId="930" />
  <row Id="12975" PostId="7905" Text="@etarion No, rather that it is the method for solving a problem of clustering in two groups of a given size. And thus without the constant, it just seems to dissolve into u-s SVM." CreationDate="2011-03-07T11:03:15.453" UserId="88" />
  <row Id="12976" PostId="7878" Text="Matt, thanks for the detailed answer. I really appreciate. I'll look into it and get back to you." CreationDate="2011-03-07T11:32:04.933" UserId="3535" />
  <row Id="12977" PostId="7879" Text="thanks steffen. I'm looking into it and trying to find if there some Ruby implementation of n-armed bandit problem. Thanks again." CreationDate="2011-03-07T11:32:58.957" UserId="3535" />
  <row Id="12978" PostId="7952" Text="@fRed, what is the coloured table? Cells with colour background? Cell text different colour? In the last scenario, the solution might be wrapping the values with appropriate tex commands." CreationDate="2011-03-07T11:36:52.637" UserId="2116" />
  <row Id="12979" PostId="7952" Text="I need the background of one row to be of a specific color. Do I have to rewrite an xtable function?" CreationDate="2011-03-07T11:42:44.543" UserId="1709" />
  <row Id="12980" PostId="7879" Text="@Imran: 1. If this is &quot;the&quot; answer, than please accept it by clicking the green check mark ;) 2. There is no need to look for such an implementation. It is really easy, I am sure you can do it yourself ! " CreationDate="2011-03-07T11:57:38.443" UserId="264" />
  <row Id="12981" PostId="7878" Text="@Matt, what if I don't have the option to reset the conversion values on a weight change? Could this algorithm be adjust to cope with this situation? Thanks for your help. " CreationDate="2011-03-07T12:10:27.070" UserId="3535" />
  <row Id="12983" PostId="7897" Score="1" Text="what do you mean by &quot;many more&quot; 100 ? 1000 ? 10000? 100000 ?" CreationDate="2011-03-07T12:33:16.153" UserId="223" />
  <row Id="12984" PostId="7879" Text="@steffen, I'd love to accept this as answer, but I may need a little more help so that I can get start implementing it ASAP (due to shortage of time). I have started reading the Chapter 2 of book you recommended. It has multiple sub-section under that chapter. I'm extremely sorry but could you please guide me which method (Action-Value Methods, Softmax Action Selection etc) is one I should go with and how would it map to my problem? I am really running short of time that's why I am going shortcut. Again sorry for this. I hope you understand the situation." CreationDate="2011-03-07T12:33:49.770" UserId="3535" />
  <row Id="12985" PostId="7775" Text="I don't really see the question. You want a function that calculates partial R2, but you already have one. Do you know package sensitivity (there are no partial R2 but pcc which is a particular case is implemented)?   " CreationDate="2011-03-07T12:39:35.443" UserId="223" />
  <row Id="12986" PostId="7953" Text="@chl +1, just as I suspected :) " CreationDate="2011-03-07T12:41:46.533" UserId="2116" />
  <row Id="12987" PostId="7951" Score="1" Text="@BB01, I tried to transform a question to the form that it is answerable. Please confirm that nothing is lost from the original form." CreationDate="2011-03-07T12:53:06.910" UserId="2116" />
  <row Id="12989" PostId="7879" Text="@Imran: I am sorry, too. Since Matt provided some code, I think his answer is more worthy to get accepted ;)." CreationDate="2011-03-07T13:19:38.043" UserId="264" />
  <row Id="12990" PostId="7932" Text="Coding is a must, but also knowing how to work with the OS not against it is important.  You must understand that sometimes splitting the work has extra costs associated with it, as accessing disks and networks carries additional costs.  You gotta understand different ways of blocking and waiting and doing interprocess communication.  I've seen great scientific code that would spent most of it's time waiting for some system calls to finish.  Befriend the sysadmin of your system, you can get a lot of help with optimization of yours systems by bringing them coffee ;)" CreationDate="2011-03-07T13:28:20.373" UserId="595" />
  <row Id="12991" PostId="7879" Text="@steffen. Great, I really appreciate that. At this I am trying his code. But I'd like to give `n-armed bandit problem` a shot too. Would you be able to help? How about starting a new question for this? Waiting..." CreationDate="2011-03-07T13:36:44.777" UserId="3535" />
  <row Id="12992" PostId="7878" Text="@Matt, I have implemented it and this works like a charm. Just curious to know about what effect would it have of not resetting the conversion values on weight change?" CreationDate="2011-03-07T14:08:41.087" UserId="3535" />
  <row Id="12993" PostId="7955" Text="@Weijlie You can use $\LaTeX$ code in the usual way, i.e. by enclosing your expression with `$` (inline) or `$$` (display). Also, please make your title shorter (e.g., &quot;Intuitive illustration of Newton's method in nonlinear regression&quot;)." CreationDate="2011-03-07T14:15:34.553" UserId="930" />
  <row Id="12994" PostId="7567" Text="@cardinal, I updated the results! What do you think? The question about the uniform distribution is another thing that i have to specify in a better way and i ll do it today or tomorrow ;)" CreationDate="2011-03-07T14:41:11.033" UserId="3342" />
  <row Id="12995" PostId="7948" Text="@chi thanks for editing my question. are there things that I should be clarifying or rewording in any future questions?" CreationDate="2011-03-07T15:00:32.627" UserId="1422" />
  <row Id="12996" PostId="7950" Text="@Dmitrij Great summary, thank you for the insight." CreationDate="2011-03-07T15:06:38.737" UserId="1422" />
  <row Id="12997" PostId="7913" Text="@Peter @chl Thanks for your suggestions. The model in my question is in fact an item response model (a Rasch or 1PL model with fixed discrimination parameter). I think the suggestion to look at individuals with aberrant performance is a good start, but I am looking for an approach that takes advantage of the additional information provided by the correlation in cheaters' responses for items on which there was cheating. You can imagine that if we used your procedure to identify cheaters, for example, they would perform well on similar difficult items." CreationDate="2011-03-07T15:22:12.183" UserId="3432" />
  <row Id="12998" PostId="7933" Score="2" Text="+1, I'll share this wonderful response (and print it to have nearby ^_^)" CreationDate="2011-03-07T15:29:58.733" UserId="2645" />
  <row Id="12999" PostId="7955" Score="3" Text="The *Mathematica* help page has a nice illustration with explanations: http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationNewtonsMethodRoot.html ." CreationDate="2011-03-07T15:31:34.527" UserId="919" />
  <row Id="13000" PostId="7516" Text="I found a great tutorial paper and accompanying spreadsheet: http://www.behan.ws/copula.pdf and  http://www.soa.org/files/xls/rsrch-copula-ex.xls" CreationDate="2011-03-07T15:33:20.310" UserId="2260" />
  <row Id="13001" PostId="7957" Text="okay so you mean like: `fitted(arima(somets,c(2,0,0))`respectively `somets-arimaresult$residuals` . I just wondered WHY this is not part of the output – is it somehow not a good idea to look at these?" CreationDate="2011-03-07T15:34:55.487" UserId="704" />
  <row Id="13002" PostId="7951" Text="@mpiktas Thank you for your valiant effort to rescue this question.  However, the changes were so substantial that significant and potentially important parts of the original question disappeared or were altered beyond recognition.  Therefore I have rolled this back to the original question.  Let's try to make sense of it as it stands." CreationDate="2011-03-07T15:35:48.263" UserId="919" />
  <row Id="13003" PostId="7951" Score="2" Text="@BB01 We can try to fix the English--that shouldn't be a problem--but there are problems with the mathematical expressions in the questions.  When $y = ab/c + x$, obviously $y$ **is** &quot;calculated from&quot; $x$.  In this case the correlation of a set of $(x,y)$ would equal $1$.  In the second question it is mysteriously redundant to write $y=P+Q+x$; you might just as well write $y=P(x)$.  These solecisms suggest you might not have properly expressed the questions that you really want answered.  Please clarify these points or we'll have to close the question as unanswerable." CreationDate="2011-03-07T15:39:35.110" UserId="919" />
  <row Id="13005" PostId="7948" Text="@analytics Your question is well stated.  @chl kindly improved some formatting, that's all.  It involved TeXifying the &quot;R^2&quot; (it was turned into $\$$R^2$\$$, which renders as $R^2$)." CreationDate="2011-03-07T15:47:26.660" UserId="919" />
  <row Id="13006" PostId="7931" Score="1" Text="These are good points.  Outliers and other data problems plague *any* dataset, no matter how large or small.  In my experience they actually are easier to identify and deal with in large datasets, because you have the power to discriminate them from the mass of data and, especially if you use robust methods, they are less likely to influence the results.  BTW, you're *always* doing &quot;data cleaning&quot; throughout any analysis. This isn't something that can be segregated and referred to a specialist to be handled once and for all. *An outlier is only an outlier in the context of a particular model.*" CreationDate="2011-03-07T15:55:24.887" UserId="919" />
  <row Id="13007" PostId="7903" Text="@Henry Here's the paper: http://pricetheory.uchicago.edu/levitt/Papers/JacobLevitt2003.pdf" CreationDate="2011-03-07T16:00:23.623" UserId="3432" />
  <row Id="13008" PostId="7815" Text="Also of interest is the related discussion of hypothesis testing with large sets of data: http://stats.stackexchange.com/q/2516/919" CreationDate="2011-03-07T16:00:55.923" UserId="919" />
  <row Id="13009" PostId="2516" Text="More discussion on large data sets appears at http://stats.stackexchange.com/q/7815/919 .  (The focus is on regression modeling there.)" CreationDate="2011-03-07T16:01:39.590" UserId="919" />
  <row Id="13010" PostId="7959" Text="For some ideas (in the context of estimating medians) see the thread at http://stats.stackexchange.com/q/346/919 ." CreationDate="2011-03-07T16:05:24.513" UserId="919" />
  <row Id="13011" PostId="7775" Text="@robin - I apologize if my question wasn't clear. I am interested in finding a package that contains this calculation (as it probably contains many other helpful functions that would be useful) and/or suggestions on how to improve the function I wrote above. It is obviously lacking any error checking, and may not be applicable for all model types." CreationDate="2011-03-07T16:07:13.997" UserId="696" />
  <row Id="13012" PostId="7903" Text="@lockedoff: Much as I recall.  It also seems to have several references to other papers related to detecting other forms of cheating." CreationDate="2011-03-07T16:38:25.780" UserId="2958" />
  <row Id="13013" PostId="7847" Text="@nico That sounds like a good argument for not using histograms when you want to compare dispersions!" CreationDate="2011-03-07T18:00:44.930" UserId="919" />
  <row Id="13014" PostId="7847" Text="@whuber: others may say it is a good argument in favour of using histograms with the same binning... but I am not going into that discussion, sorry! :P" CreationDate="2011-03-07T18:21:13.980" UserId="582" />
  <row Id="13015" PostId="7964" Text="@David, (+1) Modulo a couple of minor typos, this is nice. Expanding around something close to the mean as you've done often works better. I'm not surprised to see the higher order convergence in light of the asymptotic results. Now, if you show a similar such upper bound, that proves $(T-n\log n)/n$ is *subexponential* in the terminology of Vershynin, which has many implications regarding measure concentration. " CreationDate="2011-03-07T18:47:02.387" UserId="2970" />
  <row Id="13016" PostId="7641" Text="Thank a lot for your answer Henry." CreationDate="2011-03-07T18:49:38.760" UserId="3438" />
  <row Id="13017" PostId="7963" Text="This is an extended work of P2 algorithm. [link] http://sim.sagepub.com/content/49/4/159.abstract. The storage is still too much for my application, which runs on small sensors with a total of 10K RAM. I can consume a few hundred bytes at most for quantile estimation only." CreationDate="2011-03-07T18:50:16.110" UserId="3594" />
  <row Id="13018" PostId="7965" Score="2" Text="&quot;there is just nothing to understand here -- it is just a standard linear regression model&quot; - teehee.... like that's any consolation for mathphobics. I take it you've never had the pleasure of teaching undergraduate courses in stats for, let's say, sociology or, god help me, communications majors. " CreationDate="2011-03-07T18:52:29.843" UserId="1979" />
  <row Id="13019" PostId="7963" Text="@whuber Actually I implement the extended P2 and test it with generated samples from various distributions such as uniform and exponential, where it works great. But when I apply it against data from my application, whose distribution is unknown, sometimes it fails to converge and yields relative error (abs(estimation - actual) / actual) up to 300%." CreationDate="2011-03-07T18:55:17.727" UserId="3594" />
  <row Id="13020" PostId="7963" Score="1" Text="@sino The quality of the algorithm *compared to using all the data* shouldn't depend on the heaviness of the tails.  A fairer way to measure error is this: let $F$ be the empirical cdf.  For an estimate $\hat{q}$ of the $q$ percentile, what is the difference between $F(\hat{q})$ and $F(q)$?  If it is on the order of $1/n$ you're doing awfully well.  In other words, just what percentile is the P2 algorithm returning for your data?" CreationDate="2011-03-07T19:20:11.243" UserId="919" />
  <row Id="13021" PostId="7965" Text="@fabians This only proves my point -- confrontating people with math more complex than counting is the biggest flaw of this approach =] But I'll try to reword it." CreationDate="2011-03-07T19:33:18.507" UserId="88" />
  <row Id="13022" PostId="7946" Text="@BB01 As mpiktas mentioned, we can't help you unless you'd give some details about the data -- are those continuous or discrete data? Are -1 real measurements or NAs?" CreationDate="2011-03-07T19:44:09.853" UserId="88" />
  <row Id="13023" PostId="7968" Text="**NB** Please do not attribute these thoughts to me!  I'm just doing my best to articulate and defend the stated model, as requested.  Whether this model is appropriate, applicable, well-fit, etc., is a separate issue altogether." CreationDate="2011-03-07T19:47:27.747" UserId="919" />
  <row Id="13024" PostId="7969" Text="All you need to do is divide each gamma variate by their sum.  How, then, does underflow occur?  And how does taking the logarithm solve this problem (you can't compute the sum without exponentiating back again anyway)?" CreationDate="2011-03-07T19:54:11.230" UserId="919" />
  <row Id="13025" PostId="7969" Text="@whuber In log space, you compute the sum and then *subtract* it from each element. So, this avoids the first point of underflow. There is a bit of further processing when these dirichlets serve as mixture components and get multiplied by small numbers again." CreationDate="2011-03-07T19:58:28.250" UserId="2067" />
  <row Id="13026" PostId="7969" Text="@luis Adding the logs is mathematically incorrect: it corresponds to *multiplying* the gammas rather than adding them.  Yes, you might get working results, but they definitely will not have a Dirichlet distribution!  Again, what exactly is the nature of the original underflow and what calculations are you doing when it happens?  What are the actual values you are working with?" CreationDate="2011-03-07T20:02:15.217" UserId="919" />
  <row Id="13027" PostId="7969" Text="@whuber I might have simplified a bit too much in my description. I do forall i { t = gamma(a,b); sum += t; d[i] = log(t)}; logsum = log(sum); forall i { d[i] -= logsum; }. Previously, this underflowed if a was very small." CreationDate="2011-03-07T20:05:47.823" UserId="2067" />
  <row Id="13028" PostId="7877" Text="@steffen : Thanks for your response. Well I am afraid I don't get it still. How are the boundaries changing in the figure? What is it about the misclassified examples that causes the classifier to become smart, and do better next time? I mean let's say that there's an example that I don't know the correct class of - how would seeing that example again in later iterations help me put it in the right class? I can sense that there is something obvious here that I am not seeing - love that feeling :-)  " CreationDate="2011-03-07T20:48:17.897" UserId="3301" />
  <row Id="13029" PostId="7969" Text="@luispedro Got it: for $\alpha$ near 0 you're going to be in trouble no matter what.  Interesting problem!" CreationDate="2011-03-07T21:15:16.187" UserId="919" />
  <row Id="13030" PostId="7965" Text="@mbq This is valid criticism--especially the part about assuming linearity--but it doesn't really respond to the original question (unless your intention is to offend the hypothetical &quot;layman&quot;)." CreationDate="2011-03-07T21:21:14.493" UserId="919" />
  <row Id="13031" PostId="7972" Score="2" Text="@winchester I have the feeling that analysis of variance would be more appropriate. " CreationDate="2011-03-07T21:47:27.173" UserId="223" />
  <row Id="13032" PostId="7966" Text="+1 I think this is a great idea given the OP's constraints.  All one can hope for is an approximation, so the trick is to pick bins that have a high likelihood of being narrow and containing the desired quantile." CreationDate="2011-03-07T21:51:31.617" UserId="919" />
  <row Id="13033" PostId="7967" Text="@SheldonCooper +1 for this clear and well-balanced response." CreationDate="2011-03-07T21:57:42.710" UserId="930" />
  <row Id="13034" PostId="7968" Text="@whuber (+1) The last paragraph is very well put." CreationDate="2011-03-07T21:58:33.783" UserId="930" />
  <row Id="13035" PostId="7967" Text="I'd perhaps change the second sentence of the last paragraph to say, &quot;Even though 65 of her 66 students scored 'proficient' on the state test, they most likely would have scored the same even if they had an inept teacher.&quot;" CreationDate="2011-03-07T22:01:03.787" UserId="1764" />
  <row Id="13036" PostId="7847" Text="@nico @Ralph Maybe this has to do with a distinction between local and global information (contour) processing, but honestly I cannot really &quot;see&quot; such a difference in terms of apparent dispersion. It seems to me (but this has already been said) that binning (or window span in kernel density estimate) helps to spot multiple mode if any, not the overall shape." CreationDate="2011-03-07T22:18:13.713" UserId="930" />
  <row Id="13037" PostId="7952" Score="1" Text="@fRed xtable has already been rewritten (or perhaps pre-written) as the [latex](http://lib.stat.cmu.edu/S/Harrell/help/Hmisc/html/latex.html) function in the R package Hmisc.  Personally I swear by it for proper looking tables.  I haven't checked whether it has room for the fancy insertion trick described below, though I'd be surprised if not." CreationDate="2011-03-07T23:27:54.480" UserId="1739" />
  <row Id="13038" PostId="7980" Score="1" Text="@Berk: can any of these projections still preserve the uniformity of randomness? Again, how can I check whether the final distribution of these points are truly uniformly distributed on the surface of the sphere? Thanks." CreationDate="2011-03-07T23:35:59.223" UserId="3552" />
  <row Id="13039" PostId="7980" Text="Sorry I was just speaking hypothetically... I think the mapping functions on MATLAB would allow you to check that since they have some visualizations embedded in them. If not, I also found a nice website that talks about how to generate uniformly distributed points on a sphere in 3D using things like randomized angles etc. They have some C code on there too. [Take a look](http://paulbourke.net/geometry/spherepoints/) " CreationDate="2011-03-08T00:01:17.960" UserId="3572" />
  <row Id="13040" PostId="7816" Text="Could you explain your 2nd point? How would you use CHAID/CART/neural networks as screening tools for regression?" CreationDate="2011-03-08T00:09:19.343" UserId="1106" />
  <row Id="13041" PostId="7981" Text="@Hong, and how are copulas supposed to help if you only have the marginal distributions?" CreationDate="2011-03-08T00:19:24.700" UserId="2970" />
  <row Id="13042" PostId="7981" Score="1" Text="@cardinal, the point is that copulas are a way of specifying the additional information you need to obtain the bivariate distribution, if you already have the marginals." CreationDate="2011-03-08T00:23:28.277" UserId="1569" />
  <row Id="13043" PostId="7983" Text="@emakalic: is it possible to post the links where I can find the full text of these references? thanks." CreationDate="2011-03-08T00:30:43.960" UserId="3552" />
  <row Id="13044" PostId="7981" Text="Hi Hong. For copulas, our marginal should be uniform. However, in the question that I have asked the marginals are standard normal. Can copula still come into picture in that case?" CreationDate="2011-03-08T00:35:59.913" UserId="1102" />
  <row Id="13045" PostId="7981" Score="2" Text="@Harpreet If you make it to section 3 of @Hong's Wikipedia reference you will see that Sklar's Theorem answers your question affirmatively (http://en.wikipedia.org/wiki/Copula_%28statistics%29#Sklar.27s_theorem )." CreationDate="2011-03-08T00:41:45.693" UserId="919" />
  <row Id="13046" PostId="7922" Text="after applying weighted penalty for each class it works faster. Going through all the mathematical intricacies of SVM now. Thanks for help" CreationDate="2011-03-08T01:12:29.617" UserId="1371" />
  <row Id="13047" PostId="7984" Text="@whuber: very nice! thanks a lot for your post! &quot;$(X_1/\lambda, X_2/\lambda, X_3/\lambda)$ is uniformly distributed on the sphere.&quot; where can I find reference about its proof, or is it simply provable? " CreationDate="2011-03-08T01:12:52.643" UserId="3552" />
  <row Id="13048" PostId="7984" Text="@whuber what software did you use to generate the sphere plot?" CreationDate="2011-03-08T01:18:44.680" UserId="3572" />
  <row Id="13049" PostId="7984" Text="I bet it is mathematica... just tell from the color... :)" CreationDate="2011-03-08T01:26:36.533" UserId="3552" />
  <row Id="13050" PostId="7957" Score="3" Text="The fitted values from an ARIMA object are one-step forecasts. They are not very interesting and you don't usually need to look at them. The residuals, on the other hand, are useful for checking model assumptions and to see whether the information in the data has been adequately captured in the fitted model." CreationDate="2011-03-08T01:28:56.827" UserId="159" />
  <row Id="13051" PostId="7988" Text="@Henry, I am just wondering if your method of generating points (x,y,z) is essentially the same as whuber's method?" CreationDate="2011-03-08T01:33:20.350" UserId="3552" />
  <row Id="13052" PostId="7987" Score="1" Text="@Or Zuk, conceivably, any correction could end up having mean zero for this case." CreationDate="2011-03-08T01:43:58.647" UserId="2970" />
  <row Id="13053" PostId="7988" Text="No it isn't: whuber uses three random numbers while I use two.  Mine is a special case of &quot;generate a point on $[-1,1]$ with suitable density [proportional to $(1-z^2)^{n/2-1}$] and then step down a dimension&quot;. Here conveniently $n=2$ as this is formally a [2-sphere](http://en.wikipedia.org/wiki/2-sphere).      " CreationDate="2011-03-08T01:46:25.490" UserId="2958" />
  <row Id="13054" PostId="7937" Text="@dsimcha there really isn't a way to properly correct for multiple testing in stepwise.  See, e.g. Harrell Regression modeling strategies.  There's no way to know the right correction" CreationDate="2011-03-08T01:53:21.067" UserId="686" />
  <row Id="13055" PostId="7984" Score="2" Text="@Qiang, Here's the essence of the proof: Let $X \sim \mathcal{N}(0, I_n)$ where $I_n$ denotes the $n \times n$ identity matrix. Then for *any* orthogonal matrix $Q$, $Q X \sim N(0, I_n)$. Hence the distribution of $X$ is invariant under rotations. Let $Y = X / \|X\|_2$ and note that $Y_Q = Q X / \|Q X\|_2 = Q X / \|X\|_2$ for any orthogonal $Q$. Since $X$ is invariant to rotations, so is $Y$, and since $\|Y\|_2 = 1$ almost surely, then it must be uniformly distributed on the sphere." CreationDate="2011-03-08T01:53:59.820" UserId="2970" />
  <row Id="13056" PostId="7984" Text="@whuber, (+1), nice answer." CreationDate="2011-03-08T02:00:29.580" UserId="2970" />
  <row Id="13057" PostId="7981" Score="2" Text="@Hong, That's fine, I know what a copula is. My point is that they don't help here. If all you have is the marginal distributions and, not, say, the $(X,Y)$ pairs, all a copula will do is let you make up whatever dependence you'd like. They have no utility in terms of inference because you have no information about the joint distribution." CreationDate="2011-03-08T02:08:35.607" UserId="2970" />
  <row Id="13058" PostId="7988" Text="@Henry Or, more generally, generate uniform points on the map using any equal-area projection (yours is a cylindrical equal-area one), then project back. (+1)" CreationDate="2011-03-08T02:14:19.777" UserId="919" />
  <row Id="13059" PostId="7984" Text="@cardinal Thank you for the demonstration." CreationDate="2011-03-08T02:16:41.227" UserId="919" />
  <row Id="13060" PostId="7984" Text="@Berk The *Mathematica* code is **With[{n = 100}, &#xA;  x = #/Norm[#] &amp; /@ RandomReal[NormalDistribution[0, 1], {n, 3}]];&#xA;Graphics3D[{&#xA;  Opacity[0.75], Sphere[{0, 0, 0}, 1],&#xA;  Opacity[1], PointSize[0.025], Point /@ x&#xA;  }]**" CreationDate="2011-03-08T02:16:59.307" UserId="919" />
  <row Id="13061" PostId="7963" Text="You are right. I just measured the F(qˆ) and F(q) for the case I mentioned with relative error up to 300%. For q of 0.7, qˆ is almost 0.7, resulting in negligible error. However, for q of 0.9, qˆ seems to be around 0.95. I guess that's why I have huge error of up to 300%. Any idea why it's 0.95, not 0.9? BTW, can I post figure here and how can I post mathematical formula as you did?" CreationDate="2011-03-08T02:35:11.547" UserId="3594" />
  <row Id="13062" PostId="7990" Text="@Douglas This is one of those rare cases where a little more formality would be welcome.  Probability statements can be notoriously ambiguous and this one invites ambiguity on many levels.  Perhaps a small example would clarify.  Also, please reread your question: some words are obviously missing." CreationDate="2011-03-08T02:35:36.177" UserId="919" />
  <row Id="13063" PostId="7963" Text="@sino 0.9 --&gt; 0.95 is a shockingly large error.  If you're *absolutely* confident the algorithm is implemented without bugs, we would have to suspect that the parabolic interpolation is doing strange things.  One option might be to code a version in which linear interpolation is used just to see what effect that might have." CreationDate="2011-03-08T02:37:48.313" UserId="919" />
  <row Id="13064" PostId="7963" Text="@sino Formulas are in a significant subset of TeX, enclosed in $\$$.  E.g., &quot;$F(\hat{q})$&quot; is written as &quot;$\$$F(\hat{q})$\$$&quot;.  Figures can be linked to and/or uploaded with a tool available when editing your post, but there may be a (small) minimum reputation requirement to do that.  I've done my part in that regard: I upvoted your question!" CreationDate="2011-03-08T02:39:03.163" UserId="919" />
  <row Id="13065" PostId="7963" Text="I'll double check the implementation and get back to you soon." CreationDate="2011-03-08T02:44:16.837" UserId="3594" />
  <row Id="13066" PostId="7925" Text="@Berk U, yes, if you randomly scramble two separate $(t,m,s)$ nets independently of one another, then the resulting sets will be independent. As for a deterministic scrambling algorithm, without any notion of randomness, there really cannot be a proper notion of correlation. I'd have to think about taking even and odd entries. The standard approach would be to get some points for the first one, then generate and throw away a bunch more points, then collect your second set of points. This is related to using &quot;burned in&quot; sets of QMC points." CreationDate="2011-03-08T02:48:59.410" UserId="2970" />
  <row Id="13067" PostId="7990" Text="Is there any motivation behind why the process is the way is is?" CreationDate="2011-03-08T02:55:13.127" UserId="2965" />
  <row Id="13068" PostId="7954" Text="@robin girard: Thank you for your detailed answer. I will have a look at your links. fRed (Mines2006)" CreationDate="2011-03-08T03:12:46.143" UserId="1709" />
  <row Id="13069" PostId="7964" Text="The argument doesn't seem to generalize directly to the upper bound. Exchanging $c$ for $-c$ (and $s$ for $-s$), one can follow the same steps up to the point of calculating&#xA;$E[e^{sS}] \leq \prod_i \frac{e^{-s/p_i}}{1 - \frac{s}{p_i}}$. At this point, however, the best I can do is to use&#xA;$\frac{e^{-z}}{1-z} \leq \exp( \frac{z^2}{2(1-z)} )$, which still leaves&#xA;$$ E[e^{sS}] \leq e^{\frac{1}{2} s^2\sum_i \frac{p_i^2}{(1-s/p_i)}}$$&#xA;and I don't know what to do with this" CreationDate="2011-03-08T03:27:32.447" UserId="3500" />
  <row Id="13070" PostId="7964" Text="Interestingly enough, though, the entire argument (for the lower bound) seems to work not only for the coupon collector problem, but for _any_ sum of non-identical, independent geometric variables with bounded variance. Specifically: given $T = \sum_i T_i$, where each $T_i$ is an independent GV with success probability $p_i$, and where $\sum_i p_i^{-2} \leq A &lt; \infty$, then &#xA;$$ \Pr ( T \leq E[T] - a ) \leq e^{-\frac{a^2}{2A}}$$" CreationDate="2011-03-08T03:35:56.713" UserId="3500" />
  <row Id="13071" PostId="7945" Text="@whuber, good comments and suggestions. If I'm reading the question correctly, I believe one primary concern is uncertainty of the time-point of sampling. This may be a little different from the typical geostatistics framework where, I believe, the spacing is irregular, but still assumed known (at least to high precision). I think a rough model is, if the $n$th point of series one is at time $t_n = n t$, for fixed $t$ then the $n$th point of series 2 is at $\tau_n = t_n + \alpha + \beta t$ where $\alpha$ is probably on the order of a few milliseconds and $\beta$ is likely tiny." CreationDate="2011-03-08T03:40:43.493" UserId="2970" />
  <row Id="13072" PostId="7990" Text="@whuber and rm999: I edited a response into the question." CreationDate="2011-03-08T03:57:08.460" UserId="386" />
  <row Id="13073" PostId="7945" Text="**Typo**: That last $t$ should be an $n$." CreationDate="2011-03-08T04:00:44.373" UserId="2970" />
  <row Id="13074" PostId="7985" Text="Wow, I never really appreciated how &quot;smart&quot; the `[.data.table` function can get ... I guess I didn't realize you didn't create an `.SD` object if you didn't really need it. Nice one!" CreationDate="2011-03-08T04:18:50.483" UserId="227" />
  <row Id="13075" PostId="7963" Score="1" Text="@whuber I'm pretty confident that my implementation conforms to extended P2. 0.9 still goes to 0.95 or even larger when I simultaneously estimate 0.8, 0.85, 0.9, 0.95 quantiles. However, 0.9 goes very close to 0.9 if 0.8, 0.85, 0.9, 0.95 and **1.0** quantiles are tracked at the same time." CreationDate="2011-03-08T04:29:25.897" UserId="3594" />
  <row Id="13076" PostId="7945" Text="@Cardinal I didn't get that from the question.  I can't think of a way of estimating $\beta$ that wouldn't be quite computationally intensive.  Perhaps by breaking the time series into groups where the net effect of $\beta$ is negligible?" CreationDate="2011-03-08T05:08:24.817" UserId="919" />
  <row Id="13077" PostId="7945" Text="@whuber, @BobC, I'm making a semi-educated guess based on past experience dealing with similar problems and issues. Most approaches I've seen are computationally intensive and fail to impress. One attempt might be via something like [dynamic time warping](http://en.wikipedia.org/wiki/Dynamic_time_warping) or what Ramsay and Silverman call [curve registration](http://books.google.ca/books?id=mU3dop5wY_4C&amp;pg=PA127#v=onepage&amp;q&amp;f=false). Whether either of those would be feasible on this size dataset is unclear to me." CreationDate="2011-03-08T05:17:00.583" UserId="2970" />
  <row Id="13078" PostId="7959" Score="1" Text="This question is **[crossposted](http://math.stackexchange.com/questions/25546/algorithm-to-dynamically-monitor-quantiles)** on math.SE." CreationDate="2011-03-08T05:25:38.077" UserId="2970" />
  <row Id="13079" PostId="7983" Text="I dont have the paper on me, but this page seems to describe the algorithm (and several others) http://www.mlahanas.de/Math/nsphere.htm" CreationDate="2011-03-08T05:38:05.123" UserId="530" />
  <row Id="13080" PostId="7987" Text="Just to clarify, though, I'm not claiming I believe there is an unbiased estimator. In fact, I agree there likely *isn't*." CreationDate="2011-03-08T05:41:25.217" UserId="2970" />
  <row Id="13081" PostId="7928" Text="@whuber Thanx. I would like to try concave hull algorithm." CreationDate="2011-03-08T06:14:49.700" UserId="1261" />
  <row Id="13082" PostId="7957" Text="@Rob @ran2, I agree with Rob's comment! " CreationDate="2011-03-08T06:21:21.377" UserId="2645" />
  <row Id="13083" PostId="7928" Text="@curious PostGIS, Python, and R solutions are referenced at http://gis.stackexchange.com/q/1200/664 .  (Ignore references to &quot;convex hull&quot; algorithms and focus on &quot;concave hull&quot; and &quot;alpha shapes&quot;.)" CreationDate="2011-03-08T06:22:11.963" UserId="919" />
  <row Id="13084" PostId="7980" Score="1" Text="@Berk Uniform random points on a gnomonic projection will not be uniform on the sphere, because the gnomonic is not equal area.  The projection proposed by Henry, $(\lambda, \phi)$ --&gt; $(\lambda, \sin(\phi))$ (from longitude-latitude to a rectangle in $\mathbb{R}^2$), *is* equal-area." CreationDate="2011-03-08T06:30:25.870" UserId="919" />
  <row Id="13085" PostId="7739" Text="@cardinal - one would not be required to &quot;re-estimate&quot; the numerical integral.  You simply ignore the posterior samples for those parts of the model which are being excluded.  You only need 1 simulation from the full model, and no rank 1 updates are required - this is where I think you will save a lot of time.  One such question could be &quot;is $\beta_j$ relevant to my model, *regardless of what other parameters are in the model?* &quot;.  Very quick to decide this - just look at the marginal simulated distribution for $\beta_j$." CreationDate="2011-03-08T06:51:57.000" UserId="2392" />
  <row Id="13086" PostId="7201" Text="Hmm...In his EDA text, John Tukey specifically defined outliers without using any models at all." CreationDate="2011-03-08T06:59:38.250" UserId="919" />
  <row Id="13087" PostId="7997" Text="Are you claiming outliers are unimportant in Bayesian data analysis?" CreationDate="2011-03-08T07:00:23.227" UserId="919" />
  <row Id="13088" PostId="7155" Text="@dsimcha To date you have posted 7 questions and received 27 answers but you haven't indicated *a single one* as being acceptable.  What's the matter?  Has nobody been able to help you?" CreationDate="2011-03-08T07:02:45.533" UserId="919" />
  <row Id="13089" PostId="7723" Text="@whuber - I would suggest that &quot;vague&quot; questions can be given either &quot;vague&quot; answers, or they can specialise to a particular case - noting how it is linked to the &quot;vaguary&quot;.  But as you say, one cannot be expected to give the answer to everything.  You could specialise the &quot;how do I start?&quot; question by giving a specific example, and saying how you would start in that particular case.  I had a go in my answer with the notion of &quot;standardising&quot; a distribution.  Presumably there are other ways to &quot;start&quot;." CreationDate="2011-03-08T07:09:16.063" UserId="2392" />
  <row Id="13090" PostId="7988" Text="@whuber: Indeed.  Offtopic, but for anyone interested I have an interactive selection of world map projections [here](http://www.btinternet.com/~se16/js/mapproj.htm), some of which are equal area" CreationDate="2011-03-08T07:10:47.800" UserId="2958" />
  <row Id="13091" PostId="7954" Text="@fRed Mines2006 ! great :) where are you now ? " CreationDate="2011-03-08T07:12:03.010" UserId="223" />
  <row Id="13092" PostId="7723" Score="1" Text="@probabiliy All the more power to you!  The vaguer the question, though, the greater the likelihood that all your good work will be of no interest to the OP because you didn't read their mind correctly ;-).  That can be frustrating..." CreationDate="2011-03-08T07:12:54.350" UserId="919" />
  <row Id="13093" PostId="7877" Text="@Andy: Updated my answer accordingly. Of course you have to know the correct class to estimate the missclassified points. In every iteration, you use only a sample of the training data for actual training and the rest to estimate the quality/missclassifications." CreationDate="2011-03-08T07:47:19.923" UserId="264" />
  <row Id="13094" PostId="7982" Score="1" Text="Some questions: How many datapoints do you have (I smell a prototypical overfitting problem) ? What do you mean when you say &quot;the features in this subset are pretty much random&quot; ?" CreationDate="2011-03-08T07:53:38.713" UserId="264" />
  <row Id="13096" PostId="8001" Text="@user3614 - the &quot;jerk&quot; response is to do away with p-values, and use a Bayesian answer.  Bayes theorem in hypothesis testing automatically does the appropriate corrections you require.  Multivariate uniform prior on the effects is the way to go, simulate/integrate (whichever is easier) the effects, then calculate the proportion which fall into the rejection region.  Use this as your p-value" CreationDate="2011-03-08T08:00:17.157" UserId="2392" />
  <row Id="13097" PostId="7739" Text="@cardinal - adding to my above point, suppose you have a &quot;rejection region&quot; e.g. $\frac{|\beta_j|}{SE(\beta_j)} \leq 1$ which you are prepared to declare that the coefficient is &quot;zero&quot; and remove it from the model.  Then the $2^n$ all subsets regression on the simulated dataset reduces to the problem of an n-way contingency table, each way has 2 outcomes - in the region or not.  The &quot;best models&quot; have the highest probability in this table" CreationDate="2011-03-08T08:06:46.153" UserId="2392" />
  <row Id="13098" PostId="7957" Text="gracias...........!       " CreationDate="2011-03-08T08:38:47.180" UserId="704" />
  <row Id="13099" PostId="7879" Text="@Imran: No need for a new question, I will edit my answer and add some code (later this day)" CreationDate="2011-03-08T08:39:57.487" UserId="264" />
  <row Id="13100" PostId="7989" Text="@Qiang Li, just to clarify, since you put regression tag, are you asking how to chose between multiplicative or additive error models? If your question is from signal theory please change the tag." CreationDate="2011-03-08T08:43:24.070" UserId="2116" />
  <row Id="13101" PostId="7996" Text="@Legend, can you really fix velocity and number of vehicles and vary the area? It does not look like it is possible to do controlled experiment in your case, can you please clarify?" CreationDate="2011-03-08T08:47:37.713" UserId="2116" />
  <row Id="13102" PostId="7999" Text="@suncoolsu if efficiency is the only problem, I would consider Rcpp..." CreationDate="2011-03-08T08:56:47.220" UserId="1443" />
  <row Id="13103" PostId="7999" Text="@teucer. I agree, I am a very happy and statisfied Rcpp user. But I was looking for an R-ish way of doing this. It would be good to see the magnitude of speed-up Rcpp provides." CreationDate="2011-03-08T09:05:26.347" UserId="1307" />
  <row Id="13104" PostId="8002" Text="@mpiktas (+1) Sounds good." CreationDate="2011-03-08T09:09:05.377" UserId="930" />
  <row Id="13105" PostId="7999" Text="@suncoolsu maybe than you can store your values in a 3d array and use `rowMeans` (which is implemented in `C` I believe, so should be fast)" CreationDate="2011-03-08T09:12:14.583" UserId="1443" />
  <row Id="13106" PostId="7945" Text="It'll take me a bit to get my brain wrapped around this.  I'll start with the examples in the R packages you mentioned." CreationDate="2011-03-08T09:21:51.767" UserId="3479" />
  <row Id="13107" PostId="7981" Text="@Harpreet, even if the marginals are normal, the joint distribution can be non-normal. You can use copulas, but as @cardinal said, you need additional information." CreationDate="2011-03-08T09:38:31.507" UserId="2116" />
  <row Id="13108" PostId="7879" Text="@Steffen, so nice of you. I would be waiting." CreationDate="2011-03-08T09:40:43.043" UserId="3535" />
  <row Id="13109" PostId="8004" Text="Thanks for your help. I am pretty sure that my series is non-stationary. If I transform or detrend it, i might be able to get something stationary. But then I do not understand how to explain a trend when it's not there anymore. " CreationDate="2011-03-08T09:43:00.320" UserId="704" />
  <row Id="13110" PostId="7967" Text="@SheldonCooper +1, good overview. " CreationDate="2011-03-08T09:45:49.177" UserId="2116" />
  <row Id="13111" PostId="7968" Text="@whuber +1, the last paragraph is golden." CreationDate="2011-03-08T09:51:07.867" UserId="2116" />
  <row Id="13112" PostId="8004" Text="@ran2, if the trend is time trend or stochastic trend, then there is nothing to explain. The definite answer depends on the problem you are trying to solve. Can you describe it?" CreationDate="2011-03-08T09:54:51.453" UserId="2116" />
  <row Id="13113" PostId="8004" Text="I'll try :). Basically I would like to see to what extent my variable of interest A can be explained by a variable that has a time trend for sure (like GDP, or electricity consumption). A itself does not seem to follow a time trend (a slight downwards trend if any). Still if I use `stl()` the trend accounts for most of the music here and looks more or less like a rollercoaster. Does that help to help? " CreationDate="2011-03-08T10:05:42.627" UserId="704" />
  <row Id="13114" PostId="8004" Text="@ran2, in your case my answer applies, think of reasonable variables which can explain your variable of interest, check for stationarity and if it holds use regression analysis. If you want to include GDP though, you maybe forced to use co-integration analysis, i.e. VECM." CreationDate="2011-03-08T10:12:43.803" UserId="2116" />
  <row Id="13115" PostId="8004" Text="Thanks for your patience. Still GDP could be a possible explanatory for my variable. Probably I better use growth rates because otherwise it just represents a time trend here. The reason why I want to use a regression is because I am interested in extracting what's actually NOT explained by time trend variables like GDP." CreationDate="2011-03-08T10:18:19.693" UserId="704" />
  <row Id="13116" PostId="7952" Text="@Conjugate You're right, although it is not as straightforward as I would have thought; I've updated my response with your suggestion." CreationDate="2011-03-08T10:26:02.367" UserId="930" />
  <row Id="13117" PostId="7878" Score="1" Text="@Imran: I actually should have said that you have to reset both conversions AND views. Every time you update the weights you are trying to alter the conversion/views to a more optimal level. Then you run the program for a while to test how effective your new weighting is. This is tested by comparing conversions per view **under the new weighting scheme.** For that reason, if you don't reset conversions and views, you will be factoring in data that occurred under previous weighting schemes, which is not relevant for testing the new weighting scheme. " CreationDate="2011-03-08T10:26:17.783" UserId="3443" />
  <row Id="13118" PostId="8004" Text="@ran2, it is always best to use GDP growth instead of its real value. Note that the regression analysis can also tell you what variables **do not** explain the trend, so you might end up with the result that there are no variables which can explain your trend (or the variables you thought about do not explain the trend)." CreationDate="2011-03-08T10:26:44.393" UserId="2116" />
  <row Id="13119" PostId="7878" Score="1" Text="@Imran as you have seen, you will still get weight changes if you fail to reset conversion and views back to 0 after each weight change, and it may even converge towards the optimal weight given enough time, but it will not converge as quickly or directly." CreationDate="2011-03-08T10:37:05.587" UserId="3443" />
  <row Id="13120" PostId="7878" Text="@Matt OK, I got your point now. For the matter of accuracy, I should be resetting the conversions/views on every weight change. It'll work without resetting, though. Right?" CreationDate="2011-03-08T11:02:25.250" UserId="3535" />
  <row Id="13121" PostId="8009" Text="@Nir I have made some edits to your question. Please, could you check that I didn't alter its original meaning?" CreationDate="2011-03-08T11:34:37.730" UserId="930" />
  <row Id="13122" PostId="7996" Text="@mpiktas It seems it is a computer simulation of traffic, so anything is possible." CreationDate="2011-03-08T12:00:34.087" UserId="88" />
  <row Id="13123" PostId="8009" Text="thanks @chi. you got it right" CreationDate="2011-03-08T12:02:23.667" UserId="3506" />
  <row Id="13124" PostId="7752" Text="Thanks for your explanation, I think I now understand, although I had to reject this anyway as it doesn't fit my needs." CreationDate="2011-03-08T12:06:06.627" UserId="3104" />
  <row Id="13125" PostId="7878" Score="1" Text="@Imran to be honest, I can't really say for sure. I think so but I definitely would avoid doing it that way. If your worried about keeping track of how many total conversions you've made, you could always have a total variable just for keeping track, and a separate variable for conversions since the last weight change." CreationDate="2011-03-08T12:17:36.733" UserId="3443" />
  <row Id="13126" PostId="7878" Text="@Matt, great! I was thinking the same, separate variables for tracking and calculating weights. Thanks a lot." CreationDate="2011-03-08T12:34:01.003" UserId="3535" />
  <row Id="13127" PostId="8012" Text="@Askan, are you trying to compute $Var(\bar \tau)$? Do you want explicit expression, i.e. formula, or the algorithm for numerical computation? What are the properties of process $t_{k,i}$?" CreationDate="2011-03-08T12:39:24.697" UserId="2116" />
  <row Id="13128" PostId="7952" Text="@chl 'not straightforward' is an understatement.  That function is a beast, although a very capable one." CreationDate="2011-03-08T12:44:07.717" UserId="1739" />
  <row Id="13129" PostId="8013" Score="3" Text="If you were looking for an R package, `arules` would be worth a look. Maybe &quot;association rules&quot; is a good search term" CreationDate="2011-03-08T13:06:18.810" UserId="573" />
  <row Id="13130" PostId="7933" Text="+1, this is what I'll certainly retell my students many years to come." CreationDate="2011-03-08T13:40:09.770" UserId="2116" />
  <row Id="13131" PostId="8017" Text="adding to my answer, this probably explains why &quot;non-parametric&quot; regression, which consists of simple linear regression on &quot;knots&quot; of the data work so well - the tiny details of the noise matter very little over a &quot;small&quot; region about the regression line." CreationDate="2011-03-08T13:48:07.677" UserId="2392" />
  <row Id="13132" PostId="8017" Text="+1 for the night club and demonstration that multiplicative noise can be approximated by additive." CreationDate="2011-03-08T13:54:16.490" UserId="2116" />
  <row Id="13133" PostId="7932" Score="1" Text="Sometimes it is better to write &quot;Inefficient code&quot; if this will aid in creating data structures which anticipate additional questions down the road which will probably be asked." CreationDate="2011-03-08T14:03:35.527" UserId="3489" />
  <row Id="13134" PostId="7933" Text="@whuber, the anecdote reminded me the time when I had to transfer model from Eviews to R. The original model was done in Eviews, the result was about 20 equations. I had to present the results in the webpage with interactive interface. Since the model was work in progress, I wrote a code translating output of Eviews to R code with the same purpose that the exact model was used both in Eviews and in R. R worked very nicely, I even ended up using differencing the translated code for calculation of analytical gradient." CreationDate="2011-03-08T14:29:55.790" UserId="2116" />
  <row Id="13135" PostId="8018" Text="+1, nicely done!" CreationDate="2011-03-08T15:03:08.867" UserId="26" />
  <row Id="13137" PostId="8017" Text="@probabilityislogic: I like your answer very much. +1" CreationDate="2011-03-08T15:36:53.927" UserId="3552" />
  <row Id="13138" PostId="7953" Text="@chl thanks for your answer and the edit." CreationDate="2011-03-08T15:54:41.250" UserId="1709" />
  <row Id="13140" PostId="8021" Score="2" Text="The techniques in the previous question apply directly once you observe that the number of points within distance $r$ of the origin must be proportional to $r^3$.  Thus if you generate an independent uniform variate $u$ in $[0,1]$ along with a point $w$ on the surface of the sphere, scaling $w$ by $u^{1/3}$ does the trick." CreationDate="2011-03-08T16:04:33.050" UserId="919" />
  <row Id="13141" PostId="7987" Text="Yes agree - this is just intuition. The following paper gives conditions for the existence of unbiased estimator for a function of a univariate gaussian mean - maybe can be extended to multivariate: &#xA;http://www.stat.ncsu.edu/library/mimeo.archive/ISMS_1988_1929.pdf" CreationDate="2011-03-08T16:17:31.933" UserId="3036" />
  <row Id="13142" PostId="7932" Text="@Ralph:  +1, I absolutely agree and learned this the hard way myself.  I didn't mean to apply that you should always write efficient code no matter what the tradeoffs, just that you should know how to." CreationDate="2011-03-08T16:27:45.797" UserId="1347" />
  <row Id="13143" PostId="7985" Text="@Matthew Yes, that is indeed fast! Even if you include `dxt &lt;- data.table(dx, key='ID')` in the call to system.time(), it's faster than @Matt's solution." CreationDate="2011-03-08T16:32:02.473" UserId="3432" />
  <row Id="13144" PostId="4722" Text="@Mike I know this is an old thread, but if you're still considering that idea, take a look at this paper which shows how to create confidence bands that give X% *simultaneous* (and not just pointwise) coverage for things like quantile-quantile curves, differences in ECDFs, etc. http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-sim.pdf" CreationDate="2011-03-08T16:45:09.350" UserId="3432" />
  <row Id="13145" PostId="8002" Text="@mpiktas (+1) I did not know conversion to data.frame is costly. This was a toy example, but my data set has 1e4 rows. Thanks." CreationDate="2011-03-08T16:46:58.190" UserId="1307" />
  <row Id="13146" PostId="8013" Score="2" Text="See also the [Apriori algorithm](http://en.wikipedia.org/wiki/Apriori_algorithm) for the &quot;standard&quot; approach to this problem. " CreationDate="2011-03-08T16:53:15.873" UserId="2970" />
  <row Id="13147" PostId="8002" Score="1" Text="@suncoolsu, I do not know how exactly costly it is. I added the comment purely on the ground that any unnecessary operation is costly. On the other hand data.frame is special type in R, so there surely is additional cost of converting to it, not withstanding that passing the object to the function is effectively creating additional copy, so if speed is required this  should be preferably avoided." CreationDate="2011-03-08T17:01:38.953" UserId="2116" />
  <row Id="13148" PostId="8021" Text="@whuber: maybe I just did not get the essence of the previous techniques. Let me try what you described. Additionally, what are the ways to check the uniformity here again?" CreationDate="2011-03-08T17:10:20.297" UserId="3552" />
  <row Id="13149" PostId="8007" Score="1" Text="Is plotting it a requirement? There are pretty good ways of estimating the factors without plotting." CreationDate="2011-03-08T17:17:15.610" UserId="3369" />
  <row Id="13150" PostId="8021" Text="@Qiang Ripley's K function and chi-squared tests.  You could also check the uniformity of the radial projection of the points on the sphere's surface, the uniformity of the cube of the lengths of the points, and the independence of those two." CreationDate="2011-03-08T17:18:05.187" UserId="919" />
  <row Id="13151" PostId="7996" Text="@mpiktas: Actually @mbq is right. I am doing some computer simulations of the scenarios." CreationDate="2011-03-08T17:20:15.280" UserId="2164" />
  <row Id="13152" PostId="8023" Text="To see why averaging individual $R$ values is wrong, suppose the direction of the slope is reversed in some of the experimental conditions.  You would average a bunch of 1's and -1's out to around 0, which wouldn't reflect the quality of *any* of the fits.  To see why averaging $R^2$ (or any fixed transformation thereof) is not right, suppose that in most experimental conditions you had only two observations, so that their $R^2$ all equal $\pm 1$, but in one experiment you had a hundred observations with $R^2=0$.  The average $R^2$ of almost 1 would not correctly reflect the situation." CreationDate="2011-03-08T17:23:42.840" UserId="919" />
  <row Id="13153" PostId="7877" Text="@steffen : Thanks for explaining. The fact that not all data is used in every iteration for training was not obvious to me before. Let me chew on this for a bit, and I will get back. " CreationDate="2011-03-08T17:46:40.733" UserId="3301" />
  <row Id="13154" PostId="7945" Text="@BobC, is the rough model I gave for the timing asynchronicity close to what you have? I'm thinking it's &quot;random initial offset&quot; + &quot;linear error&quot; where the latter is due to a small constant difference in the sampling interval between your two devices. Then there is some additional small random error, say due to interrupt processing of two different uC's on top of that. " CreationDate="2011-03-08T17:53:18.873" UserId="2970" />
  <row Id="13155" PostId="8001" Text="@probabilityislogic, best not to confuse others (or yourself) with proselytizing. " CreationDate="2011-03-08T17:59:13.797" UserId="2970" />
  <row Id="13156" PostId="8001" Text="@user3614, how many genes? How many subjects? Bonferroni is usually extremely conservative in this setting. " CreationDate="2011-03-08T18:02:15.493" UserId="2970" />
  <row Id="13158" PostId="7897" Text="Probably I will have on the order of 100 features.  But I have only a few minutes to make a decison on the best feature subset." CreationDate="2011-03-08T18:34:22.567" UserId="3550" />
  <row Id="13160" PostId="8029" Text="Perhaps this question is better suited for Stack Overflow: http://stackoverflow.com/" CreationDate="2011-03-08T18:30:42.897" />
  <row Id="13161" PostId="8029" Score="2" Text="Why would you be squaring the variances in the second code snippet? Or are your variables just oddly named?" CreationDate="2011-03-08T18:58:27.177" UserId="2970" />
  <row Id="13162" PostId="8021" Text="For me, it is not so obvious what &quot;uniformly distributed&quot; means... And probably a try to define it will automagically create a generating algorithm (=" CreationDate="2011-03-08T19:33:25.990" UserId="88" />
  <row Id="13164" PostId="8028" Score="1" Text="Please invest some time in asking. &quot;Please explain me this PDF that will disappear within a few weeks&quot; is not making the Internet a better place. Thus I must close it in a current state." CreationDate="2011-03-08T19:40:56.703" UserId="88" />
  <row Id="13165" PostId="8025" Text="Heh, it is even answered in the duplicate; but let's call it a good duplicate." CreationDate="2011-03-08T19:51:18.563" UserId="88" />
  <row Id="13166" PostId="8026" Text="Thanks Henry for the answer. If I understand correctly, in the former case, you **do not** want to recognize and report the result, whereas in the latter case you **do**. Is that correct?" CreationDate="2011-03-08T19:56:23.557" UserId="3604" />
  <row Id="13167" PostId="8027" Text="@LSEactuary, I've merged your numerous accounts... please register once for good (use this link: http://stats.stackexchange.com/users/login ), you'll have a chance to use comments to discuss with answerers and earn some reputation, for instance to discuss this pivot matter on chat. In case you lost your cookie again, I still can merge your final account with this one." CreationDate="2011-03-08T20:07:57.837" UserId="88" />
  <row Id="13168" PostId="8033" Score="2" Text="Some hints are provided by other homework questions related to the normal distribution, such as http://stats.stackexchange.com/q/5504/919 ." CreationDate="2011-03-08T20:28:23.850" UserId="919" />
  <row Id="13169" PostId="2805" Text="can you explain the coefficient thing a little more elaborate for beginners? I feel your approach is worth a try because if I difference logs in my case (growth rates) the trend flattens but seasonality becomes strong. Thus the simulateneous approach seems worth trying. But what do I do with the two coefficients? in particular i mean the dummies..." CreationDate="2011-03-08T20:28:26.503" UserId="704" />
  <row Id="13170" PostId="8030" Text="Thanks Henry.  So, when I calculate the variance, then that is the figure I need for s2, rather than squaring the actual variance again?  Also, I had a look at the distribution table, but I wasn't sure what (if any) calculation I needed to perform against the table - what does k degrees of freedom mean?  Sorry for the questions - I am clearly not a statistician - more of a computer programmer.  Thanks again." CreationDate="2011-03-08T20:44:36.290" UserId="3624" />
  <row Id="13171" PostId="7945" Text="@cardinal, the initial offset is from two sources: Different actual start times (manually start Device A, then start Device B a second or two later), and differences in the RTC setting in each device (a few seconds at most).  I see no clear effects of sample timing jitter (nothing measurable), so that should be safe to ignore.  During the 1-2 hour durations of data acquisition the RTC drift is also not noticeable, though longer trial acquisitions did reveal some, but it seems more of a temperature effect that will be difficult to quantify, so it can probably be ignored too." CreationDate="2011-03-08T20:46:09.847" UserId="3479" />
  <row Id="13172" PostId="7945" Text="@whuber, I'm presently working my way through this **[variogram tutorial](http://www.goldensoftware.com/variogramTutorial.pdf)**." CreationDate="2011-03-08T20:52:23.197" UserId="3479" />
  <row Id="13173" PostId="8027" Score="1" Text="@LSEactuary, please improve the question. I fixed numerous grammatical mistakes and latexified the equations. Even in the current form the question runs high risk of closure." CreationDate="2011-03-08T20:59:04.497" UserId="2116" />
  <row Id="13174" PostId="8029" Text="@Darren Young, Excel has well documented problems with its statistical questions, so take care validating your code with it. This is for the future reference, as others pointed out in this case there was an error in your code." CreationDate="2011-03-08T21:06:33.567" UserId="2116" />
  <row Id="13175" PostId="8029" Text="@Darren Young, [this question](http://stats.stackexchange.com/questions/5913/interpreting-two-sided-two-sample-welch-t-test/5915#5915) and the answers might be of interest." CreationDate="2011-03-08T21:08:20.227" UserId="2116" />
  <row Id="13176" PostId="8021" Text="@mbq, I think to define the term, we need to have a p.d.f. of $f_{R, \Theta, \Phi}(r,\theta, \phi)=r^2$. " CreationDate="2011-03-08T21:12:41.610" UserId="3552" />
  <row Id="13177" PostId="7945" Text="@BobC, that makes it sound like a simple constant, albeit unknown, time delay, then. No? If that's the case, the variogram approach maybe a bigger hammer than you want!" CreationDate="2011-03-08T21:20:24.233" UserId="2970" />
  <row Id="13178" PostId="8033" Text="A further tip: what you actually call the *threshold* will translate to a *quantile* on the ${\cal N}(0;1)$ [PDF](http://en.wikipedia.org/wiki/Probability_distribution)." CreationDate="2011-03-08T21:31:56.620" UserId="930" />
  <row Id="13179" PostId="8029" Text="@Cardinal - I think I have mis-interpreted the formula. @mpiktas Thanks for the info.  I'll take a look when in work tomorrow." CreationDate="2011-03-08T21:40:48.007" UserId="3624" />
  <row Id="13180" PostId="8036" Score="1" Text="This question might be more appropriate for stats.stackexchange.com. " CreationDate="2011-03-08T21:10:00.563" />
  <row Id="13181" PostId="8036" Text="hmm, how do I move it" CreationDate="2011-03-08T21:17:30.663" />
  <row Id="13182" PostId="8036" Text="Did you check http://en.wikipedia.org/wiki/Confidence_interval ? " CreationDate="2011-03-08T21:19:13.130" />
  <row Id="13183" PostId="8036" Text="it will be moved automatically soon :)" CreationDate="2011-03-08T21:26:23.340" />
  <row Id="13184" PostId="8033" Text="@chl when you say $N(0; 1)$ did you mean $N(70; 8)$?" CreationDate="2011-03-08T22:01:36.387" UserId="1390" />
  <row Id="13185" PostId="8033" Text="@Gavin Ah, indeed :( " CreationDate="2011-03-08T22:17:19.893" UserId="930" />
  <row Id="13186" PostId="8026" Text="Yes: In the no-positives-in-input case, precision is meaningless; in the no-positives-predicted case, you want to report that the test has been set to be extremely negative. " CreationDate="2011-03-08T22:29:57.577" UserId="2958" />
  <row Id="13187" PostId="8030" Text="@Darren Young: Correct, you don't need to square the variances again; but you do need make sure you are using [unbiased estimates of the population variances](http://en.wikipedia.org/wiki/Variance#Population_variance_and_sample_variance). " CreationDate="2011-03-08T22:43:46.623" UserId="2958" />
  <row Id="13188" PostId="8030" Text="@Darren Young: The [degrees of freedom](http://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29) is a parameter in a [Student's t-test](http://en.wikipedia.org/wiki/Student%27s_t-test) - to complicate things, its simplistic explanation of how they come to be what they are does not apply when you are assuming unequal variances, so look especially at the section on [unequal variances](http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_sample_sizes.2C_unequal_variance). " CreationDate="2011-03-08T22:44:51.863" UserId="2958" />
  <row Id="13189" PostId="8040" Text="Sounds like something a macro might accomplish, if you can find a person skilled enough in that.  I know an inexpensive consultant that might be able to work that out for you." CreationDate="2011-03-08T23:11:13.237" UserId="2669" />
  <row Id="13190" PostId="8033" Score="1" Text="@chl @Gavin May I suggest you're both correct?  Pedagogically, the merit of standardizing and z-scores (that is, relating statistics to N(0,1)) is that you learn one reference distribution and using it habituates you to thinking in units of standard deviation.  Familiarity with that process makes light work of this question..." CreationDate="2011-03-08T23:11:20.510" UserId="919" />
  <row Id="13191" PostId="8029" Text="Excel makes a good prototyping platform for the C# code: first implement TTEST() using lower-level Excel calculations (the kind you can do easily in C#) and exploit Excel to verify their correctness.  Once that's accomplished, the port is easy." CreationDate="2011-03-08T23:13:27.543" UserId="919" />
  <row Id="13192" PostId="8044" Text="care to say why is that book not good enough?" CreationDate="2011-03-08T23:28:50.923" UserId="375" />
  <row Id="13193" PostId="8033" Text="@whuber I initially thought of retracting my initial reply (wrote too quickly, a little bit tired and using a mobile phone), but I now feel all those comments should be helpful for the question at hand -- after all this is just a question of rescaling." CreationDate="2011-03-08T23:37:06.713" UserId="930" />
  <row Id="13194" PostId="8043" Text="@csgillespie: Actually, I was not aware of latin hypercubes. The reason I am fixing two parameters at a time is because the possibilities of sampling seemed endless and each simulation takes quite a lot of time so I need to use some domain knowledge in narrowing down the values. I will look into multiple linear regression as you suggested and will get back. Thank You. +1" CreationDate="2011-03-08T23:41:00.240" UserId="2164" />
  <row Id="13195" PostId="8044" Score="2" Text="@Eduardo It's in the wrong direction :-)." CreationDate="2011-03-08T23:43:30.033" UserId="919" />
  <row Id="13196" PostId="8040" Text="@rolando You're right, but macros are unnecessary.  Excel's LINEST will do it right in the sheet, which means it will dynamically update the results when the data change: that's even better than a macro :-)." CreationDate="2011-03-08T23:46:02.403" UserId="919" />
  <row Id="13197" PostId="8021" Text="@mbq: correction, I think the p.d.f is $f_{R, \Theta, \Phi}(r,\theta, \phi)=\frac{3}{4\pi}r^2sin\theta$" CreationDate="2011-03-08T23:51:10.930" UserId="3552" />
  <row Id="13198" PostId="8044" Text="@whuber, well, you might say going from SAS to R is the right direction, but in this instance, you're right, I'm trying to go from R to SAS." CreationDate="2011-03-09T00:05:10.227" UserId="36" />
  <row Id="13199" PostId="8044" Text="@Steven Maybe you should start with the IML procedure which is the closest conceptually to R. At least that will get the juices going. http://www.psych.yorku.ca/lab/sas/iml.htm  But, I admit there is probably a need for that reverse tome." CreationDate="2011-03-09T00:19:23.110" UserId="3489" />
  <row Id="13200" PostId="8036" Score="1" Text="Do you have a set of &quot;rules&quot; by which you define a return as correct or not ?" CreationDate="2011-03-09T00:24:33.447" UserId="3489" />
  <row Id="13201" PostId="8021" Text="@whuber: is the formula above correct?" CreationDate="2011-03-09T00:25:25.973" UserId="3552" />
  <row Id="13202" PostId="8048" Text="Woah whuber, this is really, really helpful. Can you hook me up with a copy of that spreadsheet?" CreationDate="2011-03-09T00:26:53.647" UserId="3626" />
  <row Id="13203" PostId="7982" Text="I have about 40,000 datapoints, so each feature appears in about 1000 rows. (Seems large enough not to be overfitting?) I meant that there aren't groups of features that often appear together (whether feature X is missing or not is independent of what other features are missing or not)." CreationDate="2011-03-09T00:34:18.020" UserId="1106" />
  <row Id="13204" PostId="8044" Text="Stephen, what kind of programming are you going to be doing?  Do you anticipate writing a lot of custom code, or will you mostly be working with fairly standard SAS modules (with &quot;standard&quot; being highly dependent on what your company has shelled out for...)" CreationDate="2011-03-09T00:50:22.713" UserId="71" />
  <row Id="13205" PostId="7668" Text="Nope, your answer was great! I just usually try to avoid accepting an answer too quickly, and haven't visited the site in a while to come back and accept =)." CreationDate="2011-03-09T01:02:32.540" UserId="1106" />
  <row Id="13206" PostId="8049" Text="Hi Bill - is there any way you can get me a copy of that spreadsheet? It would be so helpful!" CreationDate="2011-03-09T01:03:39.857" UserId="3626" />
  <row Id="13207" PostId="7668" Text="@raegtin, that's a good approach, I think. I was more concerned that my answer was clear enough and had addressed directly enough what you were interested in. If not, I would have looked to improve it." CreationDate="2011-03-09T01:12:48.400" UserId="2970" />
  <row Id="13208" PostId="8007" Text="Not a requirement. Any way is ok." CreationDate="2011-03-09T01:17:11.643" UserId="3615" />
  <row Id="13209" PostId="8049" Score="2" Text="@Kirk I edited the above answer to show the guts of the spreadsheet.  This should be enough for you to recreate it." CreationDate="2011-03-09T01:40:00.757" UserId="2775" />
  <row Id="13210" PostId="8043" Text="@csgillespie: I updated my question with the summary I obtained after using linear regression. I am not sure if my interpretation is correct. Could you please take a look at it?" CreationDate="2011-03-09T01:41:33.630" UserId="2164" />
  <row Id="13211" PostId="8042" Text="@charles.y.zheng: +1 Thank you. I used linear regression and am now trying to interpret them. I updated my question with the results just in case you can take a look at it." CreationDate="2011-03-09T01:42:15.080" UserId="2164" />
  <row Id="13212" PostId="8042" Text="Before doing regression on the numerical values of the variates you will want to normalize them (scale them so that they lie between 0 and 1).  Otherwise the choice of units you use will affect the answer!" CreationDate="2011-03-09T01:47:54.803" UserId="3567" />
  <row Id="13213" PostId="8042" Text="@charles.y.zheng: Oh! I will do that and get back." CreationDate="2011-03-09T01:52:53.743" UserId="2164" />
  <row Id="13214" PostId="8042" Text="@charles.y.zheng: Thank you once again for your help on this. I updated my question with the new values." CreationDate="2011-03-09T02:00:02.237" UserId="2164" />
  <row Id="13215" PostId="8042" Score="1" Text="We can conclude that A is definitely playing a role.  But some other variable, like B, might turn out to have a bigger effect when we consider non-linear effects (that is, when we include additional variates A^2, B^2, etc in the data).  The problem is that the more variates you want to fit, the more data your need for your regression to be meaningful." CreationDate="2011-03-09T02:09:07.367" UserId="3567" />
  <row Id="13216" PostId="8042" Text="@charles.y.zheng: That definitely makes sense. I must admit that this data was generated on 3 runs of 20 scenarios so it might not be very stable. I am generating 20 runs of 30 scenarios to see if it changes anything. I will get back soon on this. Once again, thank you for your help. Just one small thing: I will definitely have to use some other formula if I am using additional variates like A^2 etc. right? I mean I cannot use lm because it is no longer linear. I am not sure if I should be using a non-linear regression because clearly, area, velocity and n(vehicles) seem to be independent." CreationDate="2011-03-09T02:15:49.097" UserId="2164" />
  <row Id="13217" PostId="8050" Text="@Matt - I think `Proc SGPLOT` roxxx." CreationDate="2011-03-09T02:33:19.930" UserId="1307" />
  <row Id="13218" PostId="8042" Score="1" Text="The model will be nonlinear in $A$, $B$, etc, but the regression will still be linear in the variates $A$, $A^2$, etc.  What you will want to do in R is to define a new variable &quot;A2 = A^2&quot;, then use the command &quot;lm(formula = T ~ A + S + V + A2)&quot; (for example)." CreationDate="2011-03-09T02:44:22.643" UserId="3567" />
  <row Id="13219" PostId="8001" Text="@cardinal - touchee.  Although from the question as posed, we aren't sure what *question* @user3614 is trying to answer.  p-values give the answer to &quot;how likely is the data, given the effects&quot;? - the Bayesian answer I described gives the answer to &quot;how likely is the true value of the effect sizes, given the data I have?&quot;.  I  would have a guess that the second answer is more appropriate, requiring a Bayesian approach." CreationDate="2011-03-09T03:23:29.387" UserId="2392" />
  <row Id="13220" PostId="8048" Text="@Kirk You can create it yourself in no time.  It's set up so by typing the formulas in C2:F2, you can drag them down as many rows as you like.  Then type the LINEST formula in I2:L2.  Just five simple formulas and you're done." CreationDate="2011-03-09T03:34:25.673" UserId="919" />
  <row Id="13221" PostId="8042" Text="@charles.y.zheng: Great! Thank you. I am generating my scenario files. I will get back soon." CreationDate="2011-03-09T04:02:34.390" UserId="2164" />
  <row Id="13222" PostId="8050" Text="@Matt Parker - I have not found a SAS general resource book I have found satisfactory. Do you have any suggestions or do you simply rely on internet searches?" CreationDate="2011-03-09T04:39:15.870" UserId="1036" />
  <row Id="13223" PostId="8055" Score="2" Text="This may get better answers on stats.stackexchange.com" CreationDate="2011-03-08T20:11:23.107" UserId="1657" />
  <row Id="13224" PostId="8037" Text="+1 for example of spurious regression. Will employ it in the lectures :)" CreationDate="2011-03-09T08:07:02.960" UserId="2116" />
  <row Id="13225" PostId="8060" Score="1" Text="It would also be sensible to make the value you do not give a dummy variable one of the common and specific answers, e.g. *single* and *female* rather than *unknown* and *unknown*.  Otherwise you may be tempted to report how much impact knowing your marital status or gender has. " CreationDate="2011-03-09T08:16:50.547" UserId="2958" />
  <row Id="13226" PostId="8063" Score="3" Text="Linear correlation is defined without regard to the underlying distribution.  I do not know what you mean by &quot;Pearson is not valid for non-normally distributed data.&quot;  Perhaps you are thinking of hypothesis testing?" CreationDate="2011-03-09T09:45:39.217" UserId="3567" />
  <row Id="13228" PostId="6306" Text="The Bible code episode would be a good illustration of the strengths of the Bayesian view of probability.  In particular, the Bayes factor $P(D|H)/P(D|not H)$ are insufficiently large given that we would assign a small prior probability to $P(H)$.  (H being the hypothesis there exists some mechanism whereby world events are encoded in the Bible.)" CreationDate="2011-03-09T09:58:02.527" UserId="3567" />
  <row Id="13229" PostId="8010" Text="@wok I have updated your question with your &quot;answer&quot;; was this really an answer or just a comment?" CreationDate="2011-03-09T10:11:51.537" UserId="930" />
  <row Id="13230" PostId="8050" Score="1" Text="@Matt I found The Kleinman and Horton book pretty useful.  And, like @suncoolsu I find the new SGPLOT, SGSCATTER and SGPANEL PROCS are much better than the old SAS graphics system.  And SGRENDER allows a LOT of control over graphics.  And the ODS graphics that come with each statistical PROC are very nice defaults." CreationDate="2011-03-09T10:22:31.750" UserId="686" />
  <row Id="13231" PostId="8018" Text="What an excellent answer, @probabilityislogic" CreationDate="2011-03-09T10:33:08.443" UserId="686" />
  <row Id="13232" PostId="8050" Score="2" Text="@Matt (+1) Nice response. I also came across this blog which provides illustrated examples of R/SAS: http://sas-and-r.blogspot.com/." CreationDate="2011-03-09T10:37:19.347" UserId="930" />
  <row Id="13234" PostId="8065" Text="@DrWho What are those numbers? Are these average values, what is your sample size, etc. A little bit of context would be of great help." CreationDate="2011-03-09T10:40:05.413" UserId="930" />
  <row Id="13235" PostId="7201" Text="You can define outliers without a model, but I have found such definitions to be unhelpful.  BTW, by model, I don't necessarily mean a statistical model that has been explicitly fit to the data.  Any definition of an outlier requires you to make some assumption about what sort of values you expect to see, and what sort of values you don't expect to see.  I think it is better if these assumptions (i.e. the model) are made explicit.  There is also the point that in EDA, you are exploring the data, your definition of an outlier may be very different for EDA than for fitting a final model." CreationDate="2011-03-09T11:09:06.043" UserId="887" />
  <row Id="13236" PostId="8069" Score="4" Text="@charles.y.zheng A lower *p-value* does not mean that the observed effect is stronger." CreationDate="2011-03-09T11:10:18.523" UserId="930" />
  <row Id="13238" PostId="8041" Text="In general I would not recommend attempting to define any sort of &quot;mean&quot; across values from different experiments, unless you have a clear idea of what these &quot;means&quot; are supposed to represent.  Otherwise you will usually be computing a &quot;statistic&quot; which is useless at best and misleading at worst." CreationDate="2011-03-09T11:23:37.363" UserId="3567" />
  <row Id="13239" PostId="8069" Score="1" Text="Indeed, it is important for a statistician to understand the difference between statistical and practical significance.  I interpreted the OP's question as asking about statistical significance." CreationDate="2011-03-09T11:25:52.493" UserId="3567" />
  <row Id="13241" PostId="8068" Text="@Ashok +1 for mentioning alternative measures of discrepancy. On the other hand the I did not find ready available formula to use on matrices in the link you gave. Also it does not say why I-divergence is the &quot;best&quot; measure of discrepancy. Could you please elaborate more on these points?" CreationDate="2011-03-09T11:33:26.727" UserId="2116" />
  <row Id="13242" PostId="8072" Text="@Figaro, just to clarify, margin-of-error of what? Of calculating the average?" CreationDate="2011-03-09T11:39:59.573" UserId="2116" />
  <row Id="13243" PostId="8030" Text="@I have re-calculated the functions, and for the original data above I get 17.84 as the t value, which has been confirmed using online ttest calculator.  Why then in the Excel formula would it give such a small number as 1.74463E-05&#xA;instead?  I have looked at the dist table, and with this input data the df = 8, and checked at 0.05 value, but I don't know what to do with that value?  Any ideas please?  Thanks again." CreationDate="2011-03-09T11:45:31.677" UserId="3624" />
  <row Id="13244" PostId="8072" Text="I need *a* sample to have, with 95 % probability, the same distribution as the population has (the distribution between regions and their age groups). Sorry if my terminology or explanations are subpar." CreationDate="2011-03-09T11:50:22.853" UserId="3401" />
  <row Id="13246" PostId="8072" Text="So when the question is how many people needs to be interviewed in each region and age group, this is what I want? Underlying reasoning: To assure that the poll results will not be skewed due to poor sampling, I want to know the minimum required sample size for this 5 % margin." CreationDate="2011-03-09T12:06:33.923" UserId="3401" />
  <row Id="13247" PostId="8074" Text="as a machine learner I am certainly not a saint regarding statistical correctness, but checking the assumptions AFTER performing the test seems like heresy to me." CreationDate="2011-03-09T12:11:18.917" UserId="264" />
  <row Id="13248" PostId="8071" Score="1" Text="see also this question on pearson versus spearman for non-normal data http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data" CreationDate="2011-03-09T12:14:53.490" UserId="183" />
  <row Id="13249" PostId="7742" Text="@Ashok Can you define formally what you mean by &quot;the Bayesian rule of selection has a process of selecting the posterior given the prior which is supported by Bayes theorem.&quot; ." CreationDate="2011-03-09T12:22:09.540" UserId="223" />
  <row Id="13250" PostId="8030" Text="You should get a $t$ value of about $15.959 \approx \frac{100-16.5915}{\sqrt{127.427/5 + 9.1449/5}}$ as confirmed [here](http://www.graphpad.com/quickcalcs/ttest1.cfm).  If you are assuming unequal variances as you did with `TTEST(,,,3)` you need Welch's t-test and you get 4.57 degrees of freedom here which you might round down to 4; if you assume equal variances you get 8 degrees of freedom.  Either way, you next need to find the probability that if the means were equal then $t$ would be 15.959 or more extreme.  The probability what Excel is reporting and the step you have missed." CreationDate="2011-03-09T12:28:20.250" UserId="2958" />
  <row Id="13251" PostId="8075" Text="Thanks! So let's say that each person in the poll is asked a yes or no question. How do I go about and select a sample size that produces a result which is, with 95 % certainty, within a confidence interval of 5 %-units?" CreationDate="2011-03-09T12:35:18.867" UserId="3401" />
  <row Id="13252" PostId="8069" Text="@charles.y.zheng: Thank you for the prompt reply and the clarification. I provided initially Incidence Rates. Now I added number of cases also for clarity" CreationDate="2011-03-09T12:38:50.177" UserId="2956" />
  <row Id="13253" PostId="8069" Text="@chl♦: Thank you for the prompt clarification. I provided initially Incidence Rates. Now I added number of cases also for clarity." CreationDate="2011-03-09T12:39:46.523" UserId="2956" />
  <row Id="13255" PostId="8075" Text="I suppose that doing this for each age group in each region will provide me the sample-size I need (?). " CreationDate="2011-03-09T12:44:11.383" UserId="3401" />
  <row Id="13256" PostId="8030" Text="The reason Excel is giving such a small value is that with a sample of data in the range 91-119 and another sample in the range 13-20, it is exremely unlikely that the two samples came from distributions with the same mean." CreationDate="2011-03-09T12:48:12.707" UserId="2958" />
  <row Id="13257" PostId="8068" Text="You can think of a matrix of order $m\times n$  as a vector in $\mathbb{R}^{m\times n}$  as done in matrix norm(the link given by charles.y.zheng) and use the formula $D(A\|B)=\sum(a_i \log \frac{a_i}{b_i}−a_i+b_i)$. For your second question as I already said there's huge literature on why I-divergence is the &quot;best&quot; measure of discrepancy. See for example Csiszar's Annals of Statistics paper &quot;Why least Squares and Maximum Entropy&quot;: An axiomatic approach to inference for linear inverse problems. The one which I already mentioned is actually a review of this paper. " CreationDate="2011-03-09T12:53:44.090" UserId="3485" />
  <row Id="13258" PostId="8068" Text="@Ashok, not all matrix norms treat matrices as elements of $\mathbb{R}^{mn}$. Hence ignoring matrix structure by embedding into other space might not be appropriate. Furthermore I suspect OP might not be familiar with this trick, that is why I asked you to elaborate. Your answer is fine, but it cannot be readily used. If you look around the site, the tendency is to give self-contained answers as much as that is possible. I am sorry if you found my comment offensive." CreationDate="2011-03-09T12:56:26.367" UserId="2116" />
  <row Id="13260" PostId="8073" Text="@charles.y.zheng How about for example... communication apprehension? A high apprehension has no definite difference with a very high apprehension, right? But I've seen that the variable has been correlated with other variables using Pearson's r. Is that totally okay? Thank you!" CreationDate="2011-03-09T12:57:19.547" UserId="3636" />
  <row Id="13261" PostId="8068" Text="@Ashok, I looked at the article again, it gives overview of discrepancy measures, but not which one is the &quot;best&quot;, or what &quot;best&quot; means. From your answer one can get impression (I for one got it) that the article is about pointing out the &quot;best&quot; discrepancy measure, when in fact it is a survey of discrepancy measures. Sorry for nitpicking." CreationDate="2011-03-09T12:59:26.260" UserId="2116" />
  <row Id="13262" PostId="8074" Text="@richiemorrisroe Thanks! " CreationDate="2011-03-09T13:00:13.923" UserId="3636" />
  <row Id="13263" PostId="8030" Text="Thanks Henry, I get the same t value now.  Is there a simple formula to calculate the p-value?  " CreationDate="2011-03-09T13:06:58.330" UserId="3624" />
  <row Id="13264" PostId="8030" Text="There isn't a simple formula for the cumulative distribution function of [Student's t-distribution](Student's t-distribution) but any decent Statistics library for C# or whatever you are using should have something you can apply or approximate.  I am not the person to ask on this.  " CreationDate="2011-03-09T13:12:56.093" UserId="2958" />
  <row Id="13265" PostId="8068" Text="I'm not saying this is the best. It's not a new subject that which measure is &quot;good&quot; and which is &quot;bad&quot;. It has a huge extensive literature starting from Gibbs-Boltzman-E T Jaynes. They put some axioms which they think &quot;common sense&quot; and show that those common senses leed to this measure of discrimination. So based on whether our model is built on those common senses, we can decide on which measure to deal with." CreationDate="2011-03-09T13:13:12.520" UserId="3485" />
  <row Id="13266" PostId="8030" Text="Okay, thanks for all your help." CreationDate="2011-03-09T13:14:56.560" UserId="3624" />
  <row Id="13267" PostId="8054" Text="@charles: so then (4yearMean - 1yrMean) is definitely nonparametric?" CreationDate="2011-03-09T13:16:26.010" UserId="3443" />
  <row Id="13268" PostId="8076" Score="1" Text="@robin +1, good idea to relate matrix to measure. This seems very appropriate for OP concern about using matrix norm." CreationDate="2011-03-09T13:16:26.633" UserId="2116" />
  <row Id="13269" PostId="8074" Score="1" Text="@steffen I think it's fine.  One assumption of regression is that the residuals are normally distributed.  How would you check that before running the regression?" CreationDate="2011-03-09T13:27:59.910" UserId="2310" />
  <row Id="13270" PostId="8068" Text="@robin: But there's no guarantee for the existence of the stationary probability vectors, right? I don't know why I am not able to comment below your post." CreationDate="2011-03-09T13:32:55.960" UserId="3485" />
  <row Id="13271" PostId="8068" Text="@Ashok, sorry for nitpicking again. Although you do not claim that particular measure is the best, your answer implies that it is. Your answer provides valuable insight, but I think it might be improved. " CreationDate="2011-03-09T13:37:16.700" UserId="2116" />
  <row Id="13272" PostId="8068" Text="@Ashok, commenting everywhere requires 50 reputation points. Do not worry, I think you will reach them soon." CreationDate="2011-03-09T13:39:07.277" UserId="2116" />
  <row Id="13273" PostId="8083" Text="@user3638, majority of statistical estimation methods are used for one sample." CreationDate="2011-03-09T14:18:04.040" UserId="2116" />
  <row Id="13274" PostId="7742" Text="Robin, fact is that I don't know Bayesian inference method completely. Otherwise this question would not even have arisen to me. Now I am trying to find time to learn about Bayesian. All I knew(roughly) was that using Bayes theorem if some prior and some additional information are given one can update the probabilities. I do not know this rigorously. Whereas I know MaxEnt rigorously what it means. If it is possible, please explain or lead me(i.e.,point some reference) to learn Bayesian inference rigorously. Thank you." CreationDate="2011-03-09T14:19:10.903" UserId="3485" />
  <row Id="13275" PostId="8074" Text="@Glen: In this case I cannot. But when I compare the quality of different models, I generally prefer to check the assumption (e.g. approximately normally distributed) before performing the test to reduce the tendency to relax assumptions in favor of certain test outcomes. Call it prevention of a mind trick. I guess it is just me ;)." CreationDate="2011-03-09T14:26:22.487" UserId="264" />
  <row Id="13276" PostId="7174" Text="Yes, I only have a list of the firms and after selecting them I'm going out to collect data on them. Thank you for your comments." CreationDate="2011-03-09T15:02:33.140" UserId="834" />
  <row Id="13277" PostId="8036" Text="It will be read by human, they will simply mark the document as accept (no error) or reject (there is an error)" CreationDate="2011-03-09T15:03:12.567" UserId="3625" />
  <row Id="13278" PostId="7739" Text="@probabilityislogic, your comments really don't have anything to do with all-subsets regression and I wouldn't try to force them into that framework. They appear to have more to do with *model selection* in regression. There are a multitude of such methods, both classical and modern, including threshold approaches like the one you describe. The lasso is one example and it even has a Bayesian interpretation. Usually you need a condition close to orthogonality of the design matrix to guarantee good performance (even asymptotically!)." CreationDate="2011-03-09T16:04:07.987" UserId="2970" />
  <row Id="13279" PostId="7742" Score="1" Text="@Ashok most often connection you are searching for arises from the description of convex sets with a probability measure on its extreme points (Choquet Theory). " CreationDate="2011-03-09T16:10:06.603" UserId="223" />
  <row Id="13280" PostId="8050" Text="Thanks for the pointers on those new procedures, @suncoolsu and @Peter - I've only recently been upgraded to 9.2, so I'll definitely check that out.  Also, just to clarify: I don't think &quot;SAS and R&quot; is poorly done, it has just never really helped me when I was in need.  @chl, the blog you found is actually the companion blog to this book, which is a really nice addition by the authors." CreationDate="2011-03-09T16:10:53.030" UserId="71" />
  <row Id="13281" PostId="8083" Text="A great book on bootstrapping is Bradley Efron and and Robert Tibshirani's An Introduction to the Bootstrap. It is a very easy read and covers the topics well." CreationDate="2011-03-09T16:14:30.317" UserId="401" />
  <row Id="13282" PostId="8050" Text="@Andy, nor have I, but I haven't really looked.  I can usually get what I need from searching.  But that's for putting out fires - it doesn't do much to really build my skills as a SAS coder for the long run.  Let me know if you find something good." CreationDate="2011-03-09T16:14:43.950" UserId="71" />
  <row Id="13283" PostId="8036" Text="Are you trying to determine if your proofreading process is correctly identifying the erroneous returns?  Do you have any baseline measures, for example how many returns caught by the IRS contain errors?  " CreationDate="2011-03-09T16:20:23.203" UserId="3489" />
  <row Id="13284" PostId="8010" Text="@chl  I asked the question, although I almost had the result: I was not aware at the time I asked, sorry. Having $\nu=0$ gives the result since there is a special link between $\alpha$ and $\mu$. There were more details in my first edit about this point (I edited as I did not want to give the whole solution away, with respect to the professor who gave the homework, due for today)." CreationDate="2011-03-09T16:20:54.103" UserId="1351" />
  <row Id="13285" PostId="8050" Text="Incidentally, I've also found that making snide comments about SAS and then being corrected by people who are more knowledgeable than me is also an effective method of learning about SAS, albeit a bit rough on the ego..." CreationDate="2011-03-09T16:22:19.450" UserId="71" />
  <row Id="13287" PostId="8041" Text="I would generally agree but these experiments are very much related." CreationDate="2011-03-09T16:39:51.867" UserId="3604" />
  <row Id="13288" PostId="8083" Text="Maybe you could ask some more specific question, even to get an example of bootstrap use?" CreationDate="2011-03-09T16:51:09.227" UserId="88" />
  <row Id="13289" PostId="7987" Text="Knowing the bias can help, you can correct for it to obtain an unbiased estimator.  I actually went down this route, but computing the exact bias requires that you have $u_x$ and $u_y$ -- which we don't.  So naturally I tried to use the sample mean instead to see what happens.  It doesn't appear to help.  In simulations, the corrected estimator also exhibits bias.  I'm leaning towards an unbiased estimator not existing, but I haven't come up with a good proof for it." CreationDate="2011-03-09T16:52:46.737" UserId="3602" />
  <row Id="13291" PostId="8086" Text="+1 for being gentle :-).  A nit, but maybe a revealing one: a suitable variance estimator in this situation is the number of successes themselves.  Thus one can indeed bootstrap from a single observation.  But of course that would be self-fulfilling: a Poisson bootstrapped dataset is going to look Poisson!" CreationDate="2011-03-09T16:57:50.377" UserId="919" />
  <row Id="13292" PostId="8081" Score="3" Text="Maybe you could ask about, for instance, hazards connected with using horizontal and vertical length on the same plot? This way this is a vague and low quality question that have a great chance to get closed or never answered." CreationDate="2011-03-09T16:57:59.290" UserId="88" />
  <row Id="13293" PostId="7998" Text="The samples are expensive to obtain so I can't just up the sample size until the bias goes away.  Unbiasedness is desired because I'm using the result of the estimator as the $Y$s in linear regression.  Having bias means that $Y$ will contain non-normal disturbance which is equivalent to a specification error and that leads to a mess.  I won't be able to accurately estimate the slope, the variance, construct confidence intervals, etc.." CreationDate="2011-03-09T17:00:16.730" UserId="3602" />
  <row Id="13294" PostId="8083" Text="The preceding comments addressed an earlier (more general) version of this question (which, being unanswered, I merged with this one)." CreationDate="2011-03-09T17:03:00.017" UserId="919" />
  <row Id="13295" PostId="8063" Text="This question is an extremely close relative of http://stats.stackexchange.com/questions/3730/pearsons-or-spearmans-correlation-with-non-normal-data" CreationDate="2011-03-09T17:09:01.417" UserId="919" />
  <row Id="13297" PostId="7982" Score="1" Text="Are the features missing *truly* at random?  What is the mechanism of missingness?" CreationDate="2011-03-09T17:37:59.680" UserId="919" />
  <row Id="13298" PostId="8086" Text="@whuber: Ah.  I think I was confused as to whether the question was asking about bootstrapping from a single success/failure observation, or from a single count observation (composed of several successes and failures).  But the point still stands, I think: you can't learn much about the distribution of a random variable with only one sample of it." CreationDate="2011-03-09T17:57:15.307" UserId="2975" />
  <row Id="13299" PostId="8090" Score="1" Text="Define &quot;true&quot;, please. " CreationDate="2011-03-09T18:04:52.420" UserId="2970" />
  <row Id="13300" PostId="8090" Text="@cardinal:  It's unavoidably subjective.  I want the amount of dimensions that capture &quot;most&quot; of the variance." CreationDate="2011-03-09T18:08:15.547" UserId="1347" />
  <row Id="13301" PostId="8091" Text="@shabbychef, this seems to presuppose that (a) the true structure is a linear manifold and (b) the high variance directions are the important ones. " CreationDate="2011-03-09T18:09:13.153" UserId="2970" />
  <row Id="13302" PostId="8090" Score="1" Text="@dsimcha, that's fine. I just wanted to know for sure what you were going for. There are interesting examples in multivariate statistics where, e.g., if you want to find the best number of terms to keep in a least-squares sense for PCA, you should pick a smaller rank than the true rank, i.e., the rank of the actual underlying structure from which the data were generated. " CreationDate="2011-03-09T18:13:27.673" UserId="2970" />
  <row Id="13303" PostId="8088" Text="Is `seq` in your code a typo for `sex`? If you've copy-pasted your analysis code, that might be part of the problem.." CreationDate="2011-03-09T18:26:45.183" UserId="449" />
  <row Id="13304" PostId="8016" Text="@wok Thanks for clearing that up!" CreationDate="2011-03-09T18:29:10.437" UserId="930" />
  <row Id="13305" PostId="8092" Score="3" Text="@JMS, (+1) especially for the note of caution on scree plots. I would call them &quot;standard&quot;, but also &quot;a bad idea&quot; (generally speaking). [Here](http://arxiv.org/abs/0908.2062) is a cautionary tale and also a way to pick the rank more safely, especially if MSE is your metric. This also provides an example, if I recall, where the true rank is the wrong one to pick for minimizing MSE. " CreationDate="2011-03-09T18:29:22.433" UserId="2970" />
  <row Id="13306" PostId="8092" Text="@cardinal (+1) Thx for the link to Owen &amp; Perry's article." CreationDate="2011-03-09T18:33:33.897" UserId="930" />
  <row Id="13307" PostId="8089" Text="@Raffi I think *True Negative Rate* (also called *specificity* in other settings) is was you are looking for, but see http://en.wikipedia.org/wiki/Precision_and_recall." CreationDate="2011-03-09T18:45:52.463" UserId="930" />
  <row Id="13308" PostId="8088" Text="oops, that's just a typo in the code I included above. In my actual code I didn't have the typo. Thanks for catching this though." CreationDate="2011-03-09T18:54:36.210" UserId="364" />
  <row Id="13309" PostId="8091" Score="1" Text="@cardinal You may be confusing assumption and effect.  Curvature in the manifold will be manifest as curvature in these scatterplots and the magnitude of curvature will be revealed in the magnitude of vertical variation in the plots.  In effect, shabbychef is viewing the data as residing approximately within a tubular neighborhood of a low-dimensional linear space.  That imposes no restrictions whatsoever on the shape of the data." CreationDate="2011-03-09T18:57:35.943" UserId="919" />
  <row Id="13310" PostId="8092" Text="I apparently reinvented scree plots at one point but didn't realize they were called anything.  Thanks for reminding me of them and letting me know that they are &quot;standard&quot;.  Since my goal is to produce a nice visualization, simple, standard and good enough wins out over complicated but more optimal." CreationDate="2011-03-09T19:08:11.080" UserId="1347" />
  <row Id="13311" PostId="8096" Score="3" Text="http://en.wikipedia.org/wiki/Confidence_interval#Practical_example" CreationDate="2011-03-09T20:07:07.943" UserId="919" />
  <row Id="13312" PostId="8096" Text="Does your data constitute a sample, or is it the entire population?  How big is the population, if so?" CreationDate="2011-03-09T20:42:02.277" UserId="3320" />
  <row Id="13315" PostId="8092" Score="2" Text="@dsimcha, I did a quick Google search and there is apparently an R package that implements the approach in the link I provided. There is a nice paper from about 20 years ago (maybe 30), I believe in *The American Statistician*, which clearly demonstrates the flaws of inference using &quot;standard&quot; scree plots. I'll see if I can recall the title and will post a link or reference. While standard, but (quite) flawed, is an appealing approach, especially under time constraints, I would gently caution against such a mindset. Cheers. " CreationDate="2011-03-09T20:56:17.370" UserId="2970" />
  <row Id="13316" PostId="8044" Text="@Stephen I was in an analogous situation 2 months ago with SPSS. The &quot;R for SAS and SPSS Users&quot; book was most unhelpful -- don't buy it! On the other hand, I've borrowed the &quot;SAS and R&quot; book that Peter cited, and thought it looked very useful for learning SAS by translation." CreationDate="2011-03-09T21:00:13.617" UserId="3432" />
  <row Id="13318" PostId="8100" Text="+1 (Your answer came in while I was drawing my figure...)" CreationDate="2011-03-09T21:14:35.450" UserId="919" />
  <row Id="13321" PostId="8094" Text="The problem is that the observations come in real time.  Imagine a real case where there are several workers (A-D) in a factory, and you have the times they need to complete a simple task (you get a new value with each task they finish).  Now you need to detect if one of them is being more/less productive than the others, in real time (that is, update the results with each new sample)." CreationDate="2011-03-09T22:32:46.977" UserId="3576" />
  <row Id="13322" PostId="8094" Score="1" Text="Guido: Read up on quality control statistics. I know very little about this field, but I think it is designed to do exactly what you want (while ANOVA is not)." CreationDate="2011-03-09T22:43:44.877" UserId="25" />
  <row Id="13323" PostId="8040" Text="Good to know.  Thx." CreationDate="2011-03-09T23:22:57.950" UserId="2669" />
  <row Id="13324" PostId="8080" Text="I'll second the vote for the SAS documentation, it really is pretty good (and voluminous)." CreationDate="2011-03-10T00:16:05.290" UserId="1569" />
  <row Id="13325" PostId="8103" Text="I *think* I know what you mean, David, but I'm not quite sure because &quot;support&quot; means several things in statistics.  Among them that might be relevant here are (a) values that a random variable can attain and (b) the physical extent or amount of any sample of a substance (such as a volume of soil or water) represented by a measurement.  However, neither seem exactly right for this question.  Could you perhaps indicate what you mean by &quot;common support&quot;?" CreationDate="2011-03-10T01:33:09.443" UserId="919" />
  <row Id="13326" PostId="8103" Text="I mean support in the sense of where a function is non-zero. I think for the example at hand, one has a binary variable s and some other background variables gathered on people, and the potential problem is that for people in the group with s=1 and people in the group with s=2, their range of measured values on the background variables might not overlap much. In this instance all is ok, but a referee wants a discussion that says why there is no problem about finding a region of common support for these two groups. My colleague is not a statistician so needs a pointer even if only to a textbook" CreationDate="2011-03-10T02:38:53.953" UserId="1144" />
  <row Id="13328" PostId="8083" Text="@mbq: Already add an example in the revised version." CreationDate="2011-03-10T04:03:04.167" UserId="3638" />
  <row Id="13329" PostId="8086" Text="@Aaron: whuber is right. i'm trying to bootstrap the sample to obtain different no. of successes. Anyway, thanks for pointing out (over)dispersion issue with sample size of 1." CreationDate="2011-03-10T04:06:43.680" UserId="3638" />
  <row Id="13330" PostId="8105" Text="@whuber thanks for your explanation. I have been running into convergence problems and I was surprised that so many of the variables I work with have means is &lt; 1000 (i.e. if something is &gt;1000 g it is measured in kg), and the variances are on about the same order of magnitude. So, I am realizing that I need more priors that incorporate this information even if I don't really have a good prior knowledge of its value or how it is partitioned." CreationDate="2011-03-10T04:28:55.367" UserId="1381" />
  <row Id="13332" PostId="8092" Text="@ cardinal: Never been a big fan of them myself. Best case for the OP is that it's fantastically obvious that 1-2 components won't cut it. Thanks for passing on that paper, it looks interesting. I recall learning in a linear models class about work on the performance of different approaches using scree plots (picking the &quot;elbow&quot;, eigenvalues &gt;1, etc) but I've forgotten them :/. I do remember it being disheartening." CreationDate="2011-03-10T04:55:12.220" UserId="26" />
  <row Id="13333" PostId="8108" Score="1" Text="+1 for not only answering the question, but also providing useful advice." CreationDate="2011-03-10T04:57:22.047" UserId="919" />
  <row Id="13334" PostId="8105" Text="Depending on the model, your posterior might be very close to improper using this prior" CreationDate="2011-03-10T04:57:41.907" UserId="26" />
  <row Id="13335" PostId="2805" Text="ran2, I know it may not be that clear, but I can't explain it that much better than I have already.  It is a reflection of my own communication skills more than anything.  Instead, I would suggest the basic fix that works more often than not.  That is to simply change your nominal time series variable into a % change from one period to the next and so on.  Focusing on % change instead of nominal values immediately changes a non-stationary variable into a stationary one that you can then readily regress. " CreationDate="2011-03-10T05:02:13.087" UserId="1329" />
  <row Id="13336" PostId="8108" Text="+1 - uniform for $log(\sigma)$ on a &quot;big&quot; range is often a good choice. For the variance components of a hierarchical model I think you can still get into near-impropriety of the posterior if the range is too large since you're approaching Jeffrey's again. But of course it's a simpler fix - just don't pick huge intervals :)" CreationDate="2011-03-10T05:04:25.590" UserId="26" />
  <row Id="13337" PostId="8108" Text="@JMS - in a heirarchical setting, the data do not &quot;squash&quot; the singularity at 0 (i.e. a level 2 variance could be zero).  So the prior for small values matters.  $Beta_{2}(1,1)$ is a good level 2 and higher variance prior (I think it also has been called a &quot;half Cauchy&quot;, it is similar to $F_{1,1}$-distribution).  It has &quot;fat tails&quot; and is &quot;data-robust&quot; in that, if prior and likelihood conflict, the likelihood wins.  Also $Beta_{2}(0,0)$ is the jeffreys prior." CreationDate="2011-03-10T05:58:49.967" UserId="2392" />
  <row Id="13338" PostId="8078" Text="You put a lot of domain knowledge into this though. What if you do not recognize the pattern of the time series yourself? How do you then construct a model that can, especially if it depends on inputs that are infinitely far back in the past?" CreationDate="2011-03-10T07:01:12.580" UserId="2860" />
  <row Id="13339" PostId="8102" Text="Thanks, realized this yesterday evening myself. The problem boils down to a quite simple one in the end." CreationDate="2011-03-10T07:21:52.340" UserId="3401" />
  <row Id="13340" PostId="7879" Score="1" Text="@Imran: I updated the answer accordingly. Hope it is enough to get started with coding." CreationDate="2011-03-10T07:32:37.450" UserId="264" />
  <row Id="13341" PostId="8023" Text="@whuber pardon my ignorance, but what does the # sign in your answer mean?" CreationDate="2011-03-10T07:55:02.120" UserId="1496" />
  <row Id="13342" PostId="8091" Text="@shabbyshef +1 for the decomposition idea, and of course the mentioning of the shoes :)" CreationDate="2011-03-10T07:57:12.437" UserId="2116" />
  <row Id="13343" PostId="7911" Text="chl, thank you for the explanation (and the links to the papers!), just to be sure - the confidence regions are for the **expectancy** of the bivariate distribution?" CreationDate="2011-03-10T08:54:45.937" UserId="253" />
  <row Id="13344" PostId="7258" Text="@RichardN, you have never accepted an answer -- please consider doing it." CreationDate="2011-03-10T09:28:37.860" UserId="88" />
  <row Id="13345" PostId="7911" Text="@Tal Yes, this is always in reference to a theoretical, not empirical, distribution." CreationDate="2011-03-10T09:59:59.143" UserId="930" />
  <row Id="13346" PostId="8118" Score="2" Text="More generally, assuming constant hazard rates over fixed time intervals (known as a piecewise-exponential model) you can fit fairly flexible survival models in the form of poisson GLMs - if you add interactions between the piecewise constant baseline hazard and covariates, you can estiamte time-varying effects and move away from the proportionality assumption, for example. Sources: Michael Friedman &quot;Piecewise Exponential Models for Survival Data with Covariates&quot;, Annals of Statistics&#xA;N LAIRD, D OLIVIER &quot;Covariance Analysis of Censored Survival Data Using Log-Linear Analysis Techniques&quot; JASA &#xA;" CreationDate="2011-03-10T10:18:36.653" UserId="1979" />
  <row Id="13347" PostId="8092" Text="Scree plots are unavoidably subjective. I would probably fit a CFA model (available in R with sem, lavaan or OpenMx packages) and if a four dimensions solution comes up as the best fitting, use the path diagrams and scree plots as a graphic. But don't use them without testing the model formally" CreationDate="2011-03-10T10:35:00.303" UserId="656" />
  <row Id="13348" PostId="8074" Text="@ steffen: you are correct about the heresy, but if the two procedures give the same results then its a matter of taste which to use, but if they dont then checking the asumptions and where they fail can often give useful insight into the data. Personally, i use spearman wherever possible, but this is not common practice in my field." CreationDate="2011-03-10T10:37:05.950" UserId="656" />
  <row Id="13349" PostId="8103" Score="2" Text="From your comment above and as you've tagged this with `propensity-scores`, I believe 'common support' means the same as 'overlap in the distributions of covariates' as discussed by [Rubin 2008](http://arxiv.org/PS_cache/arxiv/pdf/0811/0811.1640v1.pdf). My reading is that this paper says lack of such overlap *is* a problem. I'm no expert on propensity score analysis, but I understand Rubin to be saying that this is a problem for estimating causal effects from observational data whether or not you use propensity scores, but using propensity scores make it more obvious." CreationDate="2011-03-10T10:55:24.203" UserId="449" />
  <row Id="13350" PostId="8012" Text="@mpiktas Yes, I'm interested in $Var(\bar\tau)$. And both an explicit formula or some numerical algorithm would be fine. What I have is a time series $A(t)$, where I compute the time between two thresholds $A_k$ and $A_0$, thats what I mean by $(t_{k,i,2}−t_{k,i,1})$, $i$ meaning the i-th time this two-threshold-crossing occurs for a given $k$. In my case that's protein folding dynamics. I checked for stationarity in R by adf.test and actually, my time series are stationary. What other properties do you think of? Thanks for your efforts." CreationDate="2011-03-10T11:06:57.753" UserId="3104" />
  <row Id="13351" PostId="8087" Text="@cesar - not a full answer, but what may help you is to &quot;go backwards&quot; and learn a more basic model, and then how this one &quot;generalises&quot; that more basic model.  You may need to do this in several steps.  do you get to choose $S$? like in a mixture model?  This does look like a mixture model, where each &quot;s&quot; defines a &quot;component&quot;.  Kind of like saying &quot;I have $S$ models, but I don't know the state which each individual observation is in.  So I will average over these states&quot;" CreationDate="2011-03-10T11:17:09.947" UserId="2392" />
  <row Id="13352" PostId="8120" Text="@onestop: Thank you for the comment and the references!" CreationDate="2011-03-10T11:40:17.613" UserId="3019" />
  <row Id="13353" PostId="8118" Text="@Marco +1, nice explanation" CreationDate="2011-03-10T12:29:35.897" UserId="2116" />
  <row Id="13354" PostId="8118" Text="@mpiktas: Thanks!" CreationDate="2011-03-10T12:50:17.420" UserId="3019" />
  <row Id="13355" PostId="8118" Text="@Marco and @fabians, Thank you. Seems like a more interesting thing to look at and generate more discussion from our group!" CreationDate="2011-03-10T13:10:06.883" UserId="2961" />
  <row Id="13356" PostId="8078" Text="Infinite would certainly be tricky.  However, if you put in transforms of the data that aren't relevant to this domain, the learning algorithm will easily be able to figure that out and adjust the weights accordingly, so it's not a big problem as long as you DO have transforms of the data that are relevant.  So, having many different transforms available improves your odds of success." CreationDate="2011-03-10T13:45:12.820" UserId="2917" />
  <row Id="13357" PostId="8091" Text="@whuber, @shabbychef, I was probably a little hasty with my initial comment, especially since I was away from a computer at the time. I think I also initially misread shabby's description and I couldn't view the plots. These make a nice way to augment a standard pairs plot. " CreationDate="2011-03-10T13:48:21.037" UserId="2970" />
  <row Id="13358" PostId="8091" Text="@whuber (cont.) I do think it's still possible to be fooled. Consider if your data were noisy measurements of something shaped like a cylindrical shell with no tops. This plot should look essentially flat for the first dimension, but should also look flat (with more relative scatter) when projecting on the first two components. " CreationDate="2011-03-10T13:51:38.507" UserId="2970" />
  <row Id="13359" PostId="8092" Text="@JMS, I think I've seen things like broken-line regressions for scree plots, e.g., to try to find the &quot;elbow&quot;. That seems pretty ad hoc to me. " CreationDate="2011-03-10T14:14:25.180" UserId="2970" />
  <row Id="13360" PostId="8128" Text="Thanks mpiktas, but I am somewhat stuck at how to calculate V. Is V simply the diagonal of the covariance matrix?" CreationDate="2011-03-10T14:38:50.950" UserId="2704" />
  <row Id="13361" PostId="8126" Score="1" Text="In what way are you using **optim** (with what options, with or without supplying a gradient function, etc)?? Logistic regression is a smooth convex problem. It's readily solved using Newton's method or similar. In fact, to get an estimate of the covariance matrix, you *need* to do (something close to) this. " CreationDate="2011-03-10T14:41:20.977" UserId="2970" />
  <row Id="13362" PostId="8041" Text="@Raffi: Sorry, but your question is rather general ... can you provide how this experiments are related ? If they originate from different folds of a x-validation, an unweighted average is (of course ;)) meaningful, but in general this question is hard to answer. " CreationDate="2011-03-10T14:49:45.260" UserId="264" />
  <row Id="13363" PostId="8126" Text="I have added the info to the post" CreationDate="2011-03-10T14:51:26.757" UserId="2704" />
  <row Id="13364" PostId="8112" Text="@Hong, @Julian, while you're reading, be sure to also check out [Mikosch's critique](http://www.math.ku.dk/~mikosch/Preprint/Copula/s.pdf) of copulas. The article was published with several discussion responses, many of which are quite interesting in their own right. " CreationDate="2011-03-10T14:51:54.067" UserId="2970" />
  <row Id="13365" PostId="8128" Text="@Thomas, no, it's the diagonal matrix as you've specified it in your initial post, but where the $\pi_i$ are replaced by the estimates $\hat{\pi}_i$, i.e., the estimated probability that the $i$th response is 1 under the model. " CreationDate="2011-03-10T14:57:06.163" UserId="2970" />
  <row Id="13366" PostId="8125" Text="@mpiktas I made a little drawing of what I'm computing: http://img109.imageshack.us/i/mfpa.jpg/ I hope that is of help to you. A(t) is some  observed quantity. $A_k$ is an upper threshold and $A_0$ some fixed lower one ($0 \not\in k$). In physics we call the $y_{ki}$ first passage times. So I would say that there are high correlations for $k \neq l$, probably the correlations for $i \neq j$ are not so important. So computing this $cov(y_{ki},y_{kl})$ is in fact the problem. And thanks for the insight in stationarity. " CreationDate="2011-03-10T15:01:06.960" UserId="3104" />
  <row Id="13367" PostId="8128" Text="Ok, so for each row in the data I simply calculate the predicted probability, and multiply the square root of this vector with the matrix of independent variables? " CreationDate="2011-03-10T15:03:44.547" UserId="2704" />
  <row Id="13368" PostId="8128" Text="@Thomas, yes, that is how it is done in my code. You can check with dummy example that it really works." CreationDate="2011-03-10T15:05:38.983" UserId="2116" />
  <row Id="13369" PostId="8128" Text="Thanks a lot for this very thorough answer!!" CreationDate="2011-03-10T15:06:00.567" UserId="2704" />
  <row Id="13370" PostId="8012" Text="@Askan, I've updated the question with the graph from the comments, since it is very relevant here. Please check that I did this correctly." CreationDate="2011-03-10T15:09:45.330" UserId="2116" />
  <row Id="13371" PostId="8091" Text="@cardinal You are correct.  But if it's that noisy then it won't look four dimensional: it will appear to be truly 16 dimensional (assuming the noise affects all 16 components)." CreationDate="2011-03-10T15:23:17.383" UserId="919" />
  <row Id="13372" PostId="8023" Text="@bgbg Interaction." CreationDate="2011-03-10T15:25:23.327" UserId="919" />
  <row Id="13373" PostId="8127" Score="2" Text="T-tests, significance, sample distributions, and estimates of proportions deal with *conclusions* you draw from the data.  Finding &quot;representative cases&quot; is a matter of *selecting* cases from the dataset.  The two activities are completely different.  What, then, do you really want to do?" CreationDate="2011-03-10T15:30:39.237" UserId="919" />
  <row Id="13374" PostId="8091" Text="@whuber, at the risk of responding too hastily again, I believe that even if the noise were negligible or zero in the &quot;extraneous&quot; dimensions, you would still see such an effect." CreationDate="2011-03-10T15:32:34.140" UserId="2970" />
  <row Id="13375" PostId="8108" Text="@probabilityislogic thanks for the explanation. If I understand, the gamma is nice theoretically because its rage is $[0,\infty]$ and because it is conjugate to the normal, but in application these features are not generally required. But what is the difference between sampling from $\sigma\sim exp(U(log(L),log(U))$ and $\sigma\sim U(L,U)$? " CreationDate="2011-03-10T15:41:32.337" UserId="1381" />
  <row Id="13376" PostId="8078" Text="Imagine the following task: The first input to the net is either $0$ or $1$. Then, the net receives noise from the interval $[-0.1, 0.1]$ for any number (10, 1000, 100000) of timesteps. As soon as it receives $1$ again it has to put out either $0$ or $1$, depending on what it has seen earlier. This is the socalled &quot;Latching benchmark&quot;.&#xA;&#xA;This is quite a typical setting in sequence learning problems. The big benefit of recurrent networks is, that the whole transformation of inputs itself is learned and NOT given by a human expert or feature engineered." CreationDate="2011-03-10T15:45:26.010" UserId="2860" />
  <row Id="13377" PostId="8116" Text="This question appears identical to http://stats.stackexchange.com/q/7727/919 .  Please explain how your problem is not addressed by that discussion." CreationDate="2011-03-10T15:47:36.910" UserId="919" />
  <row Id="13378" PostId="8091" Text="@cardinal I have been looking at such plots recently of very similar surfaces :-) and have not seen anything like what you describe.  Remember, we're not looking just for &quot;flatness&quot; (that's guaranteed on average): we're looking for *small* values randomly clustered around zero.  The nature of the PCA algorithm virtually guarantees this, but the presence of outliers or small regions of large deviation would be notable.  (I am thinking, I now see, of a generalization of @shabbychef's proposal in which $|c_i|$ is plotted against $v_i$, not just $|v_i|$.  Does that make a difference to you?)" CreationDate="2011-03-10T15:53:17.500" UserId="919" />
  <row Id="13379" PostId="8012" Text="@Askan Edit your question rather than add answers, of course unless a situation when you came up with the solution." CreationDate="2011-03-10T15:53:28.130" UserId="88" />
  <row Id="13380" PostId="8012" Text="@mpiktas, @mbq. Thanks for your edits." CreationDate="2011-03-10T16:07:59.727" UserId="3104" />
  <row Id="13381" PostId="8091" Text="@whuber, interesting. Application? I cooked up a 3d example to illustrate what I've been trying to put in words. I'll try to upload a figure and link to it later on. I realize a 2d manifold in 3d probably seem a bit contrived (it is). (+1) to @shabbychef by the way. " CreationDate="2011-03-10T16:08:07.137" UserId="2970" />
  <row Id="13382" PostId="8100" Text="@whuber: I was considering drawing a figure, but was a little too lazy. You've earned my +1 for it :)" CreationDate="2011-03-10T16:24:50.087" UserId="2898" />
  <row Id="13383" PostId="8131" Text="Could you please reformat your answer (add some paragraphs)? It is quite hard to read." CreationDate="2011-03-10T16:30:52.157" UserId="264" />
  <row Id="13384" PostId="8108" Text="@probabilityislogic Not familiar with your notation, are you referring to the beta prime? If so it's an interesting choice. Not the half Cauchy though; that's just the Cauchy restricted to $(0, \infty)$. But the beta prime with $\alpha=1, \beta=1/2$ has been called the &quot;quasi Cauchy&quot; IIRC" CreationDate="2011-03-10T16:36:52.893" UserId="26" />
  <row Id="13385" PostId="7777" Text="Thanks for this.  I have read it several times.  Plenty of questions remain.  1. Functionally in R, how does one test the proportional odds assumption?  2. Are you sure that chi-squared test is right?  On this example it returns 0, meaning .. crappy fit?  But some of the t values are pretty high (InflHigh 10.1, InflMedium 5.4, ContHigh 3.7).  3. What are the plots or pairs showing?" CreationDate="2011-03-10T16:38:35.330" UserId="2849" />
  <row Id="13386" PostId="8108" Text="I should say, the convolution of the beta prime with $\alpha=1, \beta=1/2$ and the normal has been called &quot;quasi-Cauchy&quot; (I think). It's got a spike at zero and Cauchy-like tails." CreationDate="2011-03-10T16:39:03.237" UserId="26" />
  <row Id="13387" PostId="8133" Text="@Weijie, there is a Gauss-Newton numerical optimisation method and there is such a thing as non-linear regression. There is no Gauss-Newton regression, at least I never heard about it. Could you please provide the context for your citation, it is not clear for example what is artificial? " CreationDate="2011-03-10T16:55:34.753" UserId="2116" />
  <row Id="13388" PostId="8137" Text="I am also interested in this. The only way I know how to add those &quot;caps&quot; at the ends of the whiskers is to add a &quot;+ geom_errorbar()&quot; but then the line will overlap the boxplot and look quite messy." CreationDate="2011-03-10T17:59:28.127" UserId="3309" />
  <row Id="13389" PostId="8137" Text="@Casey  Returns error:geom_errorbar requires the following missing aesthetics: ymin, ymax :( Should I add extra data?&#xA;" CreationDate="2011-03-10T18:03:46.563" UserId="3376" />
  <row Id="13390" PostId="8119" Text="@teucer could you provide a reproducible example?" CreationDate="2011-03-10T18:16:51.077" UserId="1381" />
  <row Id="13391" PostId="8137" Text="@Yuriy By horizontal line, you mean arrow-like whiskers or add e.g., the mean together with the median inside the box?" CreationDate="2011-03-10T19:01:16.123" UserId="930" />
  <row Id="13392" PostId="8137" Text="@Yuriy Yes I think you'd have to add the min and max for each group to get errorbar to display correctly. Maybe there's a way to pass the min and max values from the boxplot function to geom_errorbar." CreationDate="2011-03-10T19:21:41.440" UserId="3309" />
  <row Id="13393" PostId="8132" Text="oh-- I just added King cite to my edited response. The article does indeed demolish R^2 mania (still endemic to econometrics) even where the statistic has a meaning -- for OLS regression. King notes too that that pseudo R^2 is gibberish that was manufactured to extend the thoughtlessness associated with &quot;variance explained.&quot;" CreationDate="2011-03-10T19:28:23.480" UserId="2347" />
  <row Id="13394" PostId="8135" Text="Could you clarify what you mean by &quot;the overall mean of all my independent variables should be 0&quot; and how you know this?" CreationDate="2011-03-10T20:09:53.103" UserId="449" />
  <row Id="13395" PostId="8130" Score="1" Text="Could you clarify what you means by &quot;margin of error 5%&quot;?" CreationDate="2011-03-10T20:12:00.963" UserId="449" />
  <row Id="13397" PostId="8135" Text="Basically I want to evaluate all the variables relative to average (the average of all the variables). The coefficients from the regression are relative to the omitted variable. So when I subtract the mean of all the coefficients (including the omitted variable's coefficient of 0) from each coefficient value, the adjusted values will now average 0, and each coefficient value can be viewed as the distance from average." CreationDate="2011-03-10T20:22:33.997" UserId="1090" />
  <row Id="13398" PostId="8137" Score="1" Text="@Casey Just invert the order of layers and this solution will work as wanted." CreationDate="2011-03-10T20:23:52.073" UserId="88" />
  <row Id="13399" PostId="8067" Text="I'll second the recommendation for the Kleinman book. " CreationDate="2011-03-10T20:24:49.793" UserId="1080" />
  <row Id="13400" PostId="8044" Text="On a side note, I code a little in R and mostly in Python and just joined a company that heavily uses SAS. I am making some great inroads after 5 months of getting Python included in our toolbox, displacing SAS in some areas. As you navigate the waters, we should consider starting a Wiki about &quot;How to get your company to adopt something other than SAS/SPSS for analytics&quot;. I am finding the cultural issues are far more difficult than simply evangelizing the language or performance differences." CreationDate="2011-03-10T20:29:37.607" UserId="1080" />
  <row Id="13401" PostId="7258" Text="One case where classes are not of equal importance is when you classify something into A vs. not A (as opposed to A vs. B). For A vs. B (e.g. cat vs. dog) both classes are meaningful and you may try to model both. For A vs. not A (e.g. cat vs. non-cat), the negative class includes everything, so you can't really hope to model it anyway. Another example is if cost of misclassification is different. Say in your example, if I predict a default erroneously and refuse a loan, then I just lost a little profit. If I predict non-default erroneously, I lose a much larger loan amount. " CreationDate="2011-03-10T20:30:47.620" UserId="3369" />
  <row Id="13403" PostId="8133" Text="@mpiktas I had the same reaction, but Googling turns up lots of hits.  A context would nevertheless be quite helpful." CreationDate="2011-03-10T20:59:56.303" UserId="919" />
  <row Id="13404" PostId="8133" Score="2" Text="@whuber, from what I found, the Gauss-Newton regression is non-linear regression approximation. Instead of model $y=f(x,\beta)+\varepsilon$ model with Taylor expansion of $f$ is fitted: $y=f(x,\beta_0)+\frac{\partial f}{\partial \beta}(x,\beta_0)(\beta-\beta_0)+u$. Here is the [link](http://russell.vcharite.univ-mrs.fr/EIE/fchap6.pdf) where this regression is described. Unfortunately (if you do not know French) it is French translation of [this book](http://qed.econ.queensu.ca/dm-book/)." CreationDate="2011-03-10T21:15:13.360" UserId="2116" />
  <row Id="13405" PostId="8091" Score="1" Text="@cardinal Here's a 4D example generalizing your cylinder: http://i.imgur.com/9eF8N.png .  The model is $(s,t)\to(\cos(2\pi t),\sin(2\pi t),2s/5,t^2)$ plus normal(0,.05) error added independently to all four components.  The image is a scatterplot matrix of 300 iid draws with $s$ and $t$ iid uniform(0,1), ordered by decreasing variance.  The proportions of total variance are 49.2%, 46.5%, 2.9%, 1.4%.  The two bottom rows emulate @shabby's plots.  They demonstrate the near 2D nature of the data while revealing the small nonlinearity I put into it.  Is this what you were thinking of?" CreationDate="2011-03-10T21:33:21.520" UserId="919" />
  <row Id="13406" PostId="8133" Text="@Weijie, the link I gave in previous comment answers your question quite nicely, look at the top of page 178 (the number not the actual PDF page). If no-one answers the question, I will attempt to translate." CreationDate="2011-03-10T21:35:34.013" UserId="2116" />
  <row Id="13407" PostId="8103" Text="@onestop - this is it. I'll have a look at Rubin, it may be just what my colleague needs." CreationDate="2011-03-10T21:44:48.373" UserId="1144" />
  <row Id="13408" PostId="8133" Text="@mpiktas Equation 6.06 and the discussion surrounding it are pretty revealing.  (It's interesting that the book is not as freely available in English as it is in French!)" CreationDate="2011-03-10T21:47:09.277" UserId="919" />
  <row Id="13409" PostId="8092" Score="1" Text="@JMS There is a paper by Zwick &amp; Velicer (1986) that argues for using parallel analysis (resampling eigenvalues to test whether observed values were greater than expected by chance) or their MAP criterion over scree plots or the root-one rule. In their simulations the first two methods heavily outperform the others (e.g. 92% vs. 22% accuracy) in identifying the true # of components. The R package `psych` implements both with accompanying scree-like graphics (see `fa.parallel()` and `VSS()`). Paper is &quot;Comparison of five rules for determining the number of components to retain.&quot;" CreationDate="2011-03-10T21:52:07.863" UserId="3432" />
  <row Id="13410" PostId="8140" Text="@Ralph The paper I cited in my response discussed the issue of imputation. In short, the training set should be dense enough and there's still the issue of bias that increases with the no. missing features." CreationDate="2011-03-10T22:02:16.450" UserId="930" />
  <row Id="13411" PostId="8130" Text="@onestop -- good clarification -- I removed it from the question.  I just took that variable from the following sample size calculator: http://www.raosoft.com/samplesize.html  But I don't think its relavant in this question.  Thanks!" CreationDate="2011-03-10T22:06:48.117" UserId="3658" />
  <row Id="13412" PostId="8141" Text="Beg pardons if my writing is not clear, it's a midnight in Lithuania." CreationDate="2011-03-10T22:14:35.827" UserId="2645" />
  <row Id="13413" PostId="8108" Text="@JMS - I am referring &quot;beta prime&quot;.  And it is the &quot;odds ratio&quot; of a beta distribution.  You can show using this, that its a ratio of two gamma's with same scale.  So I think (1,1) it would be the &quot;square&quot; of a cauchy distribution $\left(\frac{Z_1}{Z_2}\right)^2=\frac{\chi^{2}_{1}(1)}{\chi_{2}^{2}(1)}=\beta_2(1,1)$." CreationDate="2011-03-10T22:48:45.350" UserId="2392" />
  <row Id="13414" PostId="8108" Text="@David - if you have a highly informative likelihood, then the uniform won't be much difference.  The uniform prior depends on which scale you use.  e.g. $\sigma \sim U(L,U)$ is flat on $\sigma$, but not flat on $\sigma^2$ or $\sigma^{-2}$.  So something about using the &quot;scale&quot;, and not the variance or precision must mean something.  The jeffreys prior looks the same regardless of which you pick (invariant), as long as $L$ and $U$ are transformed accordingly." CreationDate="2011-03-10T22:57:37.090" UserId="2392" />
  <row Id="13415" PostId="8128" Text="@mpiktas - you are right about $V^2$.  Effectively what you are doing is &quot;standardising&quot; the $X$ matrix, and $Y$ vector, then doing weighted least squares on the standardised variables, then backtransforming to original scale.  You need to iterate because the standardisation depends on $\beta$" CreationDate="2011-03-10T23:14:52.970" UserId="2392" />
  <row Id="13416" PostId="8090" Text="@dsimcha - I would find a specific case where using 2-dimensions leads to a bad answer, while using 4-dimensions leads to a good answer.  This will help in getting your point across - people understand examples.  Add this to some of the more general stuff given below - &quot;eblow&quot; in the scree plot, etc." CreationDate="2011-03-10T23:21:34.230" UserId="2392" />
  <row Id="13417" PostId="8090" Text="@dsimcha - the &quot;examples&quot; would be those cases which have large scores on the third and forth components" CreationDate="2011-03-10T23:22:43.790" UserId="2392" />
  <row Id="13418" PostId="8140" Text="@chl Thanks - Did not know about that one." CreationDate="2011-03-11T01:05:23.583" UserId="3489" />
  <row Id="13419" PostId="8078" Text="I wouldn't wish to say you shouldn't ever use recurrent neural networks; quite the contrary.  However, if the task (as stated in the question) is to predict the ith from the (i-1) and (i-2) points, then you can get a better result faster by utilizing that knowledge.  I don't mean to suggest that RNN's aren't ever a good idea, but it's ok to use whatever domain knowledge you have to speed up the training process (and decrease the likelihood of training getting caught in a local minimum, etc.)." CreationDate="2011-03-11T01:59:38.667" UserId="2917" />
  <row Id="13420" PostId="8142" Score="1" Text="What is your definition of &quot;analytic form&quot;? (What people &quot;accept&quot; as closed-form solutions varies, which is why I ask.) With things like modified Bessel functions (which I wouldn't consider a &quot;closed-form expression&quot;) stuck in the denominator, it seems like there may be little hope." CreationDate="2011-03-11T03:47:42.083" UserId="2970" />
  <row Id="13421" PostId="8143" Score="2" Text="Hint: by &quot;randomly&quot; you probably mean that all such sequences have the same probability.  So: how many possible sequences of four bits are there?" CreationDate="2011-03-11T03:52:17.617" UserId="919" />
  <row Id="13422" PostId="8124" Text="Ok. I get what you mean about &quot;contradiction&quot; in the question. But, binomial distribution can also be approximated by Poisson distribution. So, does that mean $X$ still cant be Poisson? Emm... maybe i should say $X$ is approximately Poisson? And, about the &quot;high&quot; no. disproportionately in the simulations, is anything &quot;wrong&quot; with that? I mean, do i &quot;somehow&quot; need to control it? Sorry for asking so many questions, i cant help because i'm completely confused. " CreationDate="2011-03-11T04:47:59.007" UserId="3638" />
  <row Id="13423" PostId="8136" Text="Hmm, good point. I tend to think of Fisher with low cell counts only, but using it in general for 2x2 tables makes sense now that we have fancy computers =)." CreationDate="2011-03-11T07:13:32.337" UserId="1106" />
  <row Id="13424" PostId="8119" Text="@David please see the edit..." CreationDate="2011-03-11T07:57:04.200" UserId="1443" />
  <row Id="13426" PostId="7879" Text="@Steffen, thanks a lot. I TRULY appreciate your effort and help. I am sorry for the delayed response. Actually I had a death in my family and was off of last couple of days. Thanks again." CreationDate="2011-03-11T09:47:49.933" UserId="3535" />
  <row Id="13427" PostId="8137" Text="@chl By horizontal line I mean &quot;arrow-like whiskers&quot;." CreationDate="2011-03-11T09:52:07.223" UserId="3376" />
  <row Id="13428" PostId="8127" Text="Thank you for your answer. Indeed, I would like to find such cases that both data measurements (two variables) are in 99% (or 95 or 66) interval of their individual distributions. So it could be something like intersection (Boolean logic) of very significant data of two variables. Correlation between variables is 0.35275, one variable is we can say dependent on other but there are exceptions. " CreationDate="2011-03-11T10:16:13.637" UserId="3656" />
  <row Id="13429" PostId="8137" Text="@Yuriy So, maybe a combination of `geom_crossbar` + `geom_errorbar` + `geom_point` (for outlying values) should do the trick." CreationDate="2011-03-11T10:22:39.407" UserId="930" />
  <row Id="13430" PostId="8140" Text="@Ralph Also, the possibility of imputing data depends on the way missing patterns arose (as pointed by @whuber, indirectly)." CreationDate="2011-03-11T10:28:54.757" UserId="930" />
  <row Id="13431" PostId="8130" Score="2" Text="For many tests, you can calculate a sample size such that the test achieves a certain power given an assumed (fixed) effect size. In other words, you have to specifiy these things first: 1) what test do you want to use? 2) what power do you want that test to have? **conditional on** 3) an effect size that you deem interesting. 1) is something people here can probably help you with. 2) might be related to the 95% you indicated. 3) however, is something you have to provide beforehand: how different do the probabilities have to be to be considered interestingly different?" CreationDate="2011-03-11T11:16:51.953" UserId="1909" />
  <row Id="13432" PostId="8124" Text="@tsukiko - Why use an approximate distribution when the exact one is no more difficult to use?  You could use a truncated poisson for the $X_{i}$ terms, but this would basically be a substitution of $\theta=\frac{\lambda}{r+\lambda}$ in my equations, and nothing else would change." CreationDate="2011-03-11T11:34:31.000" UserId="2392" />
  <row Id="13433" PostId="8124" Text="@tsukiko - in regards to the second part of your comment, the &quot;high&quot; number is not &quot;wrong&quot; per se - I was just explaining why your numbers were higher than the theoretical values.  If you did 100 bootstraps, you may find the averages closer to the theoretical values." CreationDate="2011-03-11T11:36:52.963" UserId="2392" />
  <row Id="13434" PostId="8124" Text="@tsukiko - on looking at your question, you want to run a dispersion test for the poisson.  The theory says that $X$ is *necessarily* &quot;under dispersed&quot; because it is binomial $Var(X)&lt;E(X)$.  So you know from the start, before doing any bootstraps, that it is not dispersed according to poisson.  Further you also know by how much, a factor of $(1-\theta)$.  So if $\theta$ is small, then the approximation will be good.  But $\theta$ is not small in the illustration.  You can run bootstraps, but this will only demonstrate the maths." CreationDate="2011-03-11T11:46:30.973" UserId="2392" />
  <row Id="13435" PostId="8103" Text="@David @onestop @whuber, it has nothing to do with support of a distribution ? my answer is totally out of the point? what are propensity score ? " CreationDate="2011-03-11T12:22:46.387" UserId="223" />
  <row Id="13436" PostId="7912" Text="@Kristian My answer suggest that $\Delta^T(\Sigma^{-1}_1+\Sigma^{-1}_2)\Delta$ is maybe more meaningfull, in some sence the one you propose is not homogeneous, could be $\Delta^T\Sigma^{-1/2}_1\Sigma^{-1/2}_2\Delta$" CreationDate="2011-03-11T12:27:32.327" UserId="223" />
  <row Id="13437" PostId="8124" Text="@probabilityislogic I agree with your explanation. Tq for that. I'm using Poisson instead of binomial perhaps for the sake of studying the spatial pattern of &quot;successes&quot; in the original data." CreationDate="2011-03-11T12:47:37.070" UserId="3638" />
  <row Id="13438" PostId="8154" Text="This seems to work only for individual variables, not for dataframes. So if I do: D2 = cut(D1, quantile(D1), where D! consists of V1, V2, and V3, then instead of D2 being a dataframe of three transformed variables, it's just one single factor. (class(D1): dataframe, but class(D2): factor)" CreationDate="2011-03-11T13:54:11.547" UserId="3671" />
  <row Id="13439" PostId="8154" Text="I thought you would call cut using apply for each Vi of the dataframe and then stick them together." CreationDate="2011-03-11T14:10:48.240" UserId="1351" />
  <row Id="13440" PostId="8109" Text="heuristic: divide by 4--logit coeff/4 is approx pct-pt diff in prob. from 1 unit change. That's not same as saying &quot;person w/ βsubn charcteristic = z is x% likely to.&quot; Not only (as noted) does one have to add βsub0 &amp; subtract likelihood associated with ref class. It's also necessary to take account of predictor collinearity. B/c being southern correlates with other predictors, it won't be true that prob. of southerner voting repub is βsub0 - transformed log odds for NE plus transformed log odds for south. Better to say, &quot;all else equal, being from south predicts x pct point change in prob of&quot;" CreationDate="2011-03-11T14:15:44.570" UserId="2347" />
  <row Id="13441" PostId="8143" Text="$2^4$, right? For each entry, a bit can either be 0 or 1, so it's $2 \times 2 \times 2 \times 2$. I believe I'm looking for a permutation, but I'm having a hard time understanding how to add order to it. For $\frac{n!}{(n-r)!}$, my $n$ is 4, but it seems my $r$ would also be 4, leading to $4!$. This doesn't really seem right, but I know I shouldn't be trusting my intuition in these matters!" CreationDate="2011-03-11T14:57:58.567" UserId="3666" />
  <row Id="13442" PostId="8143" Text="@Michael Why do permutations matter?  You have just correctly counted all possible strings of four bits.  An axiom of probability is that the sum of all probabilities is 1.  Thus, the sum of your 16 equal probabilities equals 1.  Solve the equation!" CreationDate="2011-03-11T15:25:36.787" UserId="919" />
  <row Id="13443" PostId="8156" Score="1" Text="I don't understand what's on your x-axis. If it's one of A,C,T or G then as that's categorical it's a bar-chart not a histogram, but then I can't see how you can have &quot;plateau-like regions&quot;. Neither do I understand what you mean by &quot;linear dropoffs&quot;. Could you link to an example histogram?" CreationDate="2011-03-11T15:26:10.470" UserId="449" />
  <row Id="13444" PostId="8127" Text="@Dario You just answered your own question: compute the interval endpoints and make the selection.  But your references to &quot;significant&quot; and the implication that this procedure is &quot;representative&quot; are disconcerting because they suggest you are looking for an automatic or mindless way to discard the most interesting--and perhaps the most important--cases in your dataset; namely, those that *don't* lie in both intervals.  The wisdom of that approach depends on what use will be made of the selection, which you haven't revealed." CreationDate="2011-03-11T15:30:11.637" UserId="919" />
  <row Id="13446" PostId="8157" Score="1" Text="What do you mean by the fourth constraint, &quot;The magnitude of a is ..&quot;" CreationDate="2011-03-11T15:53:42.820" UserId="1499" />
  <row Id="13447" PostId="8156" Text="Not a brilliant example, but http://www.box.net/shared/f6bahdy81z.  Think of each point along the x-axis as a discrete unit along the genome that we try to determine the identity of.  We know each base exists, but we don't know whether it's an A,C,T, or G.  When we get a &quot;call&quot;, it's basically a quality score that tells us how many times the sequencing machine was able determine the identity of the base.  Another dataset tells me what bases were &quot;called&quot;.  For example, if there were a total of 100 calls for one base, and we had 98 T's called, 1 G called, and 1 A called, we would call that a T." CreationDate="2011-03-11T15:55:23.147" UserId="3672" />
  <row Id="13448" PostId="8156" Text="@mlxa What would this &quot;score&quot; mean?  How would it be used?  BTW, that plot may be deceptive due to the log scale: those aren't plateaux; they only appear to be so.  (A square root scale would be more appropriate for these data.)  The appearance of a long sequence of identical values around 185-210 shows this is not a straightforward problem: the nature of the variability, or lack thereof, is not a simple thing to characterize." CreationDate="2011-03-11T16:02:44.623" UserId="919" />
  <row Id="13449" PostId="8158" Text="@user3676 @LSEactuary Please move this into a comment or comments, using your registered account, and delete it as a &quot;reply&quot;." CreationDate="2011-03-11T16:06:04.760" UserId="919" />
  <row Id="13450" PostId="8156" Text="@whuber, also see http://www.box.net/shared/nbygq2x03u.  Why specifically would a sqrt scale be a better choice?" CreationDate="2011-03-11T16:06:22.357" UserId="3672" />
  <row Id="13451" PostId="8156" Text="@mlxa I still have the same reaction and the same questions." CreationDate="2011-03-11T16:08:05.770" UserId="919" />
  <row Id="13452" PostId="8156" Text="When the score drops to zero, it means the sequencer isn't able to determine the identity of the base.  If you've seen the double helix before, it means the sequencer can't figure out the identify of one half of a rung of the helix.  Certain regions of the genome are more difficult to characterize than others.  Bases (or x data points) with high numbers of basecalls, on the order of &gt;=100, are able to be definitively identified.  Regions with 0 basecalls are of concern because then we've got to infer from neighboring regions what the identity of the low-read region could be." CreationDate="2011-03-11T16:13:26.890" UserId="3672" />
  <row Id="13453" PostId="8157" Text="My mistake. Finished description. Thanks for the feedback." CreationDate="2011-03-11T16:18:19.080" UserId="3673" />
  <row Id="13454" PostId="8143" Text="@whuber Sorry -- I was thinking permutations because of something I failed to mention -- I'm looking for the probability of `0011` being the left half of an 8 bit sequence (e.g., `00110000`). So, this was kind of leading me to $\Pr[leftHalf = 0011] = 1/P^{8}_{4} = 0.000595$, but it didn't seem right. (My goal in this problem is to explain the behavior I'm observing in terms of probabilities. If only Proof by Example wasn't a joke!)" CreationDate="2011-03-11T16:20:13.103" UserId="3666" />
  <row Id="13455" PostId="8143" Text="@Michael The answer to your expanded question depends on whether the right and left halves are independent.  If they are, you can ignore the right half (and you're already done!).  If they are not, you need to describe how the values depend on one another." CreationDate="2011-03-11T17:46:57.730" UserId="919" />
  <row Id="13456" PostId="8156" Text="I just tried Googling 'read depth' after looking in Wikipedia w/o success and found an answer on a sister StackExchange site (first time that's happened for me!): http://biostar.stackexchange.com/questions/638/what-is-the-sequencing-depth" CreationDate="2011-03-11T17:47:16.920" UserId="449" />
  <row Id="13457" PostId="8159" Text="Is there anything you can add that would make this question less vague?  Such as the purpose or basis of the desired comparisons?" CreationDate="2011-03-11T17:48:22.673" UserId="919" />
  <row Id="13458" PostId="8143" Text="@whuber Great -- the right half is independent. So you're saying $1/16$ would give me the correct probability? Thanks for all of your help on this!" CreationDate="2011-03-11T17:49:26.317" UserId="3666" />
  <row Id="13459" PostId="8156" Text="'Straightforward algorithm'? Perhaps not, but perhaps [hidden Markov models](http://en.wikipedia.org/wiki/Hidden_Markov_model)? Not something I know much about, but i know they're used in bioinformatics (e.g. for looking for G/C-rich regions) and I know they're not that difficult to apply. I haven't thought this through yet so I'm not posting it as an answer for now!" CreationDate="2011-03-11T17:53:22.327" UserId="449" />
  <row Id="13460" PostId="8156" Text="@onestop, I'll look into them.  If you'd like, add that as an answer so I can upvote." CreationDate="2011-03-11T17:55:28.910" UserId="3672" />
  <row Id="13461" PostId="8164" Score="1" Text="Never used it, but package `png` may have what you're after: http://cran.r-project.org/web/packages/png/png.pdf" CreationDate="2011-03-11T18:08:09.037" UserId="696" />
  <row Id="13462" PostId="8160" Score="2" Text="When you say &quot;will best take into account&quot;, what do you mean by &quot;best&quot;? E.g. do you want the variables to be weighted equally, or do you want to give some variables more weight than others? Or a different way to put it, what would be the purpose of the score you'd like to get?" CreationDate="2011-03-11T18:25:35.430" UserId="3369" />
  <row Id="13463" PostId="8156" Text="Do you want the score to reflect the linearity of dropoffs? Or do you just need a number that will roughly say how many bases could not be called reliably?" CreationDate="2011-03-11T18:25:52.073" UserId="3369" />
  <row Id="13465" PostId="8167" Score="4" Text="Another option is to look at the responses and check for any inconsistencies. E.g. they responded &quot;1&quot; to &quot;how many children&quot;, but &quot;3&quot; to &quot;how many girls&quot;. Often people introduce this kind of redundancy on purpose to weed out cheaters. But if you don't have this redundancy, it will be much more difficult (although not impossible)." CreationDate="2011-03-11T18:37:44.793" UserId="3369" />
  <row Id="13466" PostId="8156" Text="@Sheldon, the biggest concern is the magnitude and linearity of the dropoff." CreationDate="2011-03-11T18:42:47.697" UserId="3672" />
  <row Id="13467" PostId="8150" Score="2" Text="you need to give more information. How much data have you collected? What kind of data is it? Are you aiming for a statistical power threshold?" CreationDate="2011-03-11T20:15:49.787" UserId="656" />
  <row Id="13468" PostId="8156" Text="I don't think stdev is a good option. It will give you some idea of the magnitude of change, but you won't know whether it's the magnitude of dropoff or magnitude of some other variability. It also won't say anything about linearity. What about something cheap and dirty, like trying to fit linear segments and seeing how many fit in a histogram, and their goodness of fit values?  How are you going to use the final score?" CreationDate="2011-03-11T20:31:24.953" UserId="3369" />
  <row Id="13469" PostId="8108" Text="@ probabilityislogic Ah cool I see the connection. $\beta_2(1,1)$ would have lighter tails than a Cauchy though, no? I think that was why $\beta_2(1, 1/2)$ was called the quasi-Cauchy since they've got the same polynomial tails. The literature goes back a ways on that specific prior but I first ran across it in Griffin &amp; Brown &quot;Bayesian adaptive lassos with non-convex penalization&quot; in the context of the exponential-gamma prior for variance parameters (which I think are a special case of $\beta_2$). " CreationDate="2011-03-11T20:41:58.703" UserId="26" />
  <row Id="13470" PostId="8173" Score="1" Text="@Aniko (+1) Good reference!" CreationDate="2011-03-11T20:47:24.903" UserId="930" />
  <row Id="13471" PostId="8156" Text="The score is used to evaluate the validity of the sequencing result for a particular genomic region.  Regions with many segments with bad scores are of concern." CreationDate="2011-03-11T21:09:39.630" UserId="3672" />
  <row Id="13472" PostId="8169" Text="I am little bit confused. My first guess was that you are trying to predict the probability of occurrence for a value of N(.,.), but your heading says &quot;classifier&quot;. What is the label/target/response variable you are trying to predict ?" CreationDate="2011-03-11T21:18:47.853" UserId="264" />
  <row Id="13473" PostId="8178" Score="3" Text="@IanS: Posting a link without an explanation does not really qualify as an answer. If you think that the OP actually looks for LDA, please elaborate this a little bit. Thank you !" CreationDate="2011-03-11T21:21:19.193" UserId="264" />
  <row Id="13474" PostId="7879" Text="@Imran: Sorry to hear that. All the best for you and your family." CreationDate="2011-03-11T21:23:15.947" UserId="264" />
  <row Id="13475" PostId="8169" Score="1" Text="I think he tries to do basically the example from chapter 1 of the Duda-Hart book. There are two classes, points from one are distributed normally with parameters x, y, points from another are normal with params a, b. What is the posterior probability of an observation given each class, and what is the decision rule." CreationDate="2011-03-11T21:23:42.083" UserId="3369" />
  <row Id="13476" PostId="8159" Text="@bit-question: What is the differene between your question and this possible duplicate (http://stats.stackexchange.com/questions/5026/what-is-the-difference-between-data-mining-statistics-machine-learning-and-ai) ?" CreationDate="2011-03-11T21:25:18.100" UserId="264" />
  <row Id="13477" PostId="8103" Text="@robin - your answer may be entirely correct, but it is not what I (or rather, my colleague) needed - and that is probably a sign of a poorly asked question! The paper which needs this reference is in social science, and won't benefit from using things like Radon-Nikodym derivatives. :) It does have to do with the support of a distribution, but also with other things which make this more than a question in distribution theory." CreationDate="2011-03-11T21:43:52.797" UserId="1144" />
  <row Id="13478" PostId="8127" Text="Aim of my whole procedure is: I will use selection for some further procedures in measurements. So at the beginning i have 1000 cases and six variables, after factor analysis, deleting some cases with one variable value and outlier test i get 53 cases and two variables. So i would like to know 99 95 or 66% confidence interval for variable 1 and 2 and select 5 to 10 cases for further work.&#xA;" CreationDate="2011-03-11T21:49:22.010" UserId="3656" />
  <row Id="13479" PostId="8156" Text="@mlxa Properties of the binomial and poisson distributions indicate the variance of a count is roughly proportional to that count, whence the square root is a variance-stabilizing transformation.  This makes it legitimate to compare variability across different heights in the plot.  The log downweights variation in the high bars whereas using the raw data downweights variation in the lowest bars. // Since you're concerned about bad scores, why not just look at the proportion of bars falling below a critical threshold?" CreationDate="2011-03-11T22:23:25.433" UserId="919" />
  <row Id="13480" PostId="8156" Text="@mlxa Importantly, the appearance of &quot;linearity&quot; of the dropoff can change dramatically depending on the vertical scale.  That's another reason to pay close attention to the issue of how best to express the counts.  But you can finesse this by focusing on how many low &quot;basecalls&quot; there are and ignoring the pattern: maybe that would be good enough." CreationDate="2011-03-11T22:27:10.500" UserId="919" />
  <row Id="13481" PostId="8169" Text="@SheldonCooper  yes!. Do you have some example. or a link to solve this in MATLAB??" CreationDate="2011-03-11T22:32:11.987" UserId="3681" />
  <row Id="13483" PostId="8157" Text="How is that `a_i` follows distribution `p_i` and is also less that `c`? That's because the distribution `p_i` is also less that `c`? In which distribution are you thinking?" CreationDate="2011-03-12T00:31:53.153" UserId="2902" />
  <row Id="13484" PostId="7897" Score="1" Text="Have you tried lasso or elastic net? packages: lasso, glmnet. Those methods can &quot;select&quot; some varibles on-the-go." CreationDate="2011-03-12T00:36:01.153" UserId="2902" />
  <row Id="13486" PostId="8157" Text="@deps_stats. Very good points. Pseudo code was not very clear. The distribution I have in mind is the poisson distribution. Each element follows a poisson distribution with a different lambda. With that in mind I guess the first condition (a_i &lt; c) is not necessary since I can just rescale a_i at the end of the generation to satisfy it." CreationDate="2011-03-12T01:43:21.940" UserId="3673" />
  <row Id="13487" PostId="8124" Text="@tsukiko - one possible way to get the poisson is perhaps to assume positive correlation between different trials.  This will not change the mean, but will increase the variance (so get closer to Var(X)=E(X)).  So a &quot;poisson&quot; approximation to that problem may be good.  But again, it may be easier to work directly with correlated bernoulli trials, by specifying a correlated structure for the $Y_{i}$ variables, rather than an independent one." CreationDate="2011-03-12T01:58:20.603" UserId="2392" />
  <row Id="13488" PostId="8124" Text="@tsukiko - you could incorporate this correlation into your bootstrap procedure by simply placing a higher re-sampling probability for results already sampled.  This amount of &quot;increase&quot; is indicative of how much correlation is though to be present." CreationDate="2011-03-12T02:01:27.603" UserId="2392" />
  <row Id="13489" PostId="8004" Text="@mpiktas: &quot;It is shown that for large part of stationary processes the central limit theorem can be applied, so classical regression analysis can be applied.&quot; -- For what small part of stationary processes can the CLT *not* be applied (so that classical regression can't be applied)?" CreationDate="2011-03-12T02:03:39.500" UserId="1106" />
  <row Id="13490" PostId="8182" Text="Have you tried princomp function?" CreationDate="2011-03-12T03:04:45.973" UserId="1808" />
  <row Id="13491" PostId="8187" Score="2" Text="Before you get into the statistical side of things, you need to wrestle with the design side--and validity.  How do we know these states didn't have radically different rates of the disease before either of the two treatments were begun?  There's a conflict between your desire to &quot;prove&quot; (strong word!) the superiority of State 1's treatment and your acknowledgment that there are confounding variables present. " CreationDate="2011-03-12T03:36:57.613" UserId="2669" />
  <row Id="13492" PostId="8187" Score="2" Text="One designs an experiment *before* collecting data, not after!" CreationDate="2011-03-12T04:02:52.987" UserId="919" />
  <row Id="13493" PostId="8004" Text="@raegtin, stationary processes which do not have second moments for example." CreationDate="2011-03-12T05:20:46.467" UserId="2116" />
  <row Id="13494" PostId="8180" Text="Also see this question on post hoc tests for ancova: http://stats.stackexchange.com/questions/2469/post-hoc-tests-in-ancova" CreationDate="2011-03-12T05:41:45.390" UserId="183" />
  <row Id="13495" PostId="8149" Score="2" Text="Great answer. However, I'd be reluctant to convert a continuous variable into a binary one. A lot of meaningful variability would be lost." CreationDate="2011-03-12T06:17:33.657" UserId="183" />
  <row Id="13496" PostId="8183" Text="(+1): Unfortunately this seems to be an orphaned project." CreationDate="2011-03-12T06:41:42.273" UserId="264" />
  <row Id="13497" PostId="8189" Text="Then what exactly is spurious about the spurious correlation between ice cream sales and swimsuit sales?" CreationDate="2011-03-12T07:37:57.590" UserId="1106" />
  <row Id="13498" PostId="8191" Score="1" Text="@Jeromy (+1) Of note, another interesting discussion on formative vs. reflective models can be found in Chapter 3 of *Measuring the Mind*, by Denny Borsboom (Cambridge, 2005)." CreationDate="2011-03-12T08:40:40.397" UserId="930" />
  <row Id="13499" PostId="8194" Score="2" Text="That appears correct. The extra question here as to why the null hypothesis is a point value not a range or inequality is more interesting! In the commonly-encountered cases at an elementary level with a single parameter of interest it always is, I think, but i believe things get a bit more complicated in the multiparameter setting. But I'm not sure so I'll leave someone better qualified to expand!" CreationDate="2011-03-12T09:09:28.283" UserId="449" />
  <row Id="13500" PostId="8187" Text="@rolando2: Thank you for the excellent comment. Permit me to clarify. Before 2004, both states had the same death rates. If I use time series there is likelihood of many unknown factors like resistance of patients changing between 2004 and 2010. So, I thought of comparing 2 neighboring states year wise." CreationDate="2011-03-12T09:21:03.890" UserId="2956" />
  <row Id="13501" PostId="8194" Text="Thanks for confirming. I will ask the additional question on a new thread since it is a different question." CreationDate="2011-03-12T09:21:42.890" UserId="3688" />
  <row Id="13502" PostId="8187" Score="1" Text="@whuber: Thank you for the very basic comment. In the medical field, in emergencies, we are forced to start managing epidemics without thinking of a study. After the epidemic is over, a cursory glance at the data shows us some important conclusion that one treatment given empirically is better than what was given in the other state. This information has to be proven statistically. Then only it will benefit other patients. Please advise me." CreationDate="2011-03-12T09:26:20.303" UserId="2956" />
  <row Id="13503" PostId="8188" Score="2" Text="Positive autocorrelation can also result from other systematic effects such as a learning process. I think it's important to remove such effects before interpreting auto-correlation as evidence of a &quot;hot streak&quot;." CreationDate="2011-03-12T09:31:11.137" UserId="183" />
  <row Id="13504" PostId="8190" Text="+1 for systematic part. I think in this case it is the best explanation of the variations in performance." CreationDate="2011-03-12T09:31:59.590" UserId="2116" />
  <row Id="13505" PostId="8199" Text="ah, didn't see robin's post!" CreationDate="2011-03-12T10:29:54.843" UserId="3485" />
  <row Id="13506" PostId="8186" Text="@Charlie, two *stationary* time series may be spuriously correlated as well. Just take simple textbook example, generate 100 independent realisations of Gaussian white noise process (very stationary process :) ), on average about 5 per cent of them will be *correlated* significantly with 5 per cent significance level. What is important - the theoretical background or deeper understanding of what's going on rather than a theoretical data-mining approach. Co-integration, Granger causality all concepts are useful, but sound model have to add some story behind the statistics it uses." CreationDate="2011-03-12T11:09:58.533" UserId="2645" />
  <row Id="13507" PostId="8195" Text="@mpiktas, I agree to the definition from wikie (+1) for this, by the way trends may be deterministic as well (not only random walks), for instance linear, so two linear trends will be indeed very correlated. Another source of spuriousness in time series (from the dirty tricks set) is the seasonal components, that are also very likely to be correlated wrongly. &quot;Correlation does not imply causation article&quot; in wikipedia is also useful. http://en.wikipedia.org/wiki/Correlation_does_not_imply_causation" CreationDate="2011-03-12T11:15:58.590" UserId="2645" />
  <row Id="13508" PostId="8195" Text="@Dmitrij, all the things you mention fall into the same category, something else is causing the relationship. That is why I cited the wikipedia." CreationDate="2011-03-12T11:45:55.043" UserId="2116" />
  <row Id="13509" PostId="8156" Score="1" Text="I'm still not clear what 'linearity of the drop-offs' means. Do you mean how how close they are to a straight line, regardless of its gradient, as I think we've been assuming? Or do you mean how close the changes in read depth are to vertical?" CreationDate="2011-03-12T12:18:08.900" UserId="449" />
  <row Id="13510" PostId="8190" Score="2" Text="might want to look at how researchers have investigated this question. A classic is Gilovich, T., Vallone, R. &amp; Tversky, A., The hot hand in basketball: On the misperception of random sequences. Cognitive Psychology 17, 295-314 (1985)." CreationDate="2011-03-12T12:31:35.603" UserId="2347" />
  <row Id="13511" PostId="8184" Text="Here's a relevant-looking ref, though I haven't read past the abstract: [Newson 1991 *Statistics in Medicine* &quot;Estimating departure from normality&quot;](http://dx.doi.org/10.1002/sim.4780100811)" CreationDate="2011-03-12T12:32:05.300" UserId="449" />
  <row Id="13512" PostId="8184" Text="Two more recent ones, both in the *Journal of Statistical Computation and Simulation*: &#xA;[Yazicia &amp; Yolacan 2007](http://dx.doi.org/10.1080/10629360600678310)&#xA;[Romão, Delgado &amp; Costa 2010](http://dx.doi.org/10.1080/00949650902740824)" CreationDate="2011-03-12T13:05:13.713" UserId="449" />
  <row Id="13513" PostId="8160" Text="@SheldonCooper: Thanks for your comment. The purpose of the score would be to determine relative 'position' of observations. In a way I'd like to be able to say what what is the standing of one particular community within the population. Initially my approach would probably avoid weights (or use data driven weights?). But it might be possible to obtain these weights from another dataset and use them later on as wellin simmilar vein to what Gordon (1995) suggests http://jech.bmj.com/content/49/Suppl_2/S39.abstract" CreationDate="2011-03-12T13:24:19.740" UserId="22" />
  <row Id="13514" PostId="8173" Text="@Aniko: Thanks a alot. Looks good - will definitely give it a go!" CreationDate="2011-03-12T13:25:43.513" UserId="22" />
  <row Id="13515" PostId="8197" Text="Perhaps it would be a better idea use the slope (before and after), to accommodate the fact that there may be a trend?" CreationDate="2011-03-12T13:28:50.383" UserId="3182" />
  <row Id="13516" PostId="8177" Text="I'll look into it, thanks! There are a few other indices and I think some of them have been used in analysis over time, so there may be other datasets that don't exhibit the problem you've highlighted." CreationDate="2011-03-12T13:31:38.010" UserId="3182" />
  <row Id="13517" PostId="8198" Text="+1, great answer" CreationDate="2011-03-12T13:38:05.197" UserId="2116" />
  <row Id="13518" PostId="8191" Text="@Jeromy: Many thanks! Very helpful answer!" CreationDate="2011-03-12T13:38:07.857" UserId="22" />
  <row Id="13519" PostId="8173" Text="@Aniko: Just one question. I understand how to generate score for one variable depending on relative position of an observation. What is the next step for combining the scores of five variables? Simple sum?" CreationDate="2011-03-12T14:03:57.553" UserId="22" />
  <row Id="13520" PostId="5131" Text="For me, he is the best statistical thinker ever.  And also one of the most interesting and distinctive writing styles I've seen.  Never seen anyone use the word &quot;common sense&quot; like he does!" CreationDate="2011-03-12T14:29:27.903" UserId="2392" />
  <row Id="13521" PostId="8184" Text="Wouldn't the likelihood ratio test give you the most powerful test by the Neyman-Pearson Lemma?  My impression has been that all of these &quot;other&quot; tests are merely easily computable approximations of the LRT (possibly with constraints on the alternative.)" CreationDate="2011-03-12T15:08:56.013" UserId="3567" />
  <row Id="13522" PostId="8198" Text="In fact, this applies to the alternative hypothesis as well." CreationDate="2011-03-12T15:10:46.153" UserId="3567" />
  <row Id="13523" PostId="8169" Text="Thank you Sheldon. @Question: (+1) after mbq's edit." CreationDate="2011-03-12T15:23:25.550" UserId="264" />
  <row Id="13524" PostId="8186" Text="Thanks, Dmitrij, of course some of the series will be falsely correlated by random chance, but that's not how I typically use the &quot;spurious correlation&quot; term; I define this term as correlation resulting from the series each containing a time trend. And I also distinguish the cross-section variant as &quot;omitted variables bias,&quot; though I do see spurious correlation (as I define it) as a special case of omitted variables bias. But I guess that's how the terms are used in economics." CreationDate="2011-03-12T15:45:25.560" UserId="401" />
  <row Id="13525" PostId="8130" Text="So if i have to give more parameters here you go: 1. **test to use** -- no idea -- do you have suggestions? 2. **power**: even after looking at the wikipedia definition - I do not know how to intelligently answer that. 3. **effect size**: Lets say 10% better" CreationDate="2011-03-12T15:55:01.763" UserId="3658" />
  <row Id="13526" PostId="8184" Score="2" Text="@Christopher, if you want to even have a notion of &quot;most powerful&quot; you need to specify a class of alternatives. Is it the class of all distributions $F$ on the real line? All of those with a density $f$ with respect to Lebesgue measure? All elliptical distributions? Etc. The Anderson--Darling (AD) test is, of course, more powerful in general than the KS test because AD can be viewed as weighting the tail observations much higher. So, if you get any unexpected deviations from normality in the tails, the AD is going to pick that up immediately." CreationDate="2011-03-12T16:00:27.453" UserId="2970" />
  <row Id="13527" PostId="8184" Score="1" Text="@charles.y.zheng, the Neyman--Pearson lemma is for comparing two point hypotheses. So, a LRT would only be (guaranteed to be) most powerful when using a *fixed* set of parameters for the normal distribution and testing against another *fixed* distribution as the alternative. I'm guessing that's not the situation of interest here." CreationDate="2011-03-12T16:02:24.720" UserId="2970" />
  <row Id="13530" PostId="8184" Text="@charles.y.zheng, please define &quot;empirical variance&quot;. The MLE of the variance of a normal is *never* equal to the (unbiased) sample variance." CreationDate="2011-03-12T16:10:05.697" UserId="2970" />
  <row Id="13531" PostId="8184" Text="@cardinal:  After some thinking I realized my mistake.  The LRT should still be the most powerful but you have to constrain the alternative for it to even make sense." CreationDate="2011-03-12T16:10:55.150" UserId="3567" />
  <row Id="13532" PostId="8184" Text="@cardinal: the MLE of the variance for the normal distribution is not an issue.  It is the sum-of-squares deviation-from-the-mean of the data divided by sample size." CreationDate="2011-03-12T16:13:09.343" UserId="3567" />
  <row Id="13533" PostId="8194" Text="This suggests you are looking for a one-tailed test.  But you might also be interested if the population correlation coefficient $\rho &lt; -0.5$.  One approach might be to consider testing instead whether $\rho^2 &gt; 0.25$." CreationDate="2011-03-12T16:16:25.770" UserId="2958" />
  <row Id="13534" PostId="8184" Score="1" Text="@charles.y.zheng, to repeat, generalized LRT (formed from considering maximization over compositive null and alternative parameter spaces) are **not** in general most powerful tests. In some cases you can get uniformly most powerful tests, which are essentially what you seem to be getting at. But those are few and far between and almost exclusively restricted to exponential families. Also UMP tests almost never exist when there are nuisance parameters present." CreationDate="2011-03-12T16:16:42.040" UserId="2970" />
  <row Id="13535" PostId="8185" Score="1" Text="Firstly notice that, in time series context you actually are dealing with *spurious relationships* (if the latter are linear than *spurious regressions*) rather than *spurious correlations*. What you will typically see in time-series analysis are auto-correlations, that are obviously not spurious. Well all your questions are actually covered by 3 responses below. By the way even more sound Granger (non)causality concept does not imply theory based causation." CreationDate="2011-03-12T16:17:08.180" UserId="2645" />
  <row Id="13536" PostId="8184" Text="@cardinal: Thanks for the clarification." CreationDate="2011-03-12T16:18:38.467" UserId="3567" />
  <row Id="13537" PostId="8189" Text="@raegtin, seasonal component that could be described by the weather conditions, temperature for instance. " CreationDate="2011-03-12T16:19:10.037" UserId="2645" />
  <row Id="13539" PostId="8186" Text="@Charlie, I just re-read your answer and cross-reference to other question and think that all 3 answers here are jointly right (+1) for this one." CreationDate="2011-03-12T16:38:26.260" UserId="2645" />
  <row Id="13540" PostId="8201" Text="@charles.y.zheng, I don't think the replication idea is appropriate. Your function $f$ is a function of the &quot;order statistics&quot; of your input $X_i$. So, why not sort the coordinates of $X_i$ and fit your $f$ that way? (Of course you're formally fitting a different function $g$ in this case, but there is a simple mapping in terms of compositions in which to define $f$ given $g$ and $X_i$.)" CreationDate="2011-03-12T16:56:34.313" UserId="2970" />
  <row Id="13542" PostId="8201" Text="@cardinal: yes, but this has the effect of favoring certain functions $f$ over others.  And since in the case of complex numbers, the ordering is not unique, it seems like this might not be the most desirable solution." CreationDate="2011-03-12T17:15:54.267" UserId="3567" />
  <row Id="13543" PostId="8201" Text="Sorry, I missed the comment on complex-valued entries. In the real-valued case, my previous comment stands and should not &quot;favor&quot; any admissible $f$ over another." CreationDate="2011-03-12T17:21:17.180" UserId="2970" />
  <row Id="13544" PostId="8205" Score="1" Text="Apparently I know enough that I am beginning to forget some.  Thanks for the reply, in any case!" CreationDate="2011-03-12T17:40:18.973" UserId="3567" />
  <row Id="13546" PostId="8177" Text="Oh plenty of people have used the CPI to make comparisons over time. But it's a terrible idea without accounting for 1) changes in the surveys used to make the composite index and 2) changes in the methodology of those surveys. Like you say I would imagine there are indices out there which try to make their measurements comparable over time, but TI doesn't." CreationDate="2011-03-12T18:33:05.017" UserId="26" />
  <row Id="13547" PostId="8187" Score="1" Text="I don't really understand which state is supposed to be superior: rolando2's comment says &quot;State 1&quot;, but the original question says &quot;State 2&quot;." CreationDate="2011-03-12T18:35:02.893" UserId="3582" />
  <row Id="13548" PostId="8195" Text="So would you say that &quot;how sweet a dessert tastes&quot; has a spurious relationship with &quot;number of calories in the dessert&quot; (assuming for the sake of example that sugar is the hidden variable influencing both sweetness and calories)? Also: we all know that correlation does not imply causation. But what about non-spurious correlation? Is *non-spurious* correlation meant to imply causation (assuming *in theory* that you've considered every possible hidden variable -- obviously you couldn't really check every possible hidden variable in real life)?" CreationDate="2011-03-12T18:39:58.340" UserId="1106" />
  <row Id="13549" PostId="8195" Text="@raegtin, yes in your example the relationship is spurious. But that does not mean that it is not useful. If the sugar is the true hidden variable and the relationship between sweetness and number of calories is linear, you can fit the regression and use it to determine the sweetness depending on the calories (note that this though takes this example too far), but you cannot say that the cause of the sweetness is the number of calories. In the time series context though the spurious relationship has worse consequences, since the usual regression does not make sense." CreationDate="2011-03-12T18:55:03.623" UserId="2116" />
  <row Id="13550" PostId="8195" Text="@raegtin, the non-spurious correlation with your given definition might imply the causation, but it does not matter, since there is no way you can calculate it. See this [question](http://stats.stackexchange.com/questions/7376/does-correlation-assume-stationarity-of-data/7378#7378) and the answers below it, I think it might be of interest." CreationDate="2011-03-12T19:01:35.850" UserId="2116" />
  <row Id="13551" PostId="8201" Text="+1 for the example of how to cross-post properly and very interesting question. When do these type of problems arise? Physics?" CreationDate="2011-03-12T19:22:59.200" UserId="2116" />
  <row Id="13552" PostId="8201" Text="@mpiktas: In my case, network analysis." CreationDate="2011-03-12T20:06:45.593" UserId="3567" />
  <row Id="13553" PostId="8213" Text="A simpler question: how would you go about detecting line segments in a two dimensional data?" CreationDate="2011-03-12T20:09:54.327" UserId="3567" />
  <row Id="13554" PostId="8201" Text="@charles.y.zheng, can you give me some reference? I can perfectly imagine that functions with intricate properties are needed, but how exactly does the regression enters into this is a bit of a mystery. I am asking out of curiosity." CreationDate="2011-03-12T20:13:58.267" UserId="2116" />
  <row Id="13555" PostId="8187" Score="3" Text="@DrWho Point taken.  But you're definitely not talking about &quot;design,&quot; then, are you?  Here's another basic point that seems well worth making here: you *cannot* &quot;prove statistically,&quot; after the fact with a sample of convenience, that one treatment was better than the other.  The treatment is confounded with the state and all the ways in which the states can differ.  **All** you can do is conclude the data are &quot;suggestive&quot; of a difference." CreationDate="2011-03-12T20:25:23.813" UserId="919" />
  <row Id="13556" PostId="8201" Score="1" Text="@mpiktas: I am a newcomer to the field so I cannot give you any references.  However, it is easy to imagine where the regression might come in.  Suppose you measure the social networks of $n$ schools.  These networks take the form of adjacency matrices $X_n$.  You also know some demographic information about the schools, like average income, ethnic composition, etc.  How are those covariates related to features of the graphs?  And by the way, the eigenvalues capture a lot of the important properties of the network, such as average degree, connectivity, and clustering." CreationDate="2011-03-12T20:29:34.827" UserId="3567" />
  <row Id="13557" PostId="8213" Text="&quot;..like the ones in the examples&quot;? What examples? Can you add a link?" CreationDate="2011-03-12T20:35:09.687" UserId="449" />
  <row Id="13558" PostId="8213" Text="The Hough transform is the obvious choice.  It should work well." CreationDate="2011-03-12T20:38:20.260" UserId="919" />
  <row Id="13559" PostId="8213" Text="I just got enough reputation meanwhile to add the image example I was referring to." CreationDate="2011-03-12T20:40:42.807" UserId="3693" />
  <row Id="13560" PostId="8196" Score="1" Text="This question is essentially the same as http://stats.stackexchange.com/q/7853/919 ." CreationDate="2011-03-12T20:42:56.403" UserId="919" />
  <row Id="13561" PostId="8168" Text="@brianjd, this definitely get more answers on stackoverflow.com" CreationDate="2011-03-12T20:44:00.380" UserId="2116" />
  <row Id="13562" PostId="8198" Text="I don't think you need to specify a simple null hypothesis in the N-P framework.  The explanation appears in my reply at http://stats.stackexchange.com/questions/7853/justification-of-one-tailed-hypothesis-testing/7856#7856" CreationDate="2011-03-12T20:45:05.433" UserId="919" />
  <row Id="13563" PostId="8184" Text="@charles: NP would work if I could specify the distribution exactly, ie: $ H_0: X_i \sim \mathcal{N}(0,1) $ and $ H_A: X_i \sim \mathcal{P}(3) $. I was hoping for something that tested $H_0: X_i \sim \mathcal{N}(\mu, \sigma^2) $ against $H_A: X_i \nsim \mathcal{N}(\mu, \sigma^2)$. " CreationDate="2011-03-12T20:59:26.970" UserId="1118" />
  <row Id="13564" PostId="8213" Score="2" Text="This is not a clustering problem.  In statistics, &quot;clusters&quot; consist of sets of objects that are mutually *closer* to one another than other objects.  Closeness does not capture circularity: that's why neither K-means nor any other clustering algorithm will work.  For this reason, this question probably fits better in the image processing or GIS sites, where you might find some experts on this issue." CreationDate="2011-03-12T22:33:07.133" UserId="919" />
  <row Id="13565" PostId="8210" Score="3" Text="You're right; sometimes the results are so obvious that applying a formal test would be overkill--and actually woudl look suspicious (&quot;what's this guy trying to hide with his statistics?&quot;).  But *by themselves,* knowing only what we have been told so far, these data are **not** convincing.  For instance, maybe the treatment was *completely without effect* but there happened to be two strains of this disease in State 2 (but not in State 1) and the second strain was much more virulent.  The point is that any &quot;proof&quot; must rest on *scientific* argument, not a statistical one." CreationDate="2011-03-12T22:42:34.847" UserId="919" />
  <row Id="13568" PostId="8213" Text="You are right in saying that closeness does not capture circularity, hence that a clustering algorithm wouldn't find circles per se in this context. But if it could find distinct &quot;objects&quot;, then indirectly the problem would be solved because the assumption would be that those are really circles (because that's the only type of objects there is supposed to be in the data). So that was what I meant by presenting it as somewhat a clustering problem, but I agree it wasn't terribly clear. Is there an image processing SE site? I didn't see it, but I would've definitely posted there if I knew." CreationDate="2011-03-12T22:58:09.250" UserId="3693" />
  <row Id="13569" PostId="8156" Text="@whuber,onestop: I apologize for being imprecise. What's most concerning is when we see near-vertical dropoffs in read depth for segments that are otherwise displaying good read depth across bases.  In particular, some segments show dropoffs from regions of roughly 1000 read depth to zero for x bases.  I've studied Markov chains previously but not HMMs, which are apparently pretty widely used in bioinformatics to make inferences about the properties of certain genomic regions.  For this question though it appears best metric may be a simple normalized score reflecting bases with 0 read depth." CreationDate="2011-03-12T23:25:35.937" UserId="3672" />
  <row Id="13570" PostId="8214" Text="I would try combining all the approaches suggested: first cluster based on distance alone, as the original poster discussed, which will give you clusters that may consist of multiple circles. Then use Hough to detect planar subsets within each cluster. Then within each planar subset again use Hough to find circles. If this last step is expensive, you might be able to do effective short-circuiting: try a few triples, guess a circle, and see if a substantial fraction of the points in your subset lies very close to that circle. If so, record that circle and remove all those points, then continue." CreationDate="2011-03-13T01:55:08.700" UserId="2898" />
  <row Id="13571" PostId="8213" Text="K-means is not going to work here. You could try spectral clustering, but it seems your circles aren't sampled densely enough for even spectral clustering to work." CreationDate="2011-03-13T02:57:52.673" UserId="3369" />
  <row Id="13572" PostId="8214" Score="2" Text="This latter idea is called RANSAC and could probably be used by itself, especially if the number of circles per image is small." CreationDate="2011-03-13T02:58:48.007" UserId="3369" />
  <row Id="13573" PostId="8210" Score="1" Text="Did you mean that &quot;State 1 performs better&quot;?" CreationDate="2011-03-13T03:19:40.953" UserId="3369" />
  <row Id="13574" PostId="7666" Text="@henry - so I take it that there is no standard variance formula for this slope - only my intuition to guide me?  presumably I could use the delta method, and get $var(\hat{\beta})\approx\frac{var(s_y^2)}{E(s_x)^2}$" CreationDate="2011-03-13T03:21:43.960" UserId="2392" />
  <row Id="13575" PostId="8187" Score="1" Text="It seems there are two separate questions here. One question is, given this data, is there a statistical test that would show which state's death rate is lower. It seems like a simple t-test is sufficient, as Fr pointed out. Another question is, given that one state's death rate is lower, does this prove that their treatment is superior. If you believe the difference in death rates is entirely due to treatment, then the same t-test would be OK. Given that before 2004 death rates were similar, this might be the case. But if there are confounding variables, then death rate data is not enough." CreationDate="2011-03-13T03:27:45.767" UserId="3369" />
  <row Id="13576" PostId="8187" Score="1" Text="At the very least you will also need to know the values of the confounding variables and do some kind of ANOVA to analyze their influence. And it may happen that even though one treatment is really superior, there won't be enough evidence in the data to support that if there are too many confounding variables." CreationDate="2011-03-13T03:29:21.037" UserId="3369" />
  <row Id="13577" PostId="7567" Text="@Maurizio, were these generated randomly? Was your scale parameter $S = 0.9$ as before? I used a sample size of 8454 and a truncation point of 500 and generated 10000 such samples. Of these 10000, the maximum observed value in the sample was 500 for 9658 of the trials, 499 for 324 trials, 498 for 16 trials and 497 for 2 trials. Based on this, I think something is still wrong with your generation procedure. Unless you used a different scale parameter." CreationDate="2011-03-13T03:54:56.910" UserId="2970" />
  <row Id="13578" PostId="8210" Text="@SheldonCooper: Thank you very much for pointing out the mistake. I have rectified my mistake." CreationDate="2011-03-13T04:17:50.687" UserId="2956" />
  <row Id="13579" PostId="8210" Text="@whuber: The two states are neighboring states. People can walk in to the other state freely. The cases are investigated epidemiologically and virologically and found to be similar. Thank you very much for the excellent comments. They are of great helpto me." CreationDate="2011-03-13T04:20:22.273" UserId="2956" />
  <row Id="13581" PostId="7853" Text="A similar question was asked afterward http://stats.stackexchange.com/questions/8196/why-is-the-null-hypothesis-always-a-point-value-rather-than-a-range-in-hypothesis/8198#8198" CreationDate="2011-03-13T06:24:56.377" UserId="223" />
  <row Id="13582" PostId="8198" Text="@whuber Sorry but what is N-P framework ? " CreationDate="2011-03-13T06:30:01.880" UserId="223" />
  <row Id="13583" PostId="8187" Text="@SheldonCooper: Thank you very much for the great addition and advising ANOVA. Could you kindly clarify &quot;some kind of ANOVA&quot;. I am a beginner in statistics." CreationDate="2011-03-13T07:26:54.480" UserId="2956" />
  <row Id="13584" PostId="8187" Text="@whuber: Thank you very much for the advice to say suggestive of a difference. It would be the best and unbiased conclusion. In medical field, sometimes unexpected results come our way and once we know a particular treatment is beneficial to the patients, it would be unethical to do a case-control study. " CreationDate="2011-03-13T07:33:41.400" UserId="2956" />
  <row Id="13585" PostId="8187" Text="@Fr.: Thank you very much for pointing out my mistake made in a hurry to post the question. I have rectified the mistake" CreationDate="2011-03-13T07:35:40.587" UserId="2956" />
  <row Id="13586" PostId="8220" Text="One example is L. Fei-Fei and P. Perona. A bayesian hierarchical model for learning natural scene categories. In CVPR, 2005. But your question is unclear. Do you mean that for each document there is a single topic? Do you mean a topic corresponds to a group of documents (a class)? Or more than one topic? How do you plan to do classification? What other topics would you need?&#xA;" CreationDate="2011-03-13T08:08:14.643" UserId="3369" />
  <row Id="13587" PostId="7666" Text="@probabilityislogic: You are correct when saying there is no standard method for $\hat{\beta}$.  I am not sure what your formula for $var(\hat{\beta})$ is, but it may be better at this stage to ask a new question.   " CreationDate="2011-03-13T08:46:29.670" UserId="2958" />
  <row Id="13589" PostId="8212" Score="1" Text="@Henry Well, your response seems more close to what was asked by the OP (so I've +1). However, I feel like this isn't a boxplot anymore, or at least it loose its interest in spotting possible outlying value. Of note, you can customize a little bit more the boxplot (see the `pars` arguments to reduce its aspect ratio (`boxwex`) and the size of the whiskers (`staplewex`))." CreationDate="2011-03-13T09:08:43.613" UserId="930" />
  <row Id="13590" PostId="8198" Score="1" Text="@robin N-P = Neyman-Pearson" CreationDate="2011-03-13T09:09:19.137" UserId="449" />
  <row Id="13591" PostId="8196" Text="You're essentially right. I somehow missed that Q completely, which is why I didn't spot this when commenting on this questioner's previous Q." CreationDate="2011-03-13T09:13:40.193" UserId="449" />
  <row Id="13592" PostId="8150" Text="@richiemorrisroe The number of observations is the data I need. Updated the question with example data set and information on possible error probability. " CreationDate="2011-03-13T10:33:25.973" UserId="3376" />
  <row Id="13593" PostId="8222" Text="this should be community wiki." CreationDate="2011-03-13T11:06:32.303" UserId="264" />
  <row Id="13594" PostId="8221" Score="1" Text="@SUBHABRATA: This question maybe sound dumb, but what does &quot;parser&quot; mean in the context of learning algorithms ? I only know &quot;parser&quot; in the context of compilers and &quot;text processing. Can you provide a link ?" CreationDate="2011-03-13T11:24:35.837" UserId="264" />
  <row Id="13595" PostId="7087" Score="1" Text="@Marina Your two accounts have been merged. Please, register as explained on the [FAQ](http://stats.stackexchange.com/faq)." CreationDate="2011-03-13T11:47:17.593" UserId="930" />
  <row Id="13596" PostId="8227" Score="2" Text="@steffen +1 but there's no need for a `for` loop here, you can contruct your dataframe inline, IMO. For the `tapply` call, use `function(x) c(mean(x),sd(x)))` and `cbind` the result as the OP asked for both statistics. Also, `ddply` from the [plyr](http://had.co.nz/plyr/) package could do this smoothly." CreationDate="2011-03-13T12:24:41.290" UserId="930" />
  <row Id="13597" PostId="8226" Text="@Marco Don't link to your local help server :-) +1 but see my comments to @steffen's response." CreationDate="2011-03-13T12:26:37.863" UserId="930" />
  <row Id="13598" PostId="8227" Text="@steffen The problem is I need the exactly the table structure I described. There is no problem with getting means and sd. The problem is with stucture." CreationDate="2011-03-13T12:35:28.860" UserId="3376" />
  <row Id="13599" PostId="8227" Text="@chl: Thank you for your comment, did not know about plyr :). I added cbind, but left the rest untouched. May another one take the credit, this answer shall remain as  a less optimal example." CreationDate="2011-03-13T12:35:58.383" UserId="264" />
  <row Id="13600" PostId="8227" Text="@Yuriy: Added cbind. If you already knew how to apply functions per group, you may reformulate your question (just for clarity ;))." CreationDate="2011-03-13T12:37:47.197" UserId="264" />
  <row Id="13602" PostId="8221" Score="1" Text="@SUBHABRATA: I just checked your profile. I suggest to take care of the other questions you have asked in the past. You did not respond to a single comment-question or answer. " CreationDate="2011-03-13T12:44:15.253" UserId="264" />
  <row Id="13603" PostId="8225" Text="@Yuriy, the commas in the result data frame mean something special, or is it just the decimal point?" CreationDate="2011-03-13T12:46:04.723" UserId="2116" />
  <row Id="13604" PostId="8226" Text="@Marco Done the thing by calling `data.frame(group=levels(factor(data$group)),mean=(aggregate(data$age, by=list(data$group), FUN=mean)$x),sd=(aggregate(data$age, by=list(data$group), FUN=sd)$x))` but I am not shure it is correct way. I am not sure what will happen then the results of binded columns will be in different order (I think it is possible). What is your oppinion?" CreationDate="2011-03-13T12:46:40.693" UserId="3376" />
  <row Id="13606" PostId="8227" Text="@steffen `cbind(&quot;mean&quot;=mperage,&quot;stdev&quot;=stperage) gives no 'group' column. Will be joining by `cbind(group=levels(factor(data$group)),&quot;mean&quot;=mperage,&quot;stdev&quot;=stperage)` correct?" CreationDate="2011-03-13T12:51:09.647" UserId="3376" />
  <row Id="13607" PostId="8227" Text="@Yuriy: The row names are the group-names .... Nevertheless I added a separate column (now using data.frame) to show this." CreationDate="2011-03-13T12:51:20.330" UserId="264" />
  <row Id="13608" PostId="8227" Text="@steffen Are you sure there will be no ordering issues when joining columns?" CreationDate="2011-03-13T12:53:52.140" UserId="3376" />
  <row Id="13609" PostId="8225" Text="@mpiktas Thank you for noting. Corrected. These were locale issues (I am russian) - we we use comma for decimal separation. " CreationDate="2011-03-13T12:59:05.770" UserId="3376" />
  <row Id="13610" PostId="8225" Score="1" Text="@Yuriy, I suspected that. All of the [Europe](http://en.wikipedia.org/wiki/File:DecimalSeparator.svg) uses comma except the British. " CreationDate="2011-03-13T13:04:26.597" UserId="2116" />
  <row Id="13611" PostId="8232" Score="2" Text="@Can't Tell: I think there is no rationale behind the chi-square distribution. You can see it as a sum of N(0,1)^2. You can also characterise it via its pdf. The reason why it is introduced is that important test statistics follow that distribution, at least approximately. For instance, the likelihood ratio." CreationDate="2011-03-13T13:21:15.473" UserId="3019" />
  <row Id="13612" PostId="8227" Text="@Yuriy, tapply orders the results by the order of the levels, so there ordering issues will not arise. @steffen, I upvoted the answer for the effort, before I knew plyr, I used to solve such problems in similar way. " CreationDate="2011-03-13T13:24:59.323" UserId="2116" />
  <row Id="13613" PostId="8232" Text="@Marco: Thanks for your comment. But I'm not very good with statistics and I don't understand what you mean by sum of N(0,1)^2 or &quot;likelihood ratio&quot;. What I was looking for was something like &quot;if you take the case X, the variables Y and Z are independent standard normal random variables and if you plot the values A and B, then that would be a Chi Square distribution.&quot; Is such an example possible(with X,Y,Z,A and B)?" CreationDate="2011-03-13T13:27:49.800" UserId="3688" />
  <row Id="13614" PostId="8232" Score="2" Text="@Can't Tell. I am not sure to get what you exactly want. But take two **independent** standard normal random variables. Square them and add them. This results in a chi-square random variable whose parameter is 2. If you had had 3 normal variables, the parameter would have been 3, etc." CreationDate="2011-03-13T13:34:21.103" UserId="3019" />
  <row Id="13615" PostId="8233" Text="This set of videos looks great. I [created a question](http://stats.stackexchange.com/questions/8239/introductory-statistics-video-courses) to popularise it and see if there are any other great quality video courses on introductory statistics." CreationDate="2011-03-13T15:26:26.937" UserId="3582" />
  <row Id="13616" PostId="7567" Text="@Maurizio, to explain the results I posted, consider that $p = \mathbb{P}(X_i = 500) \approx 4.05 \times 10^{-4}$. So, in a sample size of $n = 8454$, the expected number of outcomes with the value 500 is $8454 \cdot 4.05 \cdot 10^{-4} \approx 3.43$. The probability of seeing *at least* one such outcome is $1 - (1 - 0.000405)^{8454} \approx 0.9675$. Note how closely that matches up to the simulation above." CreationDate="2011-03-13T15:28:21.990" UserId="2970" />
  <row Id="13617" PostId="8210" Score="1" Text="@whuber: I heartily agree with your argument. I just read that quote by Ronald Fisher in [another discussion](http://meta.stats.stackexchange.com/questions/750/do-we-welcome-questions-on-data-collection-methods): &quot;To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of&quot;." CreationDate="2011-03-13T15:33:03.623" UserId="3582" />
  <row Id="13619" PostId="8227" Text="@steffen Try to replace your last three lines of code with: `t(rbind.data.frame(group=1:5, with(dat, tapply(age, group, function(x) c(mean(x), sd(x))))))` (+ rename the last two columns as `mean` and `sd`)." CreationDate="2011-03-13T15:36:16.773" UserId="930" />
  <row Id="13620" PostId="7470" Text="Agresti and Finlay mention that generalizing Fisher's exact test also makes sense when cell counts fall below 5 in any contingency table." CreationDate="2011-03-13T15:37:45.803" UserId="3582" />
  <row Id="13621" PostId="8069" Score="1" Text="The point made by chl is important: *any* p-value below your alpha level of *statistical* significance is in the safe area to reject the null hypothesis, and further distance of the alpha level will *not* indicate further *substantive* significance. This applies to the formal model behind your scatterplot and its fitted regression line, just like it will apply if you detect a non-linear relationship, either visually or through formal diagnostics, and then switch to more advanced techniques in order to model it." CreationDate="2011-03-13T15:40:44.227" UserId="3582" />
  <row Id="13622" PostId="8239" Score="1" Text="@Fr. Did you look at this question, [Statistics/Probability Videos for Beginners](http://stats.stackexchange.com/questions/1761/statistics-probability-videos-for-beginners), and the one that is linked to it?" CreationDate="2011-03-13T15:41:33.680" UserId="930" />
  <row Id="13623" PostId="8239" Text="@chl: No, I admit I had missed the [answer](http://stats.stackexchange.com/questions/1761/statistics-probability-videos-for-beginners/1763#1763) that already mentioned the Khan Academy material. But the other links are kind of unsatisfying to me because they seem recommended for specific audiences (life sciences, statisticians, Excel and R users) that deviate too far from mine. What I really like in the Khan Academy is the 'generic fit' of the lectures, and that is what I am now looking for elsewhere." CreationDate="2011-03-13T15:47:42.067" UserId="3582" />
  <row Id="13624" PostId="8235" Score="1" Text="@Yuriy Looks nice. How about outliers?! Your whiskers show the min/max, but in case there're some outliers they should show Q1/Q3 ± 1.5 x IQR, instead (at least, for a Tukey bxp). You can get outlying values (according to *this* definition) by using `boxplot.stats` (see the vector named `out`), as well as the five numbers btw, and add them with a `geom_point`. Otherwise, +1." CreationDate="2011-03-13T15:51:43.513" UserId="930" />
  <row Id="13625" PostId="6555" Text="Accepted because it noticed the main flaw in the question, messing up with FV and PV, I feel the non-reinventing-wheel-faster-way still open. More in my reply." CreationDate="2011-03-13T16:28:07.737" UserId="2914" />
  <row Id="13627" PostId="7946" Text="@Onestop You're completely right; I mean to write &quot;Pearson.&quot;  Normally I leave erroneous comments up in case others would like to learn from my mistakes, but this one is so blatant I'll just re-post it.  Thanks for the catch." CreationDate="2011-03-13T17:55:35.850" UserId="919" />
  <row Id="13628" PostId="7946" Text="	&#xA;@mpiktas You're probably correct in recommending a rank correlation coefficient but, IMHO, not for the reason given. After all, Pearson (product-moment) correlations are computable and interpretable with discrete data. (Ultimately all data are &quot;discrete&quot; anyway when represented numerically.) Rank correlations won't resolve the many ties in values, either. Your request for more details is spot on: the purpose of the study (and the nature of the data generation process) are critical determinants of the statistical (and graphical) procedures to use for data analysis." CreationDate="2011-03-13T17:56:10.687" UserId="919" />
  <row Id="13629" PostId="8213" Text="@cjauvin I thought an image processing site existed because I saw references to it on the Photography site.  It looks like it's only in the proposal stage, though: http://area51.stackexchange.com/proposals/1691/signal-image-video-processing  I apologize for misleading you." CreationDate="2011-03-13T18:00:14.753" UserId="919" />
  <row Id="13630" PostId="8243" Score="4" Text="Because $\lambda$ and $\mu$ are Lagrange multipliers, it's trivial to find derivatives with respect to them.  How do you anticipate the functional form of $L$ changing so that the derivatives with respect to the $w_i$ -- which currently are dead simple to compute -- will become complicated? // Anyway, any symbolic mathematics program should be able to do the job well: MatLab, Mathematica, etc." CreationDate="2011-03-13T18:14:23.993" UserId="919" />
  <row Id="13631" PostId="8214" Text="Thanks for the illuminating ideas! The multi-step Hough transform seems to me the most powerful and general solution, but RANSAC really looks easier to implement, and may be just enough in my context. One problem I rapidly noticed with it though is the case where you have patterns of unbalanced sizes, which obviously biases the sampling toward bigger objects. Any thoughts about this problem?" CreationDate="2011-03-13T19:08:19.727" UserId="3693" />
  <row Id="13632" PostId="7946" Text="@whuber, I must admit that my recommendation for using Spearman correlation came from some old drill: if data is discrete, use Spearman correlation. After reading the wikipedia page I now have trouble remembering why this particular advice stuck in my mind. My main intention in writing comment was to ask for more information and throw something alternative for the OP to think about. I must be careful what I throw." CreationDate="2011-03-13T19:16:11.133" UserId="2116" />
  <row Id="13633" PostId="8235" Score="1" Text="@chl Thank you for noting! Added outliers and correction of interval. Code maybe is not so beautiful, but works well. " CreationDate="2011-03-13T19:24:39.400" UserId="3376" />
  <row Id="13634" PostId="8242" Score="4" Text="You are looking for [sparklines](http://en.wikipedia.org/wiki/Sparkline). Searching in R for `??sparkline` shows that the packages `sparkTable` and `YaleToolkit` could be helpful." CreationDate="2011-03-13T19:26:29.033" UserId="307" />
  <row Id="13635" PostId="8242" Text="Looking at `YaleToolkit`, appears they are using the `grid` package to create stacked sparklines. Perhaps I can use `grid` to create the table. " CreationDate="2011-03-13T19:52:20.650" UserId="812" />
  <row Id="13636" PostId="8235" Text="@Yuriy (+1) I didn't manage to do better myself (i.e., alternating plotting layers -- not very pretty, but works the way we want)" CreationDate="2011-03-13T20:03:08.290" UserId="930" />
  <row Id="13637" PostId="8239" Text="community wiki? " CreationDate="2011-03-13T20:16:34.113" UserId="223" />
  <row Id="13638" PostId="8237" Text="@Can'tTell You can find some help about formatting here: http://stats.stackexchange.com/editing-help" CreationDate="2011-03-13T20:23:28.453" UserId="88" />
  <row Id="13639" PostId="8246" Text="Is the performance measured with the training dataset or with a held-out dataset?" CreationDate="2011-03-13T20:50:28.460" UserId="919" />
  <row Id="13640" PostId="8214" Text="Once you detect the larger circle, remove all points that belong to it from sampling." CreationDate="2011-03-13T21:06:07.230" UserId="3369" />
  <row Id="13641" PostId="8246" Text="Measured both; trend holds true for both." CreationDate="2011-03-13T21:06:47.907" UserId="3595" />
  <row Id="13642" PostId="8246" Score="1" Text="If you only do marginal calibration, It might be that before tranforming the data quadratic separation holds and after transforming the data it does not hold anymore." CreationDate="2011-03-13T21:45:18.237" UserId="223" />
  <row Id="13643" PostId="8244" Text="Excellent, thanks!" CreationDate="2011-03-13T22:16:27.293" UserId="3582" />
  <row Id="13644" PostId="8246" Text="I did not get how you have dealt with the covariance of the features." CreationDate="2011-03-13T22:41:54.957" UserId="1351" />
  <row Id="13645" PostId="8246" Text="Please show us a Q-Q plot, as well as mean, variance, skew and kurtosis of the transformed data for both labels." CreationDate="2011-03-13T22:53:08.500" UserId="1351" />
  <row Id="13646" PostId="8246" Text="Moreover, you might not get uniform distribution before taking the inverse multivariate normal cdf. See [this post](http://stats.stackexchange.com/questions/5534/transforming-arbitrary-distributions-to-distributions-on-0-1): there could be jumps." CreationDate="2011-03-13T22:57:55.950" UserId="1351" />
  <row Id="13647" PostId="8210" Text="@Fr.: Thank you very much for the remark. If post-mortem examination is not done, many important things would be missed. That is the only way we may be able to find the truth. Forensic Medicine is a challenging field in Medicine." CreationDate="2011-03-13T23:31:14.250" UserId="2956" />
  <row Id="13648" PostId="8239" Score="1" Text="The author, of course, is Salman Khan." CreationDate="2011-03-14T00:12:52.313" UserId="2456" />
  <row Id="13649" PostId="8247" Text="@chl - First of all, nice answer! This may warrant it's own question so let me know...but the descriptions I have read about ANOVA tables for regression models typically refer to three rows in the table: predictors, errors, and total. However, the `anova()` function in R returns an individual row for each predictor in the model. For instance, `anova(lm0)` above returns a row for `V1`, `V2`, and `Residuals` (and no total). As such, we get two F* statistics for this model. How does this change the interpretation of the F* statistic reported in the ANOVA table? " CreationDate="2011-03-14T00:54:46.533" UserId="696" />
  <row Id="13650" PostId="8241" Text="+1 As you see, it can get really messy in higher dimensions.  I think even your computer algebra system will choke once $d$ gets larger than 4 or 5--if it handle this problem in full generality at all." CreationDate="2011-03-14T04:45:00.833" UserId="919" />
  <row Id="13651" PostId="8240" Score="1" Text="Do you need the expected distance?  Its distribution?  The first few moments?" CreationDate="2011-03-14T04:46:18.353" UserId="919" />
  <row Id="13652" PostId="8232" Score="1" Text="@Can't What does it mean to &quot;understand&quot; a distribution?  As it stands, this question seems rather personal and idiosyncratic in that regard.  Could you rephrase it so that it might be meaningful to other people?" CreationDate="2011-03-14T04:50:10.673" UserId="919" />
  <row Id="13653" PostId="8236" Text="Is this a raster dataset, such as a (processed) remotely sensed image, or is it an irregular dataset?" CreationDate="2011-03-14T04:58:51.040" UserId="919" />
  <row Id="13654" PostId="8245" Text="Thanks! This is perfect!" CreationDate="2011-03-14T05:58:25.780" UserId="812" />
  <row Id="13655" PostId="8232" Text="@whuber: I edited the question. Do you think it is okay now?" CreationDate="2011-03-14T06:55:03.897" UserId="3688" />
  <row Id="13656" PostId="8250" Text="@Gilead: yes, the restriction $\sum_{i=1}^{n}w_{i}= 1$." CreationDate="2011-03-14T08:36:04.613" UserId="2914" />
  <row Id="13657" PostId="8252" Text="thanks for the answer, and the reference link. However I can't see how this model applies to the problem at hand. given that the data is comparative expression values (read: abundance) how can one define $a_i$,  $b_j$ and $e_{i,j}$??" CreationDate="2011-03-14T08:55:10.947" UserId="3014" />
  <row Id="13658" PostId="8247" Text="@Chase Yes, the ANOVA Table I have in mind are arranged in this way too. Feel free to ask the question; I'd love to hear what other users think of that. I generally use `anova()` for GLM comparison. When applied to an `lm` or `aov` object, it displays separate effects (SS) for each term in the model and doesn't show TSS. (I used to apply this the other way around, namely after fitting an ANOVA with `aov()`, I can use `summary.lm()` to get an idea of treatment contrasts.) However, there're subtle issues between `summary.lm()` and `summary.aov()`, especially related to sequential fitting." CreationDate="2011-03-14T09:11:21.077" UserId="930" />
  <row Id="13659" PostId="8251" Text="@Posdef. Can you pls elaborate on what kind of data are you/they looking at? I think judging by what you have quoted -- to me -- the method seems very ad hoc." CreationDate="2011-03-14T09:11:23.040" UserId="1307" />
  <row Id="13660" PostId="8253" Text="@mpiktas: which readily available package you mean to solve the problem?" CreationDate="2011-03-14T09:40:24.580" UserId="2914" />
  <row Id="13661" PostId="8253" Text="@hhh, **quadprog**, since the problem stated is quadratic optimisation problem." CreationDate="2011-03-14T09:41:44.943" UserId="2116" />
  <row Id="13662" PostId="8251" Text="@suncoolsu: it's microarray data, if you are familiar with the concept. If not, I could perhaps summarize it as; which genes are expressed, to what extent in the studied samples. Here's a better explanation: http://en.wikipedia.org/wiki/Gene_expression_profiling" CreationDate="2011-03-14T09:50:11.117" UserId="3014" />
  <row Id="13663" PostId="8251" Text="@suncoolsu Almost definitely Gene Expression Analysis data." CreationDate="2011-03-14T09:51:10.353" UserId="3704" />
  <row Id="13664" PostId="8255" Text="Right on the money. Thanks, chl." CreationDate="2011-03-14T09:57:02.423" UserId="3671" />
  <row Id="13665" PostId="8243" Text="@whuber: as far as I know, MATLAB is not symbolic. Perhaps you meant Maple? " CreationDate="2011-03-14T10:00:31.390" UserId="3014" />
  <row Id="13666" PostId="8229" Text="its nice, but somewhat tricky to export to LaTeX IME." CreationDate="2011-03-14T10:08:56.933" UserId="656" />
  <row Id="13667" PostId="8251" Text="@posdef. Ok - I wasn't sure, next-gen sequencing is also getting popular." CreationDate="2011-03-14T10:38:15.867" UserId="1307" />
  <row Id="13668" PostId="8257" Text="@kriegar: Thanks for the updates on your answer, I think I am starting to get an idea now. So if I understand correctly, the median polishing is used to assess the technical variability with regards to the probe and the chip? ... before the experiment is summed up to 1 matrix holding expression values for genes under different conditions?" CreationDate="2011-03-14T11:06:54.483" UserId="3014" />
  <row Id="13669" PostId="8247" Text="@Chase I just rediscovered this very nice response from @Gavin about the [Interpretation of R's lm() output](http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output)." CreationDate="2011-03-14T11:35:52.517" UserId="930" />
  <row Id="13670" PostId="8257" Text="@posdef from my understanding yes. For each probeset on a chip (probes of the same sequence) there are probes scattered throughout. http://plmimagegallery.bmbolstad.com/ for some pseudo images of chips. In addition to the variability within a single chip, there is variability between chips. Because of the technical variability, algorithms are run on the raw intensity values to obtain a single &quot;expression value&quot; for the probeset. The matrix of these values are then fit to determine if the genes are differentially expressed under different conditions." CreationDate="2011-03-14T11:46:37.203" UserId="3704" />
  <row Id="13671" PostId="8259" Text="@suncoolsu +1 This is a much better answer than my attempt. Thanks." CreationDate="2011-03-14T11:53:27.870" UserId="3704" />
  <row Id="13672" PostId="8259" Text="@posdef. I am wondering if there was any statistician involved in the statistical analysis of the paper." CreationDate="2011-03-14T12:08:36.623" UserId="1307" />
  <row Id="13673" PostId="8252" Text="If instead you take an abundance model like $n_{i,j} = n_i \,q_j + e_{i,j}$ or one like $log(n_{i,j}) = log(n) + log(p_i) + log(q_j) + e_{i,j}$ then you can do essentially the same thing, making the the median of each row and of each column of the residuals table equal to 0.  " CreationDate="2011-03-14T12:10:31.630" UserId="2958" />
  <row Id="13674" PostId="8225" Text="Despite not being British, I prefer dot for decimal separator." CreationDate="2011-03-14T12:17:54.077" UserId="144" />
  <row Id="13676" PostId="8124" Text="@probabilityislogic Thanks for your great comments. I agree with you about incorporating correlation into the bootstrap procedure. I'll work it out. " CreationDate="2011-03-14T12:31:50.380" UserId="3638" />
  <row Id="13677" PostId="8259" Text="@suncoolsu: thanks for your thorough reply. I think the fact that this is a pre-processing step is not well explained (or just assumed to be well known) in the paper. Speaking of which, the paper is published in 2000 (in Nature) so I presume they had at least some statistician look at their methods, if not involved in writing. But of course I can only speculate.. :) " CreationDate="2011-03-14T12:34:37.053" UserId="3014" />
  <row Id="13678" PostId="8263" Text="@richiemorrisroe: Thank you very much for the quick response! Could you briefly elaborate what the consequences of such an &quot;disaster&quot; would be? A PCA could be done on such a matrix anyway, some hints about the potential problems will be welcome." CreationDate="2011-03-14T12:50:04.447" UserId="3467" />
  <row Id="13679" PostId="8259" Text="@posdef. Ok- cool answers a lot of questions. 2000 was the time when people were still figuring out how to analyze microarray data. FDR wasn't fancy back then :-)" CreationDate="2011-03-14T12:56:33.167" UserId="1307" />
  <row Id="13680" PostId="8264" Text="@ttnphns: Thank you for the answer! Having mentioned the correlation matrix, now the n/m problem is clearer, but how do you define a &quot;reliable result&quot;? If the PCA has been performed in order to simplify and inspect the data structure, how can one notice that the results are not reliable?" CreationDate="2011-03-14T13:01:10.907" UserId="3467" />
  <row Id="13682" PostId="8260" Text="@ttnphns. I just deleted my answer to reduce unnecessary noise. Thanks for your comment. " CreationDate="2011-03-14T13:11:59.377" UserId="1307" />
  <row Id="13683" PostId="7567" Text="@cardinal, i also think that there is something &quot;wrong&quot; in the generation procedure (my goal is to validate that this generator really follows the Zipf distribution). I have to speak with the designers of the project in these days. " CreationDate="2011-03-14T13:15:15.977" UserId="3342" />
  <row Id="13684" PostId="7567" Text="@cardinal, But i have a question about your example. If I perform the chi-square test (as you have suggested to me in your posts) on a sample data generated with your R code i obtain an high value result that lead me to reject the hypothesis that the data sample is following a zipf distribution. In particular i used SCALE = 2, SIZE = 300000, TRUNCATION = 500. I'll update this answer with the chi-square results!" CreationDate="2011-03-14T13:16:33.213" UserId="3342" />
  <row Id="13685" PostId="8264" Score="1" Text="Mentioning &quot;relible&quot; simply nods towards &quot;representative sample&quot; theme. Mathematically one has error-free regression line with 2 variables and 2 points (individuals). But what if one of the latter two is outlier. No, let us have as many individuals as we can collect, to balance atypical deviates. With your correlation matrix, you just will want to be sure that high correlations in it are significant (reliable)." CreationDate="2011-03-14T13:26:27.603" UserId="3277" />
  <row Id="13686" PostId="8263" Text="@richiemorrisroe: As I have already received some hints from @ttnphns concerning the reliability problems that might occur, could you point me at some references that treat the problem and suggest the ration you mentioned?" CreationDate="2011-03-14T14:07:11.043" UserId="3467" />
  <row Id="13687" PostId="8232" Text="@Can't Much better!  I hope it brings in helpful replies." CreationDate="2011-03-14T14:55:59.180" UserId="919" />
  <row Id="13689" PostId="8243" Score="1" Text="@posdef Matlab performs symbolic differentiation: http://www.mathworks.com/help/toolbox/symbolic/brvfu8o-1.html#brvfxct" CreationDate="2011-03-14T14:59:12.750" UserId="919" />
  <row Id="13690" PostId="8253" Text="@mpiktas It's nice to see how R can do the symbolic derivatives.  But seeing how it takes an expert like yourself 2-3 pages of code to set up and test the operations, and seeing how unreadable it still is (relative to a truly symbolic environment like *Mathematica*) is a convincing demonstration of why one would not prefer R for such purposes." CreationDate="2011-03-14T15:01:53.247" UserId="919" />
  <row Id="13691" PostId="8243" Text="@whuber: cool, I didn't know that. thanks :)" CreationDate="2011-03-14T15:04:56.067" UserId="3014" />
  <row Id="13692" PostId="8268" Text="Thanks for your response, am I correct in summarising? If I don't care about the reason for prediction accuracy I could introduce multiple inputs which are various functions of other inputs? But if I want to explain what input factors influence an output prediction then multicollinearity is very important? Does this hold up for all classifier methods?" CreationDate="2011-03-14T15:18:18.130" UserId="3484" />
  <row Id="13693" PostId="8252" Text="@Henry What information is &quot;thrown out&quot; with median polish when there are no &quot;outliers&quot; (and what exactly do you mean by &quot;outlier&quot; anyway)?  After all, you can reconstruct the data *exactly* by means of the grand median, the row and column medians, and the residuals, all of which constitute the output of median polish.  If you mean the residuals are discarded, then in what sense is &quot;mean polish&quot; (equivalent to OLS) any different in this regard?" CreationDate="2011-03-14T15:21:09.797" UserId="919" />
  <row Id="13694" PostId="8266" Text="mbq - thank you for editing my formulae. Are you able to direct me on how to do this properly for future posts? I'm used to using [code] ... [/code] but couldn't find the proper method here. Thank you!" CreationDate="2011-03-14T15:37:30.540" UserId="3484" />
  <row Id="13695" PostId="8263" Score="2" Text="@ils well, i know that R will not allow you to fit this model. I got the numbers from Tabachnick &amp; Fidell, Understanding Multivariate Statistics, and a closer look at these assumptions from MacCallum et al http://notendur.hi.is/adg11/Proffraedi/Fabriger__Wegener__MacCallum_og_Strahan_(1999).pdf" CreationDate="2011-03-14T15:37:48.210" UserId="656" />
  <row Id="13697" PostId="8243" Text="@posdef And there's [sympy](http://code.google.com/p/sympy/) for Python." CreationDate="2011-03-14T15:41:18.087" UserId="930" />
  <row Id="13698" PostId="557" Text="+1 for a nice clear answer.  Paragraph 3 is essentially what I was taught as a biology undergraduate, with emphasis laid on the ease of combining continuous and categorical independent variables in an ANOVA framework.  " CreationDate="2011-03-14T15:43:11.747" UserId="266" />
  <row Id="13699" PostId="8270" Text="Why not a rotation matrix? It looks to me as a specific case of rotation though not orthogonal matrix. Consider factor analysis. We extract loadings A and oblimin-rotate the factors (axes) into oblique factors: S=A*Q where S is factor structure and Q is rotation matrix (with SS in its columns=1). Now, corresponding pattern matrix P=S*inv(Q'*Q)=A*inv(Q'), and so A=P*Q' which is backward, oblique-to-orto, rotation. The example matrix I posted was actually Q' rotating factor pattern matrix into factor loadings matrix. But my question was about whether sign of det of Q or Q' signals of reflection" CreationDate="2011-03-14T15:47:45.530" UserId="3277" />
  <row Id="13700" PostId="8269" Score="1" Text="In this case, I don't think that using `&amp;&amp;` makes sense, see `?&quot;&amp;&quot;` and [R: Logical operators: Beware of the difference between “&amp;&amp;” and “&amp;”](http://promberger.info/linux/2008/09/12/r-logical-operators-beware-of-the-difference-between-and/) ." CreationDate="2011-03-14T15:54:31.313" UserId="307" />
  <row Id="13702" PostId="8252" Text="@whuber: The residuals are kept in both cases. The mean polish takes account of how far away observations are from the centre (in a sense, it balances the weights of the residuals) while the median polish only looks at whether they are above or below the centre (in a sense, it balances the numbers of residuals).  So the weight information is unused when using the median as the centre; this can be good when some of the substantial weights/residuals are so dubious that the result for the centre cannot be trusted, but involves not using information if not.  " CreationDate="2011-03-14T15:56:28.830" UserId="2958" />
  <row Id="13703" PostId="8252" Text="@Henry If you can recover all the original data from the polish, then how is &quot;information&quot; not &quot;used&quot;?  BTW, median polish does not behave as you seem to describe: its residuals are the differences in *values*, not in ranks, of the data." CreationDate="2011-03-14T15:59:46.190" UserId="919" />
  <row Id="13704" PostId="8270" Score="1" Text="@ttnphns There are several equivalent definitions of a rotation matrix, such as a finite composition of reflections or the matrix of a linear transformation that preserves the usual inner product.  Factor analysis terminology abuses this convention by extending the term &quot;rotation&quot; to matrices which manifestly are not rotations.  The interest of a reflection lies in the fact that the parity of the number of reflections needed to express a (true) rotation is a property of that rotation.  In effect, it is an element of the quotient group O(n)/SO(n).  You need a similar context for your matrices." CreationDate="2011-03-14T16:04:33.003" UserId="919" />
  <row Id="13705" PostId="8269" Text="@ulvund I think you meant `pruned1 &lt;- subset(x, !(V5 == 1 &amp; V7 &gt; 113818477 &amp; V8 &lt; 114658477))` right? `pruned1` should exclude observations for which `V5==1 &amp; ...`" CreationDate="2011-03-14T16:07:03.260" UserId="3432" />
  <row Id="13706" PostId="8271" Score="1" Text="@RBS you sound strange. &quot;Two way&quot; ANOVA or ANCOVA needs 2 independent categorical vars (&quot;factors&quot;) by definition. Your data is one-way ANCOVA. Next, because your predictant is only 2 levels (fixed, I assume) you'd better go to Binary Logistic regression procedure instead of GLM." CreationDate="2011-03-14T16:08:08.743" UserId="3277" />
  <row Id="13707" PostId="8252" Text="@whuber: I am not disagreeing with you.  My point is merely about what information is used to calculate the $m$, $a_i$ and $b_j$;   median methods are more robust precisely because they use less (possibly distorting) information at this stage.  But as you say, the original data can be reconstructed using either method." CreationDate="2011-03-14T16:22:50.867" UserId="2958" />
  <row Id="13708" PostId="8226" Score="2" Text="@Yuriy The rows should not be out of order, but here is a way to do it one call to `aggregate()`: `aggregate(age ~ group, data=dat, FUN = function(x) c(M=mean(x), SD=sd(x)))`" CreationDate="2011-03-14T16:51:56.437" UserId="3432" />
  <row Id="13709" PostId="8150" Text="@Yuriy http://en.wikipedia.org/wiki/Sequential_probability_ratio_test might have some helpful references." CreationDate="2011-03-14T16:59:37.303" UserId="3432" />
  <row Id="13710" PostId="8266" Score="1" Text="@osknows This is a Markdown-enabled website with $\LaTeX$ support, further info can be found here: http://stats.stackexchange.com/editing-help." CreationDate="2011-03-14T17:00:26.793" UserId="930" />
  <row Id="13711" PostId="8268" Text="@osknows it is my opinion that regression (linear, generalized linear) is the domain where multicollinearity is an issue and is an issue *largely* for inference, hypothesis tests etc. As for other types of classifiers where predictive accuracy is the key interest, one should be careful about throwing the kitchen sink at the algorithm and hoping it figures it all out. If prediction is what you are concerned with, I would suggest experimenting and seeing how inclusion of correlated variables affect the accuracy. As in all things, there is no one answer and is dependent on the degree. " CreationDate="2011-03-14T17:07:36.583" UserId="2040" />
  <row Id="13712" PostId="8248" Score="1" Text="Whether this is appropriate depends on what you already know or assume about your data generating process. Subsampling the data by region (i.e. take all points in some predefined smaller window) can be dangerous if it's not homogeneous (because using a different window would have changed your conclusions). Sampling the data without regard to positioning for a training set has the effect of &quot;thinning out&quot; the observed process and invalidates conclusions you might want to draw about e.g. the range of correlations between marks or clustering/repulsion processes. " CreationDate="2011-03-14T17:10:11.683" UserId="1979" />
  <row Id="13713" PostId="8268" Text="Also, if you are concerned and assuming the variables are continuous, you can try a PCA (or related) analysis first to create features that are independent. Your decription (using a best time and a rank of that time) doesnt sounds like it would be near perfect collinearity is it? Since two runners could both have a rank of 1 and very different times depending on who else was in their race. One other idea is to run the analysis with a boosted decision tree or random forest and examine the variable importance ranking to see if both variables are important..." CreationDate="2011-03-14T17:12:26.177" UserId="2040" />
  <row Id="13714" PostId="8270" Text="@whuber, thank you a lot. I begin to understand why I'm speaking not same terms you do (you usage of terms is superior to mine). By reflection you mean bending a _point_ (vector) over a line (axis), while I was meaning a reflection of a coordinate _axis_ over origin by 180 degrees: central symmetry, that's what I meant by &quot;reflection&quot;. And wanted to know: if the determinant was negative then could I state that a (oblique) rotation of _axes_ took place and &quot;after that&quot; one _axis_ changed its direction to the opposite direction." CreationDate="2011-03-14T17:20:30.130" UserId="3277" />
  <row Id="13715" PostId="8270" Text="@ttnphns A central symmetry in an even number of dimensions is not considered a &quot;reflection&quot; by anyone, not even you(!): its determinant will be positive.  (E.g., in 2D the central symmetry is a 180 degree rotation.)  The change of direction along *one* axis is a reflection.  Indeed, if you allow that &quot;axis&quot; to be any *hyperplane* through the origin (a &quot;mirror&quot;, in effect), then you have described all the reflections." CreationDate="2011-03-14T18:12:59.697" UserId="919" />
  <row Id="13716" PostId="8239" Text="[Their team](http://www.khanacademy.org/about/the-team) seems great." CreationDate="2011-03-14T18:19:08.733" UserId="3582" />
  <row Id="13717" PostId="4681" Text="@andrew - Very nice!" CreationDate="2011-03-14T18:27:52.573" UserId="2040" />
  <row Id="13718" PostId="8281" Text="$D^{+}$ and $D^{-}$ are common names for the one-sided statistics (cf http://www.springerlink.com/content/y460k248m8w1822j/ ).  In what context does your $D'$ appear?" CreationDate="2011-03-14T18:29:37.700" UserId="919" />
  <row Id="13719" PostId="8280" Score="1" Text="It is correct that a lognormal distribution is involved, but it is unnecessary to invoke it, as my reply shows.  The underlying normal distribution *is* symmetrical of course.  The fact that a stock price (or anything else) has no upper limit does *not* imply its distribution has infinite variance.  In the Black-Scholes theory, by the way, volatility is indeed the variance parameter (for the distribution of logarithms)." CreationDate="2011-03-14T18:32:28.820" UserId="919" />
  <row Id="13720" PostId="8278" Score="3" Text="I am not sure to figure out what your data look like. Maybe it would be benefit to make it clearer. However, if your variable of interest is of the type &quot;time-to-event&quot;, then an analysis that takes time into account is definitely more informative than an analysis that only considers the number of cases. One reason for this is that censoring often occurs in such a study and a &quot;time-to-event analysis&quot; is especially designed for that setting..." CreationDate="2011-03-14T18:38:00.767" UserId="3019" />
  <row Id="13721" PostId="8281" Text="@whuber: Added context and comparison to $D^+$ and $D^-$ to the question." CreationDate="2011-03-14T18:46:31.750" UserId="220" />
  <row Id="13722" PostId="8280" Text="@Ralph, we consider the option, not the stock." CreationDate="2011-03-14T18:56:28.800" UserId="1351" />
  <row Id="13723" PostId="8167" Text="@SheldonCooper There are only a couple logic checks that I can use but they were somewhat useful. I didn't write this survey but when I write one in the future I will be sure to take your advice and add opportunities for inconsistencies. Thanks Sheldon." CreationDate="2011-03-14T19:05:58.307" UserId="1220" />
  <row Id="13724" PostId="4681" Text="I do have 2 questions if you dont mind.1)Is the proportion function simply to get the intercept correct? 2) what is the logic behind using ytest (comparing prob to a random uni draw)?" CreationDate="2011-03-14T19:18:22.853" UserId="2040" />
  <row Id="13725" PostId="8174" Text="I've incorporated your suggestions including checking for duplicate IPs and identifying the fastest respondents. In addition to duplicate IPs, I'm adding a check for the number of similar respondent's IP Addresses (matching the first three octets as shown in my original question) that took a survey within 30 minutes of a respondent as a warning flag. Thanks for your help lockedoff! I'm going to mark your answer as accepted but if anyone has any additional clever ideas please feel free to pass them along." CreationDate="2011-03-14T19:19:35.987" UserId="1220" />
  <row Id="13727" PostId="8274" Score="2" Text="+1 In short, physical reasoning: two possible outcomes, perfectly symmetric, and probabilities of all possible outcomes must sum up to 1 -- the only answer can be 1/2 (-;" CreationDate="2011-03-14T19:48:35.970" UserId="88" />
  <row Id="13728" PostId="8280" Text="@wok True, but the theory depends on the distribution of the *asset* (stock) prices.  The distribution of the option values is neither normal nor lognormal." CreationDate="2011-03-14T19:51:36.723" UserId="919" />
  <row Id="13729" PostId="8278" Score="1" Text="In addtion to @marco 's point, I think what you are looking at is a time-series and the independence assumption for observations of t-test may not hold." CreationDate="2011-03-14T20:32:12.177" UserId="1307" />
  <row Id="13730" PostId="8184" Text="@Onestop: Romão, Delgado &amp; Costa 2010 was exactly what I was looking for. I suppose I wasn't clear that I wasn't looking for a rigorous mathematical explanation of when SW was better than AD, but some conditions for when one out-preformed the other. If you submit your response as an answer, I can accept it." CreationDate="2011-03-14T20:32:34.367" UserId="1118" />
  <row Id="13731" PostId="8253" Text="@whuber, I sacrificed a bit of readability to the scalability. Function $L$ has 5 parameters and 7 constants. If we increase $n$ to 10 we will have 12 parameters and 21 constants, but my fitting function will still require 4 parameters, since constants are separated to two vectors and 1 constant. I fail to see how using symbolic environment could be advantageous for $n=10$, since typing 33 individual variables and keeping track of them should be tedious. However my experience with symbolic environments is very limited, so I would be glad if I am proved wrong." CreationDate="2011-03-14T20:39:53.937" UserId="2116" />
  <row Id="13732" PostId="8275" Text="I am familiar with marked point processes and some related theoretical tools, I should have thought of this before. Thanks a lot for the keywords, do you have maybe a few pointers for these ? Thanks also for the aggregation idea, I had a similar one, will try to do this." CreationDate="2011-03-14T20:40:39.790" UserId="3699" />
  <row Id="13733" PostId="8236" Text="Well, I think you'd call it irregular : it's recordings of crimes that happened in the UK during a given month, avaiable here : http://www.police.uk/data." CreationDate="2011-03-14T20:44:25.103" UserId="3699" />
  <row Id="13734" PostId="8253" Text="@whuber, I am curious which parts of the code are unreadable? Function `subvars` and `eqs.fun` employ some R tricks which aren't strictly necessary for this exposition. I had however lots of fun finding these tricks out, especially traversing R expression tree using recursion, so I thought that this a good opportunity to share them :)" CreationDate="2011-03-14T20:44:43.013" UserId="2116" />
  <row Id="13735" PostId="8248" Text="Yes, I know that PCA is for dimensionality reduction, this is why I was confused about how I could apply it to my dataset. The idea was to see if each event type had its own &quot;direction&quot;, or if some &quot;shared the same direction&quot;. But I guess I was simply thinking to correlation." CreationDate="2011-03-14T20:45:31.893" UserId="3699" />
  <row Id="13736" PostId="8253" Text="@mpiktas In *Mathematica*, for example: $\text{With}\left[\{n=3\}, D\left[\frac{1}{2} \sum _{i=1}^n w_i^2 \sigma _i^2-\lambda  \left(-r+\sum _{i=1}^n r_i w_i\right)-\mu  \left(-1+\sum _{i=1}^n w_i\right), \left\{\text{Join}\left[\{\lambda , \mu \}, \text{Array}\left[w_{\#}\&amp;,n\right]\right]\right\}\right]\right]$.  The output is $\left(&#xA;\begin{array}{c}&#xA; r-r_1 w_1-r_2 w_2-r_3 w_3 \\&#xA; 1-w_1-w_2-w_3 \\&#xA; -\mu -\lambda  r_1+w_1 \sigma _1^2 \\&#xA; -\mu -\lambda  r_2+w_2 \sigma _2^2 \\&#xA; -\mu -\lambda  r_3+w_3 \sigma _3^2&#xA;\end{array}&#xA;\right)$" CreationDate="2011-03-14T20:49:07.057" UserId="919" />
  <row Id="13737" PostId="8253" Text="@whuber, it occured to me that my comments are do not have a lot of statistical content in them. I think this might form a good question, i.e. how this code in R can be efficiently written using symbolic environment, but this requires an overlap of communities, users of symbolic environments, which can probably found in math.SE and users of R, which can be found in SO. Any suggestions?" CreationDate="2011-03-14T20:51:09.943" UserId="2116" />
  <row Id="13738" PostId="8253" Text="@whuber, ah nice, ability to understand summation sign. You certainly make a very good case for investing in *Mathematica* licence." CreationDate="2011-03-14T20:56:16.603" UserId="2116" />
  <row Id="13739" PostId="1112" Text="What does your data look like? Are the values bound? " CreationDate="2011-03-14T20:58:31.840" UserId="2965" />
  <row Id="13740" PostId="1112" Text="@spencer This forum welcomes (IMHO) everyone. You are perfectly fine. Can you please a bit more details about your data? Is your data non-negative? ..." CreationDate="2011-03-14T20:58:38.973" UserId="1307" />
  <row Id="13741" PostId="1112" Text="My data will be non-negative and has no bound." CreationDate="2011-03-14T20:59:48.000" UserId="1514" />
  <row Id="13742" PostId="8253" Text="@mpiktas The original question is marginal and probably should have been on SO anyway :-).  Remember, I didn't say &quot;unreadable&quot; in an *absolute* sense (because I didn't want to impugn your fine and creative work in R); I said *relative* to the single line displayed in my previous comment, taking into account your excellent remark that ease of debugging is important in such computations.  The implications for statistical analysis, IMHO, are that it helps to use the right software for the right job." CreationDate="2011-03-14T21:00:53.243" UserId="919" />
  <row Id="13743" PostId="8184" Text="@Christopher i found that ref from a simple Google Scholar search and I've only read its abstract, which doesn't in itself answer your first bullet (which seems more interesting to me than your second bullet, to which i reckon the answers are &quot;No&quot;, &quot;No&quot; and &quot;it depends&quot;). If that ref contains an answer to your 1st Q it would be great if you post it yourself, which is completely within the site's protocol, as is accepting your own answer." CreationDate="2011-03-14T21:10:49.787" UserId="449" />
  <row Id="13744" PostId="1112" Score="2" Text="Because any nondecreasing function from $[0, \infty) \to [0, 1]$ will do the trick, you have a lot of flexibility.  But some methods will be better than others, depending on the application. What is your purpose in seeking such a re-expression?" CreationDate="2011-03-14T21:12:35.277" UserId="919" />
  <row Id="13745" PostId="1112" Text="I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood." CreationDate="2011-03-14T21:14:52.980" UserId="1514" />
  <row Id="13746" PostId="1112" Text="@Spencer Exactly how are you measuring content and &quot;relevance&quot;?  E.g., on arbitrary scales, as counts, proportions, frequencies of views, correlations with other content, etc. etc.  Different types of measurements benefit from different kinds of re-expressions." CreationDate="2011-03-14T21:29:35.497" UserId="919" />
  <row Id="13747" PostId="1112" Text="I am measuring them on arbitrary scales. How old the content is. How many &quot;points&quot; a piece of content is received. Self-reported &quot;interest&quot; in the domain of the content." CreationDate="2011-03-14T21:38:14.177" UserId="1514" />
  <row Id="13748" PostId="8286" Score="1" Text="@Luther88, you case looks like a classical case of categorical data analysis. Can you please elaborate what do the fractions represent? An elaborate description of the data and problem will help us answer your question better." CreationDate="2011-03-14T21:44:48.883" UserId="1307" />
  <row Id="13749" PostId="8249" Text="i tried rotated(kpca) thinking it is the same as prcomp$rotation; which is (taken form R help(prcomp)): &quot;rotation: the matrix of variable loadings (i.e., a matrix whose columns contain the eigenvectors).&quot; However it is not. The question is however also meant very general because i'm not sure if LSA/LSI is possible using non-linear dimensionality reduction at all." CreationDate="2011-03-14T21:47:07.587" UserId="3683" />
  <row Id="13750" PostId="8271" Text="@ttnphns Assuming &quot;you sound strange&quot; means &quot;let me correct you on some points&quot; and &quot;predictant&quot; means &quot;dependent variable,&quot; I agree with what you're saying." CreationDate="2011-03-14T21:50:07.913" UserId="2669" />
  <row Id="13751" PostId="8236" Text="@Wookai 500,000,000 crimes in the UK in *one month*??  Has anarchy descended on the British Isles unreported by the press, only at last to be revealed in the police files? :-)  I could believe about 1/100th that amount--barely." CreationDate="2011-03-14T22:26:13.857" UserId="919" />
  <row Id="13752" PostId="8278" Text="@whuber: Thank you for making the question better by editing it" CreationDate="2011-03-14T22:54:26.647" UserId="2956" />
  <row Id="13753" PostId="8249" Text="sorry, but i might be missing the question; Why do you think that non-linear dimensionality reduction is not possible in LSA/LSI?" CreationDate="2011-03-14T22:54:40.337" UserId="3544" />
  <row Id="13754" PostId="8275" Score="2" Text="Peter Diggle has written a &quot;model-based geostatistics&quot;. He also has an analysis of Lancashire crime data on this page: http://www.lancs.ac.uk/staff/diggle/MADE/ that might give you some good ideas. " CreationDate="2011-03-14T22:56:37.167" UserId="1979" />
  <row Id="13755" PostId="8278" Text="@Marco: Thank you for the thought provoking comment. I have added the expected data. Our variable is survived or dead. We have taken care censoring or missing data problem will not happen. I asked the question because Meta-analysis would have the advantage of pooled Relative Risk, cumulative statistics, one study removed data etc. Kindly comment also on this." CreationDate="2011-03-14T23:06:11.593" UserId="2956" />
  <row Id="13756" PostId="8292" Score="2" Text="Works for me on win 7 x 64. Maybe you can provide the output of `sessionInfo()` ?" CreationDate="2011-03-14T23:11:27.483" UserId="3094" />
  <row Id="13757" PostId="8278" Text="@suncoolsu: Thank you for the great comment. Since my knowledge of statistics is poor, may be my direction of thinking is inappropriate. The problem with time series is the interference by unknown variables. I asked the question because Meta-analysis would have the advantage of pooled Relative Risk, cumulative statistics, one study removed data etc. Kindly comment also on this. " CreationDate="2011-03-14T23:11:47.873" UserId="2956" />
  <row Id="13758" PostId="8290" Score="1" Text="What assumptions can you make? E.g. can you assume that only one word in each phrase is responsible for the result, and the remaining words are just distractors? Or are there some &quot;word weights&quot; which get summed up to get the final result? Or something else?" CreationDate="2011-03-14T23:19:29.053" UserId="3369" />
  <row Id="13761" PostId="8184" Text="I used my university's VPN to read the article. Towards the end of the article are a list of power calculations under various conditions for quite a few GoF tests when the underlying distribution is one of maybe 30 non-normal distributions. That answered the question pretty well, showing when SW does better than AD." CreationDate="2011-03-14T23:51:55.220" UserId="1118" />
  <row Id="13762" PostId="8291" Score="2" Text="I would consider varying intercept fixed effects, see http://www.stata.com/support/faqs/stat/xtreg.html . It probably isn't a bad idea to check out those Stata FAQ's to see if they have anything pertinent to your question as well." CreationDate="2011-03-14T23:58:42.980" UserId="1036" />
  <row Id="13763" PostId="8157" Text="Let me ask something else... Are the `c`,`A`,`B` and lambdas fixed?" CreationDate="2011-03-15T00:05:37.953" UserId="2902" />
  <row Id="13764" PostId="8290" Score="1" Text="It is unknown if any of the words are meaningful.  How would I determine if the words, either individually or as a phrase, are helpful in predicting the value of Result?" CreationDate="2011-03-15T00:15:43.167" UserId="3712" />
  <row Id="13765" PostId="8284" Text="@ SheldonCooper - &quot;In image analysis it is routine to throw thousands of highly correlated features at SVM, and SVM seems to perform decently.&quot; - I would love to know more about how you do this without running out of memory. SVM have always performed quite well in prediction accuracy but with 200 input variables and over 100k samples SVM in Clementine and SPSS quickly exceeds memory limits. This may well be a another thread but would love to know how to balance SVM performance with large dimensionality &amp; samples?" CreationDate="2011-03-15T00:56:51.423" UserId="3484" />
  <row Id="13766" PostId="8284" Text="I'm not sure about SPSS. I've used libsvm with thousands of features without any problem. People have used it with millions of features (although the features in those cases are sparse, i.e. only some are active in any given instance)." CreationDate="2011-03-15T01:15:17.753" UserId="3369" />
  <row Id="13767" PostId="1112" Score="2" Text="One of the simplest transformations you could use is to convert your data into quantile scores." CreationDate="2011-03-15T01:28:43.960" UserId="3567" />
  <row Id="13768" PostId="8278" Text="Do you have, like in your previous question, the information on mortality before start of treatment?" CreationDate="2011-03-15T01:39:51.617" UserId="3369" />
  <row Id="13769" PostId="8278" Text="@SheldonCooper: Yes. I have an idea how the data would look like partially. I entered those data in the question." CreationDate="2011-03-15T01:46:33.623" UserId="2956" />
  <row Id="13770" PostId="8278" Score="1" Text="I meant the data on mortality without any treatment, or with treatment when the same drug was used in both cities. The reason I ask is that with just the current data you cannot distinguish between two possibilities. One possibility is that the drug in city 2 is better, another possibility is that the drug is no better, but something else causes the patients to do better -- maybe weather or overall hospital quality or something else." CreationDate="2011-03-15T02:01:24.110" UserId="3369" />
  <row Id="13771" PostId="8278" Text="@SheldonCooper: Thank you for asking for the clarification.The drugs are different. All other known factors are same. The cities are 1000 Kilometers apart." CreationDate="2011-03-15T02:57:30.753" UserId="2956" />
  <row Id="13772" PostId="8278" Score="1" Text="If you can just claim that all other factors are the same, then it's fine to just do a t-test year-by-year or on all 5 years together. This is OK if this is just homework. If this is a real study and real data, then keep in mind that a lot of people will refuse to just believe that all other factors are the same. You will have to convince them somehow. In your previous question you had some pre-intervention data to convince them. Do you have something similar here?" CreationDate="2011-03-15T03:25:38.717" UserId="3369" />
  <row Id="13773" PostId="8278" Text="@SheldonCooper: I am amazed at your speed of response. Thank you so much. I can get pre-invention data. I will take your advice of doing t-test. Kindly clarify one more thing for my benefit. Why Meta-analysis cannot be used? Meta-analysis would have the advantage of pooled Relative Risk, cumulative statistics, one study removed data etc in addition to relative risk year wise. I have not understood why nobody commented on that. Please be frank and tell me if I am wrong. I want to learn." CreationDate="2011-03-15T05:09:49.090" UserId="2956" />
  <row Id="13774" PostId="8279" Text="@David Reilly: I am amazed at your your commendable and authoritative discussion. One Journal from UK and 1 journal from India refused the paper because I did time series analysis and forecasting to compare what is expected and what was achieved with the new treatment." CreationDate="2011-03-15T06:05:22.147" UserId="2956" />
  <row Id="13775" PostId="8279" Text="@David Reilly: I wrote this: Selection of the model for forecast generation: Time series expert modeler of IBM SPSS Forecast v19 was used. Both exponential smoothening models and Autoregressive Integrated Moving Average (ARIMA) models were examined. Outliers were detected and prevented from influencing parameter estimates. Maximum number of Lags in ACF and PACF output checked were 24. Stationery R-square (larger values indicate better fit) was the goodness of fit measure for selecting the best fitting forecasting model to generate IRs for 2011 to 2012 for 2 states. Thank you very much." CreationDate="2011-03-15T06:06:46.373" UserId="2956" />
  <row Id="13776" PostId="8279" Text="@David Reilly: One more question to trouble you a little more. What is the statistical software you have used? I used SPSS Forecast Version 19. I generated a forecast for 2009 &amp; 2010 and showed that the observed death rate is below the Lower Confidence limit in State 1 and so it is significant." CreationDate="2011-03-15T06:19:21.573" UserId="2956" />
  <row Id="13777" PostId="8157" Text="It's not possible (in general) for the $a_i$ to satisfy these constraints *and* to have Poisson distributions.  Rescaling the $a_i$ is not generally possible, either, unless you add restrictions on $B$, $b_i$, $A$, $c$, and $D$." CreationDate="2011-03-15T06:39:56.957" UserId="919" />
  <row Id="13778" PostId="8236" Text="Wow, I'm really sorry for this &quot;typo&quot; ;) ! It's 1000 times less actually, 500'000 crimes (counting &quot;vehicule crimes&quot;, i.e. speed tickets, etc...)." CreationDate="2011-03-15T07:23:05.087" UserId="3699" />
  <row Id="13779" PostId="4373" Text="@G. Jay - this is basically a re-statement of the mean and variance being independent.  $X+Y$ is a re-scaled mean and $X-Y$ is a rescaled standard deviation." CreationDate="2011-03-15T07:24:53.357" UserId="2392" />
  <row Id="13780" PostId="8303" Text="@Leendert: There exist functions that perform automatic search. You should have a look at the **step** function. Section 5.4 illustrates that point: http://data.princeton.edu/R/glms.html" CreationDate="2011-03-15T08:43:09.950" UserId="3019" />
  <row Id="13781" PostId="8291" Text="@Andy The intercept can vary as either a fixed effect or a random effect. In the notation of your link, if it's a fixed effect then $u_i$ is treated as a constant to be estimated from the info within each group, while if it's a random effect it's assumed to have a normal distribution whose mean and variance the model estimates, and you can afterwards get 'predictions' of the individual $u_i$s which use info from all the groups and are closer to their mean than in the fixed effect models due to 'shrinkage'. " CreationDate="2011-03-15T09:45:26.097" UserId="449" />
  <row Id="13782" PostId="8294" Score="1" Text="(+1) I believe that's the overwhelmingly most common situation. Clustered SEs are *possible* in random-effect models though, and have been considered in the context of meta-analysis / meta-regression. See [Hedges, Tipton &amp; Johnson *Research Synthesis Methods* 2010](http://dx.doi.org/10.1002/jrsm.5)." CreationDate="2011-03-15T09:48:57.513" UserId="449" />
  <row Id="13783" PostId="8300" Text="@Ryan About the permutation approach, see @whuber's answer to this related question, [Look and you shall find (a correlation)](http://stats.stackexchange.com/questions/5750/look-and-you-shall-find-a-correlation), which describes the basic idea of breaking the structure or link between your two series of measurements." CreationDate="2011-03-15T09:55:29.453" UserId="930" />
  <row Id="13784" PostId="8226" Text="@lockedoff: Thank you for having completed my answer!" CreationDate="2011-03-15T10:22:20.997" UserId="3019" />
  <row Id="13785" PostId="8230" Text="@mpiktas Thanks for the update and the benchmark info!" CreationDate="2011-03-15T10:24:59.247" UserId="930" />
  <row Id="13786" PostId="8307" Text="@onestop, so there is nothing to do with forcing the data to the $[0, 1]$, and in general I do agree with standardization if the author really looks for it :) I answered before the discussion appeared, so probably delete my answer if this one is correct :)" CreationDate="2011-03-15T10:27:23.517" UserId="2645" />
  <row Id="13787" PostId="8309" Text="Well I need time to answer this (probably others will be quicker to aid you), but most insights may be taken from http://www.stat.lsa.umich.edu/~kshedden/Courses/Stat600/Notes/prediction.pdf And what is $k$ in definition of degrees of freedom, since I do miss $\lambda$ somehow." CreationDate="2011-03-15T10:55:02.297" UserId="2645" />
  <row Id="13788" PostId="8279" Text="@DrWho: I am a developer of time series software. The product name is AUTOBOX and is available from http://www.autobox.com. I hope that this comment is not perceived as being &quot;too commercial&quot; as that is not my intent at all. Please feel free to contact me via that site." CreationDate="2011-03-15T10:55:44.507" UserId="3382" />
  <row Id="13790" PostId="8309" Text="@Dmitrij: Thnx for the reply, I have updated the questions, and replaced 'k' with $\lambda$" CreationDate="2011-03-15T11:23:30.247" UserId="3722" />
  <row Id="13791" PostId="8225" Score="1" Text="The British also drive on the wrong side" CreationDate="2011-03-15T11:23:53.290" UserId="1709" />
  <row Id="13792" PostId="8312" Score="3" Text="@Peter (+1) About R packages, there're also [glmnet](http://cran.r-project.org/web/packages/glmnet/index.html) (implementation with coordinate descent algo., Friedman and coll.) and [penalized](http://cran.r-project.org/web/packages/penalized/index.html) (allows to keep some var. unpenalized). Of note, F. Harrell provides penalized ML estimation for GLMs (see `lrm`, or his RMS textbook for further info)." CreationDate="2011-03-15T11:31:19.537" UserId="930" />
  <row Id="13793" PostId="8297" Text="Shouldn't this question be made CW?" CreationDate="2011-03-15T11:33:28.333" UserId="930" />
  <row Id="13794" PostId="1206" Text="@rob forgive my ingorance, but isn't rotation normally a part of PCA as well as FA?" CreationDate="2011-03-15T11:40:16.423" UserId="656" />
  <row Id="13795" PostId="8314" Score="1" Text="@richiemorrisroe In line with your response, the [HSAUR](http://cran.r-project.org/web/packages/HSAUR/index.html) textbook deserves some attention." CreationDate="2011-03-15T11:41:44.797" UserId="930" />
  <row Id="13796" PostId="8314" Text="@chl thats totally correct, i'd forgotten about that. The books on CRAN generally are often quite good. " CreationDate="2011-03-15T11:49:47.743" UserId="656" />
  <row Id="13800" PostId="8294" Text="My understanding from the Bertrand et al. paper is that clustered standard errors also allow for heteroskedasticity at the cluster level... random effects don't address this.  [This post](http://www.stata.com/statalist/archive/2006-09/msg00782.html) suggests that clustering is standard practice in fixed effects models for that reason." CreationDate="2011-03-15T12:13:58.687" UserId="3700" />
  <row Id="13801" PostId="8309" Text="@Amit Did our mutual revisions become entangled somehow?" CreationDate="2011-03-15T12:16:52.070" UserId="930" />
  <row Id="13805" PostId="8297" Text="I'm still somewhat new to this site .. what does it mean to make something a Community Wiki (question?) ? I understand it doesn't contribute to reputation(?) but other than that, does it matter? (I have been reading the FAQ, but some of the mechanics of this site are still a bit unclear to me). Thanks." CreationDate="2011-03-15T12:48:47.440" UserId="3311" />
  <row Id="13806" PostId="8230" Text="@chl, it gave me a chance to try out this new **data.table** package. It looks really promising." CreationDate="2011-03-15T12:54:27.890" UserId="2116" />
  <row Id="13807" PostId="8230" Text="@mpiktas Yes, looks like it is." CreationDate="2011-03-15T12:57:15.000" UserId="930" />
  <row Id="13810" PostId="8315" Text="@probabilityislogic thnx a lot , but I have a doubt why do we need to assume $d_{i}^2=1$, since we have their correct values already...I have checked this formula by writing a matlab code and have not taken that assumption, but it works fine and gives correct solution" CreationDate="2011-03-15T13:21:02.827" UserId="3722" />
  <row Id="13811" PostId="8247" Text="@chl - A bit of nitpicking from me.  It is a nice answer about the *intuition* behind the F-test, and how it &quot;goes in the right directions&quot;.  But it doesn't explain the logic of why you should choose this particular test.  For example, why shouldn't we use the PRESS statistic?  You hinted at the likelihood ratio - which *does* have a logical justification - hence its applicability to all models, unlike the F-test." CreationDate="2011-03-15T13:23:22.700" UserId="2392" />
  <row Id="13812" PostId="8315" Text="The assumption is just to get the initial value of $\lambda^{(0)}$ &quot;close enough&quot; to the correct value.  If you have a better guess, then start with that.  You could even just set $\lambda^{(0)}=0$, as long as your d's are greater than zero.  The d's are not assumed 1 in the iterations, just to get the algorithm started." CreationDate="2011-03-15T13:27:24.443" UserId="2392" />
  <row Id="13814" PostId="8297" Text="@levon9 It's just that there might not be a &quot;best&quot; or good response, and that arguments can be made arbitrarily subjective. It doesn't mean, however, that the question in not interesting *per se*; asker/users won't earn rep from it, but they still get the opportunity to gain badges and the like." CreationDate="2011-03-15T13:57:58.573" UserId="930" />
  <row Id="13815" PostId="8316" Text="You seem to have answered your own question as to how to *calculate* fitted values; what to *call* them (apart from 'fitted values'), or how to describe them unambiguously in words rather than equations, would be different questions. Conditional geometric means? Geometric conditional means?" CreationDate="2011-03-15T14:28:59.377" UserId="449" />
  <row Id="13816" PostId="8315" Text="Thanks a lot for the answer. Helped me a lot. " CreationDate="2011-03-15T14:31:56.743" UserId="3722" />
  <row Id="13820" PostId="8315" Text="(+1) I would give the same numerical solution anyway." CreationDate="2011-03-15T14:44:50.980" UserId="2645" />
  <row Id="13821" PostId="8312" Text="@Peter (+1) Nice article, it seems I have to start going far beyond the author states in the question (not the first time I didn't). @chl (+1) perfect alternative suggestions too." CreationDate="2011-03-15T15:04:24.560" UserId="2645" />
  <row Id="13822" PostId="8236" Text="@Wookai That's good for you, because it means your problem is readily addressed with a GIS and may be accessible to R-based solutions." CreationDate="2011-03-15T15:06:53.297" UserId="919" />
  <row Id="13823" PostId="8321" Text="You passed the junk regression (F) test but not t tests -- it is the first sign for multicollinearity problem look here for instance http://stats.stackexchange.com/questions/4099/vif-condition-index-and-eigen-values/8223#8223" CreationDate="2011-03-15T15:07:21.890" UserId="2645" />
  <row Id="13824" PostId="8278" Score="1" Text="@DrWho Meta analysis is for combining results from multiple independent studies that have already been performed and published.  This is not such a situation.  (That's why I originally removed the meta-analysis tag from your question, because it will only mislead people who truly are concerned about meta analysis.)" CreationDate="2011-03-15T15:10:25.073" UserId="919" />
  <row Id="13825" PostId="8320" Text="Interesting.  I *always* use natural logs because many of the coefficients tend to be close to zero and then can be interpreted as proportional (relative) differences.  That's not possible in any other base of logarithm.  I see some merit in using other bases, but I think you need to clarify your response, because *prima facie* your interpretation doesn't use the value of the coefficient at all!" CreationDate="2011-03-15T15:12:00.003" UserId="919" />
  <row Id="13827" PostId="8305" Text="+1 For commenting on $R^2$." CreationDate="2011-03-15T15:22:31.257" UserId="919" />
  <row Id="13828" PostId="8301" Text="This seems a little harsh.  It sounds like the search for correlations was planned at the outset and the OP is aware of the multiple comparisons problem.  That already puts them on a better experimental, analytical, and conceptual footing than the vast majority of publications in this field." CreationDate="2011-03-15T15:25:41.100" UserId="919" />
  <row Id="13829" PostId="8321" Text="@chl has nicely answered your question in his reply at http://stats.stackexchange.com/questions/8237/logic-behind-the-anova-f-test-in-simple-linear-regression/8247#8247 : see the second paragraph." CreationDate="2011-03-15T15:38:39.933" UserId="919" />
  <row Id="13831" PostId="8295" Text="Thanks for the pointer.  Could you provide an example?  I installed the R tm package, and can create a DocumentTermMatrix from the Words column, but I don't know how to measure the correlation between the terms and the other variables from the data frame." CreationDate="2011-03-15T16:15:15.750" UserId="3712" />
  <row Id="13832" PostId="4373" Text="@probabilityislogic - I like the intuition of what you said, but I don't think it is exactly a restatement because $X - Y$ isn't exactly a rescaling of the SD: the SD forgets the sign. So independence of mean and SD follows from independence of $X+Y$, $X-Y$ (when $n = 2$), but not the other way around.  That may have been what you meant by &quot;basically&quot;.  Anyway, it's good stuff.  " CreationDate="2011-03-15T16:20:35.783" UserId="1108" />
  <row Id="13833" PostId="8236" Score="1" Text="Yes, R looks the way to go! I found a very complete report of a workshop on the spatstat module of R, that does exactly what I'm looking for : http://www.csiro.edu.au/resources/Spatial-Point-Patterns-in-R.html" CreationDate="2011-03-15T16:25:36.667" UserId="3699" />
  <row Id="13836" PostId="4546" Text="Eventually figured out I should do an inverse Markov transition step." CreationDate="2011-03-15T16:57:41.870" UserId="1893" />
  <row Id="13837" PostId="8295" Text="See some sample code above" CreationDate="2011-03-15T16:58:47.990" UserId="3489" />
  <row Id="13840" PostId="8157" Text="@whuber it makes sense. I guess the probability constraint is too restrictive for what I need. Will try to refine the constraints." CreationDate="2011-03-15T17:23:44.927" UserId="3673" />
  <row Id="13841" PostId="8157" Text="@deps_stats I simplified the constraints, it should be clearer now." CreationDate="2011-03-15T17:30:36.277" UserId="3673" />
  <row Id="13843" PostId="8129" Text="Thanks for the answer to my question. I am indeed interested in optimizing for both negatives and positives. In this case, it seems that accuracy is the way to go since it considers tp, fp, tn, and fn. However, as you mentioned above, I must be aware of the class skew. Thus, should I present accuracy along side another metric to counter this? Thanks again!" CreationDate="2011-03-15T18:20:59.870" UserId="3604" />
  <row Id="13844" PostId="8295" Text="Looks like you've got me on the right track.  I set the Result to 1 for all lines containing the word, pestology, and I get a non-zero correlation for this word: http://bit.ly/gtWn8c  How do I determine whether the correlations between the words and Result are significantly different for the two values of Source?" CreationDate="2011-03-15T19:35:25.983" UserId="3712" />
  <row Id="13845" PostId="8328" Score="1" Text="It might help to understand better how $BB'+D$ &quot;loses&quot; its positive definiteness.  This implies that either $BB'$ or $D$ (or both) are becoming non-positive definite.  That's hard to do when $BB'$ is computed directly from $B$ and even harder when $D$ is computed as a diagonal matrix with squares on its diagonal!" CreationDate="2011-03-15T19:41:30.207" UserId="919" />
  <row Id="13846" PostId="8295" Text="You need to construct confidence intervals on the correlation estimates.                             http://stat.ethz.ch/R-manual/R-patched/library/stats/html/cor.html" CreationDate="2011-03-15T20:08:15.160" UserId="3489" />
  <row Id="13847" PostId="8247" Text="@probabilityislogic Good point. My idea was originally to show the logic behind model comparison, of which the simple regression model is just a particular case (compare to the &quot;very null&quot; model), which also motivates the quick note about LRT. I agree with you, if we work along the line of a pure Neyman-Pearson approach for HT. However, I was mainly thinking in terms of the Theory of LMs, where SS have a direct geometrical interpretation and where model comparison or the single F-test for a one-way ANOVA (...)" CreationDate="2011-03-15T20:33:30.303" UserId="930" />
  <row Id="13848" PostId="8247" Text="@probabilityislogic (Con't) or a simple regression has a very intuitive approach in term of projection, subspaces, etc. I'll try to update my response to reflect this. Feel free to comment on." CreationDate="2011-03-15T20:34:07.197" UserId="930" />
  <row Id="13849" PostId="8321" Text="@Can't Could you please confirm that you're also the asker for [Logic behind the ANOVA F-test in simple linear regression](http://stats.stackexchange.com/questions/8237/logic-behind-the-anova-f-test-in-simple-linear-regression) so that we can merge your two accounts. (You will still need to register, see the [FAQ](http://stats.stackexchange.com/faq).)" CreationDate="2011-03-15T20:39:47.067" UserId="930" />
  <row Id="13850" PostId="8330" Text="How do you know which are errors? E.g. if the question was &quot;What would did you eat for breakfast?&quot; and the answers recorded were &quot;sausage, egg, bacon, sausage, tomato&quot;, are you simply not interested whether the person really ate two sausages or only one sausage?" CreationDate="2011-03-15T20:51:24.810" UserId="449" />
  <row Id="13851" PostId="6125" Text="@richiemorrisroe There's no assumption as long as you don't make inference based on PCs." CreationDate="2011-03-15T21:20:54.363" UserId="930" />
  <row Id="13852" PostId="8330" Text="Is this a question about statistical analysis or is it about obtaining a software algorithm to screen duplicate values?  Could you clarify your intent?" CreationDate="2011-03-15T21:25:53.517" UserId="919" />
  <row Id="13856" PostId="8328" Text="@whuber Typically in FA $q&lt;p$, so $BB'$ isn't ever positive definite. But (theoretically) $BB' + D$ ought to be, assuming that the $\sigma^2_j$'s are all greater than zero." CreationDate="2011-03-15T22:03:59.833" UserId="26" />
  <row Id="13857" PostId="8328" Text="This is related to this question: http://stats.stackexchange.com/questions/6364/making-square-root-of-covariance-matrix-positive-definite-matlab" CreationDate="2011-03-15T22:12:37.033" UserId="2833" />
  <row Id="13858" PostId="8310" Text="Could you explain why it is not $ \int \frac{y f(y,x)}{f(x)} dy $ ? How did the denominator of the conditional probability get pulled out of the integral?" CreationDate="2011-03-15T22:24:32.173" UserId="1118" />
  <row Id="13859" PostId="8328" Score="1" Text="@JMS Thank you.  I think my comment is still pertinent: $BB'$ can be indefinite, but should still not have any negative eigenvalues.  Problems will arise when the smallest of the $\sigma_i^2$ is comparable to numerical error in the inversion algorithm, though.  If this is the case, one solution is to apply SVD to $BB'$ and zero out the really small (or negative) eigenvalues, then recompute $BB'$ and add $D$." CreationDate="2011-03-15T22:42:31.113" UserId="919" />
  <row Id="13860" PostId="8283" Text="I am concerned about your objective function.  It will be unstable when exploring values where 'Mean Fit' and 'Sk Fit' are near zero, even if these are close to (or at) the optimum.  One wonders about the point of optimizing the sum of relative errors, too.  After all, the uncertainties in the moments get larger as the order goes up: the mean return is a *far* better estimate of the true mean than the kurtosis of the returns is an estimate of the true kurtosis. A natural objective would be something like the sum of the squared differences of the $1/k$ powers of the sizes of the $k$th moments." CreationDate="2011-03-15T22:51:33.337" UserId="919" />
  <row Id="13861" PostId="8333" Text="Could you explain what gsl_rng_uniform_pos and gsl_ran_gamma do?  I would guess the first returns a uniform random value between 0 and r and the second is related to a Gamma(1+a, b) value--maybe it's an incomplete Gamma?  Overall this looks very close to the approximation I suggested (except, in reviewing it, it's obvious I forgot to specify the divide by $\alpha$ part, which is essential!)" CreationDate="2011-03-15T22:59:04.483" UserId="919" />
  <row Id="13863" PostId="8334" Score="3" Text="Search on the &quot;change-point&quot; tag for some leads: http://stats.stackexchange.com/questions/tagged/change-point" CreationDate="2011-03-15T23:11:08.403" UserId="919" />
  <row Id="13864" PostId="8247" Text="@chl - I think the problem is with leading with &quot;intuition&quot; and then seeing if it &quot;makes sense&quot; - it gets you into trouble when you try to &quot;generalise&quot;.  this is because one is &quot;generalising&quot; the intuition rather than generalising the mathematics.  The SS in a additive model with normal errors have a logical justification - which is why they work so well.  However, if your interested in prediction or model selection - I think the logic &quot;adds in&quot; some extra pieces of information (such as an &quot;occam&quot; effect)." CreationDate="2011-03-16T00:10:32.580" UserId="2392" />
  <row Id="13865" PostId="8310" Text="Because it has no dependence on $y$.  Thus in the integral, it is just a constant." CreationDate="2011-03-16T00:12:05.217" UserId="2392" />
  <row Id="13866" PostId="8328" Text="@whuber: I agree, just wanted to clarify that slight point. Like you say I imagine the problem is very small or even negative elements in $D$. As I mentioned in my answer the problem then isn't just algorithmic but inferential too." CreationDate="2011-03-16T00:38:05.597" UserId="26" />
  <row Id="13867" PostId="8295" Text="Do you mean perform separate correlation measurements for the cheap and expensive datasets, and then measure the correlation between these correlations, like this? http://bit.ly/e9DoEk" CreationDate="2011-03-16T01:03:18.540" UserId="3712" />
  <row Id="13868" PostId="8310" Text="Ah, of course! Thank you." CreationDate="2011-03-16T02:05:10.500" UserId="1118" />
  <row Id="13869" PostId="8323" Text="Dear whuber, thanks for your answer. I will try as you suggested. Meanwhile, I also read about your post with permutation testing, and it seems a do-able way for my application as well. I do have one question tho. Suppose I ran 10000 iterations with permuted variables, and have almost normal distribution. How can I calculate the p value of my &quot;original&quot; correlation coefficient (not permuted, experimantal data)?" CreationDate="2011-03-16T02:16:02.190" UserId="3720" />
  <row Id="13870" PostId="8042" Text="@charles.y.zheng: I accepted your answer as the solution and updated my results in the following link as a new question. Can you please let me know what you think about my new results? http://stats.stackexchange.com/questions/8340/is-this-linear-regression-result-good-or-bad" CreationDate="2011-03-16T03:52:18.003" UserId="2164" />
  <row Id="13871" PostId="8325" Text="@Patrick McCann: Thank you very much. I will go though it and get back. i will have to search for the power point file mentioned in that document." CreationDate="2011-03-16T04:30:09.350" UserId="2956" />
  <row Id="13872" PostId="8340" Score="1" Text="@Legend I recommend you change the title. The answer to the question 'is this result good or bad' is subjective, and would depend on what the predictor variables are - as well as the values of the respondent. Car mechanics / salesmen / manufacturers might prefer more crashes, a car driver might prefer less. The size of the estimate of a standard error is just that - there is no reason to think that smaller is better without context. Perhaps you want to know if your results support your hypothesis or not, or if your results are valid, in which case you would be able to find useful answers here." CreationDate="2011-03-16T05:05:13.847" UserId="1381" />
  <row Id="13873" PostId="8129" Text="@Raffi: You could add the ratio of correctly classified examples of the minor class (i.e. precision or true negative rate respectively). However, I think it should be enough that you state your awareness of this problem and that you check that the model is not just predicting the major class. But this is just my opinion." CreationDate="2011-03-16T07:18:05.673" UserId="264" />
  <row Id="13875" PostId="8347" Score="1" Text="Bayesian statistics is traditional statistics - can you give a concrete example for what you mean be traditional statistics?" CreationDate="2011-03-11T19:54:53.597" />
  <row Id="13876" PostId="8347" Text="@OphirYoktan: He's talking about frequency probability versus Bayesian probability. It's even mentioned in the question's title." CreationDate="2011-03-11T19:57:38.880" />
  <row Id="13877" PostId="8347" Text="Wikipedia can clarify the distinction for people, hopefully.  Frequentist inference: http://en.wikipedia.org/wiki/Frequentist_inference  and Bayesian inference http://en.wikipedia.org/wiki/Bayesian_inference" CreationDate="2011-03-11T20:14:06.523" />
  <row Id="13878" PostId="8347" Score="4" Text="I think this question should be moved over here: http://stats.stackexchange.com/" CreationDate="2011-03-11T23:29:36.810" UserId="3078" />
  <row Id="13879" PostId="8347" Text="Hmm, maybe.  I certainly tried framing it in terms of a skeptical assessment of Bayesian statistics.  I would probably frame my query differently towards a devoted statistics crowd." CreationDate="2011-03-11T23:51:45.120" />
  <row Id="13880" PostId="8347" Text="@MindDetective: It's definitely an interesting question (I'd love to see an answer from a professional statistician). But even when framed as a skeptical assessment of Bayesian statistics, it's still more about statistical methodology than it is about skepticism. You're far more likely to get a good answer from a statistician than you are from a skeptic." CreationDate="2011-03-12T03:04:43.957" UserId="3078" />
  <row Id="13881" PostId="8347" Text="I was hoping there were a few that might be both around! :)" CreationDate="2011-03-12T03:07:57.447" />
  <row Id="13882" PostId="8347" Text="@MindDetective: I think it would be more appropriate here if it were specifically about the conclusions Bem reached (with skepticism fostered by the chosen statistical analysis), rather than about statistical analysis (with Bem's paper as an example)." CreationDate="2011-03-12T03:09:04.107" UserId="3078" />
  <row Id="13883" PostId="8347" Text="Voted to close as off-topic, sorry! :-)" CreationDate="2011-03-12T12:04:07.393" />
  <row Id="13884" PostId="8347" Text="I wasn't interested in discussing Bem's study. The topic was about the vaunted benefits of Bayesian statistics, NOT about precognition.  Additionally, I think we should be skeptical of the claims that statisticians make as well as those of scientists or homeopathy proponents or politicians.  Is there really no room to assess the claims of applied mathematics here?  If it isn't interesting or answerable by anyone who frequents the site, that's fine.  But off topic?  That leaves a narrower definition of on-topic than I like, personally." CreationDate="2011-03-12T14:11:26.987" />
  <row Id="13885" PostId="8347" Text="I also voted off-topic; I think this would fit better on stats.SE." CreationDate="2011-03-13T01:28:57.460" />
  <row Id="13886" PostId="8347" Score="2" Text="I asked a [question on meta](http://meta.skeptics.stackexchange.com/questions/200/are-applied-mathematics-questions-on-topic) about whether this should be on-topic." CreationDate="2011-03-13T01:46:18.177" />
  <row Id="13887" PostId="8308" Text="When I read the question I wanted to suggest that strategy. But I preferred to wait for other suggestions because something stopped me. What if one of the $X_i$'s is negative?" CreationDate="2011-03-16T07:46:48.060" UserId="3019" />
  <row Id="13888" PostId="8320" Text="@whuber sorry what does *prima facie* mean? First face??" CreationDate="2011-03-16T07:56:52.083" UserId="449" />
  <row Id="13889" PostId="8308" Text="@Marco, read the footnote in wikipedia for *geometric mean*. If one goes for the geometric means, he or she assumes that all $X_i$'s are strictly positive (even zero would be not suitable here). Real life data when in levels is mostly positive ^_^ And even if you do some negatives (like gains and loses) split the two and make them positive again ^_^" CreationDate="2011-03-16T07:57:19.170" UserId="2645" />
  <row Id="13890" PostId="8347" Text="The funny thing about Bayesian Probability theory, is that it has the theoretical apparatus to explain why people are skeptical of the theory!  And it also tells you under what conditions this will happen!" CreationDate="2011-03-16T09:25:04.930" UserId="2392" />
  <row Id="13891" PostId="8349" Text="@Lizzan Are you interested in displaying each error type or just the overall error rate?" CreationDate="2011-03-16T09:33:09.690" UserId="930" />
  <row Id="13892" PostId="8286" Text="@Luther88 For the sake of clarity, I have moved the comments you left in your question." CreationDate="2011-03-16T09:39:13.817" UserId="930" />
  <row Id="13893" PostId="8351" Score="1" Text="@ran2, it would be easier to judge if you insert a plot of your data. For adf-test you have an option to test the trend stationarity, also consider alternative tests KPSS, PP. It may happen that you have some deterministic parts. You may use HP and more general Kalman Smoothing for trend extraction, loess do the same locally, so it bares in mind only a window of data, by default, I think, it fits linear trend. So loess for seasonal smoothing HP, Kalman not for seasonal but cyclical smoothing. Baxter-King though could do both. Looking for the data screening though to add more." CreationDate="2011-03-16T10:15:42.880" UserId="2645" />
  <row Id="13894" PostId="6125" Text="@chl, indeed. those inferences can be tricky." CreationDate="2011-03-16T10:17:47.520" UserId="656" />
  <row Id="13895" PostId="8303" Text="I'm sorry but my post has been edited so that it no longer asks my question. I have 35 (26 significant) explanatory variables in my logistic regression model. I need the best possible combination of 8, not the best subset, and at no point was I interested in a stepwise or all subsets style approach. There is no wiggle room in this 8. I just thought someone may have know how I could fit all combinations of 8 explantory variables and it could tell me which maximises the likelihood (sorry about the R-squared brain fart but AIC isnt relevant either since I have a fixed number of parameters, 8).   " CreationDate="2011-03-16T10:25:05.953" UserId="3721" />
  <row Id="13896" PostId="8349" Text="@chl Both, in two different charts." CreationDate="2011-03-16T10:26:57.137" UserId="3735" />
  <row Id="13897" PostId="8349" Text="@Lizzan I think sth like Cleveland's dotplot can do well; can you provide a snapshot of what your data look like and how they are arranged (e.g., just some records with corresponding error type)?" CreationDate="2011-03-16T11:01:40.703" UserId="930" />
  <row Id="13898" PostId="8349" Text="@chl I added some example data, I hope it's what you're looking for." CreationDate="2011-03-16T11:56:18.433" UserId="3735" />
  <row Id="13899" PostId="8359" Text="(+1) nice simulation example." CreationDate="2011-03-16T13:31:59.110" UserId="2645" />
  <row Id="13900" PostId="8355" Text="Are you sure he has enough data points to produce meaningful spectral analysis? Though I also do like to work in the frequency domain from time to time :) (+1) for the link though, will add to my collection of 'yet to read' things.  " CreationDate="2011-03-16T13:33:36.630" UserId="2645" />
  <row Id="13901" PostId="8358" Text="@mbq: Thank you for your time in editing this" CreationDate="2011-03-16T13:40:38.203" UserId="2956" />
  <row Id="13902" PostId="8354" Text="So the story is about the influence of additional data point? I more used to the impulse response for the time series data, in statistical context all influence would be described by marginal effect or (better choice) beta coefficient from standardized regression. Well I really need more context to judge the question and answer, but this one is nice, I think (+1 not yet but awaiting)." CreationDate="2011-03-16T13:43:19.573" UserId="2645" />
  <row Id="13904" PostId="8279" Text="@IrishStat: Thank you very much for the very useful reply. Your answer was very useful. I am still reading it to understand it completely because I am a beginner in Statistics with negligible knowledge of Mathematics. I am sorry I missed your reply for so long. I read about you in your website. I must congratulate you for being so active and prompt at 72. " CreationDate="2011-03-16T13:54:11.310" UserId="2956" />
  <row Id="13905" PostId="8351" Text="@ran2, what is `y` in your second code snippet?" CreationDate="2011-03-16T14:13:07.770" UserId="2116" />
  <row Id="13906" PostId="8320" Text="@onestop http://www.google.com/search?q=define+&quot;prima+facie&quot;" CreationDate="2011-03-16T14:26:24.310" UserId="919" />
  <row Id="13907" PostId="8279" Text="@IrishStat: Thank you very much for the very useful reply. I was struggling to find the software. I did visit your website and read about you. I must appreciate your prompt hardwork at the age of 72. When the Brain is used more, it performs better because it learns to do so." CreationDate="2011-03-16T14:27:01.097" UserId="2956" />
  <row Id="13908" PostId="8344" Score="1" Text="@stevejb There isn't a specific question here yet: do you want to see how the influence function is computed?  Do you want a specific empirical example?  A heuristic explanation of what it means?" CreationDate="2011-03-16T14:29:38.583" UserId="919" />
  <row Id="13909" PostId="8332" Text="Let me second that in the main part of algorithms, you never want to actually invert a matrix. You may need to at the very end to get the standard estimates though. See this blog post http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/" CreationDate="2011-03-16T14:42:07.323" UserId="3713" />
  <row Id="13911" PostId="8359" Text="thanks for the nice example. Probably I still got a problem with the terminology. I see that the H0 of unit root is rejected – so what does stationary mean if there can be a time trend like you have shown?" CreationDate="2011-03-16T15:10:10.763" UserId="704" />
  <row Id="13912" PostId="8351" Text="@mpiktas: sorry, y should by x, it's the linear decreasing trend. Just changed because I thought an explanatory called y would be misleading and forgot it up there. now it's fine." CreationDate="2011-03-16T15:14:49.930" UserId="704" />
  <row Id="13913" PostId="8350" Score="3" Text="I'm not sure how this answers the question. Frequentists of course disagree with desideratum 1 from this list, so the rest of the argument doesn't apply to them. It also doesn't answer any of the specific questions in the OP, such as &quot;is Bayesian analysis more powerful or less error-prone than a frequentist analysis&quot;." CreationDate="2011-03-16T15:18:47.323" UserId="3369" />
  <row Id="13914" PostId="8364" Text="I'm not sure that asking about type 1/type 2 errors implies anything about a repeated sampling framework. It seems that even if my null hypothesis cannot be sampled repeatedly, it is still meaningful to ask about the probability of type 1 error. The probability in this case, of course, is not over all the possible hypotheses, but rather over all possible samples from my single hypothesis." CreationDate="2011-03-16T15:24:33.810" UserId="3369" />
  <row Id="13915" PostId="8359" Score="2" Text="@ran2, adf test is designed to test whether the stochastic trend is present. Although series with deterministic trend are non-stationary, regression analysis with them is &quot;safe&quot;. Also detrending is meaningful. With stochastic strend the usual detrending methods will give misleading results." CreationDate="2011-03-16T15:29:42.307" UserId="2116" />
  <row Id="13916" PostId="8295" Text="Look at conf.int from corr.test for each of the correlations. If the ranges overlap then that suggests that the two values of Source might be combined. But I would also use business knowledge and determine if it makes sense to combine them.  " CreationDate="2011-03-16T15:35:54.233" UserId="3489" />
  <row Id="13917" PostId="8364" Text="It seems to me that the general argument is this: although making a type 1 (or 2) error *can* be defined for a 'one shot' inference (Type 1 vs 2 is just part of a typology of mistakes I can make) unless my making this mistake is embedded in repeated trials neither error type can have a frequentist probability." CreationDate="2011-03-16T15:53:19.883" UserId="1739" />
  <row Id="13918" PostId="8364" Text="What I'm saying is that making a type 1 (or 2) error is *always* embedded in repeated trials. Each trial is sampling a set of observations from the null hypothesis. So even if it is difficult to imagine sampling a different hypothesis, repeated trials are still there because it is easy to imagine sampling a different set of observations from that same hypothesis." CreationDate="2011-03-16T15:59:02.640" UserId="3369" />
  <row Id="13919" PostId="8340" Text="@David: +1 Thank you for the suggestion. I changed it to a comparison question. Hope this is ok. I should have qualified my question with a simulation tag. I am evaluating my scenarios from a driver's perspective so less crashes is better. My original hypothesis was that the output (time at which 80% of the cars crash) does not depend on the three parameters. But from my results, I am tempted to say that I can reject my hypothesis but how strongly I can reject is still something I am looking into but thank you for the insight." CreationDate="2011-03-16T15:59:18.350" UserId="2164" />
  <row Id="13920" PostId="8364" Text="You are, I think backing off to from N-P to Fisher when you ask whether it doesn't still make sense to think of the p value as simultaneously a measure of divergence from some null, as a way to define a type 1 error, and as something that doesn't require a sampling framework.  I'm not sure how to have all these at once, but perhaps I'm missing something." CreationDate="2011-03-16T16:01:22.653" UserId="1739" />
  <row Id="13921" PostId="8364" Text="oops, we overlapped.  In the terms of my original response to the question you're saying that you *can* imagine embedding in a repeated framework.  OK, then Type 1 and perhaps Type 2 makes good sense." CreationDate="2011-03-16T16:04:43.383" UserId="1739" />
  <row Id="13922" PostId="8364" Text="I say that it does require a sampling framework, but that this sampling framework is easy to have. As a frequentist, I don't know what it means to &quot;sample a hypothesis&quot;. But I do know what it means to &quot;sample observations from a hypothesis&quot;. So I do this sampling repeatedly and use it to define type 1 errors. So I do use a sampling framework (of course), but this is sampling of observations only, not sampling of hypotheses." CreationDate="2011-03-16T16:07:06.720" UserId="3369" />
  <row Id="13923" PostId="8312" Text="@chl: +1 for glmnet, that's a great package." CreationDate="2011-03-16T16:07:21.307" UserId="2817" />
  <row Id="13926" PostId="8346" Text="@Dimitrij Celov: +1 for your time. I am sorry about my question. I fixed all the mistakes (parameters-&gt;variables) and giving more meaningful names to the variables. And sorry about not linking to my previous question (somehow I thought I did). I also clarified that this question is based on simulation and not on real-life empirical values. The values I am using are just values that are believed to be ok as there has been no real experiment. I am designing a protocol but before doing that, I wanted to understand the dynamics of my environment. I am still reading your post to digest everything." CreationDate="2011-03-16T16:16:51.563" UserId="2164" />
  <row Id="13927" PostId="8358" Score="1" Text="@DrWho Is this intended as a replacement for your previous question at http://stats.stackexchange.com/q/8187/919 ?" CreationDate="2011-03-16T16:17:24.123" UserId="919" />
  <row Id="13928" PostId="8343" Text="@user3727: +1 I am not sure I am doing this right but when I do `resid(my_model)` on my second one I am getting a NULL. At this point, I am not sure I should be considering the previous attempt because the experiment design was pretty careless on my part." CreationDate="2011-03-16T16:23:45.193" UserId="2164" />
  <row Id="13930" PostId="8295" Text="How do I run cor.test with the document-term matrix containing all of the words in the corpus and the result?  I can do it for a single word, but not for all of the cheap words and cheap results because &quot;'x' and 'y' must have the same length&quot;." CreationDate="2011-03-16T16:44:32.240" UserId="3712" />
  <row Id="13931" PostId="8295" Text="Sounds like it has to do with specific &quot;R&quot; syntax.  I would open it up as a new question." CreationDate="2011-03-16T17:01:50.620" UserId="3489" />
  <row Id="13932" PostId="8370" Text="I also know people are researching these ideas. Therefore, heatmap may be the only way to go." CreationDate="2011-03-16T17:05:50.787" UserId="1307" />
  <row Id="13933" PostId="8369" Score="1" Text="@charles.y.zheng. Logistic regression seems ok, but it has the additional constraint of row and column total being fixed. This may not be the case with Poisson data. I believe the answers won't differ much." CreationDate="2011-03-16T17:08:02.410" UserId="1307" />
  <row Id="13934" PostId="8371" Score="5" Text="Quota sampling is distinct from stratified sampling in precisely this respect: unlike selection in a stratified sample, the lack of randomness in quota sampling means we cannot assure lack of bias.  So: maybe you can't erect CI's for estimates from quota sampling, but you *can* establish intervals for the bias.  Unfortunately, except when the sample is a substantial proportion of the population, the bias extends from almost -100% to almost +100%.  This is why quota samples have long been discredited (&quot;Dewey defeats Truman&quot;, etc.).  *Caveat emptor.*" CreationDate="2011-03-16T17:45:15.717" UserId="919" />
  <row Id="13935" PostId="8346" Text="@Legend, the first regression you made was based on 20 points, now they are not doubled but 10 times bigger, 210 points you use in the second regression (at least F-test's degrees of freedom state that). What I am confused about is the joint probability that the certain *believed to be ok* combination of variables should be introduced in your *artificial* sample. To make &quot;meaningful&quot; comparisons do the 2 step in the post (compute beta coefficients), &quot;&quot; implies that conclusions will be subjective and not practical (but you may still try to do formal estimation)." CreationDate="2011-03-16T17:51:19.287" UserId="2645" />
  <row Id="13936" PostId="8367" Text="@Zach Do you suggest to rely on RFs to perform feature selection, and then apply a GLM -- in this case, there's risk of overfitting or over-optimism --, or to use RFs (with standard measures of var. importance, or all-relevant selection) as a standalone tool?" CreationDate="2011-03-16T18:05:55.290" UserId="930" />
  <row Id="13937" PostId="8303" Text="@Leendert You can revert to the previous version of your post, or combine both edits. I'm sure @mpiktas was of good intention when trying to improve its appearance and just didn't notice the No. parameters." CreationDate="2011-03-16T18:09:44.197" UserId="930" />
  <row Id="13938" PostId="8333" Text="I edited my answer to include more detail now." CreationDate="2011-03-16T18:15:41.033" UserId="2067" />
  <row Id="13939" PostId="8176" Text="Unfortunately I don't think the paper you cite addresses this problem.  The paper deals with when you have a set of non stochastic variables A, and finding the smallest element in A through sampling.  In the context of this problem, each element in A would be a random variable, and therein lies the kicker.  You have to find an unbiased estimator of the mean of the smallest random variable in A." CreationDate="2011-03-16T18:35:15.983" UserId="3602" />
  <row Id="13940" PostId="8333" Text="@luis Thank you: but what is &quot;r&quot;?  (Note that the recursion is bounded: at most one recursive call will be made, because a &gt; 0 implies 1.0+a &gt; 1.)" CreationDate="2011-03-16T18:35:46.950" UserId="919" />
  <row Id="13941" PostId="8333" Text="r is the random number generator (where you are getting the random numbers from)." CreationDate="2011-03-16T19:11:57.097" UserId="2067" />
  <row Id="13942" PostId="8372" Text="@whuber. Thank you, I will give it a try and update you with the results. If possible, could you also elaborate on how is it different from a heatmap hierarchical clustering with Euclidean distance metric? Thanks in advance." CreationDate="2011-03-16T19:26:38.590" UserId="1307" />
  <row Id="13943" PostId="8373" Text="Please don't [cross-post](http://quant.stackexchange.com/questions/740/time-series-regression-with-overlapping-data)." CreationDate="2011-03-16T19:43:39.823" UserId="1657" />
  <row Id="13944" PostId="8372" Text="@suncoolsu Your comment came while I was writing. Heatmap with HC will give close results, but HC can be done simultaneously on both rows and columns which might slightly alter the two separate views that are proposed here. Here is my [R code](https://gist.github.com/873226) as an attempt to replicate the above results, if you're interested." CreationDate="2011-03-16T20:20:48.903" UserId="930" />
  <row Id="13945" PostId="8364" Text="fwiw, I don't know what 'sample a hypothesis' might mean, and I'd guess that no Bayesian does either (except as part of some MCMC *implementation* detail). I guess that because you identify the use of random variables so tightly with the existence of random sampling you're thinking that Bayesian must sample their parameters/hypotheses.  (After all, they do treat them as random). But for Bayesians the two are decoupled and the priority reversed: the random variables represent uncertainty, which *may or may not* be caused by a sampling process." CreationDate="2011-03-16T20:24:20.850" UserId="1739" />
  <row Id="13946" PostId="8372" Text="@chl, thanks for the R code, I was going to ask @whuber for more details, now I do not have to:)" CreationDate="2011-03-16T20:25:04.470" UserId="2116" />
  <row Id="13947" PostId="8372" Text="@whuber +1, nice answer. I do not understand though, why the second graph shows some pattern when the data is random?" CreationDate="2011-03-16T20:27:09.273" UserId="2116" />
  <row Id="13948" PostId="8376" Text="@chl +1, thanks for the links." CreationDate="2011-03-16T20:27:59.070" UserId="2116" />
  <row Id="13949" PostId="8323" Score="1" Text="@Ryan (If you want to address @whuber directly, don't forget to prefix its username with `@` so that he can be notified.) I don't really like to intervene like this, but what is your &quot;original&quot; correlation about? I mean, a perm. test will inform you about the prob. of observing such a stat. ($r$ in your case) at least as extreme as the one you observed in your data (and with 10,000 draws, this is largely enough to conclude at a 5% level); then reporting your $p$-value is just a matter of computing the No. times you observed $r&gt;r_\text{obs}$, where $r_\text{obs}$ = your empirical corr." CreationDate="2011-03-16T20:41:57.497" UserId="930" />
  <row Id="13950" PostId="8367" Text="@chl: I was suggesting using the RFs to perform feature selection, and then apply the GLM.  I agree there's risk of over fitting, but the OP said he needed exactly 8 variables." CreationDate="2011-03-16T20:47:11.347" UserId="2817" />
  <row Id="13951" PostId="8367" Text="@Zach &quot;exactly 8 variables&quot;... Then, you are somewhat soft-thresholding the potential variables of interest based on a measure of variable importance (which thanks to being based on permutation and double resampling is supposed to be bias-free) and then reinject them into an GLM. IMHO, you break the control on overfitting exerted through bagging. This is also described in [ESLII](http://bit.ly/bW3Hr4) from Hastie et al.: feature selection, if any, must be included in the cross-validation procedure (where cross-validation include the assessment of model performance)." CreationDate="2011-03-16T20:57:02.553" UserId="930" />
  <row Id="13952" PostId="8333" Text="@luis Ah, this is clever: the product of a $\Gamma(\alpha+1)$ and an independent $B(\alpha,1)$ variate turns out to be a $\Gamma(\alpha)$ variate.  I edited my reply so it points to your solution and explains why it works." CreationDate="2011-03-16T20:58:43.347" UserId="919" />
  <row Id="13953" PostId="8372" Score="1" Text="@mpiktas The data are perfectly random and mutually independent, *but the orderings of the rows and columns are not.*  In some sense this is a multivariate analog of what happens when you sort a sequence of iid variates: they are still random, but the sequence now has a pattern!" CreationDate="2011-03-16T21:19:20.240" UserId="919" />
  <row Id="13954" PostId="8364" Text="I was referring to the text of your reply saying that &quot;it implies that you can put your inference problem into a repeated sampling framework&quot;. I thought you meant sampling hypotheses somehow. I understand now this is not what you meant. On the other hand, sampling data from the hypothesis seems to be readily possible. If that's the case, would it be fair to summarize your argument as saying that in most cases there is no strong reason to use Bayesian statistics?" CreationDate="2011-03-16T21:36:09.947" UserId="3369" />
  <row Id="13955" PostId="8372" Text="@whuber, thanks for explanation, I now understand the your original point: reordering can bring out patterns where none exist. But if we know in advance that reordering is not important, your second graph then should not display any patterns? " CreationDate="2011-03-16T21:41:40.690" UserId="2116" />
  <row Id="13956" PostId="8374" Score="1" Text="The title mentions that there is a gap in one variable, but from the body it seems that the gap is in both variables for which you are trying to compute the correlation. So what data exactly is missing?" CreationDate="2011-03-16T21:44:49.643" UserId="2116" />
  <row Id="13957" PostId="8372" Text="@mpiktas I guess I haven't been clear enough.  Reordering the rows and columns will help us see patterns.  *I recommend it* (at least when there is no inherent *a priori* order to the rows or columns).  However--this was purely an incidental aspect of the example I created--you *will* find patterns even in perfectly random data.  Therefore you need to test the patterns you find for statistical significance." CreationDate="2011-03-16T21:54:55.577" UserId="919" />
  <row Id="13958" PostId="8377" Text="The two together don't tell you anything more than the second one would alone!  The main effects are uninteresting and misleading when there is interaction present.  The second model tells you all you need to know.  Describe your variables more (what are they, what are their ranges like, are they both numeric or are they factors?) and someone might help you think about what that interaction means/looks like." CreationDate="2011-03-16T22:04:05.460" UserId="485" />
  <row Id="13959" PostId="8371" Text="Pollsters are claiming that quota sampling, which is the industry standard in France, is also widespread in Britain and Southern European states. I wonder if Britain really uses quota sampling, and why use quota sampling in the first place if the bias is irremediable?" CreationDate="2011-03-16T22:30:31.307" UserId="3582" />
  <row Id="13960" PostId="8350" Text="@sheldoncooper - if a frequentist disagrees with desideratum 1, then on what basis can they construct a 95% confidence interval?  They must require an additional number." CreationDate="2011-03-16T22:32:44.523" UserId="2392" />
  <row Id="13961" PostId="8375" Text="A couple questions: First, are you wedded to using a discrete time model? What you want to do may be simpler in a parametric survival model. Second, would you be able to provide an example of what you mean by churn; ow maybe just an example of what your data look like?" CreationDate="2011-03-16T22:33:30.203" UserId="3265" />
  <row Id="13962" PostId="8371" Score="1" Text="@Fr Quota sampling is easier to carry out than random sampling: the pollster can specify what kinds of subjects are needed and how many to interview, send interviewers onto the street with these instructions, and easily check whether the job was correctly done: no need to do the difficult work of establishing a sampling frame, drawing a probability sample from it, hunting down its members, chasing after nonrespondents, *etc*.  Quota sampling is a textbook poster child for outmoded, known-bad sampling methods: see http://www.math.uah.edu/stat/data/1948Election.pdf for instance." CreationDate="2011-03-16T22:40:31.810" UserId="919" />
  <row Id="13963" PostId="8382" Score="3" Text="http://en.wikipedia.org/wiki/Winsorized_mean" CreationDate="2011-03-16T22:59:33.277" UserId="919" />
  <row Id="13964" PostId="8350" Text="@sheldoncooper - and further, sampling probabilities would have to be re-defined, because they too are only 1 number.  A frequentist cannot reject desideratum 1 without rejecting their own theory" CreationDate="2011-03-16T23:12:45.030" UserId="2392" />
  <row Id="13965" PostId="8377" Text="@Brett There seems to be a mystery here, though.  According to the first model all group means are negative.  According to the second model, group means with a=1 are strongly positive.  It makes me wonder whether the independent variables have been coded the same way in both calculations..." CreationDate="2011-03-16T23:13:51.483" UserId="919" />
  <row Id="13966" PostId="8377" Text="a ranges from -1.37 to 2.12, b ranges from -1.03 to 1.3. Both variables are numeric. (I must refrain from posting what exactly a and b represent in realworld, since the data belongs to a research group, and I just offered to help out to improve my knowledge) " CreationDate="2011-03-16T23:14:37.640" UserId="3745" />
  <row Id="13967" PostId="8377" Text="@Sarah That information is helpful, thanks." CreationDate="2011-03-16T23:16:33.650" UserId="919" />
  <row Id="13968" PostId="8350" Score="1" Text="I'm not sure what additional number you are referring to. I'm also not sure what is the computation of $p(H_1|...)$ that you've introduced. The procedure in standard statistical tests is to compute $p(E_1, E_2, ... | H_0)$. If this probability is low, $H_0$ can be rejected; otherwise it cannot." CreationDate="2011-03-16T23:21:12.617" UserId="3369" />
  <row Id="13969" PostId="8350" Score="1" Text="&quot;they cannot reject desideratum 1 without rejecting their own theory&quot; -- what do you mean by that? Frequentists have no notion of &quot;plausibility&quot;. They have a notion of &quot;frequency of occurrence in repeated trials&quot;. This frequency satisfies conditions similar to your three desiderata and thus happens to follow similar rules. Thus for anything for which the notion of frequency is defined, you can use laws of probability without any problem." CreationDate="2011-03-16T23:25:31.443" UserId="3369" />
  <row Id="13970" PostId="8378" Text="Thanks Lockedoff, especially for helping me sort out my own thinking w.r.t the difference between the conditional and joint independence model" CreationDate="2011-03-16T23:26:27.203" UserId="3732" />
  <row Id="13971" PostId="8363" Text="Think the link is broken, did you mean [this categorical one here](http://biology.ucf.edu/~pascencio/classes/Methods/R%20demos/)? That was helpful, thanks!" CreationDate="2011-03-16T23:30:46.357" UserId="3732" />
  <row Id="13972" PostId="8347" Text="Why is this not CW? It seems like a ripe situation for devolving into an argumentative, subjective discussion. At the very least, it seems like they wrong situation for awarding or taking reputation based on the answers." CreationDate="2011-03-16T23:33:45.247" UserId="2970" />
  <row Id="13973" PostId="8347" Score="1" Text="I think this question can potentially have a &quot;good&quot; or &quot;correct&quot; answer. E.g. if someone could say &quot;for every frequentist test with type 1 error $\alpha$ and type 2 error $\beta$, there exists a Bayesian test with type 1 error $\alpha$ and type 2 error $\beta - x$&quot;, this would be a good answer. Or something like &quot;every frequentist test is equivalent to a Bayesian test with uninformative prior&quot;. I.e. this doesn't have to be a religious war between frequentists and bayesians. I'm only arguing because I don't understand how the replies relate to the specific questions in OP." CreationDate="2011-03-16T23:56:15.910" UserId="3369" />
  <row Id="13974" PostId="8350" Text="Just to clarity, in frequentist interpretation the notation $p(...|H_0)$ does not of course represent conditioning on a random variable $H_0$. Rather, it's a convenient shortcut for $p_{H_0}(...)$, the probability of $...$ under model $H_0$." CreationDate="2011-03-16T23:59:17.937" UserId="3369" />
  <row Id="13975" PostId="8383" Text="Thanks for the book recommendations link, &quot;Statistics in Plain English&quot; sounds like a very inviting title :) I am gonna check it out. " CreationDate="2011-03-17T00:02:13.180" UserId="3745" />
  <row Id="13976" PostId="8380" Score="2" Text="Suppose the exact variable values are $x_i$, the exact output value is $y$, observed variable values are $\hat{x}_i$ and the observed output value is $\hat{y}$. Is it true that in your model $y = \sum_i w_i x_i$ (i.e. the exact output is a linear function of the exact variable values)? Then, if you had access to $x_i$ and $\hat{y}$, this would be standard regression. But the issue is that you don't have $x_i$, but only $\hat{x}_i$, so you want to build a model of the form $\hat{y} = \sum w_i \hat{x}_i$, even though the true model is $y = \sum_i w_i x_i$?" CreationDate="2011-03-17T00:17:57.700" UserId="3369" />
  <row Id="13977" PostId="8358" Text="@whuber: I am amazed at your keen observation. Yes. Now we could recover more data from a crashed hard disk. I thought that increase in the sample size will increase the power of the study and yield better inferences." CreationDate="2011-03-17T00:44:49.570" UserId="2956" />
  <row Id="13978" PostId="8334" Text="Thanks for the leads. They are very informative." CreationDate="2011-03-17T01:13:19.383" UserId="2997" />
  <row Id="13979" PostId="8357" Text="Thankyou very much. I will do the search and the reading." CreationDate="2011-03-17T01:14:59.023" UserId="2997" />
  <row Id="13980" PostId="8375" Text="@JWM. Churn means the customer canceled their account. Since I know only the month the customer canceled, I think discrete time is appropriate. I believe the same issue with time varying predictors exists if the model is discrete or continuous time (Cox or AFT) No?  " CreationDate="2011-03-17T01:22:40.360" UserId="2040" />
  <row Id="13981" PostId="8358" Score="1" Text="@DrWho It might be best to merge these two questions, then.  That can help garner more focused answers for you.  Consider editing your previous one in light of the information in this one." CreationDate="2011-03-17T02:05:09.637" UserId="919" />
  <row Id="13982" PostId="8391" Text="+1 Nicely explained." CreationDate="2011-03-17T02:07:25.920" UserId="919" />
  <row Id="13983" PostId="8350" Text="@sheldon - if you reject desideratum 1, then presumably you require more than one number to represent the plausibility of an event.  This means you cannot represent $P(\dots|H_0)$ by a single number.  But this is what a frequentist does! calculates the likelihood (i.e. 1 number).  Therefore, a frequentist cannot reject desideratum 1.  For frequentist &quot;plausibility&quot;=&quot;frequency&quot; I would have thought." CreationDate="2011-03-17T02:25:39.263" UserId="2392" />
  <row Id="13984" PostId="8394" Text="@A.C. that is a nice package that you have written, but I am interested in estimating the sample SE, and these functions appear to give variance estimates for the meta analysis effect sizes, whereas I would like to infer the variance of the population (e.g. scaled to the original data). Could you provide an example of how the functions in the `compute.es` package could be used to replicate the equations and function that I have written above?" CreationDate="2011-03-17T02:47:25.173" UserId="1381" />
  <row Id="13985" PostId="8367" Text="@chl: Doesn't the r package &quot;boruta&quot; cross validate by running the random forest several time?  Are you basically saying we need &quot;meta-cross-validation,&quot; where you do variable selection and and the fit your model on random subsets of the data?" CreationDate="2011-03-17T02:49:16.123" UserId="2817" />
  <row Id="13986" PostId="8022" Text="+1. This is one of the techniques recommended by the comp.graphics.algorithms FAQ [&quot;Uniform random points on sphere&quot;.](http://www.cgafaq.info/wiki/Uniform_random_points_on_sphere)" CreationDate="2011-03-17T02:51:57.870" UserId="3754" />
  <row Id="13987" PostId="8252" Text="@Henry OK.  I understand &quot;robust&quot; and &quot;information&quot; in a different sense, I guess." CreationDate="2011-03-17T02:57:18.610" UserId="919" />
  <row Id="13988" PostId="8358" Text="@whuber: Thank you for the advice. But I am afraid, many would think that the query is already answered and may not be interested in discussing further. Moreover the statistical approach is likely to change totally. Incorrect approach would make such important and pains-taking data a waste. Kindly advise me." CreationDate="2011-03-17T02:58:29.033" UserId="2956" />
  <row Id="13989" PostId="8372" Text="@whuber. Thank you  for the insights. You are a great resource. Saves a lot of time for me." CreationDate="2011-03-17T03:14:43.753" UserId="1307" />
  <row Id="13990" PostId="7689" Text="As shown by the numerous answers (none being accepted by Scott Weinstein btw), the question of which diversity index is optimal seems to be trickier than it appears at first sight. In this context, I thought that a link towards that paper would be of some interest, as it aims at clarifying the topic in an axiomatic way. But you are right, it is maybe far too theoretical compare with what Scott was looking for. (And sorry for replying so late!)" CreationDate="2011-03-17T03:49:03.893" UserId="3459" />
  <row Id="13991" PostId="8353" Text="@IrishStat: Thank you very much for the clarification. Now I am able to recover lost data from a crashed hard disk. I have posted complete data in a new question as I felt the statistical approach and calculations are likely to change but increased sample size will increase the power of the study. The revised question is at http://stats.stackexchange.com/questions/8358/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-cas      This new question has additional data from 1994 to 2003. I  apologise for the inconvenience." CreationDate="2011-03-17T04:05:18.403" UserId="2956" />
  <row Id="13992" PostId="8353" Text="@IrishStat: AFS university book at http://www.autobox.com/AFSUniversity/afsuFrameset.htm is excellent for a beginner like me. Entertainment section is a great collection. Can it be downloaded? I wish it can be so that I can read it even when offline." CreationDate="2011-03-17T04:43:15.267" UserId="2956" />
  <row Id="13993" PostId="8383" Text="I strongly second plotting the data. See http://eagereyes.org/criticism/anscombes-quartet" CreationDate="2011-03-17T04:45:39.617" UserId="287" />
  <row Id="13994" PostId="8325" Text="@Patrick McCann: Thank you very much for the simple but useful reference." CreationDate="2011-03-17T05:40:23.563" UserId="2956" />
  <row Id="13995" PostId="8353" Score="1" Text="@DrWho: Simply right click on http://www.autobox.com/AFSUniversity/afsuFrameset.htm and then select &quot;open link in new window&quot; and then left click on  &quot;Entertainment&quot; which will bring up the text. Higlight the text and copy to the clipboard and then paste to a word doc and you are good to go. I will look at your revised data and try to answer your &quot;new question&quot;" CreationDate="2011-03-17T06:06:37.597" UserId="3382" />
  <row Id="13996" PostId="8323" Text="@chl, thanks for your answer. Although, what do you mean by my &quot;original&quot; correlation? Also, you said that reporting a p-value is a matter of computing the # of times I observe r&gt;robs. For example let's say my observed correlation value is 0.7, and in 10,000 draws, only 10 of them were larger than 0.7. Then, my p-value is 10/10,000=0.001?" CreationDate="2011-03-17T06:31:23.207" UserId="3720" />
  <row Id="13997" PostId="8350" Text="&quot;if you reject desideratum 1, then presumably you require more than one number to represent the plausibility of an event&quot; -- sure, not that I care. &quot;This means you cannot represent $P(\dots|H_0)$ by a single number.&quot; -- I disagree with that. For a frequentist $P(\dots|H_0)$ is not plausibility; in fact, frequentist doesn't care about plausibility at all. It's just frequency. " CreationDate="2011-03-17T07:03:09.263" UserId="3369" />
  <row Id="13998" PostId="8398" Text="@Rob, @mohamed, (+1) nice to hear from the author of the function ^__^ From `example(accuracy)` it seems that x should be supplemnented first to see Theil's U and ACF1 see you do use `forecasterrors()` if `!missing(x)`. I think, for dr.Abed it is not intuitively clear. " CreationDate="2011-03-17T07:03:18.477" UserId="2645" />
  <row Id="13999" PostId="8350" Text="&quot;For frequentist &quot;plausibility&quot;=&quot;frequency&quot; I would have thought.&quot; -- again, I disagree. My understanding is that frequency is the only thing frequentists care about (hence the name). The notion of &quot;plausibility&quot; for a frequentist is at best some vague intuition which sometimes helps and sometimes doesn't, but definitely not something equivalent to probaiblity of an event." CreationDate="2011-03-17T07:07:01.040" UserId="3369" />
  <row Id="14000" PostId="8376" Text="@chl thanks for the links! I read Hastie et al. It is something which might be very useful. " CreationDate="2011-03-17T07:08:19.573" UserId="1307" />
  <row Id="14001" PostId="8379" Text="(+1) nice spider fish anecdote! " CreationDate="2011-03-17T07:15:01.163" UserId="2645" />
  <row Id="14002" PostId="8401" Text="typo in the last paragraph? $C(w)$ instead of $c(w)$?" CreationDate="2011-03-17T07:30:46.813" UserId="2116" />
  <row Id="14003" PostId="8399" Score="1" Text="Please copy the rest of the question here." CreationDate="2011-03-17T07:47:13.073" UserId="88" />
  <row Id="14004" PostId="8371" Text="@whuber: Thanks for the details. I knew about the &quot;Dewey defeats Truman&quot; story. I still doubt that British polling institutes are indeed using quota sampling, as claimed by the previously cited document, and would have to check. It would be interesting to know what are the different industry standards across the largest countries. I know for a fact that most of the data I download from [ICPSR](http://www.icpsr.umich.edu/icpsrweb/ICPSR/) is Random Digit Dialling." CreationDate="2011-03-17T07:49:56.230" UserId="3582" />
  <row Id="14005" PostId="8379" Text="whuber is wise.  With a gap that big, I think it is almost never justified to give any importance to a single measure of correlation." CreationDate="2011-03-16T22:59:14.990" UserId="3748" />
  <row Id="14006" PostId="8363" Text="Yes, it seems that the space in the URL breaks it." CreationDate="2011-03-17T07:52:05.093" UserId="3467" />
  <row Id="14007" PostId="8367" Score="1" Text="@Zach My point was that bagging, or in particular RFs, prevent from overfitting (to a certain extent) provided you remain in the same toolchain. If you use the results from RFs and then go see how another model would perform with the same data, then you break the CV loop. But why not use directly RFs for classification? Another solution would be to use RFs on a training sample and then apply the GLM on an held-out sample (that could potentially be cross-validated too). " CreationDate="2011-03-17T07:56:45.867" UserId="930" />
  <row Id="14008" PostId="8323" Text="@Ryan Let's say you estimate a correlation of 0.36 on *your* sample -- this is what I call the observed or original correlation. Now you permute the rows of one your block (this would break the link between the two blocks) and compute your cross-correlations (between each variable of each block), iterate say 10,000 times, and then yes you can compute the empirical p-value as you described. The same strategy could be used to assess the significance of the first canonical correlation in a CCA, BTW." CreationDate="2011-03-17T08:02:04.257" UserId="930" />
  <row Id="14009" PostId="8353" Text="@IrishStat: Thank you very much. It is appreciable that you are working late night," CreationDate="2011-03-17T08:12:00.750" UserId="2956" />
  <row Id="14010" PostId="8401" Text="You don't *have* to use Monte Carlo methods for high-dimensional integrals just because a closed form doesn't exist. You *could* use methods based on [sparse grids](http://en.wikipedia.org/wiki/Sparse_grid). I don't know the pros and cons myself though." CreationDate="2011-03-17T08:26:08.960" UserId="449" />
  <row Id="14011" PostId="8304" Text="Thank you for clearing my confusion about `which` and `subset`. I was doing it more complicated than it was. The tip about my successive subsetting is also speeding up my analyses, which is great." CreationDate="2011-03-17T08:31:42.497" UserId="3705" />
  <row Id="14012" PostId="8392" Text="Although there are statistical aspects to this Q there are also sysadmin aspects, suggesting the folks over at serverfault.SE might be able to contribute. Is this perhaps a rare case where cross-posting a Q on more than one SE site would be acceptable? " CreationDate="2011-03-17T08:33:56.737" UserId="449" />
  <row Id="14013" PostId="8269" Text="Yes I wanted to exclude the observations that fulfilled the conditions. So this was exactly what I meant, thank you. The function works beautifully now." CreationDate="2011-03-17T08:41:09.897" UserId="3705" />
  <row Id="14014" PostId="8392" Text="@onestop: &quot;sysadmin&quot;-aspects in the sense of &quot;datacollection&quot;-aspects ? In this case I agree." CreationDate="2011-03-17T08:44:33.530" UserId="264" />
  <row Id="14015" PostId="8401" Text="@mpiktas - thanks for pointing it out&#xA;" CreationDate="2011-03-17T08:53:06.423" UserId="3572" />
  <row Id="14016" PostId="8401" Text="@onestop - you are right sparse grids work a little better in my case, though I'd also like to test this idea out." CreationDate="2011-03-17T08:53:33.050" UserId="3572" />
  <row Id="14017" PostId="367" Text="I second this. Also, there is quite a lot of suggestions for further reading which I think are useful in the book." CreationDate="2011-03-17T08:55:43.243" UserId="199" />
  <row Id="14018" PostId="8272" Text="I haven't even noticed that V5 was a character variable, thank you for pointing it out. Somehow it still worked, but it's good to know in downstream analyses. I am still confused about `&amp;&amp;`, but I am saving that for later. " CreationDate="2011-03-17T08:56:49.540" UserId="3705" />
  <row Id="14019" PostId="8401" Text="It sounds a bit like finance." CreationDate="2011-03-17T09:13:34.817" UserId="1351" />
  <row Id="14020" PostId="8392" Text="@steffen : Seems to me the question of whether it's *possible in principle* requires system admin knowledge. If it *is* possible, the best way to do it from the available info is then suitable q for statisticians and data miners. Of course sysadmins and statisticians/data miners aren't mutually exclusive sets!" CreationDate="2011-03-17T09:19:07.540" UserId="449" />
  <row Id="14021" PostId="8398" Score="2" Text="I've added an example. If `x` is omitted, only in-sample results can be computed as explained in the help function. It would be possible to add an in-sample version of Theil's U as shown in the example, but I've never needed it and I can't imagine why anyone else would. But I can add it to the next version if it is useful to someone." CreationDate="2011-03-17T09:34:01.287" UserId="159" />
  <row Id="14022" PostId="8304" Text="@Caroline, you are welcome." CreationDate="2011-03-17T10:14:41.333" UserId="2116" />
  <row Id="14023" PostId="8377" Text="@whuber Not really a mystery, I don't think.  For one thing, the intercept varies.  Also, the interaction term can do strange things." CreationDate="2011-03-17T10:29:01.317" UserId="686" />
  <row Id="14024" PostId="8383" Text="I agree with @MichaelBishop - plotting is essential, especially when there is an interaction" CreationDate="2011-03-17T10:30:03.610" UserId="686" />
  <row Id="14025" PostId="8401" Score="1" Text="It could be applied to finance. We are using it for the optimal management of power systems though :)" CreationDate="2011-03-17T10:41:22.277" UserId="3572" />
  <row Id="14027" PostId="8350" Text="So on what basis can a &quot;frequentist&quot; as you describe it make decisions if they do not adopt a notion of plausibility (or some other synonym of the word)?  The world is uncertain, so how does the frequentist take account of this?  Can you define what a confidence interval is without either adopting a notion of &quot;likelihood&quot; or &quot;prediction&quot;? (which are all equivalent to the notion of plausibility).  It seems by your notion would only be able to describe the past, or what is observed, making it useless for the problems people want statistics for. (more later)" CreationDate="2011-03-17T11:26:57.807" UserId="2392" />
  <row Id="14028" PostId="8350" Text="...(cont'd) people use statistics to make inferences about what is unknown.  Frequencies are very important, but they can only be observed, they cannot be used for prediction without adopting some connection between &quot;likelihood&quot; and &quot;frequency&quot;.  This all falls under the cloud of &quot;plausible reasoning&quot;.  It could be my use of terminology and particular words that are obscuring my point.  And you still haven't answered my earlier question &quot;on what basis can a confidence interval be used without a notion of plausibility?&quot;" CreationDate="2011-03-17T11:33:48.533" UserId="2392" />
  <row Id="14029" PostId="8350" Text="@sheldon - just looking at your earlier comment, you seem to think that by writing $P(H_0|\dots)$ I have somehow made $H_0$ &quot;random&quot;.  This is not true.  This just means I want to know the extent to which the evidence I have (on the right hand side of &quot;|&quot;) implies the hypothesis I am testing(I am using the word &quot;implies&quot; in *exactly the same way as in deductive logic*)." CreationDate="2011-03-17T11:39:22.837" UserId="2392" />
  <row Id="14030" PostId="8347" Text="@sheldon - I would agree here (+1 from me), but at the same time it is a &quot;loaded&quot; question, in that it is essentially asking for a value judgment on behalf of the person answering the question.  And there is nothing wrong with challenging either side - it should make both stronger (as long as people &quot;keep their cool&quot; and don't make things personal, stay on issue).  I think that's one of the best things about this forum - you can express your ideas, and see what other people think." CreationDate="2011-03-17T11:46:08.353" UserId="2392" />
  <row Id="14031" PostId="8403" Score="2" Text="To Peter's good comments, I'll add that it's hard to imagine any audience/consumer of this research who is going to give it any credibility with these sample sizes.  Can you say something about the context and purpose of the research?" CreationDate="2011-03-17T11:53:50.043" UserId="2669" />
  <row Id="14032" PostId="8364" Text="Riddle me this: how does one decide &quot;what is random?&quot;  E.g. suppose you have an urn, someone is sampling &quot;at random&quot; from the urn.  Also suppose an &quot;intelligent observer&quot; is also present, and they know the exact contents of the urn.  Is the sampling still &quot;at random&quot; even though the &quot;intelligent observer&quot; can predict with certainty exactly what will be drawn?  Has anything about the urn changed if they are no longer present?" CreationDate="2011-03-17T11:54:32.277" UserId="2392" />
  <row Id="14033" PostId="8354" Text="@dmitrij - That is what was implied (or what I inferred) from the link - it is about the robustness properties of a statistic.  Influence functions are slightly more general than 1 data point - you can redefine the delta function to be a sum of them (so many observations).  I would think of it as a &quot;cheap Jacknife&quot; to some degree - because you don't require re-fitting of the model." CreationDate="2011-03-17T12:02:30.443" UserId="2392" />
  <row Id="14034" PostId="8344" Score="1" Text="If you look up Frank Critchley's 1986 paper &quot;influence functions in principal components&quot; (can't remember the exact name of the paper).  He defines the influence function for ordinary regression here (which may or may not prove my answer wrong)." CreationDate="2011-03-17T12:05:27.743" UserId="2392" />
  <row Id="14035" PostId="783" Text="Rosnow &amp; Rosenthal's is a very useful, eye-opening quote that is *almost* correct.  " CreationDate="2011-03-17T12:15:01.923" UserId="2669" />
  <row Id="14036" PostId="8405" Text="Could you show a reproducible example of what you can do, and a mock-up of what would you like to achieve? You are mentioning very powerful functions (e.g. `summaryBy`), and most likely you can do what you want with them, but it is hard to understand what you are missing." CreationDate="2011-03-17T12:33:26.443" UserId="279" />
  <row Id="14037" PostId="8408" Text="Perfect, thanks a lot!" CreationDate="2011-03-17T12:37:03.863" UserId="3757" />
  <row Id="14038" PostId="8405" Text="@Aniko: I've updated the question with example output." CreationDate="2011-03-17T12:41:50.770" UserId="2824" />
  <row Id="14039" PostId="8377" Text="@Peter Due to the ANOVA presentation I had mistakenly imagined a and b might be binary categorical variables, that's all.  You are right that there is no mystery when a and b have the ranges indicated by @Sarah." CreationDate="2011-03-17T12:58:34.957" UserId="919" />
  <row Id="14041" PostId="8319" Text="@amit - Go team!" CreationDate="2011-03-17T13:21:11.733" UserId="2392" />
  <row Id="14042" PostId="8410" Text="@Aniko: Thanks. I'm working through your answer now. The first `g` in the first formula must be `G`. Why do you have a third data set (`B`), if `A` is your continuous and `G` is your factor/categorical?" CreationDate="2011-03-17T13:29:22.217" UserId="2824" />
  <row Id="14043" PostId="7126" Text="@sesqu anyway, parametric modeling is a good way to compress the density so +1 for that" CreationDate="2011-03-17T13:34:29.427" UserId="223" />
  <row Id="14044" PostId="8265" Text="@bayer, +1 I like the way you turn the problem with &quot;the more likely an event is to occur the more bits for that event. But you should notice that often the tails of the distribution are somehow &quot;precious&quot; (in wind power forecast, there are relatively rare errors or events that cost a lot in terms of money)" CreationDate="2011-03-17T13:37:26.147" UserId="223" />
  <row Id="14045" PostId="8364" Text="@SheldonCooper The short version: IF (you can [put your problem in a resampling framework] AND if the behavior of your estimator over many such samples is of relevance AND if you are not particularly interested in making probability statements about particular events AND you don't mind a plethora of competing principles) THEN there's no particular reason to change what you're doing.  I don't know whether the antecedent describes  'most cases' but it doesn't describe most of my cases." CreationDate="2011-03-17T13:47:19.080" UserId="1739" />
  <row Id="14046" PostId="8272" Text="@Caroline You can check everything easily in R: `&quot;1&quot;==1` returns true (T) but `&quot; 1&quot;==1` returns false (F). `c(T,F,T) &amp; c(F,F,T)` returns `F,F,T` as expected but `c(T,F,T) &amp;&amp; c(F,F,T)` returns `F` because it only compares the first element in each vector." CreationDate="2011-03-17T14:24:25.843" UserId="3432" />
  <row Id="14047" PostId="8382" Text="I guess that could help but I would really need to change how to take the winsorized mean when i'm close to saturation and zero so it really doesn't solve my problems.. I was looking for an approach with more mathematical foundations. I mean, this problem seems like such a common thing that someone must have done work on it already. " CreationDate="2011-03-17T14:50:23.823" UserId="3749" />
  <row Id="14048" PostId="8382" Text="@umps I don't understand the problem.  The computation is simple.  You only need to compute what proportion of your data are 0 and what proportion are 127 (&quot;saturation&quot;).  Let $q$ be the larger of those two proportions.  If $q\ge.50$ you're out of luck no matter what.  Otherwise, let $\bar{m}$ be the mean of the middle $1-2q$ of the data (lying between the $q$ and $1-q$ quantiles), $x_{-}$ be the $q$ quantile, and $x_{+}$ be the $1-q$ quantile.  The Winsorized mean equals $(1-2q)\bar{m}+q(x_{-}+x_{+})$.  I don't understand the reference to &quot;more mathematical foundations,&quot; either." CreationDate="2011-03-17T14:59:55.773" UserId="919" />
  <row Id="14049" PostId="7777" Text="Thanks for your extensive answer suncoolsu.  I'm in a similar situation and have a couple of questions.  1.  I also get 0 for every model using your chi-sq test equation.  2. The wikipedia page on AIC says &quot;the preferred model is the one with the minimum AIC value&quot;, but you said, &quot;AIC should be high for a good fit.&quot;  I'm trying to reconcile these accounts." CreationDate="2011-03-17T15:02:41.583" UserId="3388" />
  <row Id="14050" PostId="8410" Text="I fixed the problem with `g` - it should have been `G`. I used `B` because I wanted to have a third variable inside the table for the second example. Also, I kept `cut`-ting `A` to make rows of it to mirror your table." CreationDate="2011-03-17T15:54:21.373" UserId="279" />
  <row Id="14052" PostId="8129" Text="@Steffen Thanks! I think I will go that route then, i.e., the present just accuracy and that the model is not predicting just the major class." CreationDate="2011-03-17T15:59:01.207" UserId="3604" />
  <row Id="14054" PostId="8358" Score="1" Text="@whuber: I agree that this question was answered but the new data (17 years ) may present analysis opportunities unavailable with the smaller data set (6 years). IrishStat 2 hours ag" CreationDate="2011-03-17T13:39:30.443" UserId="3382" />
  <row Id="14057" PostId="8380" Text="Thanks @SheldonCooper - your notation helps.  I think you have almost everything right.  I have two data sets $D_1$ and $D_2$.  $D_1$ contains a vector of observed output variables $Y_1$ and a matrix with corresponding exact variable values $X_1$.  $D_2$ contains a different matrix of exact variable values $X_2$ and a corresponding matrix of observed noisy variables $\hat{X}_2$.  I do _not_ have any $Y$ vector in $D_2$, but I want to build a linear model that takes a noisy observation vector $\hat{x}$ and estimates $y$ (no exact $x$ will be available to predict $y$)." CreationDate="2011-03-17T16:26:07.587" UserId="3744" />
  <row Id="14058" PostId="8415" Score="1" Text="@Pantera, what about the second model and residual analysis (and what analysis made you sure that no anomalies are present)? what estimator do you use for AR(1) with deterministic trend (the first model)?" CreationDate="2011-03-17T16:27:35.730" UserId="2645" />
  <row Id="14060" PostId="8392" Score="2" Text="More to the point: why is there any reason to suppose such data would give reliable information about course usage of software?  There are so many possible confounders that the results could at best be suggestive, at worst be woefully wrong.  This sounds more like a *management* problem to me.  For instance, one could collect these data, fire a summary back to the professors, and threaten to charge them license fees unless they confirm non-usage.  *That* might get some responses!" CreationDate="2011-03-17T16:51:49.280" UserId="919" />
  <row Id="14061" PostId="8265" Text="But then you can use a heavy tailed distribution for that, right?" CreationDate="2011-03-17T16:53:48.000" UserId="2860" />
  <row Id="14062" PostId="8418" Text="What do you mean by &quot;isolate speed bins&quot;? Do you mean that you want to fit a model for each bin?" CreationDate="2011-03-17T16:59:49.270" UserId="287" />
  <row Id="14064" PostId="8418" Text="Exactly. Apologies if I'm being naiive!" CreationDate="2011-03-17T17:18:59.693" UserId="3761" />
  <row Id="14065" PostId="8416" Text="Hi Celov, and thank for your comments. The first model (short-dun model with the lagged dep.variable) has no anomalies, i.e. no first or higher order autocor. residuals, no uncond. or cond. heterosc. and the residuals are normally distributed. The model is estimated with OLS. If the second model, y(t) = alpha + beta*Time + residual, is estimated with OLS, then the residuals are autocorrelated. However, if we estimate the model by using autoregressive least square and model the autocorrelation, we solve the autocor. problem. the gamma (growth rate) in the short-run mod is &lt; gamma in model 2." CreationDate="2011-03-17T17:26:31.617" UserId="3762" />
  <row Id="14067" PostId="8332" Text="@JMS: The values of D matrix are getting smaller smaller as the number of iterations increases. Maybe this is the problem as you pointed out." CreationDate="2011-03-17T17:41:02.167" UserId="3499" />
  <row Id="14068" PostId="8420" Score="1" Text="This question would be of interest at http://gis.stackexchange.com because most of the examples will be maps.  A nice example (comparing GPD and GPD per capita) was quoted in a recent discussion of cartograms http://gis.stackexchange.com/q/7406/664" CreationDate="2011-03-17T17:42:38.767" UserId="919" />
  <row Id="14069" PostId="8416" Text="@Rob Hyndman, @Pantera, Rob could actually add some nice ideas here. But as for autoregressive process with trend, I would not go for OLS directly (in fact the parameter estimates will differ if you go for lagged dependent variable regression and estimate $AR(1)$ process). The choice is quite large: maximum likelihood, Yule Walker equations, Burgs (well OLS is among methods for $AR(p)$ process though :)). The autoregressive residuals cures are more for classical regressions, in my opinion." CreationDate="2011-03-17T17:45:07.977" UserId="2645" />
  <row Id="14071" PostId="8416" Text="I also modelled the y(t)-series by estimating an ARIMA(4,1,0) incl. a constant (it's yearly data), but the constant was not significant. Well, the scientific objective is to estimate the trend - and I agree with you, the deterministic long-run trend is only valid in a limited interval. " CreationDate="2011-03-17T17:54:18.823" UserId="3762" />
  <row Id="14074" PostId="8332" Score="1" Text="@Andy Amos: I'd bet money on it. Like @whuber points out it's almost impossible that $BB'$ would have negative eigenvalues if you're computing it directly, &amp; the zeros (from being rank deficient) should be taken care of by adding $D$ since its positive diagonal - unless some of those elements are really small. Try generating some data from a model where $\sigma_i^2$ are pretty large and $\sum_q B_{iq}^2 \approx \sigma_i^2$. The more data the better so that  the estimates should be accurate and stable. That will at least tell you if there's a problem in your implementation." CreationDate="2011-03-17T18:17:43.603" UserId="26" />
  <row Id="14075" PostId="8328" Score="1" Text="It's got to be small elements in $D$; $I_q + B'D^{-1}B$ should be well-conditioned otherwise since $q&lt;p$" CreationDate="2011-03-17T18:19:41.340" UserId="26" />
  <row Id="14076" PostId="8401" Text="&quot;And how can I tell that the stationary distribution has been reached?&quot; You can't - ever, really - unless you know the target distribution (and its normalizing constant). Convergence rates can be established for some very special cases which almost certainly don't apply here. Lack of convergence *can* be demonstrated but is often not apparent by inspection or in the various diagnostic tests. This is a vitally important thing to remember when you're using MCMC, particularly if you are new to the game." CreationDate="2011-03-17T18:24:08.060" UserId="26" />
  <row Id="14077" PostId="8424" Text="Thanks, I would've never have tried that! I will give it a go." CreationDate="2011-03-17T18:38:27.360" UserId="3761" />
  <row Id="14078" PostId="8424" Text="Brilliant, it works wonderfully. Thanks again." CreationDate="2011-03-17T18:45:23.360" UserId="3761" />
  <row Id="14079" PostId="8425" Score="3" Text="@awshepard: Try http://stats.stackexchange.com. They also cover data mining, which I would say this falls under. Also, a &quot;which algorithm should I use&quot; question typically requires a lot more information about the size and shape of your data than you've given here." CreationDate="2010-08-27T22:47:31.723" UserId="455" />
  <row Id="14080" PostId="8373" Text="Josh, which exchange would be better for this question?" CreationDate="2011-03-17T19:30:12.743" UserId="1349" />
  <row Id="14081" PostId="8417" Text="Thank you for your answer. I'm quite new here at stackexchange. How can I move my question to SO?" CreationDate="2011-03-17T19:45:27.507" UserId="3757" />
  <row Id="14082" PostId="8403" Text="@rolando2 &quot;credibility&quot; is usually associated with *statistical significance,* which is only indirectly related to sample size, as @Peter suggests in the first sentence.  If the effect size is expected to be large, why use a large sample?" CreationDate="2011-03-17T19:48:42.950" UserId="919" />
  <row Id="14084" PostId="7777" Text="@dfrankow and @Sam Swift. I am sorry, been a bit busy writing some papers. Ok - if you get a p-value = 0, it means the model is NOT a good fit as goodness of fit test fails. Regarding the problem of AIC, wikipedia and I are using a different convention for it. I am using the one that is used by BDR and WV. (cf. Extending linear Models in R, by Dr. Julian Faraway)" CreationDate="2011-03-17T20:16:04.923" UserId="1307" />
  <row Id="14085" PostId="8429" Text="The question is: is there are test to decide if the variable should be modeled as a mixed effect or random effect? Otherwise you could do the test you described and then test it with a chi-square dist (I'm not sure what the appropriate test would be)." CreationDate="2011-03-17T20:16:38.043" UserId="3727" />
  <row Id="14086" PostId="8380" Text="&quot;Latent variable regression&quot; returns a lot of hits on google, but I didn't find a tutorial or some standard model." CreationDate="2011-03-17T20:21:19.753" UserId="3369" />
  <row Id="14087" PostId="8429" Score="2" Text="@MudPhud: Modelling a variable as a fixed or as a random effect should actually be decided before the analysis, when the study is planned. It depends, in particular, on the scope of your conclusions. Random effects allow more generalisability. It could also avoid some technical difficulties. For example, the asymptotics might break down when the number of parameters grow up, as it is the case when a categorical variable with a lot of levels is considered as a fixed variable." CreationDate="2011-03-17T20:24:22.043" UserId="3019" />
  <row Id="14088" PostId="8429" Text="@ocram: I agree, but when I tried explaining this to my PI he just turned around and asked for a p-value of some kind. I want to include this analysis in a manuscript, but he won't put it in if there isn't a more concrete justification." CreationDate="2011-03-17T20:28:15.333" UserId="3727" />
  <row Id="14089" PostId="8328" Text="@JMS That's right; and the matrix inversion formula in the edit does not seem to offer any protection against ill-conditioning in $D$ itself." CreationDate="2011-03-17T20:30:38.543" UserId="919" />
  <row Id="14090" PostId="8429" Score="1" Text="@MudPhud: To the best of my knowledge, there is no p-value for such a decision. If interest centers on the effect of the specific levels chosen then it should be considered as fixed. If the available factor levels are seen as a random sample from a larger population and that inferences are wanted for the larger population, the effect should be random. " CreationDate="2011-03-17T20:39:58.723" UserId="3019" />
  <row Id="14091" PostId="8430" Text="@lockedoff: Thank you for completing my answer. Also, sometimes people use a mixture of chi-squares instead of a chi-square distribution for the test statistic." CreationDate="2011-03-17T20:46:22.353" UserId="3019" />
  <row Id="14092" PostId="8430" Text="@ocram +1 for your comment on deciding whether to treat the variable as random or fixed separately from the analysis. @MudPhud If your PI doesn't understand the issue and insists on a p-value, then maybe just show him the result of the test of the random effect (which you would include anyway in the write-up)." CreationDate="2011-03-17T20:55:05.683" UserId="3432" />
  <row Id="14093" PostId="8430" Text="Thanks for the code. When I ran it the result is none of the bootstrapped LRTs are greater than the observed, so this means that I can stick to the lm without the random effects or even the original variable thrown in." CreationDate="2011-03-17T21:05:12.383" UserId="3727" />
  <row Id="14094" PostId="8430" Text="@MudPhud: Did you get any errors? Try typing `lrt.sim` to make sure they're not all zeros, in which case the most likely culprit would be that you don't have the package `lme4` installed." CreationDate="2011-03-17T21:08:14.157" UserId="3432" />
  <row Id="14095" PostId="8430" Text="@lockedoff They're not 0, just very small (~1e-6) compared to the observed (63.95). " CreationDate="2011-03-17T21:17:13.137" UserId="3727" />
  <row Id="14096" PostId="8425" Score="1" Text="What you want to do is called &quot;online clustering&quot;. An &quot;online&quot; algorithm in general means you can add data to it incrementally without retraining from scratch. FLAME is not online; you can add data incrementally to it, but in general the solution will be different than if you retrained. I don't know what would be a standard online clustering algorithm. People often use custom hacks, e.g. add incrementally for 10 iterations, then retrain." CreationDate="2011-03-17T21:23:39.573" UserId="3369" />
  <row Id="14097" PostId="8391" Score="1" Text="+1, nice answer. Of course there's still an integral hiding in there ;)" CreationDate="2011-03-17T21:34:03.043" UserId="26" />
  <row Id="14098" PostId="8356" Text="Strictly speaking, that it is a $-5/3$ slope" CreationDate="2011-03-17T21:39:46.980" UserId="2958" />
  <row Id="14099" PostId="8422" Text="@Vladimir The boxplot is supposed to summarize the AUC? If so, why is the min value = 0.5?" CreationDate="2011-03-17T22:22:08.290" UserId="930" />
  <row Id="14100" PostId="8364" Text="The issue I have with the &quot;repeated&quot; nature of frequentists is that in order to work, the conditions must remain the same.  But if the conditions remain the same, you should be able to pool your data sets together and get a better estimate.  The frequentist ignores the past information precisely under the conditions when it is reasonable to take it into account." CreationDate="2011-03-17T22:46:38.817" UserId="2392" />
  <row Id="14101" PostId="8401" Text="@JMS Thank you for this. It was definitely helpful. One thing: I could actually sample from the target distribution though it would be extremely costly. Assuming that I could, what kind of test could I run in order to show that the distribution has converged?" CreationDate="2011-03-17T23:19:03.623" UserId="3572" />
  <row Id="14102" PostId="8401" Text="@Berk U, in principle you could use such a test. But it's extremely unlikely that this would make much practical sense." CreationDate="2011-03-17T23:53:31.173" UserId="26" />
  <row Id="14103" PostId="8380" Text="Looking at this question may be useful: http://stats.stackexchange.com/questions/7772/is-it-possible-to-fit-a-multivariate-regression-model-where-the-independent-varia" CreationDate="2011-03-18T00:09:34.060" UserId="3369" />
  <row Id="14104" PostId="8434" Text="Do you have an example to follow the idea?" CreationDate="2011-03-18T00:33:25.243" UserId="3681" />
  <row Id="14105" PostId="8367" Text="@chl: That makes a lot of sense, thanks for educating me!" CreationDate="2011-03-18T00:58:16.713" UserId="2817" />
  <row Id="14106" PostId="8392" Text="The data is already collected, and continues to be collected in near real-time... I'm just wondering if an analysis would give me accurate results, and if so, how do I do the analysis? Yes, I am a systems admin. Threatening the professors either doesn't work or is too painful, politically." CreationDate="2011-03-18T00:59:06.250" UserId="2814" />
  <row Id="14107" PostId="8435" Score="2" Text="@Mav, can you pls be a bit more specific, multivariate logistic regression is a special case of GLMM. AFAIK." CreationDate="2011-03-18T02:31:35.560" UserId="1307" />
  <row Id="14108" PostId="8434" Text="@darkcminor - I am happy to elaborate, but your question asked about quantifying the error.  So if you could add some info into your question about the rule you are using to classify, then that would make it much easier for me." CreationDate="2011-03-18T03:03:04.137" UserId="2392" />
  <row Id="14109" PostId="8438" Text="He's using an ADC, so he's got ***both*** truncation (i.e., saturation) *and* quantization effects. It's straightforward to show that if the noise is zero-mean Gaussian, then the expected value of each sample is something like $\sum_{i=1}^m q^\star_i \cdot (\Phi((q_{i} - \mu)/\sigma) - \Phi((q_{i-1}-\mu)/\sigma)$ where $q^\star_i$ is the quantization value of the $i$th bin, $q_i$ is the $i$th bin boundary and $\Phi(\cdot)$ is the standard normal cdf. If he wants something unbiased he has to find a way to get that $\mu$ outside of the $\Phi(\cdot)$ functions!" CreationDate="2011-03-18T03:52:50.037" UserId="2970" />
  <row Id="14110" PostId="8434" Text="I updated the question, pleae let me know if it is clear. Thank you" CreationDate="2011-03-18T03:56:17.177" UserId="3681" />
  <row Id="14111" PostId="7126" Text="If you want to store data (an empirical distribution) instead of your predictive distribution, under conditional independence arithmetic coding over the quantiles wrt/ some prior is about optimum. This will take up much more space than the predictive distribution that was asked for, but again, binning can help you find the right compromise." CreationDate="2011-03-18T06:34:36.970" UserId="2456" />
  <row Id="14112" PostId="8435" Text="What do you mean by 'multivariate' in 'multivariate logistic regression'? '*Multivariate*, i.e. more than one outcome variable which you wish to include in the same model? *Multivariable*, i.e. logistic regression with more than one covariate? *Multinomial*, i.e. outcome has more than two categories? " CreationDate="2011-03-18T07:38:20.440" UserId="449" />
  <row Id="14113" PostId="8439" Score="1" Text="Data is not enough -- you need to have some hypothesis to test with a statistical test, like &quot;People hate when administration website has magenta background&quot; or &quot;Younger people use adm. services at night more often&quot;." CreationDate="2011-03-18T07:56:15.193" UserId="88" />
  <row Id="14114" PostId="8436" Text="@AVB, did you try logistic transformation: $y=\log(p/(1-p))$? For $p\in(0,1)$, $y$ will be in $(-\infty,\infty)$. Also variance properties might improve and you will not have to worry about estimating the asymptote of S-curve. " CreationDate="2011-03-18T08:03:51.130" UserId="2116" />
  <row Id="14115" PostId="8409" Text="Thank you so much for your post. I very much like your stratified MC idea. I was wondering if you knew about any resources on it so that I could read up on it? Anything with an example in action would be much appreciated." CreationDate="2011-03-18T09:31:54.700" UserId="3572" />
  <row Id="14116" PostId="8355" Text="Thanks for the link, indeed it's some work to go through that, but I'll try to go through it as far as I can. Though if already a question concerning the direction of this read... Is it entirely focused on estimating frequency?" CreationDate="2011-03-18T09:50:22.963" UserId="704" />
  <row Id="14117" PostId="8436" Text="Can you give more information about the data you are working on? How many regressors you have etc. And is it actually a time series? Are you regressing on the whole time series at once, on time windows, ... ?" CreationDate="2011-03-18T09:58:49.573" UserId="2860" />
  <row Id="14119" PostId="8312" Text="@chl Thanks!  One of the problems with R is keeping track of packages (there are so many!) and which are best.  The task views do help" CreationDate="2011-03-18T10:42:38.583" UserId="686" />
  <row Id="14120" PostId="8409" Text="This is an idea that I have had, so I don't know of any resources.  It could be put into the MCMC framework, by adding another layer which defines the strata.  And one of my main weaknesses is doing stuff - I'm a thinker at heart!  I know this doesn't help you very much and I apologise for that.  But surely I'm not the only person who has thought of this.  Anyone who has done &quot;design based&quot; statistics and &quot;Bayesian statistics&quot; would surely have thought of this." CreationDate="2011-03-18T11:00:02.577" UserId="2392" />
  <row Id="14121" PostId="8355" Text="That is the example used, but if you look at the functional form of the model, it is an extremely general model (incorporates all regression models, OLS and GLMs, trend and seasonal analysis, etc.).  Although you would need to fill in the specific details.  It does go through an example of removing trend on page 146." CreationDate="2011-03-18T11:03:09.923" UserId="2392" />
  <row Id="14122" PostId="8419" Text="@darkcminor - I have updated my answer in using the example data.  Hope you enjoy (its actually quite a good classifier!)" CreationDate="2011-03-18T11:06:02.807" UserId="2392" />
  <row Id="14124" PostId="8448" Text="@Jim: I'm not sure that this gives me what I need. I want continuous variable A (age) in BINS by categorical G (see example in my question). Your solution groups A by G." CreationDate="2011-03-18T12:15:06.900" UserId="2824" />
  <row Id="14125" PostId="8410" Text="@Aniko: Thanks. The first part of the answer (after I excluded `B`) is returning what I need. I'm having difficulty getting the mean/media/sd instead of the counts. I don't know how to use the second part of the answer without `B`." CreationDate="2011-03-18T12:46:12.587" UserId="2824" />
  <row Id="14126" PostId="8410" Text="Do you want the mean/median of A by bins of A? Then just use `A` where I used `B`. I thought that means/medians got to imply the presence of a third variable." CreationDate="2011-03-18T13:07:12.420" UserId="279" />
  <row Id="14127" PostId="8421" Text="I sincerely appreciate your response.  However, the purpose of StackExchange is to provide answers, not references.  Statisticians seem particularly prone to this reference problem.  Is there any detail you can add about how to use residuals.lrm?  For example, an example command and an example of interpreting the graph for the house.plr example?" CreationDate="2011-03-18T13:50:44.420" UserId="2849" />
  <row Id="14128" PostId="8440" Text="Upvoted for mentioning &quot;multinomial logit model&quot;" CreationDate="2011-03-18T13:53:21.660" UserId="2658" />
  <row Id="14129" PostId="8454" Text="@IrishStat: With your experience, you may be able to offer some explanation" CreationDate="2011-03-18T13:57:19.080" UserId="2956" />
  <row Id="14130" PostId="8410" Text="@Aniko: Yes, I want means, etc. of A by bins of A by outcome. I replaced `B` with `A` but did not get the output I need. I've done it again and this time I think it is giving me what I need. Thanks." CreationDate="2011-03-18T14:02:57.420" UserId="2824" />
  <row Id="14131" PostId="8454" Text="What is the source of this quotation?" CreationDate="2011-03-18T14:04:52.463" UserId="919" />
  <row Id="14132" PostId="8446" Text="Why do you need a *formula?*  There are other ways to describe distributions." CreationDate="2011-03-18T14:10:05.220" UserId="919" />
  <row Id="14133" PostId="8454" Text="@whuber: I really appreciate your speed of response. Please read http://stats.stackexchange.com/questions/8187/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-dea" CreationDate="2011-03-18T14:23:47.497" UserId="2956" />
  <row Id="14134" PostId="8446" Text="I wrote formula because when i spoke with my supervisor he told me &quot;formula&quot;, but any other type of description is accepted :) " CreationDate="2011-03-18T14:28:33.083" UserId="3342" />
  <row Id="14135" PostId="8417" Text="@Steven: You're welcome.  I'm not sure how to move questions; I think moderators can do it." CreationDate="2011-03-18T14:36:03.263" UserId="1657" />
  <row Id="14136" PostId="8457" Text="Thanks Bayer   I understand Binomial but it is early AM and I am getting old. I asked this question as I am a retired nuclear engineer who is wondering whether we should look more at the overall 'expectation value' than individual probabilities if we want to use probabilistic criteria for designing these plants." CreationDate="2011-03-18T14:36:05.930" UserId="3777" />
  <row Id="14137" PostId="8373" Text="@Vishal: I think it's better suited here, since it's about statistics, not programming." CreationDate="2011-03-18T14:38:11.127" UserId="1657" />
  <row Id="14138" PostId="8453" Text="You are confusing me with using $R^2$ (proportion of data variability explained by the model, not the size of a cause) and estimated marginal effect represented by the slope coefficient (even in a pairwise regression without intercept term) in this strange (non-sensible?) fashion. If you are interested in impacts you may consider beta-coefficients or elasticity coefficients that really do have acceptable interpretation." CreationDate="2011-03-18T14:45:21.020" UserId="2645" />
  <row Id="14139" PostId="8458" Text="Okay, so let's take warming and pirates.  For example's sake, If the correlation between warming and pirates observed is -.8, (those predominantly tropical swashbucklers just can't seem to stand the heat), the r^2 is 64%.  Let's say the slope of the datasets is something like 1:-10.  Can you say &quot;Of the 10 fewer pirates we saw in a year where the avg temp increased 1 degree, the retirement of about 6 of those brigands could be related to the temperature increase&quot;" CreationDate="2011-03-18T15:07:14.517" UserId="3776" />
  <row Id="14140" PostId="8446" Text="@Maurizio You have already provided a fine description by means of your plot of the empirical density :-).  What else do you want?  The point is that everything depends on *how the description is intended to be used,* but you haven't yet disclosed that crucial information." CreationDate="2011-03-18T15:08:12.910" UserId="919" />
  <row Id="14141" PostId="8457" Score="3" Text="@bayer, I am not voting this down (though I'm a bit tempted), but the independence assumption strikes me as wholely inappropriate in this circumstance and would lead to absurd inferences!" CreationDate="2011-03-18T15:12:26.463" UserId="2970" />
  <row Id="14142" PostId="8454" Score="1" Text="@Dr You have taken this quotation, which is at http://stats.stackexchange.com/questions/8187/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-dea/8279#8279, out of context, likely leading to a misinterpretation.  The root cause is vague punctuation and grammar in the original, which can taken to mean &quot;Confounding variables, although *their nature may currently be* unknown, can be proxied by both ARIMA structure and/or detectable Interventions.&quot;  You really should be asking this question in a comment to @IrishStat on that thread." CreationDate="2011-03-18T15:13:42.653" UserId="919" />
  <row Id="14143" PostId="8458" Text="How are there more than one slope? If you regress pirates on global temperatures and find a coefficient of -10, then, for every one degree increase in global temperatures, the number of pirates falls by 10. If the global temperature increase by 0.5 degrees last year, then we expect that there were 0.5*(-10) more (or 5 fewer) pirates." CreationDate="2011-03-18T15:17:30.233" UserId="401" />
  <row Id="14144" PostId="8458" Score="2" Text="The regression only tells you the strength of the possible association, it says precisely nothing about causation.  It can't even tell you the direction of the causal relationship.  In this case, there are multiple confounding variables, so interpreting the regression in this way is very risky and should be accompanied by plenty caveats.  " CreationDate="2011-03-18T15:23:17.493" UserId="887" />
  <row Id="14145" PostId="8447" Score="2" Text="As a reader, I would find that a good way to represent changes in RR. Usually time would be in the horizontal axis, so I would be tempted as a reader to turn the page 90 deg when I'm reading it. " CreationDate="2011-03-18T15:28:18.010" UserId="812" />
  <row Id="14146" PostId="8457" Score="1" Text="I'm with @cardinal; independent failures is a ridiculous assumption. What if, say, the plants are near each other and in an area of high tectonic activity..." CreationDate="2011-03-18T15:30:39.463" UserId="26" />
  <row Id="14147" PostId="8457" Score="1" Text="@cardinal is absolutely right: this is the crux of the matter.  Engineers have used these kinds of independence assumptions without considering the possibility that *everything could go wrong at once* due to a common cause (such as an earthquake).  That's (apparently) why multiple backups have failed in some Japanese installations." CreationDate="2011-03-18T15:34:03.987" UserId="919" />
  <row Id="14148" PostId="8456" Text="What program are you using to make the plots?" CreationDate="2011-03-18T15:44:15.773" UserId="3567" />
  <row Id="14149" PostId="8457" Text="You are right. I should have pointed that out." CreationDate="2011-03-18T15:57:33.647" UserId="2860" />
  <row Id="14150" PostId="8456" Text="@sciencefairfuels Try to be more concise, like this. Otherwise you have a small chance that someone would try to decipher the question, and thus a little chance for an answer." CreationDate="2011-03-18T16:09:50.850" UserId="88" />
  <row Id="14151" PostId="8457" Text="Just to clarify...Engineers do indeed consider 'common-mode' failures. That is why there are batteries for power systems in case of total site power loss. However, again assumptions have been made about how long it takes to restore power. In the case of the Japan accident, those assumptions were obviously wrong. " CreationDate="2011-03-18T16:27:23.227" UserId="3777" />
  <row Id="14152" PostId="8380" Text="@SheldonCooper Okay so I guess I would need to first use linear regression to build a model for predicting $y$ given $x$, and then a separate model for estimating the latent variable $x$ given $\hat{x}$.  Any suggestions for how to build this second model with my $D_2$ data set consisting of a matrix of exact variable measurements $X_2$ and corresponding noisy variable measurements $\hat{X}_2$?" CreationDate="2011-03-18T16:44:13.293" UserId="3744" />
  <row Id="14153" PostId="8457" Text="I am speculating that the 'design tsunami' was less than the 10 meter wave that actually happened. This would have probably been chosen based on historically based probability data. So, if the design probability criteria for the plant was forced lower by considering the combined probabilities of hundreds of plants that would have lead to more stringent design. Does that make sense ? It eventually boils down to overall economics as to what is possible." CreationDate="2011-03-18T16:54:12.257" UserId="3777" />
  <row Id="14154" PostId="8438" Text="Yes, I have both truncation and quantization. That is correct, I will look into the truncation further to see if I can find anything. " CreationDate="2011-03-18T18:07:09.950" UserId="3749" />
  <row Id="14155" PostId="8462" Text="oh, great. now the sketch is there. :)" CreationDate="2011-03-18T18:21:22.320" UserId="1531" />
  <row Id="14156" PostId="8462" Text="'Periodic chart'?" CreationDate="2011-03-18T18:50:54.010" UserId="449" />
  <row Id="14158" PostId="8464" Score="1" Text="Being picky: A histogram never has blank spaces between the bars. When it does it is called a bar chart" CreationDate="2011-03-18T19:27:09.450" UserId="3489" />
  <row Id="14159" PostId="8466" Score="1" Text="Please check your statements.  In the first line what is $V$? By definition a Student T has zero mean and its SD is determined by $\nu$, so where do $\mu$ and $\lambda$ come from?  What, in the first two lines, is the relationship between $X$ and $Z$?  Usually Student's T is used when mean and SD are being estimated from data, but your &quot;standardization&quot; presupposes you *know* $\mu$ and &quot;$\text{std}(X)$&quot;.  How?  And why is there even a question here, when by definition the &quot;log likelihood ... for $Z$&quot; is the logarithm of the density, which you have explicitly written down?" CreationDate="2011-03-18T19:35:12.430" UserId="919" />
  <row Id="14160" PostId="8464" Score="1" Text="@os @Ralph A *histogram* represents probability (or frequency) by means of graphical *area,* whereas a &quot;bar chart&quot; (which is what we're looking at here) uses *length*.  Of course histograms can have blank spaces between bars, when there's no probability or frequency there!" CreationDate="2011-03-18T19:38:27.217" UserId="919" />
  <row Id="14161" PostId="8465" Score="1" Text="The point of the question seems to be that the x-axis summarizes a *periodic* or *cyclic* variable." CreationDate="2011-03-18T19:40:12.580" UserId="919" />
  <row Id="14162" PostId="8463" Text="How big are those generators? I would have guessed either a Skycrane or Mi-26 could haul them in, at the very least in pieces. " CreationDate="2011-03-18T19:44:03.560" UserId="2970" />
  <row Id="14163" PostId="8438" Text="@AVB &quot;Exact opposite&quot; in what sense?" CreationDate="2011-03-18T19:51:34.520" UserId="919" />
  <row Id="14164" PostId="8181" Text="thanks caracal so much, I really appreciate it" CreationDate="2011-03-18T20:14:08.467" UserId="3682" />
  <row Id="14165" PostId="8446" Score="1" Text="What kind of data is this ? We often find chronological/time series data having &quot;unusual pdf's&quot; and after suitable filtering/modelling the resultant series has a much more manageable density. Please describe your original data and perhaps post it to the web for guys like me to ponder over." CreationDate="2011-03-18T20:19:23.060" UserId="3382" />
  <row Id="14166" PostId="8465" Text="@whuber: yup! (thanks for the detailed answer, though, JD)" CreationDate="2011-03-18T20:28:18.870" UserId="1531" />
  <row Id="14167" PostId="8467" Text="to me, &quot;time series&quot; implies either a cycle, or linear time. I'm interested in something specific to cyclic data." CreationDate="2011-03-18T20:28:59.183" UserId="1531" />
  <row Id="14168" PostId="8467" Text="Seems I misunderstood your question - so the data is aggregated over the hours of a day, or or the days of the week for a number of days or weeks? In that case it is a kind of histogram. " CreationDate="2011-03-18T20:36:19.530" UserId="1766" />
  <row Id="14169" PostId="8467" Text="@owe, hmm... I think you're correct (looking at the wiki entry on histogram), but ideally something specific to time would be better." CreationDate="2011-03-18T20:45:34.090" UserId="1531" />
  <row Id="14170" PostId="8462" Text="@onestop: that's a decent alternative to &quot;cyclic&quot;. If you turn the comment into an answer, I might give it to you." CreationDate="2011-03-18T20:52:32.227" UserId="1531" />
  <row Id="14171" PostId="8465" Text="doh! thanks @whuber, I didn't realize that. " CreationDate="2011-03-18T20:54:18.047" UserId="29" />
  <row Id="14172" PostId="8464" Text="I take back my earlier comment. Given what I now realize from the discussion, this could be a histogram showing frequency by time or day buckets. " CreationDate="2011-03-18T20:56:42.537" UserId="29" />
  <row Id="14173" PostId="8470" Text="Why not work with logarithms of probabilities?" CreationDate="2011-03-18T21:46:29.873" UserId="3567" />
  <row Id="14174" PostId="8470" Text="Technically, you could compute $p(data)$, which is the normalization factor. However, the easiest way to compute it is usually to compute the results for all classes and sum them up. Note that this works for any number of classes, not just two." CreationDate="2011-03-18T21:46:55.920" UserId="3369" />
  <row Id="14175" PostId="8464" Text="as I said on Owe's answer, I think technically, this is correct, but a name that indicates the relationship to time would be nice. " CreationDate="2011-03-18T21:57:24.423" UserId="1531" />
  <row Id="14176" PostId="8469" Score="1" Text="Henry Normally we try to provide useful advice and guidance on homework questions without actually doing the homework for people.  The lack of an answer for ten days was restraint on the part of this community, not ignorance." CreationDate="2011-03-18T22:14:51.747" UserId="919" />
  <row Id="14178" PostId="8462" Score="1" Text="An aside: Sometimes it is better to plot this kind of chart around a circle. For example, there might be an interesting pattern from 10pm to 2am, but, the way that your chart is cut at midnight, you might not notice this pattern. See here: http://stackoverflow.com/questions/2076370/most-underused-data-visualization/2083230#2083230" CreationDate="2011-03-18T22:35:54.690" UserId="401" />
  <row Id="14179" PostId="8454" Text="@whuber: Thank you for the clarification. I struggled on the internet for a day to get a clarification." CreationDate="2011-03-18T23:14:17.527" UserId="2956" />
  <row Id="14180" PostId="8447" Text="@Ming-Chin Kao: Thank you for the comment. Is it wrong to use a forest plot? The reason is that journals put a limit of 5 figures/tables. One forest plot will have the matter of 1 table and 1 graph along with confidence limits. In a graph with time on horizontal axis, can you kindly guide me how to draw RR with confidence limits?" CreationDate="2011-03-18T23:21:37.517" UserId="2956" />
  <row Id="14181" PostId="8279" Text="@IrishStat: You made a statement that &quot;Confounding variables are unknown in nature&quot;. Why confounding variables are unknown in Nature? or did I comprehend it wrongly?&#xA;&#xA;confounding" CreationDate="2011-03-18T23:27:10.380" UserId="2956" />
  <row Id="14182" PostId="8466" Text="I have revised the question. My question is how to find the distribution of $Z$ and $f_Z(z)$ in the above case? Does this make sense now?" CreationDate="2011-03-18T23:29:58.030" UserId="862" />
  <row Id="14183" PostId="8463" Text="There are (at least) two reasons for inadequate backup power in my understanding...1. the tidal wave took out the fuel tanks for the standby generators (inadequate tidal wave protection) 2. Inadequate batteries to keep essential equipment running until replacement power was available (likely impractical).   Both of these situations are part of a large and complex probabilistic safety analysis of multiple scenarios. However, the bottom line is...the lower your probability criteria is....the more stringent your design will be (cont'd)" CreationDate="2011-03-18T23:33:25.567" UserId="3777" />
  <row Id="14184" PostId="8463" Text="As a former nuclear reactor designer I am unaware of anyone who ever considered the 'total reactor population of the world' when estimating risk. The last few days have made me wonder whether this should not be the case in future. This is what prompted my question." CreationDate="2011-03-18T23:34:14.847" UserId="3777" />
  <row Id="14185" PostId="8463" Score="1" Text="Why would one use such a strict criteria ? Because the consequences of such (potentially) low frequency events are so large we need to try to eliminate them completely. Again, economics will limit just how much we can do in this regard." CreationDate="2011-03-18T23:39:52.297" UserId="3777" />
  <row Id="14186" PostId="8459" Text="Any chance for a &quot;before&quot; and &quot;after&quot; visualizations? I'm really not sure (even after going as far as trying your algorithm on some artificial data) what do you mean by &quot;denoising&quot; here?" CreationDate="2011-03-18T23:41:52.700" UserId="2658" />
  <row Id="14187" PostId="8380" Text="If you wanted to do that, then the simplest thing would be to build a regression for every $x_i$ separately. I.e. for $x_1 = c_1 \hat{x}_1 + o_1$, you collect all observations that link them and do a regression. Then repeat for $x_2$, etc. If appropriate, you may replace linear regression with nonlinear. But this is not a good approach. It can deal with bias, but it will not deal with variance. Suppose you have two variables, $x_1$ and $x_2$, that are basically the same, but the variance in observing $x_2$ is much larger. Then it makes sense for a regression to weigh $\hat{x}_1$ heavily ..." CreationDate="2011-03-19T00:26:26.453" UserId="3369" />
  <row Id="14188" PostId="8380" Text="... and ignore $\hat{x}_2$, since they supply the same information but the variance in $\hat{x}_1$ is smaller. But your two-stage model wouldn't know that. Thus I think a combined model would be much better. But I don't know of a standard model to do it. Try retagging your question with the &quot;r&quot; tag to see if R experts know a standard way to do latent variable regression." CreationDate="2011-03-19T00:32:15.700" UserId="3369" />
  <row Id="14189" PostId="8463" Text="@JPresley: If I had to do such a calculation, I think it's more of a &quot;when&quot; than an &quot;if&quot; situation.  As a simple model, I would use a Poisson Distribution for the &quot;when&quot;, and maybe a Lognormal Distribution (Gamma??, Exponential??) for the magnitude of the problem.   That's why several layers of backups/contingencies are necessary." CreationDate="2011-03-19T01:05:10.780" UserId="2775" />
  <row Id="14190" PostId="8474" Score="1" Text="@Rob, Can you explain the part about probability coverage?" CreationDate="2011-03-19T02:39:34.887" UserId="2040" />
  <row Id="14191" PostId="8468" Text="@dsimcha, Why not utilize an approximate standard error for the R-squared value on the hold out. For example using CI.Rsq in the psychometric package? " CreationDate="2011-03-19T03:11:28.837" UserId="2040" />
  <row Id="14192" PostId="8447" Text="I got an answer to make a bar graph in Excel. http://www.ph.ucla.edu/epi/rapidsurveys/RScourse/chartconfinterval.pdfBut Excel will not generate publication quality graphs or eps files. Can any other software generate print quality graph?" CreationDate="2011-03-19T03:47:32.243" UserId="2956" />
  <row Id="14193" PostId="8447" Text="@mbq: Thank you very much for the editing to make the question clearer." CreationDate="2011-03-19T03:49:47.013" UserId="2956" />
  <row Id="14194" PostId="8463" Text="@bill_80: I am not sure but I think you may be confusing my question with the probability/risk associated with a single reactor (or site).There is already a well established set of probabilistic models dealing with this. There are also 'deterministic' models looking at what backups are needed. If I have read you wrong, I would need to see an example of your proposed calculation to understand it. My stats knowledge is somewhat 'rusty'." CreationDate="2011-03-19T04:29:38.713" UserId="3777" />
  <row Id="14195" PostId="8364" Text="+1 For a thoughtful analysis and attempt to respond directly to the OP." CreationDate="2011-03-19T04:34:27.517" UserId="919" />
  <row Id="14196" PostId="8446" Text="@Irish See http://stats.stackexchange.com/q/7542/919" CreationDate="2011-03-19T04:38:15.160" UserId="919" />
  <row Id="14197" PostId="8470" Score="1" Text="@Matt Those numbers are not probabilities at all: they are probability *densities.*  It makes no sense to force a bunch of them to sum to 1.  (In fact, you can make them individually be *greater* than 1 merely by changing the units of measure for weight or whatever.)  See the related question and analysis at http://stats.stackexchange.com/q/4220/919 ." CreationDate="2011-03-19T04:48:23.903" UserId="919" />
  <row Id="14198" PostId="7946" Text="@BB01 I doubt these are data for a &quot;population.&quot;  They appear to be measurements of some *process* and therefore constitute a finite sample of that process.  If you aim is to draw conclusions about the process--or to generalize *anything* from the data--you must treat these data as a sample." CreationDate="2011-03-19T04:51:38.890" UserId="919" />
  <row Id="14199" PostId="8437" Text="Thanks for the suggestion.  I'm reading up on GAMMS." CreationDate="2011-03-19T04:51:44.287" UserId="3748" />
  <row Id="14200" PostId="8457" Text="@cardinal...what sort of 'absurd inferences' are you thinking of ? Please expand on that. Is this problem not similar to estimating the overall probability of a single plane crashing given an individual crash probability, p in a population of n airplanes ? I realize that everything is not 'totally independent', but wouldn't @bayer's suggestion provide a reasonable first estimate ?" CreationDate="2011-03-19T04:54:04.673" UserId="3777" />
  <row Id="14201" PostId="8208" Text="(1) The Spearman (rank) correlation can be computed just fine even with non-discrete data.  (2) My point about machine numbers was that viewing data as discrete or not is a modeling decision, whence it is unwise to base one's statistical judgment solely on whether, *for some other purpose,* the data have been considered discrete.  (3) Your second paragraph is difficult to follow because you seem to be advising against the use of sample statistics to make any estimates at all.  (It's unlikely the OP has a complete census anyway.)" CreationDate="2011-03-19T04:56:40.963" UserId="919" />
  <row Id="14202" PostId="8447" Text="I got partial answer for Print quality graph. In SPSS, Graph&gt;Legacy dialogs&gt;High-Low will generate this graph" CreationDate="2011-03-19T05:00:05.757" UserId="2956" />
  <row Id="14203" PostId="8470" Text="@whuber that's right, the question is about probability densities. I'm wondering if I can apply a function or constant(s) (such as the normalization factor mentioned above by SheldonCooper) to transform the probability density &quot;5.3778E-4&quot; from the posterior value in the example to something like &quot;99.99%&quot;, or is that not practical?" CreationDate="2011-03-19T07:10:33.753" UserId="3784" />
  <row Id="14204" PostId="8483" Text="Can you give us some rough idea of your sample size?" CreationDate="2011-03-19T08:01:40.723" UserId="449" />
  <row Id="14205" PostId="8462" Score="2" Text="@Charlie an alternative that is often easier to both plot and read is to repeat part of the data and plot a bit more than a complete period, e.g. to plot over 30 hours from midnight to midnight and on to 6am, so the info from midnight to 6am appears twice - much as New Zealand or Alaska appear twice on some maps. I got this idea via Stata guru Nick Cox but I can't find the ref at present. [Tukey (1972)](http://www.edwardtufte.com/tufte/tukey) favoured two complete cycles." CreationDate="2011-03-19T08:21:20.940" UserId="449" />
  <row Id="14206" PostId="8469" Text="@whuber: I realise that, which I waited ten days.  But I do think that questions that have an answer should eventually get one.  Happy to discuss on meta if you prefer. " CreationDate="2011-03-19T09:20:36.697" UserId="2958" />
  <row Id="14207" PostId="8446" Text="@whuber, i'm apologize because this is a completely &quot;new world&quot; for me! I'll include the &quot;fine&quot; description also in this Q&amp;A, so we have a more detailed topic, but what do you mean with &quot;_how the description is intended to be used_&quot;? i find this unusual pdf and, as wrote by IrishSee i hope to find a more manageable density." CreationDate="2011-03-19T09:37:13.213" UserId="3342" />
  <row Id="14208" PostId="8485" Text="googling &quot;Gibbs sampling&quot; isn't a bad way to get a range of views on the subject.  I think its a good way to start, because you tend to approach it with a &quot;skeptical mind&quot; - you can't take googles word for granted, so you need to find a range of views.  Of course you may need a reputable source at a later stage when you try to implement.  But starting with the &quot;reputable source&quot; is not always the best idea, because they may be quite attached to a particular way of doing something - i.e. they know the &quot;right way&quot; and &quot;all others are wrong or ineffective&quot;." CreationDate="2011-03-19T10:26:09.000" UserId="2392" />
  <row Id="14209" PostId="8449" Text="@ttnphns, i can try with Q-Q plots, but to compare the distributions i should know the theoretical one, isn't it? or i'm wrong?" CreationDate="2011-03-19T10:30:12.517" UserId="3342" />
  <row Id="14211" PostId="8349" Text="Is each item monolithic, or does it consist of several sub-items, each of which is either right or wrong? An example of monolithic is a word (like &quot;century&quot;), which may have several spelling errors (e.g. &quot;sentery&quot; has two). An example of non-monolithic is something like purchasing data, an item is &quot;name:computer, price:$500&quot;. In this case, you don't care how many spelling error there are, you just count how many sub-items are wrong." CreationDate="2011-03-19T11:05:35.847" UserId="3369" />
  <row Id="14212" PostId="8446" Text="@Irish See, i have update the information!" CreationDate="2011-03-19T11:33:25.187" UserId="3342" />
  <row Id="14213" PostId="8476" Score="2" Text="@Gary - a p-value doesn't tell you to reject or not any more than a Z-score.  They are just numbers.  It is only the decision rule which determine accepting or rejecting.  This decision rule could equally well be defined in terms of a Z-score (e.g the $2\sigma$ or $3\sigma$ rule)" CreationDate="2011-03-19T12:09:16.853" UserId="2392" />
  <row Id="14214" PostId="8473" Text="@sheldon - if the sample size is large, then the standard deviation will be small, hence the Z-score will be high.  I think you may discover this if you tried a numerical example." CreationDate="2011-03-19T12:11:17.427" UserId="2392" />
  <row Id="14215" PostId="8483" Text="[This might help.](http://stats.stackexchange.com/questions/3386/how-to-deal-with-not-binary-categorical-variables-in-logistic-regression-spss)" CreationDate="2011-03-19T12:37:55.013" UserId="1351" />
  <row Id="14216" PostId="8422" Text="yep, seems strange, because several ROC should be below 0.5. Digging up what's wrong..." CreationDate="2011-03-19T12:39:09.847" UserId="3345" />
  <row Id="14217" PostId="8446" Text="@Maurizio What are you trying to learn from you data?  What kind of description would be &quot;more manageable&quot;?  For example, if you want to use it to predict future results, a simple formula is not the best choice.  If you want to relate your data to a scientific hypothesis you might need to estimate some properties of the distribution; to this end, having a formula can help.  If you're trying to understand why your data are not uniform, as expected, then having a formula for them will be next to useless." CreationDate="2011-03-19T13:26:24.213" UserId="919" />
  <row Id="14218" PostId="8469" Text="@Henry This discussion already occurred on meta at http://meta.stats.stackexchange.com/questions/12/how-should-we-deal-with-obvious-homework-questions .  That would be a good place to share your thoughts.  Also take a look at http://meta.stats.stackexchange.com/questions/725/what-to-do-with-homework-questions-posed-as-non-homework ." CreationDate="2011-03-19T13:29:54.990" UserId="919" />
  <row Id="14220" PostId="8470" Text="@Matt It is *practicable* (it can be done) but its meaningfulness is doubtful.  However, *ratios* of densities do make sense, as @ProbabilityIsLogic has suggested in a reply.  For instance, you might re-express all the results as ratios relative to the largest one.  They won't sum to unity--that is the part that is meaningless--but they will all lie between 0 and 1." CreationDate="2011-03-19T13:58:21.733" UserId="919" />
  <row Id="14221" PostId="8486" Text="A decibel is on a log scale, which immediately suggests looking at the log of the other variable.  Any other consideration, to be applicable and relevant, would have to consider the nature of your data and the objective of your comparison." CreationDate="2011-03-19T14:03:50.873" UserId="919" />
  <row Id="14222" PostId="8485" Score="1" Text="(+1) Questions that are easily answered by Googling are usually not welcome, but this one IMO attempts to capitalize on the collective wisdom of a community in a way that the Google ranking cannot do.  It would be interesting to see what sources people *really* found useful for learning this material." CreationDate="2011-03-19T14:05:36.143" UserId="919" />
  <row Id="14223" PostId="8483" Text="Could you clarify what you mean by &quot;allocate&quot;?  Some possible meanings are (1) where to collect data (experimental design), (2) how to partition data into training and testing sets, (3) how to identify the dependent variables (model selection)." CreationDate="2011-03-19T14:09:45.530" UserId="919" />
  <row Id="14225" PostId="8457" Score="1" Text="I think in this particular case, assuming independence could mean miscalculating the relevant probabilities by several orders of magnitude. I am not an expert in either nuclear power plant design nor aircraft design and logistics. However, I would hazard a guess that the dependence structure of the risk to Japanese power plants is substantially stronger than for calculating probabilities of aircraft crashes. The nuclear power plants are simultaneously subject to the same risk factors: Off the top of my head: (a) earthquake, (b) tsunami, (c) electrical power grid, (d) common manufacturer,..." CreationDate="2011-03-19T14:18:30.860" UserId="2970" />
  <row Id="14226" PostId="8457" Score="1" Text="...(e) other geographical/geophysical design risks. Just the fact that they are all in close proximity of one another makes them have a common risk factor for simultaneous attack, let's say. The point is that you have to consider all these possible risk factors and for a lot of these, conditional on one of them occuring, the probabilities of multiple plant failures *simultaneously* jumps to close to one. Plane crashes seem a bit different as they are largely automonous and operate more widely geographically. Now, if air-traffic control over New York (TRACON) went down completely, all of a..." CreationDate="2011-03-19T14:24:39.987" UserId="2970" />
  <row Id="14227" PostId="8457" Text="...sudden there is a simultaneous increase in risk across all airplanes in that space. Bad weather would also historically raise the risk of all planes in the airspace, though these risks have gone down dramatically over the last 20 years or so due to improved technology. So, I would *not* argue that plane crashes are independent either, but I'd guess they might tend to be &quot;more&quot; independent than nuclear reactor failures in someplace like Japan. Thoughts, @J Presley?" CreationDate="2011-03-19T14:28:27.270" UserId="2970" />
  <row Id="14228" PostId="8455" Score="1" Text="The nuclear situation in Japan is a Black Swan event. According to N.N. Taleb, Black Swan are events of very low probability but have a very high impact. His assertion is that such probabilities are uncomputable, and any computed probabilities have very little bearing on real life. " CreationDate="2011-03-19T15:23:51.117" UserId="2833" />
  <row Id="14229" PostId="8455" Text="http://en.wikipedia.org/wiki/Black_swan_theory" CreationDate="2011-03-19T15:30:01.267" UserId="2833" />
  <row Id="14230" PostId="8455" Score="1" Text="Taleb, [ *cringe* ]." CreationDate="2011-03-19T15:30:31.290" UserId="2970" />
  <row Id="14231" PostId="8499" Text="Agreed -- it is folly to assign a probability to so-called Fourth Quadrant events such as this, or to even think we can predict them. All we can do is to make the system robust to their negative effects through redundancies etc." CreationDate="2011-03-19T15:32:00.000" UserId="2833" />
  <row Id="14232" PostId="8455" Text="@cardinal, one often wishes the conveyor of such ideas wasn't a guy like Taleb (whose personality can be overbearing). But I wouldn't dismiss the ideas because of the man. " CreationDate="2011-03-19T15:37:40.947" UserId="2833" />
  <row Id="14233" PostId="4497" Score="1" Text="It's not clear from your answer, but &quot;Why Tables are Really Much Better than Graphs&quot; is a satire." CreationDate="2011-03-19T15:41:42.567" UserId="46" />
  <row Id="14234" PostId="8455" Text="I've read each of his books. Though interesting, I'd say, few, if any, ideas are *his*. He's been quite successful at popularizing them, though. I've also read a bit of the literature that he cites. Some of it I feel he misrepresents for his own purposes. That perturbs me." CreationDate="2011-03-19T15:47:09.387" UserId="2970" />
  <row Id="14235" PostId="8455" Text="@cardinal, no disagreements there. I don't think his ideas are truly original, but they have been associated with him due to the popularity of his books. Nevertheless, as someone who does quantitative modeling and straddles the divide between the deterministic and stochastic, I cannot help but agree with many of his observations." CreationDate="2011-03-19T15:50:29.797" UserId="2833" />
  <row Id="14236" PostId="8462" Text="An example of some of the circle charts in R, http://goo.gl/WoyVp . I agree with onestop's point as well though and is probably just as easy if not easier to read." CreationDate="2011-03-19T16:04:49.887" UserId="1036" />
  <row Id="14237" PostId="8471" Text="those are all cool, but I was looking for something for the whole class of charts. We went with histogram...." CreationDate="2011-03-19T16:11:14.357" UserId="1531" />
  <row Id="14238" PostId="8349" Text="@SheldonCooper : non-monolithic, they are items with a name, identification number and lots of other details." CreationDate="2011-03-19T16:12:27.587" UserId="3735" />
  <row Id="14239" PostId="8484" Text="Even though we wound up going with histogram in case we decide to do other kinds of similar charts that aren't time related, I think this is the most direct answer to the question that I asked. (We're actually using histogram/time, but maybe I'll change it to histogram/cycle....)" CreationDate="2011-03-19T16:14:13.810" UserId="1531" />
  <row Id="14240" PostId="8488" Text="Thanks @Henry, these are really useful! I'm not actually limited to only two charts, it was just how I had thought of it previously. If you have other ideas on how to show the data they would be very welcome!" CreationDate="2011-03-19T16:16:33.823" UserId="3735" />
  <row Id="14241" PostId="8486" Text="The variable is measured in terms of a percentage of what?" CreationDate="2011-03-19T16:19:50.490" UserId="3489" />
  <row Id="14242" PostId="8502" Text="What kind of model is it? Linear Regression?" CreationDate="2011-03-19T16:43:32.650" UserId="1351" />
  <row Id="14243" PostId="8502" Text="Are the samples A and B exactly the same size?" CreationDate="2011-03-19T16:44:04.083" UserId="1351" />
  <row Id="14244" PostId="8502" Text="It is Linear regression, and the sizes, in theory, should be the same size." CreationDate="2011-03-19T16:45:41.847" UserId="3491" />
  <row Id="14245" PostId="8503" Text="One of the potential issues I have is not necessarily having access to both datasets at the time that this calculation needs to be performed." CreationDate="2011-03-19T16:47:02.737" UserId="3491" />
  <row Id="14246" PostId="8485" Score="1" Text="That is the problem. Google returns way too many results and not all the papers or tutorials are clear enough." CreationDate="2011-03-19T17:04:02.873" UserId="3788" />
  <row Id="14247" PostId="8464" Text="A histogram displays the distribution of a continuous variable.  A bar chart displays the distribution of a discrete variable." CreationDate="2011-03-19T17:15:02.923" UserId="46" />
  <row Id="14248" PostId="8418" Text="Why are you converting a continuous variable to discrete?  There are very few situations where this is appropriate.  It can very easily be manipulated to show spurious relationships." CreationDate="2011-03-19T17:27:55.737" UserId="46" />
  <row Id="14250" PostId="801" Score="1" Text="I agree - the idea that there is one &quot;optimal&quot; bin width is a huge simplifying assumption." CreationDate="2011-03-19T17:32:41.800" UserId="46" />
  <row Id="14251" PostId="8505" Score="2" Text="How do you plan on taking logarithms of any counts that are zero?" CreationDate="2011-03-19T18:47:51.487" UserId="919" />
  <row Id="14252" PostId="8506" Score="3" Text="Actually, $\mathbb{E}(\log(Y) | X) \neq \log(\mathbb{E}(Y | X))$ **ever** unless $\mathbb{P}(Y = f(X) | X ) = 1$ for some $\sigma(X)$-measurable function $f$, i.e., $Y$ is fully determined by $X$.&#xA;" CreationDate="2011-03-19T18:50:55.873" UserId="2970" />
  <row Id="14253" PostId="8505" Score="2" Text="Definitely not equivalent. An easy way to see this is to look at what would happen if you observed zero counts. (Comment created before seeing @whuber's comment. Apparently this page didn't refresh appropriately on my browser.)" CreationDate="2011-03-19T18:52:33.110" UserId="2970" />
  <row Id="14254" PostId="8507" Score="1" Text="Actually, the affine transformation from the unit circle to the ellipse is given by the *square root* of the &quot;shape&quot; parameter.  E.g., the matrix {{4,0}, {0,4}} will only double the size of the circle." CreationDate="2011-03-19T19:23:16.087" UserId="919" />
  <row Id="14256" PostId="8463" Text="@JPresley: My answer was too long for a comment.  So, I added some things to my original answer.   See the above." CreationDate="2011-03-19T20:01:30.973" UserId="2775" />
  <row Id="14257" PostId="8504" Score="2" Text="I think that if the ellipsoid is of the form $(x-\bar{x})^TS^{-1}(x-\bar{x})=g^2$, then g should be given as radius." CreationDate="2011-03-19T20:16:10.987" UserId="339" />
  <row Id="14258" PostId="8470" Text="@whuber thank you for the insights. I am not expecting the numbers to sum to 1, in fact it's easier to compute a ratio or percentage when the classifications are mutually exclusive and cover the whole set (like male/female and spam/not-spam). The problem I'm dealing with is when you have classifications that can overlap and are not exhaustive (i.e. a sample could match multiple classifications, or none)." CreationDate="2011-03-19T20:33:39.670" UserId="3784" />
  <row Id="14260" PostId="8464" Text="@whuber - I'm not clear. Are we being picky because the OP's diagram was drawn with no marked frequency on the y-axis? My point was that if you do a Google search, you will see many definitions of &quot;Histogram&quot; which adamantly insist that the bars never touch. In practice, it really does not make a difference, but most published histograms will show the bars adjacent. – Ralph Winters 0 secs ago edit " CreationDate="2011-03-19T20:41:28.420" UserId="3489" />
  <row Id="14261" PostId="8490" Text="How many tests do you have in total, what is the measurement (grading) technique, and are you certain that the 'same test, different subject' option is viable given the nature of your test data?" CreationDate="2011-03-19T22:14:10.707" UserId="3582" />
  <row Id="14262" PostId="8473" Text="Not really. Suppose you sample from N(0, 1). Then your std will be about 1 regardless of sample size. What will get smaller is the standard error of the mean, not standard deviation. p-values are based on SEM, not on std." CreationDate="2011-03-19T22:19:59.413" UserId="3369" />
  <row Id="14263" PostId="8510" Text="+1 very nice links" CreationDate="2011-03-19T22:20:13.720" UserId="3788" />
  <row Id="14264" PostId="8511" Score="1" Text="The usually excellent UCLA statistical computing pages have made a rare error here -- there shouldn't be any parentheses in the expression for pseudo-$R^2$, i.e. it should be $1-L_1/L_0$. (Sorry for not answering your queries as I'm about to head for bed -- I'm sure someone else will have answered this before I'm awake enough to do so.)" CreationDate="2011-03-19T23:08:15.783" UserId="449" />
  <row Id="14265" PostId="8358" Text="@IrishStat: Can you spare a few minutes of your valuable time for this. I can guess how busy you would be." CreationDate="2011-03-19T23:23:32.947" UserId="2956" />
  <row Id="14266" PostId="8457" Text="@cardinal..your points are all valid...however, I feel I am being generally misunderstood here. The Fukushima Site (ie 6 reactors) was designed according to some probability criteria. For argument assume it is 10e-6/year. I am talking about now considering this reactor along with all the other reactor sites in the world designed to a similar (not necessarilly the same) probability criteria. This is a given. Now, with all these separate design probabilities of accidents we have an overall probability of an accident (or multiple accidents at separate sites) for any reactor anywhere in the world." CreationDate="2011-03-19T23:57:59.603" UserId="3777" />
  <row Id="14267" PostId="8457" Text="Don't confuse my question with anything to do with the fact that 4 reactors at the Fukushima site were damaged. This is essentially one 'event' in my discussion." CreationDate="2011-03-20T00:00:08.057" UserId="3777" />
  <row Id="14268" PostId="8499" Text="@energynumbers: I don't fully agree. The Tsunami was not unanticipated, the 'level' of the Tsunami was unanticipated. The plant was 'apparently' designed for a 7 meter Tsunami based on historical probabilty information. This was considered acceptable based on some probabilistic arguments by someone. If the criteria was more stringent than a 'less probable' wave hejght would have been required in the design...this is my point..." CreationDate="2011-03-20T00:06:09.717" UserId="3777" />
  <row Id="14269" PostId="8473" Text="@sheldon - The Z-score is (observed-mean)/(standard deviation).  But the mean and standard deviation are of the observed statistic, not of the population from which components of it were drawn.  My slack terminology has been caught here.  However, if you are testing the mean, then the appropriate standard deviation in the Z-score is the standard error, which gets smaller at the same rate as the p-value." CreationDate="2011-03-20T00:08:50.863" UserId="2392" />
  <row Id="14270" PostId="8495" Text="Thanks for that!  (and thanks to the other answerers too)." CreationDate="2011-03-20T00:09:10.460" UserId="386" />
  <row Id="14271" PostId="8455" Text="@Gilead...seems that we are getting a bit philosophical here...are you suggesting that there are certain numerical limits to the validity of probability calculations. If so....what is the numerical value at which we say....sorry...that's too low" CreationDate="2011-03-20T00:11:57.963" UserId="3777" />
  <row Id="14272" PostId="8455" Text="@Presley, to my understanding, the notion of low probabilities goes beyond numbers -- it's a notion related to epistemology and what is knowable. Exact thresholds cannot be defined because low probability events in a complex system cannot be modeled reliably. For instance, in &quot;beyond design basis&quot; accidents, there is often very difficult to know or predict the probabilities because the no. of past samples could well be very low or none. Therefore to use a numerical probability (e.g. 1 in 1e6) as a metric for risk becomes meaningless, in my opinion." CreationDate="2011-03-20T00:45:31.643" UserId="2833" />
  <row Id="14273" PostId="8455" Text="Again forgive me for quoting the man, but this is what he calls the Fourth Quadrant. He gave a talk to the American Statistical Association about this, and when he's not being controversial, you can sort of pick out some useful ideas from what he's saying. See here for more information: http://stats.stackexchange.com/questions/2906/what-is-the-communitys-take-on-the-fourth-quadrant" CreationDate="2011-03-20T00:47:40.410" UserId="2833" />
  <row Id="14274" PostId="8455" Text="Anecdote:where I work, one of our lab managers (an older guy) tells us in our annual safety training: &quot;if an experienced guy who's been at the job 30 years tells you he pipettes with his mouth and nothing's ever happened to him in 30 years, disregard his advice. With safety, past experience is not predictive of future outcome. Humans are notoriously bad at predicting low-probability events. What's worse is that these days, due to modern engineering controls, lab accidents happen so rarely we tend to downplay their probabilities.&quot;..." CreationDate="2011-03-20T01:17:59.837" UserId="2833" />
  <row Id="14275" PostId="8455" Text="... so instead of estimating probabilities, our labs try to set loss limits and add robustness to system, e.g. instead of benzene (a controlled substance), toluene is used a sub (less carcinogenic). You can't inhale what you don't work with. Granted, adding robustness isn't always easy or possible (or even economically viable), so it's not a general solution. Nevertheless, I feel that it is important to be aware of the existence of low probability high impact events in our design thinking and build robustness toward the negative ones." CreationDate="2011-03-20T01:27:17.473" UserId="2833" />
  <row Id="14276" PostId="8455" Text="@Gilead...everything you say has merit. However, you mention the term 'less carcinogenic'. This , in itself, is based on probabilistic analysis of low probability values. An overall rationale for building anything, be it a bridge, highway or dam or what have you needs a rational basis. Generally, probability has come into that rationale somewhere along the line." CreationDate="2011-03-20T04:35:59.647" UserId="3777" />
  <row Id="14277" PostId="8470" Text="@charles.y.zheng I could use logarithms of the probability densities and work with results that weren't as small, but I don't think that would get me to a percentage, would it? As I understand it, using sums of logarithms is just to avoid rounding errors when your floating point numbers get too small. The results stay in order, so you can see which result is the best, but they aren't comparable to anything else." CreationDate="2011-03-20T05:12:35.603" UserId="3784" />
  <row Id="14278" PostId="8474" Text="Suppose $P(-a &lt; X &lt; b)=p$ and $P(a&lt;X&lt;b)=q$ where $a,b&gt;0$. Then $P(a^2 &lt; X^2 &lt; b^2)=q\ne p$." CreationDate="2011-03-20T05:50:27.037" UserId="159" />
  <row Id="14279" PostId="8476" Text="@probabilityislogic I agree with you. Indeed, you could construct some test based on $Z$ score threshold but it does not allow you to explicitly define a test size in the classical sense (i.e. in terms of probability). This sort of criteria might be trouble some if your distribution has thick tails. When you construct a test, you explicitly define a test size and thus the $p$-value immediately tells you whether you accept or reject, which is the point I was trying to make." CreationDate="2011-03-20T08:01:39.787" UserId="3786" />
  <row Id="14280" PostId="8517" Text="+1 for nice R code." CreationDate="2011-03-20T09:38:04.277" UserId="1351" />
  <row Id="14281" PostId="8518" Text="@EnergyNumbers: Thank you for the stimulating questions. 1. Time lag between case recording to death is up to 6 months. Cases recorded in 1 state are followed up for 6 months. No case was ever lost to follow up. Case recorded in December 2007 and dying in May 2008 was recorded as 2007 case only by the state. 2. When a journal refuses a paper with a comment, they don't give us a chance to explain. So I want to send to another journal.  " CreationDate="2011-03-20T09:55:32.943" UserId="2956" />
  <row Id="14282" PostId="8518" Text="@EnergyNumbers: 3. The native place of the patient and the detials of prior treatment were recorded at admission it self. This work was done with the help of an International Organizations. Migrated cases were taken as belonging to the state of their origin. 4. I would be extremely happy to clarify any other point." CreationDate="2011-03-20T09:55:55.297" UserId="2956" />
  <row Id="14283" PostId="8518" Text="@EnergyNumbers: I wrote like this. Selection of the model for forecast generation: Time series expert modeler of IBM SPSS Forecast v19 was used. Both exponential smoothening models and ARIMA models were examined. Outliers were detected and prevented from influencing parameter estimates. Maximum number of Lags in ACF and PACF output checked were 24. Stationery R-square (larger values indicate better fit) was the goodness of fit measure for selecting the best fitting forecasting model. " CreationDate="2011-03-20T10:06:20.297" UserId="2956" />
  <row Id="14284" PostId="8518" Text="@EnergyNumbers: CFR from 1994 to 2005 were used for forecasting CFR along with upper and lower confidence limits from 2006 to 2010 and compared with the actual observed values till 2010 and the results were interpreted. Significance at 99% confidence level and objective endpoint (death) were used to minimize the effect of the confounding variables and make the conclusions more reliable. The data from the same state were compared. Multiple areas-India, Both states were studied independently and conclusions were drawn. Please tell me if my methodology was wrong." CreationDate="2011-03-20T10:07:23.337" UserId="2956" />
  <row Id="14285" PostId="8518" Text="Please ignore Multiple areas-India in the above comment.That came in by mistake" CreationDate="2011-03-20T10:18:48.617" UserId="2956" />
  <row Id="14286" PostId="8490" Text="I would have 4 tests for example, and 3 subjects per time of day. Tests notes are normal (test result given as a stdev), and these tests are supposed to be quite equivalents (testing the same thing, reading performance for example, even though they aren't obviously identicals." CreationDate="2011-03-20T10:22:35.473" UserId="3791" />
  <row Id="14287" PostId="8511" Text="@dfrankow A somewhat related question was asked here, [Logistic Regression: Which pseudo R-squared measure is the one to report (Cox &amp; Snell or Nagelkerke)?](http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox-s)." CreationDate="2011-03-20T10:52:52.690" UserId="930" />
  <row Id="14288" PostId="8519" Text="Thanks, Dmitrij. If I understand correctly, both coxtest and jtest are essentially modified nested tests. Step1: Run the model with the combined pool of regressors from model1 and model2. Step2: Test each of model1 and model2 separately as subsets of the &quot;supermodel&quot;. Am I right?  Also, on the note of IC measures, is there any way to statistically compare the AIC/BIC differences between models 1 and 2? Note: I am NOT trying to pick the &quot;best&quot; model, but you are right in that I am essentially trying test whether two models have the same fits." CreationDate="2011-03-20T11:17:42.543" UserId="3671" />
  <row Id="14289" PostId="8506" Text="@cardinal. Very well put." CreationDate="2011-03-20T12:34:40.743" UserId="1307" />
  <row Id="14290" PostId="8476" Text="@gary - not really, for the p-value makes no reference to alternatives.  So it can't be used to directly compare alternatives.  For example, take $H_0:\mu=0$ vs $H_A:\mu=-1$.  The the p-value for $H_0$ stays the same $5\times 10^{-7}$.  So you say &quot;reject the null&quot; which means &quot;accept the alternative&quot; and declare $\mu=-1$.  But this is absurd, nobody would do this, but the p-value rule you use here does this.  Put another way, the p-value rule you described is not invariant with respect to what is called the &quot;null hypothesis&quot; (resolution coming)" CreationDate="2011-03-20T12:50:14.553" UserId="2392" />
  <row Id="14291" PostId="8476" Text="(cont'd)  The resolution of the apparent absurdity is note that the p-value is not an &quot;absolute&quot; test, but a relative one, defined with an implicit alternative hypothesis.  In this case, the implicit alternative is $H_{imp}:\mu=5$.  You can see this by noting that if I calculate the p-value of $H_A$ I get $1\times 10^{-9}$, which is smaller than the p-value for $H_0$.  Now in this example, the &quot;implicit alternative&quot; is easy to find by intuition, but it is much harder to find in more complex problems, where nuisance parameters or no sufficient statistic." CreationDate="2011-03-20T12:59:21.907" UserId="2392" />
  <row Id="14292" PostId="8525" Text="@EnergyNumbers Good remark ! " CreationDate="2011-03-20T13:46:08.567" UserId="223" />
  <row Id="14293" PostId="8519" Text="@user, you do not need to add to a supermodel anything, just supply `jtest` or `coxtest` with non-nested fits from Step1. Information criterion for non-nested will be a nice guide to what model is more statistically suitable (parsimonious), but for the hypothesis testing I would just go for any $LR$ (actually the log likelihood is a part of any info-criterion) tests myself. Conclusions will be somewhat close, but since there are two deterministically given penalty functions it is a bit tricky to compare them statistically." CreationDate="2011-03-20T15:38:34.603" UserId="2645" />
  <row Id="14294" PostId="8522" Text="@Paolo I removed the link to your company as it is not necessary to understand the question and answer it." CreationDate="2011-03-20T15:55:16.953" UserId="930" />
  <row Id="14295" PostId="8522" Text="@ chl. OK. Tx. Relevance to the question." CreationDate="2011-03-20T16:01:43.197" UserId="3804" />
  <row Id="14296" PostId="8532" Text="@Galit. Tx for the fast answer. The key point is that I need to have this as an EMBEDDED functonality in an existing SW solution. So with Spotfire (very nice) and Tableau (even better) the only option is an OEM agreement that (if available) is  prohibitively expensive..." CreationDate="2011-03-20T16:04:07.547" UserId="3804" />
  <row Id="14297" PostId="8514" Score="3" Text="I would suggest you check out the visualization tag over at the gis stack exchange site for examples http://gis.stackexchange.com/questions/tagged/visualisation" CreationDate="2011-03-20T16:07:31.427" UserId="1036" />
  <row Id="14298" PostId="8525" Score="3" Text="Many mapping software platforms have the capabilities mentioned in this post. The distortions based on attributes when it comes to maps are frequently referred to as cartograms. See http://gis.stackexchange.com/q/7406/751 . That being said, bars placed happenstance in a map aren't any easier to visualize than colors. When the bars aren't side by side they are difficult to make relative comparisons, which isn't as big a problem with a color scale." CreationDate="2011-03-20T16:12:51.097" UserId="1036" />
  <row Id="14300" PostId="8529" Text="thanks. Of course you're right. " CreationDate="2011-03-20T16:13:36.953" UserId="30" />
  <row Id="14301" PostId="8181" Score="1" Text="@mhh I'm glad you find it helpful. If this answers your question, you can consider accepting it: &quot;When you have decided which answer is the most helpful to you, mark it as the accepted answer by clicking on the check box outline to the left of the answer. This lets other people know that you have received a good answer to your question. Doing this is helpful because it shows other people that you’re getting value from the community. (If you don’t do this, people will often politely ask you to go back and accept answers for more of your questions!)&quot; (from the FAQ)" CreationDate="2011-03-20T16:28:46.037" UserId="1909" />
  <row Id="14302" PostId="8455" Text="I apologize for my error: toluene is not less carcinogenic, it is actually not classified as carcinogenic at all. I agree that probability has to come in somewhere (after all, one can't design for every possibility). My only concern is that by assigning an actual number to that probability (a reductionist approach), an incorrect idea of the actual risk is implicitly conveyed. That is all." CreationDate="2011-03-20T16:31:42.170" UserId="2833" />
  <row Id="14303" PostId="8534" Text="@Henry: Thank you very much for these explanations/experiments! If I could vote twice, I would do!" CreationDate="2011-03-20T16:36:49.570" UserId="3019" />
  <row Id="14305" PostId="8527" Text="I'm sorry, I didn't mean to derail your thread. The events in Japan had me thinking about how much the mathematics we use actually correspond to risk in reality, which prompted my remarks (which did not really address your question)." CreationDate="2011-03-20T16:47:33.327" UserId="2833" />
  <row Id="14306" PostId="8464" Text="@Ralph I wouldn't waste time being picky :-).  The difference between a bar chart and a histogram is fundamental, but it's not surprising that the Web is full of confusion on this point because superficially the two look similar.  Please refer to a reputable textbook, such as the Freedman, Pisani, &amp; Purves *Statistics* (any edition)." CreationDate="2011-03-20T17:10:23.667" UserId="919" />
  <row Id="14307" PostId="8532" Text="@Galit: yes, Spotfire Silver is very cool, very attractive visually, and it's available on a one-year trial. I fell in love just looking at the demos; it's easy to believe that anyone could become addicted. But it seems only to know about Oracle and MSSQL databases. It looks like it won't run on, say, the usual Linux server which often has MySQL only: Oracle would be uncommon and MSSQL impossible. Have you any insights as to how Spotfire is going to address those issues?  " CreationDate="2011-03-20T17:35:46.280" UserId="3811" />
  <row Id="14308" PostId="8534" Text="+1 for this lengthy one, deleted mine wrong guess with negative default pm = 0 parametrization. " CreationDate="2011-03-20T17:40:53.187" UserId="2645" />
  <row Id="14309" PostId="8533" Score="3" Text="How does that work?  Doesn't the GLM estimate *expectations*, which are not necessarily integral?" CreationDate="2011-03-20T17:50:30.090" UserId="919" />
  <row Id="14310" PostId="8528" Text="If I understood you correctly, your final distribution is basically as good as your &quot;mock/randomly generated values&quot; for the objects. So - do you think you've sampled the distribution of your &quot;objects&quot; well?" CreationDate="2011-03-20T19:06:00.430" UserId="2658" />
  <row Id="14311" PostId="8514" Score="1" Text="Along that same line, you might want to check out the gradients on www.0to255.com . " CreationDate="2011-03-20T19:08:53.050" UserId="3811" />
  <row Id="14312" PostId="8455" Score="1" Text="@Gilead I did not know about Taleb but I read a book by Jean Pierre Dupuy in 2004  http://fr.wikipedia.org/wiki/Jean-Pierre_Dupuy, the book is called &quot;Pour un catastrophisme éclairé&quot; and if I reamember, it talks exactly about that. Jean Pierre Dupuy is Teaching at Stanford and at Polytechnic but if I think he mentionned Ivan Illich while talking about that. Does Taleb mentionned sources of inspirations? " CreationDate="2011-03-20T19:51:40.200" UserId="223" />
  <row Id="14313" PostId="8545" Text="@Marco; The Chow test assumes that there are no gaussian violations in either the sub-models or in the composite model. Have you verified the gaussian assumptions?&#xA;" CreationDate="2011-03-20T20:59:32.487" UserId="3382" />
  <row Id="14314" PostId="8007" Text="This model you've devised:&#xA;&#xA;Dorder=nitems∗Dfetch+nparts∗Dassemble&#xA;&#xA;Are there good reasons to believe that's how it works, or is that just a first stab at it?&#xA;&#xA;How many distinct items are there, relative to the number of orders for which you have the time the total order took?" CreationDate="2011-03-20T21:24:35.893" UserId="3794" />
  <row Id="14315" PostId="8524" Text="@chl's thorough answer to a Q last December is relevant here http://stats.stackexchange.com/questions/5171/testing-paired-frequencies-for-independence/5258#5258" CreationDate="2011-03-20T21:25:51.043" UserId="449" />
  <row Id="14316" PostId="8505" Text="OK, I obviously should say, log(1+count).  Obviously not equivalent, but wondering if there was a relationship, or if they can handle similar phenomena." CreationDate="2011-03-21T00:45:25.157" UserId="3799" />
  <row Id="14317" PostId="8538" Text="Douglas Bates seems to agree, at least with respect to the hazy mapping of R-square definitions in mixed models to their non-mixed analogs: https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q1/003363.html" CreationDate="2011-03-21T01:01:44.953" UserId="364" />
  <row Id="14318" PostId="8476" Text="@probabilityislogic what you seem to be describing is the power of the test. I've made no suggestions to &quot;accepting the alternative&quot; which is not the same as &quot;rejecting the null.&quot; In fact, the implicit alternative is not what you described but rather the two sided test $H_1:\mu\neq0$. The only point I was making is that the $p$-value seems more rigorous and a choice of $Z$ score threshold, which may or may not come from a normal distribution, seems more arbitrary." CreationDate="2011-03-21T02:35:58.220" UserId="3786" />
  <row Id="14319" PostId="8531" Text="Thank you Gail.....that is exactly the solution I wanted. By the way ...is there a general series (power,taylor or other) expansion for (1-p)^n   that you know of ?" CreationDate="2011-03-21T02:37:36.930" UserId="3777" />
  <row Id="14320" PostId="8463" Text="@bill_080....thank you for the hard work you put into this. I just found this site and I am amazed at the level of responses one gets here" CreationDate="2011-03-21T02:45:34.610" UserId="3777" />
  <row Id="14321" PostId="8543" Text="@IrishStat: Thank you for your time and expertise. I can see how much valuable time you have spent to do the calculations. Since I am a beginner in Statistics, I require some time to understand your calculations and method. I will get back at the earliest. Kind Regards" CreationDate="2011-03-21T03:15:12.613" UserId="2956" />
  <row Id="14322" PostId="8208" Text="@whuber Yes of course Spearman's $\rho$ can be computed with continuous data. I didn't think that was in question.I'd have a really hard time believing that someone could make a more meaningful inference with continuous data using a rank correlation than they could with a plain old Pearson's $r$." CreationDate="2011-03-21T03:38:11.233" UserId="2660" />
  <row Id="14323" PostId="8544" Text="Yes, I will have a look at it! Thx!!" CreationDate="2011-03-21T05:59:18.837" UserId="3019" />
  <row Id="14325" PostId="8558" Text="Hi Dmitrij... I knew you'd spot this. Actually, your response to the original question set this train rolling. I thought this would make an interesting stand-alone question. The problem I'm grappling with is that statistical tests (including Cox's Test) are frequentist and so you can test the differences between two models at some predefined level of significance. But AIC/BIC are likelihood based, and it seems to me that the numbers cannot be directly compared except, as you point out, by rule of thumb. Since IC measures are scale-dependent, an absolute value (2) can be problematic, no? " CreationDate="2011-03-21T06:16:05.817" UserId="3671" />
  <row Id="14326" PostId="8558" Text="@user, The absolute value of $2$ is not problematic. You may go for relative probability suggestion, so you will be probably more confident with this than some nice value of $2$. By _scale effect_ you mean when the criterion is less biased in small samples and consistent in large? Try consistent $BIC$ instead and $AICc$ for small samples will be also a good alternative. Rule of thumbs are still usable." CreationDate="2011-03-21T06:35:08.087" UserId="2645" />
  <row Id="14327" PostId="8477" Text="What is $D(0,1)$? Space of [càdlàg](http://en.wikipedia.org/wiki/Cadlag) functions?" CreationDate="2011-03-21T08:04:20.010" UserId="2116" />
  <row Id="14328" PostId="8545" Text="thank you for your response. yes, the response variable and the residuals of my dataset show some normal distribution." CreationDate="2011-03-21T08:05:33.533" UserId="3814" />
  <row Id="14329" PostId="8547" Text="thank you for your hint, too! I found this package, but I thought it deals exclusively with time series data. thus, I can use it for cross-sectional data as well. I will give it a try... " CreationDate="2011-03-21T08:09:09.927" UserId="3814" />
  <row Id="14330" PostId="8489" Text="@probabilityislogic +1, nice answer to somewhat convoluted question. " CreationDate="2011-03-21T08:13:59.703" UserId="2116" />
  <row Id="14331" PostId="8551" Text="@Adam Kane, what kind of answer you are expecting? The first half of a bell curve is the first half of the bell curve. Even if it had specific name, I do not understand how would that might help." CreationDate="2011-03-21T08:18:41.820" UserId="2116" />
  <row Id="14332" PostId="8543" Text="@IrishStat: I am extremely happy with your excellent guidance. It is taking a long time for me to understand your calculations. For state 1, the model was ARIMA (1,0,0). For state 2 what was the model? Is it the same? Can you kindly let me know? " CreationDate="2011-03-21T08:20:43.663" UserId="2956" />
  <row Id="14333" PostId="8551" Text="@Adam Kane, if we take first half of a bell curve, then we truncate the original distribution. So [truncated normal distribution](http://en.wikipedia.org/wiki/Truncated_normal_distribution) might be something you are after." CreationDate="2011-03-21T08:23:54.407" UserId="2116" />
  <row Id="14334" PostId="8547" Text="@marco try package [segmented](http://cran.r-project.org/web/packages/segmented/index.html). You might search this site also, the structural breaks came up several times in last few months." CreationDate="2011-03-21T08:26:40.037" UserId="2116" />
  <row Id="14335" PostId="8528" Text="@AVB: not so much the distribution of the object but the distribution of the scores for one particular object. I guess my goal is to: A) Make sure the score calculation is sound and not-biased towards different properties of the objects in question; and B) To be able to say something about the significance of the score calculated from real experimental data" CreationDate="2011-03-21T08:40:29.523" UserId="3014" />
  <row Id="14336" PostId="8550" Text="Thanks for your answer, Gary. I am familiar with bootstrapping methods, however I am not sure how (read: where in the analysis) do you suggest I implement bootstrapping. Which original samples are we talking about? " CreationDate="2011-03-21T08:42:58.803" UserId="3014" />
  <row Id="14337" PostId="8543" Text="@IrishStat: While trying to search for more information on OLS model, I chanced upon your very useful document http://www.autobox.com/pdfs/Pickett.pdf published in THE JOURNAL OF BUSINESS FORECASTING, SPRING 2005. Now I know why it is taking so much time for me to understand what you wrote in a few minutes. i am so happy my call for help through this website  attracted the attention of a great and dedicated man." CreationDate="2011-03-21T08:48:59.717" UserId="2956" />
  <row Id="14338" PostId="2897" Text="@Mark could you provide a reference for this result?" CreationDate="2011-03-21T10:01:54.277" UserId="3786" />
  <row Id="14339" PostId="8512" Text="@raegtin, this depends heavily how do you get estimated probability distribution. At the moment the questions in my opinion is too vague, although it is an interesting one. " CreationDate="2011-03-21T10:03:18.423" UserId="2116" />
  <row Id="14340" PostId="8550" Text="Original samples refer to the $N=1000$ samples you used to construct your empirical distribution. You mentioned using this distribution to compute statistics; let's say for simplicity the mean. The way to use the bootstrap would be to compute a mean with the initial $N$ points, let's call it $\hat{\mu}_N$. Now we take $M$ bootstrap samples and compute $M$ estimates for the mean, $\{\hat{\mu}_i^*\}_{i=1}^M$. You can then order this set and the order statistics will give you confidence intervals on $\hat{\mu}_N$" CreationDate="2011-03-21T10:15:05.480" UserId="3786" />
  <row Id="14341" PostId="8560" Text="@Peter +1 for emphasizing the need for both statistical and substantive expertise." CreationDate="2011-03-21T10:26:40.033" UserId="930" />
  <row Id="14343" PostId="8547" Text="above I have added first results but I struggled with the interpretation... please, see the example." CreationDate="2011-03-21T11:24:20.320" UserId="3814" />
  <row Id="14344" PostId="8446" Text="@whuber, I want to relate my data to a scientific hypothesis. (I don't want understand why my distribution is not uniform as some week ago). Now I want to say for example that the obtained distribution satisfies some properties..how can I proceed?Thank you." CreationDate="2011-03-21T11:53:58.903" UserId="3342" />
  <row Id="14345" PostId="8563" Text="As the [disambiguation page on wikipedia](http://en.wikipedia.org/wiki/Precision) demonstrates, *precision* is an ambiguous term. Can you put it into context?" CreationDate="2011-03-21T12:17:14.890" UserId="977" />
  <row Id="14346" PostId="8563" Text="@John Precision in what sense? For example, *reliability* of scores will probably have a different meaning in Psychometrics, depending on the model we consider, as compared to other scientific fields." CreationDate="2011-03-21T12:25:28.527" UserId="930" />
  <row Id="14347" PostId="8543" Text="@DrWho: The model for STATE1 is a Transfer Function Model because it includes causal variables ( # of cases plus the Interventions that were Detected ) and an ARIMA component (1,0,0). The model for STATE2 was similar but had an ARIMA component (0,0,0) which means it was a simple multiple regression model (OLS) . Regression(OLS) is a particular case of a Transfer Function." CreationDate="2011-03-21T12:43:08.207" UserId="3382" />
  <row Id="14348" PostId="8479" Text="@Gary, thanks, i'll try to use MLE!" CreationDate="2011-03-21T12:44:33.433" UserId="3342" />
  <row Id="14349" PostId="8543" Text="@IrishStat: Thank you very much. Every time you reply, I learn something more to my great advantage. I am still struggling to understand the meaning of your approach. Best Regards" CreationDate="2011-03-21T12:47:36.423" UserId="2956" />
  <row Id="14350" PostId="8561" Text="I thought the blocks in the regression commands just ran sequential models. Is there any reason why running the models separately will not suffice? " CreationDate="2011-03-21T13:03:29.140" UserId="1036" />
  <row Id="14351" PostId="8527" Text="@J Presley, with all due respect, I *don't* believe this is the question you were initially asking, **nor** do I believe it is the answer that you should really, *truly* want. Sure, it's a convenient one, in particular, because it's not &quot;messy&quot;. **[Here](http://maptd.com/map/earthquake_activity_vs_nuclear_power_plants/)** is an interesting map I just found showing nuclear power-plant site distribution and locations of high tectonic activity. Sorry, I don't mean to sound argumentative. I do note the same tendency I often see to revise the question *post hoc* to obtain a simpler answer." CreationDate="2011-03-21T13:20:13.060" UserId="2970" />
  <row Id="14352" PostId="8527" Text="@J Presley, also, just a note on site participation: If you truly believe that **bayer** has provided the answer you desired, then it would be polite to mark that answer as the accepted one instead of posting your own and accepting it." CreationDate="2011-03-21T13:21:58.783" UserId="2970" />
  <row Id="14353" PostId="8543" Text="@DrWho; I would be more than willing to help you. Please contact me privately at dave@autobox.com. After you and I have resolved your &quot;difficulties&quot; I would like you to share your newfound knowledge and understanding to the list as others may also have your &quot;difficulty&quot; and you are the only one to speak up about it." CreationDate="2011-03-21T13:39:41.107" UserId="3382" />
  <row Id="14354" PostId="8563" Text="Precision as the &quot;quality&quot; that assures that repeated measurements (on the same subject) will lead to the same result." CreationDate="2011-03-21T13:43:19.357" UserId="3823" />
  <row Id="14355" PostId="8567" Text="I assumed you were talking about R package; correct me if I'm wrong." CreationDate="2011-03-21T13:50:02.423" UserId="88" />
  <row Id="14356" PostId="8563" Text="@John The NIST is *the* authority on such issues.  Its guidance is comprehensive and clear: http://physics.nist.gov/Pubs/guidelines/appd.1.html .  (For the record: the NIST prefers not to use &quot;precision&quot; but accepts it when used with disambiguating language.  See D.1.2.)" CreationDate="2011-03-21T14:08:13.873" UserId="919" />
  <row Id="14357" PostId="8559" Text="You are proposing a *quadrat count analysis.*  There is much about this in the ecology literature.  For an introduction see http://oai.cwi.nl/oai/asset/10611/10611A.pdf .  (Stick to peer-reviewed papers, preferably those that don't shy away from explicitly invoking probability models, and avoid the GIS and geographical literature, which tends toward the overly simple and uncritical.)" CreationDate="2011-03-21T14:14:59.017" UserId="919" />
  <row Id="14358" PostId="8476" Text="@Gary - the p-value is no more rigorous just because it is a probability.  It is a monotonic 1-to-1 transformation of the Z-score.  any &quot;rigor&quot; that is possessed by the p-value is also possessed by the Z-score.  Although if you are using a two sided test then the equivalent is the absolute value of the Z-score.  And in order to compare $H_1:\mu\neq 0$ to the null, you have to take a &quot;minimax&quot; approach: which is to pick the sharp hypothesis which is most supported by the data and consistent with $H_1$.  Unless you can demonstrate how to calculate $P(X|\mu\neq 1)$" CreationDate="2011-03-21T14:18:41.133" UserId="2392" />
  <row Id="14359" PostId="8563" Text="That document is not up to date. The 2008 revision of VIM does give a definition for &quot;precision&quot;. Nonetheless, NIST does raise valid questions...&#xA;&#xA;But when I say &quot;precision&quot; I could also say &quot;accuracy&quot;. What is &quot;accuracy&quot; for a measurement?" CreationDate="2011-03-21T14:25:04.000" UserId="3823" />
  <row Id="14360" PostId="8563" Text="@John *op. cit.*, section D.1.1.1; *viz.,* &quot;closeness of the agreement between the result of a measurement and the value of the measurand //&#xA;&#xA;    NOTE //&#xA;&#xA;       1. &quot;Accuracy&quot; is a qualitative concept. // &#xA;&#xA;       2. The term &quot;precision&quot; should not be used for &quot;accuracy.&quot; &quot;&#xA;&#xA;" CreationDate="2011-03-21T14:40:23.037" UserId="919" />
  <row Id="14361" PostId="8474" Text="@Rob:  Definitely true but not what I was suggesting.  Of course if the lower and upper bound of your confidence interval on Pearson correlation have different signs then your lower bound on the confidence interval for R^2 is zero. " CreationDate="2011-03-21T14:50:14.030" UserId="1347" />
  <row Id="14362" PostId="8563" Text="@whuber: Let me rephrase my question: I want to use &quot;precision&quot; and &quot;measurement&quot; in the same sentence. Should I say: &quot;precision is an important **quality** of measurement&quot; or, instead of quality, should I use another synonym? Characteristic?" CreationDate="2011-03-21T14:53:31.620" UserId="3823" />
  <row Id="14363" PostId="8563" Score="1" Text="@John This sounds like a question more suitable for the 'english' SE site.  I think the NIST would say precision is a quality, characteristic, attribute, feature, or property of the *measurement process,* not of the measurement itself (much as &quot;painful&quot; might describe the creation of a work of art but--one hopes--does not apply to the art work itself!).  As far as English usage goes, both &quot;quality&quot; and &quot;characteristic&quot; tend to invoke a sense of being *inherent* or *essential* whereas &quot;attribute,&quot; &quot;feature,&quot; and &quot;property&quot; are more generic.  Which to use depends on what you want to emphasize." CreationDate="2011-03-21T15:01:22.910" UserId="919" />
  <row Id="14364" PostId="8563" Text="Thanks, whuber." CreationDate="2011-03-21T15:06:31.873" UserId="3823" />
  <row Id="14365" PostId="8543" Text="@IrishStat: Great offer which I could never have dreamt of. I am  really sorry for bothering you repeatedly. The reason is tousands of lives can be saved in State 2 in future if I can present the data as you advised. I am getting confused with the letters you have used.  X1 is the number of cases and Y is the number of deaths. X2 is the empirically identified point of anomaly.&#xA;What do X3, X4,X5,X6, A and T stand for?  I am a Brain specialist doctor and poor in Statistics and Mathematics. I am trying to learn little by little. I have contacted so many statisticians but nobody could help me. " CreationDate="2011-03-21T15:15:45.110" UserId="2956" />
  <row Id="14366" PostId="8568" Score="1" Text="just a quick idea: it seems for this question to belong to the realm of Bayesian approach (so may be there will be such a reply from an expert in that field). From the point of view of a regular $OLS$ user, why not to re-run the same estimation with a new dataset (it is hardly believable that calculations take too long time, though Bayesian tricks could be quicker here)?" CreationDate="2011-03-21T16:07:45.647" UserId="2645" />
  <row Id="14368" PostId="8380" Text="@SheldonCooper Thanks - I tagged with R as you suggested.  I did come up with an approach for this problem but had been holding off on presenting it to avoid biasing responses.  I guess I'll reveal it now: it is available [here](http://www.stanford.edu/~mbloem/research/bloem_ATIO2009_robust_DD_pred.pdf).  What do you think of the approach I tried?  See section III and the appendix for the details." CreationDate="2011-03-21T16:09:51.133" UserId="3744" />
  <row Id="14369" PostId="8490" Score="1" Text="@Wam It sounds like a simple random intercept model would answer both of your questions. It will give you a intra-subject correlation coefficient and allow you to test the difference in test score across times with a standard error that has been adjusted for the within-subject correlation. Especially because you expect test forms to be positively correlated, you should not just compare the mean of time 10:00 to the mean of 15:00 because your p-value will be too small. Try searching for &quot;random intercept&quot; or &quot;random effect&quot; -- there are lots of examples worked out for R." CreationDate="2011-03-21T16:10:15.303" UserId="3432" />
  <row Id="14370" PostId="8562" Text="Is the redundancy just in the meaning of the predictor variable or is it in the actual data such that each variable repeats the same baseline data?" CreationDate="2011-03-21T16:13:52.897" UserId="601" />
  <row Id="14371" PostId="8528" Text="You might consider taking more samples from your objects. " CreationDate="2011-03-21T17:43:34.817" UserId="1146" />
  <row Id="14373" PostId="8574" Text="The second (what are some common techniques...). Most texts present a model and then describe how to do inference. I'm looking for sort of the inverse, where the focus is on ways to fit a model and then comparing them in applications, if that makes sense. There are a few of these sort of books for MCMC where they compare different samplers and describe where they're useful &amp; some of the pitfalls (eg Gamerman &amp; Lopes)." CreationDate="2011-03-21T18:07:31.660" UserId="26" />
  <row Id="14374" PostId="8574" Text="Also, thanks for the references thus far. The Hastie et al book is pretty close, actually. It's been a while since I had it off the shelf; thanks for the prompt :)" CreationDate="2011-03-21T18:10:05.450" UserId="26" />
  <row Id="14377" PostId="8580" Text="Could you please explain why you use these lines of code `deviance = bsxfun(@minus,xtest,mu);  %tbc&#xA;deviance = bsxfun(@rdivide,deviance,sigma); %tbc&#xA;deviance = deviance .^ 2; %tbc`" CreationDate="2011-03-21T18:33:28.377" UserId="3681" />
  <row Id="14378" PostId="8580" Text="@darkcminor: the `xtest` that I have, and the sample data you provide are drawn from $\mathcal{N}(0,1)$. The classifier is always going to prefer the $\mathcal{N}(5,1)$ class to the $\mathcal{N}(6,1)$ class in this case! Instead, why not try comparing to two classes, one with mean $-1$ and the other with mean $1$." CreationDate="2011-03-21T18:34:28.097" UserId="795" />
  <row Id="14380" PostId="8580" Text="Guess not getting the complete idea, I understand N(5,1) will be preffered to N(6,1), How could you incorporate Prior probabilities, as they are lets say `( p(x∣c1))/(p(x∣c2)) = .90/.10` Do you just multiply?, I guess I am missing a logarithm here, Guess log(.90/.10)" CreationDate="2011-03-21T18:40:54.157" UserId="3681" />
  <row Id="14381" PostId="8580" Text="if $\log(P(x = c_1) / P(x = c_2))) &gt; 0$, then call it class $1$, otherwise call it class $2$." CreationDate="2011-03-21T18:47:30.737" UserId="795" />
  <row Id="14382" PostId="8581" Text="@Coronier Sorry, I doubt there is a comprehensive guide for understanding psychology articles that use statistical modeling. But the required background should all be at the level of a master's degree in statistics. If your program will pay for it, consider getting an M.A. in stats. The next best option for your purposes might be to re-take the stats department's version of multivariate stats -- usually they provide notes with the mathematical background for PCA, clustering, trees, etc. You will need a background in linear algebra and basic mathematical statistics regardless." CreationDate="2011-03-21T18:56:46.390" UserId="3432" />
  <row Id="14383" PostId="8581" Text="Please ask more specific questions." CreationDate="2011-03-21T19:05:52.777" UserId="88" />
  <row Id="14384" PostId="8581" Score="4" Text="I'm a Psychology PhD student as well, and I made the choice to take a significant amount of math in my undergraduate years because there were so many Psychology PhDs who have no idea how a PCA (for example) was computed. The very first thing you need to do is work your way through any decent linear algebra textbook. What's a decent linear algebra textbook? Gilbert Strang's is the bomb, and he has video lectures of his linear algebra course on MIT's website to boot. You can even get them on iTunes." CreationDate="2011-03-21T19:07:52.730" UserId="2660" />
  <row Id="14386" PostId="8563" Text="@John We're leaving this thread open for a short while in the hope that you have a way to clarify it that is of statistical interest.  If your interest is primarily in the meaning of &quot;quality&quot; in this context we'll migrate it to the English site for you.  If you don't do anything, we'll just close it." CreationDate="2011-03-21T19:35:18.120" UserId="919" />
  <row Id="14387" PostId="8566" Text="@baris_a: I do not know about the &quot;naive Bayes&quot; procedure. Nonetheless, intuitively, the answer is yes. To ensure this, you could simulate some data sets, test your &quot;train logistic classifier&quot;, and compare its performance with the &quot;non-train logistic classifier&quot;..." CreationDate="2011-03-21T20:11:40.883" UserId="3019" />
  <row Id="14388" PostId="8563" Text="@whuber: You can close it. &quot;Characteristic&quot; is what I was looking for." CreationDate="2011-03-21T20:31:54.797" UserId="3823" />
  <row Id="14389" PostId="8588" Score="1" Text="@edi I am not sure - can you provide more details?" CreationDate="2011-03-21T21:03:47.653" UserId="2750" />
  <row Id="14390" PostId="8540" Text="I would recommend the same thing at such low sample size. It might not be entirely rational, but I just disregard almost any statistical test running on less than 30 observations." CreationDate="2011-03-21T21:22:28.043" UserId="3582" />
  <row Id="14391" PostId="8586" Text="Are you comparing *variances* (as asserted in the first line) or *means* (as indicated in the third line)?" CreationDate="2011-03-21T21:24:08.930" UserId="919" />
  <row Id="14392" PostId="8588" Score="2" Text="@edi That's a good comment but it doesn't have enough information to qualify as a reply." CreationDate="2011-03-21T21:24:48.600" UserId="919" />
  <row Id="14393" PostId="8586" Text="@whuber I had confused myself; I have clarified my question. " CreationDate="2011-03-21T21:28:06.110" UserId="2750" />
  <row Id="14394" PostId="8586" Text="@Abe It seems now you have reversed &quot;means&quot; and &quot;variances&quot; but there's still a contradiction!  (Unless perhaps you want to compare the variances of the means.)  What do you mean by &quot;variable&quot;: the means or the underlying values on which they are based?" CreationDate="2011-03-21T21:31:31.583" UserId="919" />
  <row Id="14397" PostId="8586" Text="@whuber sorry, is that better?" CreationDate="2011-03-21T21:33:37.790" UserId="2750" />
  <row Id="14398" PostId="8586" Text="@Abe The new title helps a lot.  But just to be clear: are the sigma-squareds the variances of the *means* or of the underlying *variables* upon which the means are based?" CreationDate="2011-03-21T21:35:53.633" UserId="919" />
  <row Id="14399" PostId="8586" Text="@whuber the underlying variables; I have removed the term 'mean' for clarity." CreationDate="2011-03-21T21:37:56.880" UserId="2750" />
  <row Id="14400" PostId="8589" Text="@shabbychef thanks for pointing me in this direction. It only took me about 5 min to download, read the section chapter, and compute; very much appreciated, but I am going to hold of on accepting your answer in hopes that other methods will be suggested - as this one is rather limited (only tests at alpha = 0.05, and there may be other options for large samples as in the present case)" CreationDate="2011-03-21T22:00:00.830" UserId="2750" />
  <row Id="14401" PostId="8591" Text="For the theoretical part google for &quot;function of random variables&quot;. For example http://www.ece.uah.edu/courses/ee420-500/500ch4.pdfjCNFXYR3RhUEfHvQlffhq4wn_ZObeQQ&amp;sig2=IB6UZa5fOokXx9c2vtWsEg&amp;cad=rja, but there are many links" CreationDate="2011-03-21T17:14:28.803" />
  <row Id="14402" PostId="8591" Text="I'd try asking this on http://math.stackexchange.com." CreationDate="2011-03-21T17:14:52.863" />
  <row Id="14403" PostId="8589" Text="+1; nice reference.  (Thanks to @Abe for supplying the link.)" CreationDate="2011-03-21T22:08:29.253" UserId="919" />
  <row Id="14404" PostId="8581" Score="1" Text="The question is so broad that it will not get a satisfying answer in a few paragraphs. Statistics are like questions: it gets easier if you break it down to several manageable components." CreationDate="2011-03-21T22:11:17.190" UserId="3582" />
  <row Id="14405" PostId="8474" Text="@dsimcha. No. If you set the lower bound to zero you will change the probability coverage of the CI." CreationDate="2011-03-21T22:12:51.947" UserId="159" />
  <row Id="14406" PostId="8581" Text="I can only agree with the above comments. Either you'll have to focus on a particular issue, or you just need to work through some textbooks or online handouts first. A decent textbook that covers basic concepts for multivariate statistics with illustrations is *Mathematical Tools for Applied Multivariate Analysis*, by Carroll and Green (AP, 1997, Rev. Ed.). Another one is *Applied Multivariate Statistics and Mathematical Modeling*, by Tinsley and Brown (AP, 2000)." CreationDate="2011-03-21T22:23:09.973" UserId="930" />
  <row Id="14407" PostId="8589" Text="@shabbychef I've already +1 but was very unlucky with the R package -- OS X build doesn't include the aforementioned function :(" CreationDate="2011-03-21T22:26:53.230" UserId="930" />
  <row Id="14408" PostId="8514" Text="Some of the maps packages for R have built-in colour codes that prevent this kind of issue, but is that what you were asking about?" CreationDate="2011-03-21T22:32:53.023" UserId="3582" />
  <row Id="14409" PostId="673" Text="Some data analysis does not rely on frequentist statistics." CreationDate="2011-03-21T22:41:42.453" UserId="3582" />
  <row Id="14411" PostId="8589" Text="@chl AFAIK the package is just a convenient bundle for the functions that are available at http://www-rcf.usc.edu/~rwilcox/Rallfun-v13" CreationDate="2011-03-21T22:42:51.283" UserId="1909" />
  <row Id="14412" PostId="8589" Text="@Abe: for values of $\alpha \ne 0.05$, Wilcox outlines the method for constructing the cutoff values of $l$ and $u$, but I agree it would be nice if they were available as rough functions of $\alpha$." CreationDate="2011-03-21T22:46:40.157" UserId="795" />
  <row Id="14413" PostId="8589" Text="@shabbychef I am confused - in the book it says 'the method can be applied only with $\alpha=0.05$, modifications based on other $\alpha$ values have not been derived&quot;? His 2002 paper abstract hints otherwise, but I am not able to access it." CreationDate="2011-03-21T22:50:21.157" UserId="2750" />
  <row Id="14414" PostId="8589" Text="'The method just described is based on a strategy similar to Gosset's derivation of Student's t: Assume normality and then make adjustments so that for small sample sizes, accurate probability coverage is obtained.' " CreationDate="2011-03-21T22:55:11.863" UserId="795" />
  <row Id="14415" PostId="8593" Text="Please correct my train of thoughts: some fields might focus on, say, biochemical exposure, and hence want to use p &lt; 0.001 as to prevent any Type I error that might lead to health hazard. Also, along this [article](http://www.radford.edu/~jaspelme/611/Spring-2007/Cowles-n-Davis_Am-Psyc_orignis-of-05-level.pdf) from *Am Psych*, I also remember a great study in the *Am J of Sociol* or one of the soc sci journals that I follow. My favourite is, of course, [Ziliak and McCloskey](http://www.deirdremccloskey.com/articles/stats/preface_ziliak.php)." CreationDate="2011-03-21T23:19:48.753" UserId="3582" />
  <row Id="14416" PostId="8596" Text="I understand the distinction, although I am probably making the mistake routinely. But my question is, is there any conventional usage, somewhere out there, of p &lt; .0001 for instance? Or, to put it provocatively, is the p &lt; .05 cult universal?" CreationDate="2011-03-21T23:35:40.427" UserId="3582" />
  <row Id="14417" PostId="7083" Text="Ouliers in the error series will &quot;deflate the ACF&quot; thus yielding an ALICE IN WINDERKlAND effect. All ACF's are subject to this thus one must ensure no anomalies" CreationDate="2011-03-21T23:44:25.510" UserId="3382" />
  <row Id="14418" PostId="8596" Text="The 'cult' of P&lt;0.05 may be nearly universal, but it is not possible to be confident about any assertions on this point because apparent exceptions are quite likely to be the result of unknowing hybridisation of Fisher and Neyman-Pearson methods. In basic pharmacological research papers there is almost never an explicit statement regarding the use of Neyman-Pearson error rates." CreationDate="2011-03-22T00:17:23.340" UserId="1679" />
  <row Id="14419" PostId="8593" Text="What you describe here sounds backwards.  I'd be worried about Type II errors, saying something's not there when it is, with biochemical exposure.  In that case I might set alpha higher, not lower." CreationDate="2011-03-22T00:42:57.933" UserId="601" />
  <row Id="14420" PostId="8601" Text="Ok - I goofed on the UCL and LCL values. I'm used to java code with semi-colons. I fixed the post but still get 0... :( Could you please elaborate on what you mean by the mean of the empty set? Sorry for my bad english." CreationDate="2011-03-22T00:50:04.893" UserId="3832" />
  <row Id="14421" PostId="8514" Text="I am using this on a custom map, and the obvious approach (to divide each value by the total population) does give me a value between 0 and 1 (I then use this value to choose the &quot;intensity&quot; of the color). The problem is that there are values that are too far appart, so some states look completely colored and some have almost no color at all. I know statistically speaking this is correct but I want to make the data representation more relevant and easier to understand." CreationDate="2011-03-22T01:05:36.590" UserId="3803" />
  <row Id="14422" PostId="8593" Text="I was working under the assumption that the test would of the form: &quot;Let's assess if pregnancy is related to HRT&quot; (in that case, a Type I error is more serious than a Type II error, but perhaps this design is nonstandard)." CreationDate="2011-03-22T01:26:36.180" UserId="3582" />
  <row Id="14423" PostId="8596" Text="Thanks for the example. I am less and less impressed by pharmacological research, for many (not all scientific) reasons…" CreationDate="2011-03-22T01:28:12.387" UserId="3582" />
  <row Id="14424" PostId="8601" Text="Instead of telling me everything that wrong with current code, could you give hint at how to approach from beginning?" CreationDate="2011-03-22T01:31:39.940" UserId="3832" />
  <row Id="14425" PostId="8588" Text="Oh, I´m sorry. Just though on doing a Levene-test, but get the p-value by permutation of the observations. " CreationDate="2011-03-22T01:32:58.577" UserId="1050" />
  <row Id="14426" PostId="8588" Text="Though on something like [Anderson 2001](http://rparticle.web-p.cisti.nrc.ca/rparticle/AbstractTemplateServlet?calyLang=eng&amp;journal=cjfas&amp;volume=58&amp;year=2001&amp;issue=3&amp;msno=f01-004)" CreationDate="2011-03-22T01:45:21.067" UserId="1050" />
  <row Id="14427" PostId="8455" Text="@robin, I don't think Dupuy was referenced, but his work sounds interesting. Benoit Mandelbrot's work however was mentioned as a source of inspiration." CreationDate="2011-03-22T01:56:14.653" UserId="2833" />
  <row Id="14429" PostId="8598" Text="A quick note related to your question about coverage probability: We have some unknown quantity that we cannot observe, in this case, the mean. We can construct a confidence interval, which gives us a rough idea of where the true value of the parameter is. Depending on what points we sample from in our population, we will get different confidence intervals. Coverage probability is the proportion of those intervals that contain the true value of the parameter. See Wikipedia: http://en.wikipedia.org/wiki/Coverage_probability" CreationDate="2011-03-22T02:32:13.877" UserId="1118" />
  <row Id="14430" PostId="8601" Text="The zeros stem from the fact that mean(LCL &gt; 0 &amp; UCL &lt; 0); should be mean(LCL &lt; 2 &amp; UCL &gt; 2) (the number of times the Ci includes 2, the true mean, divided by m)." CreationDate="2011-03-22T02:43:18.057" UserId="26" />
  <row Id="14431" PostId="8596" Text="You shouldn't take my comment about basic pharmacological research as a specific criticism of that field, it is just my own particular discipline and thus the one with which I am most experienced. I am confident that you would find many areas in basic research with exactly the same shortcomings with respect to hybridized P values and error rates." CreationDate="2011-03-22T03:39:49.950" UserId="1679" />
  <row Id="14432" PostId="8481" Text="Thanks for this! Could you elaborate a little more about the relationship between the effectiveness of MCMC and the correlation between the variables? I had always thought that MCMC would provide an advantage over exisiting numerical integration techniques when the variables are correlated - though I cannot provide a good reason for why it would. " CreationDate="2011-03-22T03:40:53.377" UserId="3572" />
  <row Id="14433" PostId="8539" Text="I am trying to break something; I know the vales I am getting are statistically correct, but I want to make it easier for the users to understand the data. It's a UI decision." CreationDate="2011-03-22T04:04:23.073" UserId="3803" />
  <row Id="14434" PostId="8581" Text="@Jeromy @chl Please consider promoting your comments to replies.  @Coronier If you could focus this question a little, such as asking about specific kinds of equations or the kind of &quot;understanding&quot; you seek, this thread might be more fruitful for everyone." CreationDate="2011-03-22T05:23:13.543" UserId="919" />
  <row Id="14436" PostId="8581" Text="thank whuber. I've deleted my comment and added it as an expanded answer." CreationDate="2011-03-22T05:53:24.897" UserId="183" />
  <row Id="14438" PostId="8586" Text="Anwsers (including mine :) )  to that question http://stats.stackexchange.com/questions/841/comparing-the-variance-of-paired-observations/7013#7013 might help." CreationDate="2011-03-22T07:10:57.740" UserId="223" />
  <row Id="14439" PostId="8589" Text="@caracal Thanks. I guess the Win/Linux versions available on RForge are more up to date." CreationDate="2011-03-22T07:20:54.263" UserId="930" />
  <row Id="14440" PostId="8611" Text="@berkay Could you clarify what precisely are those measurements (time to perform a given task?) and what is the data structure, that is what are the different conditions and what's the number of records? Also you added two different tags? Are you expecting two distinct solutions, in Matlab and Gnuplot?" CreationDate="2011-03-22T07:27:18.597" UserId="930" />
  <row Id="14441" PostId="8610" Score="3" Text="@Brett (+1) There was also an interesting thread about the causality issue in this related question, [From a statistical perspective, can one infer causality using propensity scores with an observational study?](http://stats.stackexchange.com/questions/3400/from-a-statistical-perspective-can-one-infer-causality-using-propensity-scores-w)." CreationDate="2011-03-22T07:36:38.977" UserId="930" />
  <row Id="14442" PostId="8559" Text="Thanks for the info !" CreationDate="2011-03-22T07:53:37.717" UserId="3699" />
  <row Id="14444" PostId="8448" Text="@Jim: Thanks. The updated version of your answer gives me what I need." CreationDate="2011-03-22T08:45:58.393" UserId="2824" />
  <row Id="14445" PostId="8593" Text="+1 for the side note and the link" CreationDate="2011-03-22T09:04:03.737" UserId="2645" />
  <row Id="14446" PostId="8410" Text="@Aniko: Please could you indicate how I could get a $\chi^2$ p-value for each of the two outcomes per bin? I need to know if there is a difference in the counts for each of the two outcomes in each bin.  (I've asked Jim as well, as the two solutions are different.)" CreationDate="2011-03-22T09:06:29.850" UserId="2824" />
  <row Id="14447" PostId="8448" Text="@Jim: Please could you indicate how I could get a $\chi^2$ p-value for each of the two outcomes per bin? I need to know if there is a difference in the counts for each of the two outcomes in each bin. (I've asked Aniko as well, as the two solutions are different.)" CreationDate="2011-03-22T09:07:29.123" UserId="2824" />
  <row Id="14448" PostId="8614" Score="1" Text="@Dmitrij Please put your comment as an answer, this might help the OP to refine his question or engage into a more constructive dialogue. (As you said this has become a long-lasting series of questions now). @Imelza You already asked 5 questions, and got some comments on them (see e.g., [here](http://stats.stackexchange.com/questions/7238/matlab-interpretation-of-xcorr2-cross-correlation-function) and [here](http://stats.stackexchange.com/questions/8116/cross-correlation-in-matlab)) but you didn't address any of them. Any reason why?" CreationDate="2011-03-22T09:17:06.593" UserId="930" />
  <row Id="14449" PostId="8410" Text="@SabreWolfy Look at the `test=TRUE` option in `summary.formula()`." CreationDate="2011-03-22T09:22:44.703" UserId="930" />
  <row Id="14450" PostId="8448" Text="@Jim Finally, the `Hmisc` library is not needed here?..." CreationDate="2011-03-22T09:24:15.670" UserId="930" />
  <row Id="14451" PostId="8612" Score="4" Text="One more: `m %*% diag(1/colSums(m))`" CreationDate="2011-03-22T09:33:43.893" UserId="1909" />
  <row Id="14452" PostId="8410" Text="@chi: Thanks for the tip." CreationDate="2011-03-22T09:52:10.220" UserId="2824" />
  <row Id="14453" PostId="8613" Text="Thanks, the ressources you provide are great. Btw, your blog is totally absorbing (I'm an I/OP student and useR, it's like a revelation for me)." CreationDate="2011-03-22T09:57:36.050" UserId="3827" />
  <row Id="14454" PostId="8410" Text="@chi: Adding `test=TRUE` (to the first formula above) gives output for the first bin only." CreationDate="2011-03-22T10:03:42.893" UserId="2824" />
  <row Id="14455" PostId="8621" Text="@Peter Not sure that sample size will not be an issue, as the OP specified that &quot;The dataset is not in same size as shown below&quot; -- stripchart with 10^6 points will be messy :)" CreationDate="2011-03-22T10:28:25.740" UserId="930" />
  <row Id="14457" PostId="8618" Text="@Dmitrij Celov: Great and simple answer. You have an appreciable knowledge at an young age.&#xA;" CreationDate="2011-03-22T10:31:03.347" UserId="2956" />
  <row Id="14458" PostId="8619" Score="1" Text="@probabilityislogic: Thank you very much for making it clear. " CreationDate="2011-03-22T10:45:32.300" UserId="2956" />
  <row Id="14459" PostId="8613" Text="@Coronier It's great to meet another person combining R with I/O Psych." CreationDate="2011-03-22T10:46:09.040" UserId="183" />
  <row Id="14460" PostId="8602" Text="Thank you so much for this - it makes sense :). I get ~0.92 as my answer." CreationDate="2011-03-22T11:00:26.743" UserId="3832" />
  <row Id="14461" PostId="8619" Text="(+1) nice suggestions, though more not-so-dirty-tricks probably possible." CreationDate="2011-03-22T11:04:45.023" UserId="2645" />
  <row Id="14462" PostId="8359" Text="hmm what I still don't get: statistics.com defines non-stationarity as &quot;depending on time&quot;. and here we seem to have a stationarity variable with a time trend. How can it be stationary if it depends on time? Confuses me...  " CreationDate="2011-03-22T12:11:35.793" UserId="704" />
  <row Id="14463" PostId="8410" Text="No, adding `test=TRUE` does not give output for the first bin only, it adds (to the first row) an _overall_ p-value testing whether the distribution across the bins is the same in the two groups. You can control exactly what test is performed (`catTest` parameter) - for example, here you might want to run an ordered test, because the bins are ordered - but it will be one p-value per variable. It does not make sense to do a separate test for each bin." CreationDate="2011-03-22T12:38:08.530" UserId="279" />
  <row Id="14464" PostId="8610" Text="I agree with the general premise of this answer, but when one matches based on the propensity scores it is not the same as plopping all the covariates into the model (and hence is not just a dimension reduction technique). It is not the same if one weights by propensity scores either." CreationDate="2011-03-22T12:46:28.680" UserId="1036" />
  <row Id="14465" PostId="8606" Text="@berk - interesting question.  you should add &quot;hats&quot; to your estimates, so that it is clear which quantities you are talking about (hypothesis refers to true values - others places refer to estimates).  Also, are these upper and lower bounds &quot;plugged&quot; back into the optimization routine to get a new estimates of upper and lower bounds? so that $[\hat{\mu}_{UB}^{(j)},\hat{\mu}_{LB}^{(j)}]=F[\hat{\mu}_{UB}^{(j-1)},\hat{\mu}_{LB}^{(j-1)}]$  where $F[.]$ is the optimization routine." CreationDate="2011-03-22T12:54:30.280" UserId="2392" />
  <row Id="14466" PostId="8627" Text="@Andy +1 for this well-balanced answer. " CreationDate="2011-03-22T13:18:03.550" UserId="930" />
  <row Id="14467" PostId="8528" Text="@John: you mean increasing the number from 1000 to, say 10K? I was thinking about that as well, I wasn't sure how well it would pay off though, considering the computation time. Any ideas on that?" CreationDate="2011-03-22T13:41:20.253" UserId="3014" />
  <row Id="14468" PostId="8550" Text="Thanks again for the explanation. I am not sure if this is useful however, when I try to estimate $P(x &gt; x_{obs})$ where $x \in \hat{F}_{1000}(x)$. If you mean that I can go from the estimate $\hat{\mu}_{N}$ to a probability calculation, then I certainly interested.." CreationDate="2011-03-22T13:49:09.047" UserId="3014" />
  <row Id="14469" PostId="8610" Text="@ Andy. Indeed.  That is an important point and my answer referred to propensity score adjustment.  Stratifying and matching will produce different results that the ANCOVA model and can have some undesirable properties (which you hint at in your reply below)." CreationDate="2011-03-22T14:02:07.150" UserId="485" />
  <row Id="14470" PostId="8610" Text="@Andy.  I just noticed that last sentence.  Adjusting for a propensity score and ANCOVA models produce equivalent estimates of effects.  I've read results of simulations that show this clearly.  I agree, matching and stratifying are different animals, but propensity score adjustment and the equivalent ANCOVA will get you the same effect estimate.  You get more power via the reduction of covariates, but that's it.  " CreationDate="2011-03-22T14:05:32.667" UserId="485" />
  <row Id="14471" PostId="8623" Text="@IrishStat: Great explanation and a memorable quotation. You kept up your seniority and expertise. Can you kindly expand your statement &quot;knowledge waiting to be discovered&quot; with reference to my earlier question http://stats.stackexchange.com/questions/8358/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-cas" CreationDate="2011-03-22T14:08:10.447" UserId="2956" />
  <row Id="14472" PostId="8628" Text="@schenectady: It is the human tendency. Robert Millikan was no exception. I am very happy that so many new things have been enlightened and the philosophy behind a statistical model is emphasized." CreationDate="2011-03-22T14:13:44.200" UserId="2956" />
  <row Id="14474" PostId="8623" Score="1" Text="@DrWHO: The identification of the LEVEL SHIFT at 2014 which remedied a very bad looking residual plot is an example of &quot;knowledge waiting to be discovered&quot; as it unveiled the apparent delay between a policy change date  and it's full implementation/realization date.The statement that a permanent level(step) shift was fully realized in 2004 (year 11 of 17) reflects the de facto date where as the de jure date was a few years before." CreationDate="2011-03-22T14:30:55.710" UserId="3382" />
  <row Id="14475" PostId="8610" Text="@Brett, I don't know the correct lingo, but if when you say adjustment you mean including the propensity score on the RHS of a regression equation then yes I agree that it is equivalent to the ANCOVA solution. I'm not quite sure what undesirable properties I hinted at, although they definitely exist (that may be getting off topic though for this question)." CreationDate="2011-03-22T14:31:45.390" UserId="1036" />
  <row Id="14476" PostId="8631" Score="2" Text="Chapter 3 would be of interest to understanding rates, http://goo.gl/H4lcD , (Rothman, 2002)" CreationDate="2011-03-22T14:36:00.113" UserId="1036" />
  <row Id="14478" PostId="8631" Score="1" Text="[Cancer Prevalence References](http://surveillance.cancer.gov/prevalence/references.html) from the US National Cancer Institute" CreationDate="2011-03-22T14:51:37.027" UserId="449" />
  <row Id="14479" PostId="7689" Text="@wfoolhill , no need to apologize! Welcome to the site and I hope you continue to contribute." CreationDate="2011-03-22T14:55:19.060" UserId="1036" />
  <row Id="14480" PostId="8606" Text="@probabilityislogic Thanks! I added some more detail to the question to help clear things up. In this case, the upper and lowerbounds are discarded from the routine. " CreationDate="2011-03-22T15:00:30.053" UserId="3572" />
  <row Id="14482" PostId="8528" Text="I don't know of an easy way to quantify the difference. It might be instructive to look at a bunch of histograms or Kernel Densities (with your eyes) with different numbers of points. " CreationDate="2011-03-22T15:06:36.290" UserId="1146" />
  <row Id="14483" PostId="8611" Text="@chl, these are the arrival times of the packets measured in our network. the two dataset is not big, but the sizes are different as mentioned. i just want to show the differences of these time measurements in the same plot. Thanks. " CreationDate="2011-03-22T15:36:39.413" UserId="3270" />
  <row Id="14484" PostId="8621" Text="@Peter Flom, i dont have any y axis, only the time measurements belong to networks which varies..." CreationDate="2011-03-22T15:39:17.010" UserId="3270" />
  <row Id="14485" PostId="8621" Score="2" Text="@berkay Ah so stripchart may make sense in this case! You can even superimpose individual (jittered) points onto a boxplot for each series of measurement. Either way, your y-axis = your measurement units, the x-axis = your two groups or a dummy variable coding them." CreationDate="2011-03-22T15:52:06.873" UserId="930" />
  <row Id="14486" PostId="8630" Score="2" Text="Strictly, PC2 is the linear combination *orthogonal to PC1* which describes the next most variance in the data." CreationDate="2011-03-22T15:54:27.070" UserId="2958" />
  <row Id="14487" PostId="8621" Text="@chl Yes, jittering works well: good suggestion." CreationDate="2011-03-22T16:36:55.423" UserId="919" />
  <row Id="14488" PostId="8638" Score="2" Text="For reference you might want to add the line y=x, not a linear fit.  This is more relevant to the need to demonstrate the two datasets differ." CreationDate="2011-03-22T17:30:42.393" UserId="919" />
  <row Id="14489" PostId="8637" Text="@whuber (+1) I like the first, and especially the last one (pretty much the same as the *Elements of Statistical Learning*, by Hastie and coll., but with other examples and a discussion of biplots and correspondence analysis)." CreationDate="2011-03-22T17:31:06.097" UserId="930" />
  <row Id="14490" PostId="8625" Score="1" Text="The answer is that you are summing the deviations of values around their mean and that sum is zero.  But what, precisely, are $x$, $m$, and $m_i$?  How are $m$ and $m_i$ related to $\mu$ and $\mu_i$?  The quality of answers will depend on how accurately we guess but you're forcing us to do an awful lot of guessing!" CreationDate="2011-03-22T17:35:55.287" UserId="919" />
  <row Id="14491" PostId="8638" Text="@whuber: good catch; my mistake;" CreationDate="2011-03-22T17:41:38.520" UserId="795" />
  <row Id="14492" PostId="8637" Text="+1 for mentioning Rencher's book. It's fantastic, and also has a chapter for matrix algebra and required background maths." CreationDate="2011-03-22T17:43:32.017" UserId="1356" />
  <row Id="14494" PostId="8642" Score="3" Text="'Which method minimizes type II error for a given type I error?' might be a more interesting question." CreationDate="2011-03-22T19:02:34.917" UserId="449" />
  <row Id="14495" PostId="8635" Text="When you say 'their peer behaviour' is that the child's peers or the parents' peers?" CreationDate="2011-03-22T19:04:43.933" UserId="449" />
  <row Id="14496" PostId="8629" Score="2" Text="@ schenectady while i sympathesise with your viewpoint, for me at least, the R code provides a bridge that i can use to greater my understanding of the equations and maths concerned. That being said, i wholeheartedly agree with the need for problems, statistics and maths in general is something that can only be learned by doing. " CreationDate="2011-03-22T19:13:21.127" UserId="656" />
  <row Id="14497" PostId="8639" Text="It takes only the most casual acquaintance with WinBUGS to answer this question as stated (i.e., does WinBUGS actually process data?  Yes, of course), so I'm wondering whether some other question is intended." CreationDate="2011-03-22T20:09:38.720" UserId="919" />
  <row Id="14498" PostId="7314" Text="&quot;...conditional probability tables used in this approach does not correspond to stochastic or deterministic relationships in Winbugs&quot;: How not?" CreationDate="2011-03-22T20:14:48.713" UserId="919" />
  <row Id="14499" PostId="8437" Text="	&#xA;Package recommendations: **amer** — Additive mixed models with lme4, **gamm4** — Generalized additive mixed models using mgcv and lme4, **mgcv** (gamm function, via glmmPQL in MASS package), from [here](glmm.wikidot.com/faq)" CreationDate="2011-03-22T20:28:54.970" UserId="3748" />
  <row Id="14500" PostId="8621" Text="@chi ... ooops, I missed that.  Yeah, a strip chart with a lot of data would be a mess" CreationDate="2011-03-22T20:35:52.440" UserId="686" />
  <row Id="14501" PostId="6772" Text="You also might want to think about measurement error.   e.g., if we're talking about proportion of people who agree with &quot;A,&quot; and 2% of everyone says the opposite of their true feelings, then most of the people who say &quot;A,&quot; don't really agree with &quot;A.&quot;&#xA;" CreationDate="2011-03-22T21:50:32.097" UserId="3748" />
  <row Id="14502" PostId="8623" Text="@IrishStat: Thank you for the clarification. It is very difficult to convince policy makers, doctors and public that a particular treatment can have drastic changes in disease outcome. It takes decades. This Level shift was seen in 2004 reflects the delay in accepting something new. Is it better to leave the Level shift or treat it as an outlier for the calculations of Case Fatality Rates of State 1 while dealing with the question http://stats.stackexchange.com/questions/8358/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-cas" CreationDate="2011-03-22T21:51:28.647" UserId="2956" />
  <row Id="14503" PostId="1472" Text=":Shane ; DONE !" CreationDate="2011-03-22T21:56:37.080" UserId="3382" />
  <row Id="14504" PostId="8623" Score="1" Text="my comment above should have been LEVEL SHIFT at 2004 . Sorry about the confusion." CreationDate="2011-03-22T22:03:11.143" UserId="3382" />
  <row Id="14505" PostId="2199" Text="I actually found Tabachnick rather unclear, even for subjects i knew quite a bit about. The introductory stuff on univariate stats and data cleaning was very good though" CreationDate="2011-03-22T22:09:36.597" UserId="656" />
  <row Id="14506" PostId="8645" Text="@Irish This form of response steps over the line between being helpful and using this site for overt marketing.  Please don't take my deletion to reflect any opinion about you or your software.  Your expertise and experience are apparent and quite welcome on this site, but please share them in ways that clearly show your objectivity." CreationDate="2011-03-22T22:53:57.353" UserId="919" />
  <row Id="14508" PostId="8597" Text="what do you mean by forecast?  can you please elaborate via an example, thanks." CreationDate="2011-03-23T00:12:32.123" UserId="2664" />
  <row Id="14509" PostId="8640" Text="The ratio of the variances is the *square* of the Pearson correlation between $X$ and $Y$. I'm not sure what you mean by the &quot;full-rank&quot; case either." CreationDate="2011-03-23T00:24:55.283" UserId="2970" />
  <row Id="14510" PostId="8650" Text="@mbq, what about standard error in this case? as plotted error bars." CreationDate="2011-03-23T01:14:58.033" UserId="3270" />
  <row Id="14512" PostId="8550" Text="I guess I'm a little confused by what you're seeking so I'll do my best to use the example you've posed. For a fixed value, let's say $y$, we wish to estimate $F(y)$. Then we take the $N=1000$ samples and get some value $\hat{q}$. Now take $M$ replicants of the original sample and you repeat the same computation yielding $M$ values of $\hat{q}^*$, giving a CI on $\hat{q}$. The key point is that the original sample can get you the statistic and the bootstrap gives the CI. Perhaps the real question is what you want to do with the empirical distribution." CreationDate="2011-03-23T01:33:07.130" UserId="3786" />
  <row Id="14513" PostId="8648" Text="Thanks for your answer. So, my problem lies in (how I calculated) the midpoint distribution. Please excuse my ignorance if I am asking an obvious question, but how the midpoint distribution is different to the [sum of normally distributed random variables](http://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables)? Is it because we are in a bi-variate case?" CreationDate="2011-03-23T02:05:54.243" UserId="3843" />
  <row Id="14514" PostId="8651" Score="1" Text="There are many ways to create a bivariate distribution in which both marginals are Weibull.  Thus the &quot;spearman correlation between the two&quot; distributions is not well-defined.  It would help to quote a source for the statement in your second paragraph." CreationDate="2011-03-23T02:56:55.857" UserId="919" />
  <row Id="14516" PostId="8648" Score="1" Text="@jorges The sum will be normal and therefore is symmetric about a single mode.  When the two means are sufficiently far apart (compared to their SDs), the mixture distribution is *bimodal*.  In no circumstance is the mixture normal except in the degenerate cases (equal means and equal SDs).  This is also apparent from the formula for $\varphi_m$: this average of exponentials cannot be written as the exponential of a quadratic form." CreationDate="2011-03-23T03:09:24.660" UserId="919" />
  <row Id="14517" PostId="8375" Text="@B_Miner: I apologize for still not quite grasping your concern. You should be able to estimate your model well enough. Though, because you will likely have measurement error in your TVC and dependent variable, you should probably use a lag of the TVC in the model. Otherwise, you risk using a covariate value which is measured after the occurrence of the event as a predictor for that event. You can see the time-consistency issue. While I know your goal is prediction, if you want reasonable confidence intervals, you will want to estimate robust standard errors (at least)." CreationDate="2011-03-23T04:04:08.973" UserId="3265" />
  <row Id="14518" PostId="8503" Score="1" Text="@dmcnelis, actually this IS **a crucial assumption**, that all you have at hand are two (probably coming from poorly specified model) got by $OLS$ estimates of the parameters. I have had several suggestions what to do (testing changes in parameters, if a structural break occurs, weighting on the basis of relative usefulness (forecasts accuracy) of past data to forecast next years values, etc.), but under such an assumption any suggestions a la $(k\cdot B + A)/(k+1)$ are just fine here, where $k$ is your subjective opinion. " CreationDate="2011-03-23T07:06:13.413" UserId="2645" />
  <row Id="14519" PostId="8359" Score="1" Text="@ran2, stationary (weak stationarity) means $EX_t=const$ and $cov(X_t,X_{t+h})=r(h)$, for all $t$ and $h$. If these properties are violated then the process is not stationary. However if the process is $Y_t=f(t)+X_t$, where $X_t$ is stationary and $f$ is some function, it is called trend-stationary. The analysis of trend-stationary and stationary processes is similar, so that is why sometimes trend-stationary processes are called stationary, although strictly this is not the case." CreationDate="2011-03-23T07:15:04.280" UserId="2116" />
  <row Id="14520" PostId="8617" Text="@mbq: Thank you for editing it to make it better." CreationDate="2011-03-23T07:24:15.160" UserId="2956" />
  <row Id="14521" PostId="8065" Text="@chl: The first paragraph shows the Incidence Rates of 2 diseases (cases per million per year) being compared&#xA;The second paragraph gives the actual number of cases of 2 diseases being compared" CreationDate="2011-03-23T07:32:43.703" UserId="2956" />
  <row Id="14522" PostId="8303" Text="@ Everyone: Thank you very much. In the end I used many different thing in the hope they would all give similar answers. And they did. I used the BMA, bestglm and glmnet packages as well as the step function. Fitted models with all of them, and there was not discrepancy in what BMA with maxcol = 9 and step deemed the best model. All the experts in the field around me seemed very content with the variables, and felt that it was quite progressive. So thanks for all the input. I really used it all. " CreationDate="2011-03-23T08:22:11.670" UserId="3721" />
  <row Id="14523" PostId="8359" Text="many thx for the elaborate help with the terminology. I guess I got it now :)" CreationDate="2011-03-23T08:39:17.050" UserId="704" />
  <row Id="14524" PostId="8625" Text="@whuber: You are totally right, I revised my question." CreationDate="2011-03-23T08:46:51.753" UserId="760" />
  <row Id="14525" PostId="8623" Score="1" Text="@DrWHO: In answer to your question &quot;Is it better to leave the Level shift or treat it as an outlier for the calculations of Case Fatality Rates of State 1 while dealing with the question&quot;. If you don't treat it then one can simply say STATE1 had a Level Shift Change at 2004 while STATE2 did not thus they are different but one can not place a probability on that statement. After treating STATE1 for the Level Shift one has normalized the data for a status change at 2004. The normalized data ( cleansed data ) can then be compared with STATE2's normalized data without loss of generality." CreationDate="2011-03-23T09:08:03.083" UserId="3382" />
  <row Id="14526" PostId="8657" Text="@dole Can you work on improving your question? As it stands, it is hard to tell what is being asked, unless you are just looking for some kind of census data." CreationDate="2011-03-23T09:44:13.523" UserId="930" />
  <row Id="14527" PostId="8639" Text="whuber, you are right, I could not express myself clear enough. I've edited the question. " CreationDate="2011-03-23T10:13:15.903" UserId="3280" />
  <row Id="14528" PostId="8652" Text="@david: regarding the article you mentioned, it's an interesting observation, however it doesn't come as a surprise to me. I find a significant portion of statistical concepts and common practices as confusing and convoluted (even though I have a strong background in mathematics, and have taken a number of courses in mathematical statistics). I personally feel a lot of the concepts would have been much easier to understand if they were taught visually and using examples, instead of long and convoluted verbal explanations. " CreationDate="2011-03-23T10:23:45.070" UserId="3014" />
  <row Id="14529" PostId="8662" Text="not that it matters dramatically but when using histograms it usually helps to have the relative frequency instead of absolute frequency on the y-axis. " CreationDate="2011-03-23T10:27:46.970" UserId="3014" />
  <row Id="14530" PostId="8662" Text="that is, to provide 120000/15000000=0.008 instead 120000 on vertical axis?" CreationDate="2011-03-23T10:53:12.343" UserId="2820" />
  <row Id="14531" PostId="8660" Text="I think, it is also not applicable here, since the author assumes, that he may not observe $X$ at all, just the parameters." CreationDate="2011-03-23T11:03:15.567" UserId="2645" />
  <row Id="14533" PostId="8668" Text="@schenectady: Good idea suggesting the qqplot. However, I think that your explanation of the technique is a little vague/hard to understand. Can you provide some exemplary R-code ? This would increase the value of the answer drastically." CreationDate="2011-03-23T12:07:24.930" UserId="264" />
  <row Id="14534" PostId="8668" Text="I expect that somebody encountered the picture like mine and investigated the underlying distribution, because the values have physical basis." CreationDate="2011-03-23T12:14:40.020" UserId="2820" />
  <row Id="14535" PostId="8668" Text="I am investigating the physical background of the sample distribution - how it is distributed and why." CreationDate="2011-03-23T12:18:49.317" UserId="2820" />
  <row Id="14536" PostId="8660" Text="@dmitrij - so someone is going to just give the estimates of parameters without any errors associated to them?  Seems an odd thing to do, for it does not require much extra &quot;space&quot; and it greatly helps the prediction.  Instead of sending 2p numbers, you send 2p(p+1) numbers.  You only need 2p(p+1) numbers to fit this model." CreationDate="2011-03-23T12:31:22.273" UserId="2392" />
  <row Id="14537" PostId="8663" Text="@sabre - you question would be much easier to understand if you included the model equations that you are using." CreationDate="2011-03-23T12:51:52.100" UserId="2392" />
  <row Id="14538" PostId="8657" Text="@dole doug: What country? Which elections? What have you found in your searching to date?" CreationDate="2011-03-23T13:26:13.317" UserId="3794" />
  <row Id="14539" PostId="8666" Text="@caracal: Thanks -- I'll need to look through your answer carefully. In STATA one can just run `logit` and `logistic` and get odds ratios and confidence intervals easily. I am somewhat frustrated that this appears to be so complicated and non-standard in `R`. Can I just use `exp(cbind(coef(x), confint(x)))` from fabians' answer below to get the OD and CI? I'm not clear what your answer is providing?" CreationDate="2011-03-23T13:28:21.447" UserId="2824" />
  <row Id="14540" PostId="8663" Text="@probabilityislogic: I've added examples to my question." CreationDate="2011-03-23T13:40:20.083" UserId="2824" />
  <row Id="14541" PostId="8667" Score="1" Text="+1 for @fabian's suggestion. An inferior way to do this that usually yields similar intervals is to compute the interval on the logit scale and then transform to the odds scale: &#xA;`cbind(&#xA;	exp(coef(x)),&#xA;	exp(summary(x)$coefficients[,1] - 1.96*summary(x)$coefficients[,2]),&#xA;	exp(summary(x)$coefficients[,1] + 1.96*summary(x)$coefficients[,2])&#xA;)`. There's also the delta method: http://www.ats.ucla.edu/stat/r/faq/deltamethod.htm" CreationDate="2011-03-23T15:19:45.803" UserId="3432" />
  <row Id="14542" PostId="8662" Text="@mbaitoff: Your comments to the answer of schenectady indicate, that you are less interested in getting the name of the distribution but in finding out WHY the values are distributed this way. Is this correct ?" CreationDate="2011-03-23T15:34:31.170" UserId="264" />
  <row Id="14543" PostId="8671" Text="(+1) The second term, being the transpose of the first, must also be zero :-)." CreationDate="2011-03-23T15:57:55.313" UserId="919" />
  <row Id="14544" PostId="8596" Text="No worries, I can easily imagine that this shortcoming travels well across fields of inquiry." CreationDate="2011-03-23T16:26:51.130" UserId="3582" />
  <row Id="14545" PostId="8669" Score="1" Text="Why not regress 0 on $1, X_1, \ldots, X_{11}, Y_1, \ldots, Y_4$?  The difference $a_0 - b_0$ will not be identifiable, but in retrospect that's obvious.  And of course you cannot obtain an $R^2$ for either of your models, because you have absolutely no information about the variation of $Z$; however, the $R^2$ for the regression involving all $X$ and $Y$ together is informative: it tells you how closely the two models can typically be made to agree (although the MSE is much more useful in that regard)." CreationDate="2011-03-23T16:42:26.517" UserId="919" />
  <row Id="14546" PostId="8666" Text="@SabreWolfy I wasn't sure what OR you are referring to: originally, I thought you meant the OR from the classification table that compares actual category membership with predicted membership (the `cTab` part in my answer). But now I see you probably just mean the $exp(b_{j})$: like fabians explained, $exp(b_{j})$ equals the factor by which the predicted odds change when $X_{j}$ increases by 1 unit. This is the ratio of the odds &quot;after the 1-unit-increase&quot; to &quot;before the 1-unit-increase&quot;. So yes, you can just use fabians answer." CreationDate="2011-03-23T17:04:02.193" UserId="1909" />
  <row Id="14547" PostId="8662" Text="@mbaitoff - one problem with the way you have presented this question is that we have to guess at what the values are.  If you could supply some actual data points that would be helpful" CreationDate="2011-03-23T17:12:42.077" UserId="3489" />
  <row Id="14548" PostId="8671" Text="@whuber, yes, that too :)" CreationDate="2011-03-23T17:16:21.607" UserId="2116" />
  <row Id="14549" PostId="8375" Text="@JVM apparently I have failed to explain myself well. I added a response in the question." CreationDate="2011-03-23T17:38:50.820" UserId="2040" />
  <row Id="14550" PostId="8648" Text="@whuber and cardinal: While intuitively I understand what you say, I seem to be having a serious problem with concepts. I am not really a statistician, and I think I am mixing &quot;sum of random variables&quot; with &quot;mixture distribution&quot;. I am not really contesting what cardinal and you responded, but rather trying to understand the difference and when to use which, as I might be making similar mistakes elsewhere. I'll go back to do some reading and see if I can clear myself up. Thanks for your answers." CreationDate="2011-03-23T17:42:08.620" UserId="3843" />
  <row Id="14551" PostId="6923" Score="1" Text="@cardinal Gene Golub told me that the SMW identity was so numerically unstable that it shouldn't be used in practice - I wasn't going to argue with him! ;o)  I had a look at the Given's rotation method (Gentleman's paper); it looks interesting, must get round to implementing it at some point!" CreationDate="2011-03-23T18:15:51.167" UserId="887" />
  <row Id="14553" PostId="8687" Score="1" Text="+1 Nice applications.  They remind me strongly of [influence diagrams](http://sspp.proquest.com/archives/vol3iss1/0610-036.grosskurth-print.html) (and maybe they are meant to)." CreationDate="2011-03-23T18:37:39.160" UserId="919" />
  <row Id="14554" PostId="8687" Text="@whuber (+1) It seems so, although I'm not so familiar with them." CreationDate="2011-03-23T18:47:02.310" UserId="930" />
  <row Id="14555" PostId="8662" Text="@steffen: Actually, both questions are of much importance. I expect that to derive the physical background from guessed distribution is easier than to derive the distribution from the physical background, since the measured process is specific (an amplitude measurements of inducted seismic waves)." CreationDate="2011-03-23T18:49:27.143" UserId="2820" />
  <row Id="14556" PostId="8682" Score="1" Text="@Michael I like your first paragraph and about the last one, there's a nice summary about [Problems Caused by Categorizing Continuous Variables](http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous) on Frank Harrell's website. Should you have some more time to invest in your response, you could expand on the &quot;why the difference&quot; (`summary.formula` offers either standard $\chi^2$ test or trend test in case the variable on the RHS is ordinal; the $p$-value associated to Age treated as continuous in the GLM is derived from a Wald test)." CreationDate="2011-03-23T19:30:38.030" UserId="930" />
  <row Id="14557" PostId="8662" Score="1" Text="@mbaitoff, I'm not sure it would quite fit your application, but in related application areas, magnitudes of waves that undergo (many) random reflections between source and receiver are modeled by a Rayleigh distribution or one of its generalizations, e.g., Rice or Nakagami-$m$ distributions. " CreationDate="2011-03-23T20:01:00.453" UserId="2970" />
  <row Id="14558" PostId="8689" Text="@suprvisr I tried to reword a little bit your question. I'm afraid it looks like you're asking two very different questions. For the second one, a lot of resources is available on this site, but also on [CRAN](http://cran.r-project.org/)." CreationDate="2011-03-23T20:11:42.393" UserId="930" />
  <row Id="14559" PostId="8674" Text="(+1): Didnt know that package before." CreationDate="2011-03-23T21:53:11.160" UserId="264" />
  <row Id="14560" PostId="6342" Text="You'll have a much better idea what you're doing if you plot your data, e.g. 'scatterplot(A, age)' and 'plotMeans(O, binnedAge, error.bars = &quot;se&quot;)'  You'll need 'library(Rcmdr)' for 'plotMeans()'." CreationDate="2011-03-23T22:11:40.327" UserId="3748" />
  <row Id="14561" PostId="8418" Text="Hadley, do you mean speed? It's biologically relevant to categorize the continuous variable into discrete bins. Between certain speeds there are locomotory gait transitions, so I am investigating each speed bin as to compare, say, walks with walks, rather than walks with runs." CreationDate="2011-03-23T22:12:12.260" UserId="3761" />
  <row Id="14562" PostId="8702" Text="Do your boxplots take into account the fact that your repeated measures are within individuals?  If so, they are probably pretty useless for letting you guess about the outcome of the mixed model." CreationDate="2011-03-23T23:12:29.660" UserId="601" />
  <row Id="14563" PostId="8367" Text="@chl: idea #2: the 'caret' package has a function called 'rfe' that does recursive feature selection (aka backwards elimination).  Would it make sense to run this function, using logistic regression as your model, and stop when you get to 8 variables?" CreationDate="2011-03-23T23:34:28.207" UserId="2817" />
  <row Id="14566" PostId="8708" Text="Please do not simultaneously crosspost. See here: http://math.stackexchange.com/questions/28752/is-there-a-program-or-app-that-will-let-you-sort-items-using-manual-choices-and-d" CreationDate="2011-03-24T02:19:58.977" UserId="2970" />
  <row Id="14570" PostId="8623" Text="@IrushStat: Thank you for the additional information." CreationDate="2011-03-24T03:23:10.037" UserId="2956" />
  <row Id="14571" PostId="6923" Text="@Dikran, perhaps my comment that a full update would be necessary on occasion was a little too indirect. Yes, I generally agree with this statement. Of course, I think you *should* have argued with Gene, if for no other reason than I think he would have enjoyed it. He was rarely shy about his opinions. RIP." CreationDate="2011-03-24T03:26:57.920" UserId="2970" />
  <row Id="14572" PostId="8662" Text="@Ralph Winters: I have edited the post to provide a link to file with raw data values." CreationDate="2011-03-24T04:55:12.057" UserId="2820" />
  <row Id="14573" PostId="8697" Text="Thanks for the illumination, that's quite a detailed answer!" CreationDate="2011-03-24T06:20:06.900" UserId="1537" />
  <row Id="14575" PostId="8669" Text="There is one problem with the method you have suggested tho, i.e. all my co-eff &amp; intercept comes out as zero..." CreationDate="2011-03-24T06:57:23.303" UserId="3859" />
  <row Id="14576" PostId="8708" Text="My first attempt was to opt for close, but now I think that the point of view of this side can lead to different solutions. Maybe on of the cases where crossposting is allowed ? E.g., this seems to be related with active learning." CreationDate="2011-03-24T07:26:59.030" UserId="264" />
  <row Id="14577" PostId="8712" Score="1" Text="(-1): this is not an answer and it is not even funny. The OP provided an example for the general task of automatically inferring an order from a set of examples and did not ask for solving the example problem only." CreationDate="2011-03-24T07:29:24.987" UserId="264" />
  <row Id="14578" PostId="8713" Score="1" Text="Downvote, since this can be seen as linear regression with non-linear basis functions. You can still solve it in closed form." CreationDate="2011-03-24T08:17:25.133" UserId="2860" />
  <row Id="14579" PostId="8693" Text="How does renaming something make it linear? I don't understand, if the identity newv = b * c holds, it's not linear at all. I am confused." CreationDate="2011-03-24T08:21:17.657" UserId="2860" />
  <row Id="14581" PostId="8704" Score="1" Text="It seems to be a homework exercise so I have added a tag. The exercise is simple though, first find the pdf of $I^2$ then the pdf of the product (try to do it yourself, what are the difficulties solving the exercise?). By the way solution could be found in H.Hsu. nice book for self-study http://www.amazon.com/Schaums-Outline-Probability-Variables-Processes/dp/0070306443 " CreationDate="2011-03-24T08:41:21.667" UserId="2645" />
  <row Id="14582" PostId="8714" Text="Honestly, I suggest using a dedicated statistics software, because it is too easy to make mistakes and not be able to recognize that there is a mistake. With a line-based statical software like R, at least you get warned that something is not working, and get a hint, were the mistake might be. Excel will just as likely as not produce some result anyway. " CreationDate="2011-03-24T09:57:19.977" UserId="1766" />
  <row Id="14583" PostId="8721" Text="@robin, it would be interesting to know what in particular causes difficulties to the author, since this one is really a standard pdf computation example I did with my students during random processes practice." CreationDate="2011-03-24T11:23:51.230" UserId="2645" />
  <row Id="14584" PostId="8721" Text="@Dmitrij yes but I gues I am not the one to be asked this. I gave the general idea and try to point out to @H_S that this is connected to computing the sum of random variable and that it can be generalized to other group. I gave the formulas and link to the details. I guess 90% of the good question of this site could be asked to student and it should not stop us to try to answer and try to give a point of view that can help them to reach a better understanding of the problem they are facing. I think this Q is way way more interesting than &quot;what is standard deviation&quot;. " CreationDate="2011-03-24T11:50:51.720" UserId="223" />
  <row Id="14585" PostId="8704" Text="@Dmitrij @H_S I found the question makes sence. Many other people that are not student of Dmitrij may encounter that problem. It is certainly not an advanced research question but I found this is a very good question for this site. " CreationDate="2011-03-24T11:58:46.393" UserId="223" />
  <row Id="14586" PostId="8721" Text="@robin, I agree with you and will give (+1) for this general idea and links, I hope the author will solve by himself starting from denoting $Z = I^2$, then cdf of $F_Z(z)=P\{Z\leq z\} = P\{-\sqrt{z} \leq I \leq \sqrt{z}\} = ...$ for $z &gt; 0$ and, and $0$ for $z \leq 0$. For the product of 2 independent RV  you gave the crucial hint (but, @robin, it is $|t|$ in denominator). So @H_S, I hope to see yours complete solution here for +1 from me too." CreationDate="2011-03-24T12:13:19.287" UserId="2645" />
  <row Id="14587" PostId="8722" Score="3" Text="@Beatrice, its a standard exercise on pairwise regression, well the first part is in any introductory text book, the second part comes from the definition of $Corr(X,Y) = Cov(X,Y)/\sqrt{Var(X)Var(Y)}$. " CreationDate="2011-03-24T12:23:56.020" UserId="2645" />
  <row Id="14588" PostId="8710" Text="I would suggest searching for references in 'functional data analysis' and 'measurement error.'" CreationDate="2011-03-24T12:28:57.700" UserId="3567" />
  <row Id="14589" PostId="8717" Text="Crossposted at math.SE as per comment suggesting that over there: http://math.stackexchange.com/questions/28779/what-will-be-minimum-variance-unbiased-estimator-for-this" CreationDate="2011-03-24T12:29:20.293" UserId="2970" />
  <row Id="14590" PostId="8721" Text="@Dmitrij thanks for the complement. I use the notation $\mathbb{R}_+$ for positive real otherwise you are right that there should be absolute values" CreationDate="2011-03-24T12:31:24.400" UserId="223" />
  <row Id="14591" PostId="8521" Score="1" Text="You might want to make your question more specific." CreationDate="2011-03-24T12:31:45.577" UserId="3567" />
  <row Id="14592" PostId="8699" Score="1" Text="This is theoretically incorrect simply because none of the X &amp; Y variables are dependent variables. Hence, I doubt if doing a regression would be the right thing to do." CreationDate="2011-03-24T12:55:41.403" UserId="3859" />
  <row Id="14593" PostId="8727" Text="Nice answer! In the last hint: could it be necessary to transform categorical attributes in numerical ones?  I think in missing data patterns analysis the db is a 0/1 db (0 non missing, 1 missing)." CreationDate="2011-03-24T13:10:39.723" UserId="2719" />
  <row Id="14595" PostId="8727" Text="@Simone Yes, we can work with a binary indicator for missingness or low variance (with a constant cut-off), but I guess you could also use a numerical summary of &quot;variable sparsity&quot; as defined above (which would work for any kind of variables)" CreationDate="2011-03-24T13:15:47.730" UserId="930" />
  <row Id="14596" PostId="4681" Text="@B_Miner 1) The other way around - to get the proportion right, you need to set the intercept correctly - so adjust the intercept until you get the proportion which you are expecting. 2)  The logic of ytest is that we need to get a dichotomous 0 or 1 outcome.  So so we compare each sample from the uniform distribution to the probability (prob) to get our dichotomous outcome.  The 'runis' doesn't have to be drawn from the random uniform distribution - a binomial or other distribution may make more sense for your data.  Hope this helps (sorry for the delay in reply)." CreationDate="2011-03-24T13:34:42.930" UserId="1991" />
  <row Id="14597" PostId="8722" Score="3" Text="@Beatrice Have you made an attempt? Is there something you don't understand? Not to be rude, but we aren't here to complete you homework for you." CreationDate="2011-03-24T13:38:06.277" UserId="26" />
  <row Id="14598" PostId="8701" Score="1" Text="@Thomas McCullagh and Nelder is pretty good, although somewhat mathematically dense. I figured out how to use GLM's from the first 100 pages or so. Also, Gelman and Hill discuss GLM's in their Applied Regression Models and this book is pretty good for taking you from the very basics right through to Multilevel models using lme4 and Bugs. " CreationDate="2011-03-24T13:40:44.380" UserId="656" />
  <row Id="14599" PostId="8708" Score="2" Text="The OP might be best served by asking on the theoretical computer science stackexchange.  This question in particular concerns constructing a total order given a directed acyclic graph, and is well-studied in the computer science literature." CreationDate="2011-03-24T13:56:18.903" UserId="3567" />
  <row Id="14600" PostId="8722" Text="@Beatrice Please read the info for the &quot;Homework&quot; tag and restate your question accordingly.  After editing it, you can re-open it." CreationDate="2011-03-24T14:02:55.833" UserId="919" />
  <row Id="14601" PostId="8717" Text="@amul28 You probably want to limit your search to *linear* estimators; otherwise, this question might be difficult to answer." CreationDate="2011-03-24T14:05:28.407" UserId="919" />
  <row Id="14602" PostId="8710" Text="@AGage GPS units *require* time information to work, accurate to the nanosecond, and are capable of reporting it.  Could you not just collect the timestamps from the GPS then?" CreationDate="2011-03-24T14:07:39.950" UserId="919" />
  <row Id="14603" PostId="8669" Text="@Santosh That's the point, isn't it?  *Your problem is not solvable because you have no observations of $Z$!*  It's possible you could make progress by constraining the coefficients to be nonzero, perhaps through a Bayesian hierarchical model.  That at least would indicate whether some linear combinations of the $X$ tend to agree with some linear combination of the $Y$.  But that's as far as you can get." CreationDate="2011-03-24T14:13:51.083" UserId="919" />
  <row Id="14604" PostId="8727" Text="Sorry, but maybe I didn't get it. I should transpose my db and then add 1 for each missing value and 0 otherwise. It works for a missing values analysis. In my case, I should compute the mean for each attribute and then for each value put 0 or 1 due to a numerical summary (eg if it is the most frequent value I choose 1 else 0?). Thanks." CreationDate="2011-03-24T14:18:57.090" UserId="2719" />
  <row Id="14605" PostId="8715" Text="+1 Good thorough answer: it provides options and useful comments." CreationDate="2011-03-24T14:22:12.527" UserId="919" />
  <row Id="14606" PostId="8721" Text="@robin, +1 for idea of using groups. It stil seems a bit complicated, since it requires some comfortability in algebraic methods, when Dmitrij's idea really is more elementary. On the other hand this gives a deeper understanding of the topic. I also corrected some small typos and changed formatting, I hope you do not mind. " CreationDate="2011-03-24T14:27:07.177" UserId="2116" />
  <row Id="14607" PostId="8702" Text="Is this structure the correct way to account for individual repeated measures?   bwplot(specfactor ~ log10(feature), random=~1|IndividualID, subset=Frfactor==&quot;low&quot;, data=animals)  It doesn't generate an error but it also doesn't change my boxplot results." CreationDate="2011-03-24T14:44:41.890" UserId="3761" />
  <row Id="14608" PostId="8727" Text="@Simone I originally meant recode as 0/1 depending on whether your variables are below or above your fixed threshold for deciding of low-variance, or just use a numerical value (e.g., the frequency ratio cited above, or the CV you proposed) that reflects the amount of variability present in here (a numeric variable with 2 or 3 unique observed values or a categorical variable with only one modality present would be what I call a poorly discriminative variable, it is not necessarily uninformative) -- this is for examining structured patterns of sparsity, if any." CreationDate="2011-03-24T14:49:34.040" UserId="930" />
  <row Id="14609" PostId="8630" Text="Are you trying to estimate $Var(A+2B+5C)$?" CreationDate="2011-03-24T14:50:08.363" UserId="1670" />
  <row Id="14610" PostId="8729" Text="I realize that these are two slightly different questions, but I want them to be together to give some background to what I am using the results of the `scale()` function." CreationDate="2011-03-24T14:52:09.670" UserId="2635" />
  <row Id="14611" PostId="8729" Text="@celenius `center=true` just means remove the mean, and `scale=TRUE` stands for divide by SD; in other words, with both options active, you're getting standardized variables (with mean 0, unit variance, and values expressed in SD units). " CreationDate="2011-03-24T14:58:31.323" UserId="930" />
  <row Id="14612" PostId="8728" Text="@richiemorrisroe Here's another link, with Mplus: http://www.statmodel.com/discussion/messages/9/258.html?1237657036. I didn't find anything for SPSS specifically. Your suggestion of writing some SYNTAX is the best way to go, IMO." CreationDate="2011-03-24T15:01:22.460" UserId="930" />
  <row Id="14613" PostId="8729" Text="@chl Ah! Thank you. That is a much clearer explanation to me than the help file. Does it make a difference how a variable is standardized for clustering with kmeans though? I would assume not, but don't know for sure. " CreationDate="2011-03-24T15:01:49.490" UserId="2635" />
  <row Id="14614" PostId="8727" Text="let me stop bothering you: each value of an attributes gets 0 if it's below a treshold otherwise 1? I was wondering if I should replace each value for each attribute in a different way. If we moved giving the same value for an attribute it would be nice to substitute them with a frequency ratio to obtain a sort of lattice of cluster, ie attributes with similar frequency ratio. Am I right? Thanks." CreationDate="2011-03-24T15:10:25.677" UserId="2719" />
  <row Id="14615" PostId="8693" Text="@bayer: newv is a new variable. The new equation is a linear function of three variables (b, c, newv), where the coefficients provide a linear relationship.  Neither equation is a linear combination of just two variables." CreationDate="2011-03-24T15:17:35.260" UserId="2775" />
  <row Id="14616" PostId="8729" Score="2" Text="@celenius I can suggest this article, [Standardizing Variables in K-means Clustering](http://www.springerlink.com/content/l438152634256311/) (Steinley, 2004) to get an idea of the effects of different kind of transformations. The utility of scaling depends on the data you have, but usually one reason to use standardized (scaled) variables is to avoid obtaining clusters that are dominated by variables having the largest amount of variation." CreationDate="2011-03-24T15:18:09.887" UserId="930" />
  <row Id="14617" PostId="8699" Text="@Santosh: If your above equation is correct (not overspecified), then not all of those variables are independent.  One is a linear combination of the others.    " CreationDate="2011-03-24T15:27:30.170" UserId="2775" />
  <row Id="14618" PostId="8713" Text="Agree with bayer." CreationDate="2011-03-24T15:29:41.993" UserId="2775" />
  <row Id="14619" PostId="8687" Text="@chl thanks for the link to the Boorsboom paper, he's always pretty good. I should really check out the qgraph package" CreationDate="2011-03-24T15:42:13.857" UserId="656" />
  <row Id="14620" PostId="8694" Text="Really interesting suggestion whuber, and one that feels intuitively correct to me even though I'm not very skilled w/ stats (yet). I will try it (as soon as I find out how, exactly, to run the test).  And thanks for the note, Peter Flom." CreationDate="2011-03-24T15:43:48.833" UserId="3871" />
  <row Id="14621" PostId="8679" Text="@ Henry the Guardian Data Blog is a good place to go for some of these kinds of statistics http://www.guardian.co.uk/news/datablog" CreationDate="2011-03-24T15:44:50.827" UserId="656" />
  <row Id="14622" PostId="8699" Text="@Santosh: Is this a homework problem?" CreationDate="2011-03-24T16:49:10.163" UserId="2775" />
  <row Id="14623" PostId="841" Text="@drknexus it is important because the dependence makes the calibration of Fisher test difficult. " CreationDate="2011-03-24T17:33:38.163" UserId="223" />
  <row Id="14624" PostId="8714" Text="Everyone thank you for your help!" CreationDate="2011-03-24T17:34:04.710" UserId="3887" />
  <row Id="14625" PostId="8717" Text="@whuber, or rather, *functions* of linear estimators. I've posted an answer on math.SE where this question originated. " CreationDate="2011-03-24T17:44:33.110" UserId="2970" />
  <row Id="14626" PostId="8717" Text="@cardinal I don't follow.  Since $X_1, X_2, \ldots, X_n$ are individually linear estimators, the most general function of a set of linear estimators is just--any function of the data.  You seem to be making no effective distinction.  Do you mean functions of a *single* linear form in the data?" CreationDate="2011-03-24T17:46:40.397" UserId="919" />
  <row Id="14627" PostId="8729" Text="@celenius Why is collinearity a problem?  It shouldn't affect k-means at all." CreationDate="2011-03-24T17:49:39.270" UserId="919" />
  <row Id="14628" PostId="8717" Text="@whuber, just slightly sloppy wording on my part, as I suppose it could be interpreted in at least two ways. &quot;Functions of *a* linear estimator&quot; would probably have been clearer. " CreationDate="2011-03-24T17:51:21.560" UserId="2970" />
  <row Id="14629" PostId="8717" Text="@whuber, looks like your edit to your comment and my response to the original one got crossed in the ether. " CreationDate="2011-03-24T18:00:36.060" UserId="2970" />
  <row Id="14630" PostId="8663" Text="What exactly does &quot;summary.formula&quot; do?  What test?" CreationDate="2011-03-24T18:09:37.897" UserId="702" />
  <row Id="14631" PostId="8729" Text="@whuber I assumed that the more collinear variables I include, the more that it biases the clustering towards those collinear dimensions." CreationDate="2011-03-24T18:18:28.917" UserId="2635" />
  <row Id="14632" PostId="2897" Text="@Gary: unfortunately I don't know a (published or online) reference; it's part of the folklore of my field, spelled out in courses but written off as &quot;simple and well-known&quot; in papers.  The proof is easy, though.  Given the tail estimate, the moment estimate follows from integration by parts (i.e. $\mathbb{E} |X|^p = \int_0^\infty p t^{p-1} \mathbb{P}(|X| &gt; t) dt$) and Stirling's formula. Given the moment estimate, the tail estimate follows by applying Markov's inequality and optimizing over $p$." CreationDate="2011-03-24T18:33:45.033" UserId="89" />
  <row Id="14633" PostId="8729" Score="1" Text="@celenius It shouldn't introduce bias.  Take, for instance, the extreme of perfect collinearity in 2D: all observations are of the form $(a+t x, b+t y)$ for constants $a,b,x,y$.  K-means uses Euclidean distances and the distance between two such points given by $t$ and $s$ is just $|s-t|$: it's really a 1D problem and there's no special weighting given to any of the data." CreationDate="2011-03-24T19:31:46.477" UserId="919" />
  <row Id="14634" PostId="8717" Text="@Cardinal Thanks for the clarification." CreationDate="2011-03-24T19:33:18.830" UserId="919" />
  <row Id="14637" PostId="8734" Text="If you can, don't use a table :) A graphic is, IMO, almost always easier to read than a table (he obvious exception being if you don't have many numbers). Journals and their reviewers don't always agree, unfortunately...." CreationDate="2011-03-24T20:26:29.767" UserId="26" />
  <row Id="14638" PostId="8737" Text="Hi, I don't normally see folks use repeated measures in such a fashion. My question is to see if one can. All the compounds are in the same units (eg ng/g). One would not _routinely_ compare compound X with compound Y (though one occasionally does). Following the Diet X FA interaction, a posthoc test could compare compound X between diets, compound Y between diets, etc. My main impetus in asking this question is that none of the compounds are independent of each other. Thus, if compound X varies among diets, then chances are other compounds do as well. What is a better way than 30 ANOVAs?" CreationDate="2011-03-24T20:30:05.207" UserId="3886" />
  <row Id="14640" PostId="8739" Text="Yes, PCA or Factor Analysis is frequently used with this type of data. But those answer different questions, do they not? I had thought that someone would suggest MANOVA, though the idea of having 30 variables is daunting... I was hoping that repeated measures would be valid, as it is easier to interpret, allows one to compared individual compounds between treatments (post hoc), and it can (?) handle nonindependent variables. I will looking into using MANOVA" CreationDate="2011-03-24T20:39:57.523" UserId="3886" />
  <row Id="14641" PostId="8739" Text="I was suggesting using PCA as a possible data reduction strategy, then running a MANOVA on the PC scores, if the PCA is helpful.  If you could reduce the # of dependents in MANOVA, you could improve your power.  Whether or not PCA is helpful, would really depends on how much overlap there is in the Fatty Acid measurements.  There are some problems with PCA too, though.  For example a PC that explains little variance in the X's may have a strong relationship with Y and by standard conventions, you might throw it away." CreationDate="2011-03-24T20:46:24.070" UserId="485" />
  <row Id="14643" PostId="8708" Text="This topic might belong here but it is not yet possible to determine exactly what question is being asked.  Please clarify it and consider posting it on a site where it might be more relevant." CreationDate="2011-03-24T21:11:23.473" UserId="919" />
  <row Id="14644" PostId="8706" Text="+1 Specifically, in a &quot;linear model&quot; the dependent variable $y$ is a linear function of the *parameters* but not necessarily of the data." CreationDate="2011-03-24T21:14:52.440" UserId="919" />
  <row Id="14645" PostId="8693" Text="@bayer See the reply by @Charlie.  In the present example, *both* models are linear (whether or not R views them as such) because in both of them `a` is a linear function of the four coefficients." CreationDate="2011-03-24T21:16:44.623" UserId="919" />
  <row Id="14646" PostId="8713" Text="I reluctantly agree with @bayer to downvote, because this answer is wrong, but it does serve nicely to illustrate the source of the original confusion.  The second equation may be quadratic in the variable $x$, but it is still a linear model.  In another context it would be correct to say the second model is nonlinear, but not here!" CreationDate="2011-03-24T21:18:55.290" UserId="919" />
  <row Id="14647" PostId="8737" Text="@P auritus: You ask &quot;What is a better way than 30 ANOVAs?&quot;  A single MANOVA with 30 dependent variables or a single MANOVA with the first K principal components.  Either approach takes into account the interrelationships among your FAs.  In fact, just as an aside, a MANOVA on the PCA scores would give you the same result as the MANOVA with the 30 original variables if you use all 30 principal components." CreationDate="2011-03-24T21:26:23.117" UserId="485" />
  <row Id="14648" PostId="8739" Text="Thanks! As a further aside, I had thought that was one of the advantages of Partial Least Squares (compared with doing regression following PCA) – the first component automatically has the highest correlation with Y, the 2nd component the second highest correlation, etc..." CreationDate="2011-03-24T22:05:45.750" UserId="3886" />
  <row Id="14649" PostId="8739" Text="Have a look at Haldi and Ling (1998) &quot;Some cautionary notes on the use of principal components regression.&quot;  in The American Statistician.  Bottom line, relationships with variables outside of the PCA can be hidden in even the last PC." CreationDate="2011-03-24T22:12:20.487" UserId="485" />
  <row Id="14650" PostId="8737" Text="Appreciate the help. I have already been using multivariate stats on the data – discriminant function analysis. But out of curiosity I had decided to run a repeated measures. Shall I assume that it is the merely the least preferable option compared to your better alternatives, or is it an invalid option entirely?" CreationDate="2011-03-24T22:17:28.237" UserId="3886" />
  <row Id="14651" PostId="8739" Text="Yes, I remember that paper from sci.stat.math, posted by the author himself..." CreationDate="2011-03-24T22:19:35.707" UserId="3886" />
  <row Id="14652" PostId="8734" Score="3" Text="@JMS Good point, but Tables are useful for summarizing detailed characteristics of statistical units (cross-classified by a factor interest, e.g. clinical diagnosis or whatever), with variables of different types (continuous, nominal, and ordinal), and other results derived from statistical modeling *per se* (confusion matrix, regression coef. etc.) that won't fit into Figures (or not always if you think of Gelman's approach for showing reg. coef. as dotcharts). We need both; the question is when do we really need a Figure instead of a Table, IMO." CreationDate="2011-03-24T22:37:54.610" UserId="930" />
  <row Id="14653" PostId="8527" Text="@cardinal..thanks for your comments. However, please don't try to second-guess me. I was involved with Nuclear Reactor Safety Analysis 30 years ago and I understand the thinking that went into it. Sites are designed on 'individual failure probabilities'. I was 'wondering aloud' the 'mathematical' effect of increasing the total population. As I mentioned , we can debate the 'real-world applicability of all of it." CreationDate="2011-03-24T22:38:06.427" UserId="3777" />
  <row Id="14654" PostId="8531" Text="I have officially accepted Galit's (sorry I called you Gail) answer although Bayer's answer came close to the answer I was expecting to my original question." CreationDate="2011-03-24T22:41:29.327" UserId="3777" />
  <row Id="14655" PostId="8740" Score="1" Text="@whuber My very big +1 because it is really a lot of good advices. In the same vein, I like to show to students that it is really pointless to summarize data gathered from surveys (or votes) as % with a lot of decimals without considering sample size (which impacts standard error)." CreationDate="2011-03-24T22:46:31.110" UserId="930" />
  <row Id="14656" PostId="8736" Text="@AVB Why do you suggest to omit a decimal for the statistic of interest if you agree to show them in the CIs (i.e., if it's meaningless for 12, why does it make sense for 123.4)?" CreationDate="2011-03-24T22:50:38.373" UserId="930" />
  <row Id="14658" PostId="8663" Text="@AndyF In this particular case, a classical $\chi^2$ test (and not a trend test) as the factor `Age` as not been specified as an ordered one." CreationDate="2011-03-24T23:06:59.843" UserId="930" />
  <row Id="14659" PostId="8744" Score="2" Text="Is the cluster size also specified? Then, as stated, the problem seems to be unsolvable to me. Consider the following case with $n = 4, p = 1$: $X = \{-1, 0.99, 1, 1.01\}$. If you want 2 clusters, you get either different sizes or not &quot;spatially cohesive&quot;. Or do you want something like, &quot;as spatially cohesive as possible&quot; - minimizing the maximal intra-cluster distance or so? The other solution would be to allow any divisor of $n$ as cluster size - but then there's always the trivial solution of $n$ clusters of size $1$." CreationDate="2011-03-24T23:46:56.207" UserId="2898" />
  <row Id="14660" PostId="8736" Text="@chl: it doesn't make much sense, but omitting it might be misleading. If I put in 123.4, someone like you will see the extra digits and just disregard them, no harm done. If I put in 120, many readers will think this is accurate to 3 digits - bad." CreationDate="2011-03-24T23:59:49.963" UserId="2658" />
  <row Id="14661" PostId="8745" Score="1" Text="In fact I would like to know a general way to compute any query,  is this possible knowing the probabilities?, so for example how to compute queries like `P(D=true|A=true,S=true), P(D=true|X=true,S=true)`, ETC... " CreationDate="2011-03-25T00:01:48.900" UserId="3681" />
  <row Id="14662" PostId="8744" Text="Good point. Ideally I would like something that is &quot;as spatially cohesive as possible&quot; while satisfying the equal cardinality constraint. But I'd be interested to hear about any procedures that make other tradeoffs here as well, since maybe they can be adapted." CreationDate="2011-03-25T02:16:06.167" UserId="3891" />
  <row Id="14663" PostId="8744" Text="Would splitting the data by quantiles suffice? If the values are not monotonic relative to each other, I fail to see how else they could be 'spatially cohesive'." CreationDate="2011-03-25T02:24:24.673" UserId="2635" />
  <row Id="14664" PostId="8662" Text="The real interest in these data lies in the dozen or more spikes: the amount of data is large enough that those are *real*, in the sense that they are evidence of actual local modes.  There seems to be a rich set of data here with a wealth of information that would be overlooked were a simple parametric formula used to summarize their distribution." CreationDate="2011-03-25T02:40:35.563" UserId="919" />
  <row Id="14665" PostId="8744" Score="3" Text="There has been some recent research on constrained clustering.  Google http://www.google.com/search?q=constrained+k-means ." CreationDate="2011-03-25T02:42:23.800" UserId="919" />
  <row Id="14666" PostId="8730" Text="Thanks VERY much for all your input on this... given me lots to think about" CreationDate="2011-03-25T03:20:19.833" UserId="3878" />
  <row Id="14667" PostId="8710" Text="It is not coming from a GPS unit, but a simulation. There is no timestamp with the data - only a timestamp from when i request the data (or when I receive it), otherwise i would have used that ;-)" CreationDate="2011-03-25T03:22:50.690" UserId="3878" />
  <row Id="14668" PostId="8734" Text="@chi Fair. I did say almost always :). Things like big n-way tables are impossible to (completely) reproduce graphically. It depends on the forum I'd say. Tables have the benefit of being complete, sure, but does your reader actually *absorb* all that extra information? If there are too many parameters to fit in a graph, I'd contend that a table is often at least difficult to read. However, I do think complete results should be accessible (online, appendix, etc) if for nothing else but reproducibility. In that case I'd also like data &amp; code though! Wandered OT, sorry.." CreationDate="2011-03-25T03:57:49.303" UserId="26" />
  <row Id="14669" PostId="8734" Text="Also I think regression coefficients and confusion (correlation, covariance, ...) matrices are usually better suited to a graphical display, dotplots or similar for the former and heatmaps or graphs for the latter. " CreationDate="2011-03-25T04:00:10.473" UserId="26" />
  <row Id="14670" PostId="8662" Text="@whuber♦: You're right, those spikes are not random outliers, they seem to represent groups of outliers with some seriously deviant properties. Since the background of the raw data acquisition is indeed group-based, this makes sense. However, raw data represents the maximums of the registered wavelets, and anomalous spikes in wavelets can distort the distribution. I'm now calculating some more robust statistics of the wavelets." CreationDate="2011-03-25T04:25:31.587" UserId="2820" />
  <row Id="14671" PostId="8734" Text="@JMS I agree with your point, but in this case there is a figure limit, some other cases there are figure charges. Also, in this case if readers glance over the table and focus on the figures that are presented, then they won't waste time trying to figure out the point of an esoteric figure. But I fully support reproducibility, and while I am at it, I could (if I get around to it) add a visualization of the table to the code that is attached." CreationDate="2011-03-25T05:40:12.460" UserId="1381" />
  <row Id="14672" PostId="8734" Text="@chi that is a nice list. The present case falls under 'other results derived from statistical modeling' " CreationDate="2011-03-25T05:41:25.127" UserId="1381" />
  <row Id="14673" PostId="8745" Score="1" Text="If you could work out the joint probabilities then you could work out the conditional probabilities.  So for `P(D=true|A=true,S=true)`, it would help to know  `P(D=true,A=true,S=true)` and `P(D=false,A=true,S=true)` " CreationDate="2011-03-25T06:21:31.630" UserId="2958" />
  <row Id="14674" PostId="8745" Text="But the Bayes formula says something like the sum / something, is it enough to know those probabilities? Could you please post an example?" CreationDate="2011-03-25T06:37:20.560" UserId="3681" />
  <row Id="14675" PostId="8745" Text="`P(D=true|A=true,S=true) =` `P(D=true,A=true,S=true) /` `(P(D=true,A=true,S=true) +` `P(D=false,A=true,S=true))` `=` `P(A=true,S=true|D=true) * P(D=true) /` `((P(A=true,S=true|D=true) * P(D=true) +` `P(A=true,S=true|D=false) * P(D=false))`" CreationDate="2011-03-25T06:45:54.923" UserId="2958" />
  <row Id="14676" PostId="8745" Text="and for example when you do:&#xA;`(P(A=true,S=true|D=true)` and `P(A=true,S=true|D=false)` these expressions can be replaced even more with simpler expressions?" CreationDate="2011-03-25T06:50:36.167" UserId="3681" />
  <row Id="14677" PostId="8745" Text="If we take all variables not independent... is it the same procedure?, for example `P(D=true|S=true,B=true)`, and what happens in a case  like  `P(X=true|T=true,B=true)`??" CreationDate="2011-03-25T07:14:03.913" UserId="3681" />
  <row Id="14678" PostId="8744" Text="Just one not tested idea. In clustering, a so-called Silhouette statistic is used often. It shows you how well an object is clustered and what is the best other, neighbour cluster it could be enrolled in. So, you could start with K-MEANS or other classification with different cluster _n_'s. Then move objects not very well classified (accorging to the statistic) to their best neighbour clusters with lesser _n_ until you obtain equal _n_. I expect iterations: move some objects, recalculate the statistics, move some objects, etc. That will be trade-off process." CreationDate="2011-03-25T07:31:23.477" UserId="3277" />
  <row Id="14679" PostId="8367" Text="@Zach That's a good idea as you'll get a measure of predictive accuracy inside the CV loop. Two other bibliographic pointers that I'd recommend are: Frank Harrell's textbook (*[Regression Modeling Strategies](http://bit.ly/fPqH1n)*), and Ambroise's article (also cited in the `caret` package), [Selection bias in gene extraction on the basis of microarray gene-expression data](http://bit.ly/eEFCVn). My opinion is that some methods are very good for doing FS when you have a lot of predictors and no a priori hypothesis (like genes, biomarkers, voxels, etc.), that is without substantive grounds." CreationDate="2011-03-25T07:54:00.067" UserId="930" />
  <row Id="14680" PostId="8727" Text="@Simone I didn't try what I proposed :( There are specific metric for binary variables (like Jaccard), but I guess it would be better to use directly your measure of sparsity which, as you suggested, is variable-specific. I was partly thinking of filtering methods used in genomic studies (but see the [Bioconductor](http://www.bioconductor.org) project), where genes whose level of expression is somewhat unreliable are filtered during clustering, often in an iterative manner, as described in e.g., [Microarray learning with ABC](http://bit.ly/eX7uhx). But most basic filters remain univariate." CreationDate="2011-03-25T08:04:00.963" UserId="930" />
  <row Id="14681" PostId="8751" Text="Okay, thanks for the useful answer.  Perhaps I should have rephrased my question, but you answered it perfectly :)" CreationDate="2011-03-25T08:13:31.357" UserId="179" />
  <row Id="14682" PostId="8756" Text="Hi @ejs.  You would need to code &quot;sample&quot; the same way as any other categorical variable - dummy coding or effect coding or whatever." CreationDate="2011-03-25T09:52:39.860" UserId="686" />
  <row Id="14683" PostId="8756" Text="Regarding interactions .... yes, they can be difficult to interpret.  I like a graphical approach to showing what they mean. " CreationDate="2011-03-25T09:53:16.680" UserId="686" />
  <row Id="14684" PostId="8755" Text="With great difficulty.  The problem is that almost all the information you have is location indpendent. The only location information you have is the data about the estimated intercept. You know that the point $(\bar{x}, \bar{y})$ lies on the regression line, so knowing the mean of $x$ should give you the mean of $y$." CreationDate="2011-03-25T10:25:22.393" UserId="2958" />
  <row Id="14685" PostId="8759" Text="@richiemorrisroe: **PART 1** First of all, thank you for your answer. Indeed, frequentist methods correct for multiple testing. Hence, the problem of biased treatment effect estimate cannot come from there. At an interim analysis, the test is based on the current information, using the current sample size, not the overall planned sample size. So the problem does not come from there either." CreationDate="2011-03-25T10:28:13.137" UserId="3019" />
  <row Id="14686" PostId="8759" Text="@richiemorrisroe: **PART2** I agree that stopping early may mean that the treatment is &quot;more effective than one hopped&quot;. In that sense, the estimate treatment effect would be larger than expected. But, according to me, this does not make it biased... Instead, according to me, in some sense, &quot;our hope was biased&quot;." CreationDate="2011-03-25T10:30:07.987" UserId="3019" />
  <row Id="14687" PostId="8755" Text="@Henry Your point seems valid. However the question is from an exam, so I doubt that it's **not** possible to find an answer. In this case, how do I get the mean of $x$?" CreationDate="2011-03-25T11:43:58.293" UserId="1205" />
  <row Id="14688" PostId="8755" Text="Perhaps, having found what you call &quot;the mean $y$ for several $x$&quot;, you are now being asked for the confidence intervals around those estimated values of $y$.  Might that be possible?" CreationDate="2011-03-25T11:50:27.060" UserId="2958" />
  <row Id="14689" PostId="8755" Text="@Henry I don't think so. I updated my question to reflect a practical example (with numbers). The thing is, it says that the regression is used to estimate the mean." CreationDate="2011-03-25T12:11:20.480" UserId="1205" />
  <row Id="14690" PostId="8742" Score="1" Text="I think your notation may be a bit off, or your probabilities are inconsistent.  For instance, there is only one link for Asia (A), and that leads to tuberculosis (T).  So this means that $P(A)\geq P(T)$.  But you have assigned $P(A)=0.01&lt;0.0104 = P(T)$.  So this is inconsistent.  I think this is actually $P(T|A)=0.0104$.  But then the rest is unclear, are these conditional probabilities?  Is $P(D)$ conditional on $B$ or $E$? or just $B$? or just $E$?" CreationDate="2011-03-25T12:31:27.943" UserId="2392" />
  <row Id="14691" PostId="8753" Text="(+1): Did not know about conjoint analysis before (just read this article:http://en.wikipedia.org/wiki/Conjoint_analysis). Question: So basically one collects preference/purchase data for all or a well chosen subset of combinations and then performs an analyis afterwards to determine the variable importance. Is this correct ?" CreationDate="2011-03-25T12:37:05.693" UserId="264" />
  <row Id="14692" PostId="8755" Text="@slhck - is the standard deviation (0.482) of x or y?" CreationDate="2011-03-25T12:37:43.760" UserId="2392" />
  <row Id="14693" PostId="8765" Score="1" Text="This question is probably more suited to stats.stackexchange.com. You might look at [Hastie, et al.](http://www-stat.stanford.edu/~tibs/ElemStatLearn/) for some ideas and some ways of evaluating them." CreationDate="2011-03-25T02:43:29.837" UserId="2970" />
  <row Id="14694" PostId="8763" Text="@probabilityislogic: Thank you! If I understand it well, &quot;bias&quot; should not be taken in a statistical sense. I think this makes sense because Piantadosi speaks about the &quot;bias&quot; of an estimate and not of an estimator..." CreationDate="2011-03-25T13:10:20.780" UserId="3019" />
  <row Id="14695" PostId="8755" Score="1" Text="@slhck These data suffice to compute the *range* of the desired confidence interval, *but they do not determine the mean of y.* To see this, consider what would change if you were to add a constant $c$ to all the $x_i$ and simultaneously add $-0.00024 c$ to all the $y_i$: the answer is, *nothing.*  Consequently you cannot estimate the mean of $y$.  Thus, if the original question is answerable at all, understanding the vague &quot;upper and lower 95% and 99%&quot; is crucial. I would guess these would be confidence limits for the mean of $x$. (See the replies for why I'm wrong. :-)" CreationDate="2011-03-25T13:23:26.063" UserId="919" />
  <row Id="14696" PostId="8747" Text="There might be a bit of text analytics involved. Are &quot;Title&quot; and &quot;Descriptions&quot; true independent variables, or do they just supply information about image?" CreationDate="2011-03-25T13:28:05.917" UserId="3489" />
  <row Id="14697" PostId="8763" Text="@ocram - What I meant by &quot;biased&quot; is the usual statistical term $E(\mu-\hat{\mu})^{2}=var(\hat{\mu})+Bias(\hat{\mu})$ where $\mu$ is the &quot;true value&quot; and $\hat{\mu}$ is the &quot;estimator&quot;.  If the second term (the bias) depends on the sample size, then you would expect that stopping early would increase the bias, because it has decreased the sample size, relative to if the experiment continued.  But from what you say, it sounds like &quot;bias&quot; should be interpreted as &quot;error&quot; from Piantadosi's perspective." CreationDate="2011-03-25T14:01:10.663" UserId="2392" />
  <row Id="14698" PostId="8763" Score="1" Text="This argument says nothing about the bias, only the hypothesis testing aspect of the problem, which nobody questions." CreationDate="2011-03-25T14:12:23.137" UserId="279" />
  <row Id="14699" PostId="8768" Text="At first I thought this too, but when I sat down to prove it, I couldn't: I could only show that the resulting estimate actually is *unbiased.*  (New intuition: the positive bias from a conditional stop balances a *negative* bias from carrying the experiment to completion.)  So: can you present a more rigorous demonstration?" CreationDate="2011-03-25T14:26:41.410" UserId="919" />
  <row Id="14700" PostId="8767" Text="Please see my comment to Aniko's reply, because I would ask the same question of you: can you provide a more rigorous demonstration?" CreationDate="2011-03-25T14:27:41.520" UserId="919" />
  <row Id="14701" PostId="8768" Text="@whuber I'll try to write it up, but the point is that Piantodosi's statement is only about what happens when you _do_ stop early. There is not completion to balance it." CreationDate="2011-03-25T14:56:01.417" UserId="279" />
  <row Id="14702" PostId="8769" Score="5" Text="Check out the third picture: http://stats.stackexchange.com/questions/3947/help-me-understand-support-vector-machines/3954#3954" CreationDate="2011-03-25T14:57:27.437" UserId="88" />
  <row Id="14703" PostId="8768" Text="@Aniko But the definition of bias is a difference between the expected value of the estimator and the true value of the estimand.  That expectation is assessed *before the experiment begins.*  Your comment now clarifies that you are talking about bias *conditional on early termination.*" CreationDate="2011-03-25T15:00:08.090" UserId="919" />
  <row Id="14704" PostId="8763" Text="@Prob I have to agree with @Aniko: it is obvious that when the null is true, there is a positive probability of early termination, in which case the estimate of the effect will be nonzero.  Thus the expectation of the estimated effect, *conditional* on early termination, is positive, whereas the unconditional expectation is zero.  (Notice that the OP is addressing *estimation*, not hypothesis testing.)" CreationDate="2011-03-25T15:02:37.760" UserId="919" />
  <row Id="14705" PostId="8768" Text="@whuber Yes, that's what the original statement claims as well. Your point that there will be an opposite bias conditional on completing the study is also valid. The whole message should be that once you start doing interim monitoring, funny things start happening to your ability to estimate the effect size." CreationDate="2011-03-25T15:07:29.680" UserId="279" />
  <row Id="14706" PostId="8768" Score="1" Text="@Aniko It should be possible to adjust for the bias when early termination occurs.  We seem therefore to be discussing the naive use of a standard estimator, intended for fixed-size random samples, in conditionally terminated experiments, where such estimators do not have their desired properties. (+1, by the way.)" CreationDate="2011-03-25T15:17:30.743" UserId="919" />
  <row Id="14707" PostId="8771" Score="1" Text="You have framed this as an estimation problem (find the best) but are suggesting you want to do hypothesis testing.  In the absence of a prior distribution, the best guess you can make of the level that is best is the one that appears the best in testing: it's that simple.  In other words, choose the ad with the highest click-through rate." CreationDate="2011-03-25T15:22:47.867" UserId="919" />
  <row Id="14708" PostId="8763" Text="@whuber - there is no difference between estimation and hypothesis testing here, just replace $H$ by the proposition &quot;the true value of $\mu$ is in some small interval $(a,a+da)$&quot;.  The estimate depends on $S$ only through the data $D$ and prior $I$.  So while this may be true before you see the data (that $S$ matters), $S$ is irrelevant after the data has been observed.  $S$ only gives you information about $D$ and $I$, not directly about $\mu$." CreationDate="2011-03-25T15:41:53.407" UserId="2392" />
  <row Id="14709" PostId="8763" Text="(cont'd) for if $S$ did depend directly on $\mu$ then somehow you would need to know the true value in order to stop.  But this defeats the whole purpose of doing the test: if you already know the true value, why do the test? you won't learn anything about $\mu$ that you didn't already know...  Hence the stopping rule will generally be based on an estimate of $\mu$ and an estimate of its accuracy.  But these are known functions of the data and the prior information, so knowing $S$ doesn't tell you anything about $\mu$ that you didn't know already from the data and prior information." CreationDate="2011-03-25T15:47:51.640" UserId="2392" />
  <row Id="14710" PostId="8768" Text="@whuber Sure, you can adjust for this bias, but first you have to recognize that it exists. And then you have to sell to the investigator that even though clearly 5 out of 10 patients responded, the estimated rate of response is 40% (numbers made up) after adjusting for the bias due to early stopping." CreationDate="2011-03-25T15:52:23.277" UserId="279" />
  <row Id="14711" PostId="8768" Text="@Aniko, @whuber: Thank you for this discussion !" CreationDate="2011-03-25T16:00:03.023" UserId="3019" />
  <row Id="14712" PostId="8773" Text="+1 Very good!  I didn't think that the CI on the intercept would reveal the mean, but of course it does: the CI depends on the squared mean, as you point out.  Once we are reminded of that, the rest is algebra." CreationDate="2011-03-25T16:06:25.443" UserId="919" />
  <row Id="14713" PostId="8706" Text="1st one is linear ? really  - the one to the power of x ?" CreationDate="2011-03-25T16:12:59.510" UserId="3870" />
  <row Id="14714" PostId="8713" Text="can r handle non linear functions ? I know it has the same internal function for any regression ?" CreationDate="2011-03-25T16:14:23.253" UserId="3870" />
  <row Id="14715" PostId="8768" Text="@Aniko, @whuber: you can find an illustration in my reply " CreationDate="2011-03-25T16:31:25.563" UserId="2958" />
  <row Id="14716" PostId="8734" Text="Good question. Just looked at the latest APA Publication Manual and was surprised at how little guidance (or how few rules) it offered on this.  " CreationDate="2011-03-25T16:47:34.003" UserId="2669" />
  <row Id="14717" PostId="8766" Text="Sounds interesting, I'll have to look into that!" CreationDate="2011-03-25T16:48:57.120" UserId="887" />
  <row Id="14718" PostId="8747" Text="@ralph-winters If I understand correctly they are all independent.  Some will work better with others, but they all have an affect on the Percent who purchased.  Image and title usually have the biggest impact though." CreationDate="2011-03-25T16:56:51.997" UserId="3892" />
  <row Id="14719" PostId="8773" Text="@probabilityislogic Nice, thank you very much. Concerning the positive correlation and the negative slope, I double checked, it's what the data says. Given the above numbers, is it even valid to say that there's a dependency between $x$ and $y$?" CreationDate="2011-03-25T17:11:46.547" UserId="1205" />
  <row Id="14720" PostId="8718" Score="1" Text="This article may be of interest, http://dx.doi.org/10.1093/esr/jcp006 , apparently comparing effects between Logistic models is considerably more complicated than in the OLS case!" CreationDate="2011-03-25T17:17:36.223" UserId="1036" />
  <row Id="14721" PostId="8777" Text="Before asking these questions, did you search this site for &quot;PCA&quot; or explore the &quot;PCA&quot; tag?  Most of your questions are already answered there." CreationDate="2011-03-25T17:20:01.107" UserId="919" />
  <row Id="14722" PostId="8763" Text="@prob I think you have your hands full in the face of examples like that provided by @Henry!  Are you really maintaining that an unbiased estimator remains unbiased conditional upon early termination of the experiment?  Or is that just a misreading of what you're saying?  (Note, too, that event $S$ is not independent of the data, so it definitely *does* give information about $\mu$.)" CreationDate="2011-03-25T17:30:05.077" UserId="919" />
  <row Id="14723" PostId="8777" Score="1" Text="@whuber I think the OP is looking for the use of PCA as a way to account and adjust for population stratification when modeling a given outcome (continuous phenotype or case/control studies) and DNA markers (SNP). I gave a reference here: http://stats.stackexchange.com/questions/1708/variation-in-pca-weights/2848." CreationDate="2011-03-25T17:42:03.467" UserId="930" />
  <row Id="14724" PostId="8684" Score="1" Text="(+1) nice solution indeed, *however* the author still needs to store a bit more than just $\hat{\beta_1}$, but the idea is great, since allows fast updating with the smallest storage and requirements for the data. I personally would only go for this trick with the models that were at least statistically shown not to be inadequate." CreationDate="2011-03-25T17:54:45.337" UserId="2645" />
  <row Id="14725" PostId="8684" Score="1" Text="@Dmitrij Those are good points.  I presume by &quot;statistically adequate&quot; you mean the goodness of fit has been tested and found acceptable." CreationDate="2011-03-25T18:39:13.813" UserId="919" />
  <row Id="14726" PostId="8380" Text="multivariate regression would be more appropriate as this would accommodate the correlation in the uncertainty between variables - see my answer to the question below." CreationDate="2011-03-25T19:08:43.650" UserId="887" />
  <row Id="14727" PostId="8777" Score="1" Text="GWAS certainly can be done without principal components. In the absence of population stratification, all you need is thousands of $t$-tests or thousands of chi-squared tests." CreationDate="2011-03-25T19:31:19.370" UserId="449" />
  <row Id="14728" PostId="8777" Text="@onestop (+1) I'll consider you answered the 2nd question, that I even didn't consider in my own response." CreationDate="2011-03-25T19:34:31.437" UserId="930" />
  <row Id="14729" PostId="8774" Text="@suprvisr What is $x$ precisely? Let me try two educated guesses: (a) You're working with genotype data and a binary outcome (e.g., you have a two-way table of observed genotypes, given as the frequency of one of the two possible alleles on the DNA strand, by disease or exposure status or whatever might be coded as 0/1); (b) `independence.test` refers to sth like `independence_test` in the [coin](http://cran.r-project.org/web/packages/coin/index.html) package. Could  you confirm and update your question accordingly?" CreationDate="2011-03-25T19:41:32.357" UserId="930" />
  <row Id="14730" PostId="8734" Text="@JMS No prob. I was not criticizing; just a way to say that researchers often need to have numbers to refer to (as e.g., when comparing current work to earlier studies), which are helpful for meta-analysis or replication. I agree that good figures are to be preferred, when possible (never seen any confusion matrix displayed this way, though :-) @rolando2 The APA guidelines seem to focus on formatting issues and emphasize the need for reporting effect size measures (rather than NSHT) in the last 5 years; I'm not aware of the subtle issue of how best to convey statistical results." CreationDate="2011-03-25T20:45:04.280" UserId="930" />
  <row Id="14731" PostId="8687" Text="chl, as usual, thank you for the pointers - I'll look more into them in the future :)" CreationDate="2011-03-25T21:32:31.413" UserId="253" />
  <row Id="14732" PostId="8660" Text="@Probability (+1) Your proposal is reasonable and well-founded.  (In fact, I suspect the results would closely emulate the procedure I describe, provided equal weights are used.)  I wonder about one thing, though: we could interpret the OP's desire for weighting to be an expression of a more complex model in which the parameters are expected slowly to change over time.  Weighting is an *ad hoc* but simple and potentially effective way to cope with that.  Without overly complicating the situation or the solution, how could a slow temporal shift be accommodated within a Bayesian framework?" CreationDate="2011-03-25T21:33:08.830" UserId="919" />
  <row Id="14733" PostId="8678" Text="Hi Whuber, very interesting connections!  once I'll delve into this more, this gives me a lot to look into - Thanks!" CreationDate="2011-03-25T21:33:40.727" UserId="253" />
  <row Id="14734" PostId="8757" Text="The `cast` functions is really neat." CreationDate="2011-03-25T21:34:11.490" UserId="1537" />
  <row Id="14735" PostId="8757" Score="1" Text="@Matthias, yes it is. I say a prayer to Hadley Wickham everytime I use it :)" CreationDate="2011-03-25T21:47:27.817" UserId="2116" />
  <row Id="14736" PostId="8775" Text="@henry, you are taking the perspective of &quot;pre-data&quot; world.  What you say is true, that the stopping rule matters *but only before you consider the data*.  This is because the stopping rule provides information about the data, but not about the true probabilities.  So once the data is in, the stopping rule no longer matters.  Note that the true probabilities are unknown in the actual experiment.  So you also need to consider situations when the probabilities are, say $P(+)=\frac{1}{4}$ and $P(-)=\frac{3}{4}$, as well as any other possible combination." CreationDate="2011-03-25T21:59:20.370" UserId="2392" />
  <row Id="14737" PostId="8775" Text="@henry - So I take your example as stating that $P(H|S,I)\neq P(H|I)$.  This is certainly true!  My answer also conditions on $D$ though.  This is because, if you tell me the stopping rule, but not whether you actually did stop, I can figure this out from the data set that I actually have.  In fact, I can figure out  if *any stopping rule* would have actually stopped, once I know the data." CreationDate="2011-03-25T22:07:48.510" UserId="2392" />
  <row Id="14738" PostId="8763" Text="@whuber - not really, because @henry is only conditioning on the fact that we stopped, and not on the data that was actually observed.  @henry's example is a &quot;pre data&quot; example.  He is averaging over different data sets that might have been observed, but were not (given the data we actually did observe).  He should be averaging over the different stopping rules that could have lead to the data set that was actually observed." CreationDate="2011-03-25T22:14:57.367" UserId="2392" />
  <row Id="14739" PostId="8773" Text="@slhck - given the numbers, it is valid to say an error has been made in either calculating the slope or in calculating the correlation, or both (unless the correlation is $r^2=0.117$ and not $r$).  But putting this aside, given the absolute value of the slope, the t-stat is just over 1 - not very strong evidence of a slope.  Correlation does not imply dependence in a physical sense, only a logical one.  So $x$ and $y$ may be *physically dependent* (but on a small scale), but given the data, it is pretty safe to assume *logical independence*." CreationDate="2011-03-25T22:24:34.927" UserId="2392" />
  <row Id="14740" PostId="8726" Text="Any particular reason for the down vote?  Seems a bit harsh to me - what is wrong with my answer?" CreationDate="2011-03-25T22:45:01.563" UserId="2392" />
  <row Id="14741" PostId="8754" Text="I think this is a perfect question to &quot;flesh out&quot; the differences between Bayesian and Frequentist methodology" CreationDate="2011-03-25T22:51:29.207" UserId="2392" />
  <row Id="14742" PostId="8767" Text="I defer to Aniko--he does a better job than I could. But if you agree that &quot;desk drawer effect&quot; results in bias, the logic here is identical. There is bias in favor of data supportive of hypothesis -- in the former case b/c the not-supportive data are not reported, in the latter b/c some fraction of not-supported data is necessarily not being collected: Ending the trial early when results look good excludes that part of the &quot;bad results&quot; distribution populated by trials that will produce their bad results *late*. Maybe this bias can be adjusted for--but there is bias in need of adjustment." CreationDate="2011-03-25T23:29:35.137" UserId="2347" />
  <row Id="14743" PostId="7538" Text="The folks over at StackOverflow may know better than CV." CreationDate="2011-03-26T00:48:45.000" UserId="1118" />
  <row Id="14745" PostId="8771" Text="@whuber. Hello. How is my question different than an experiment to see which of ten fertilizers produces the greatest yield in a classic DOE?  A one-way-anova with a continuous distribution is used to test if each mu_i differs from the max of the other mu (instead of simply differing because of sampling variation) - how is this different? If I had only two versions (coded 1/0), I would compare them  - potentially in a logistic regression framework. I could do the same here but would have issues with all the comparison." CreationDate="2011-03-26T01:30:46.237" UserId="2040" />
  <row Id="14746" PostId="8785" Text="	&#xA;Thanks for the detailed answer. Is it the standard formula for JS? and why did you mention 3 are same between two distributions? Is it important? Do I need to calculate this as well when I perform JS? How did you get 2/5? 1/5?" CreationDate="2011-03-26T01:39:37.197" UserId="3900" />
  <row Id="14747" PostId="8771" Text="@B The answer is that you're asking a different question.  You're not trying to decide whether there are significant differences among different options; you simply need to make the best decision possible, based on these data, concerning which is the best.  Suppose you had to answer to the head of advertising, who wants your decision: which is &quot;the best performing ad?&quot;  Are you going to tell her you don't know because some statistical test suggests the difference between best and second best could be due to chance?  Her reply will be &quot;so what?  We will use the best performed in your tests.&quot;" CreationDate="2011-03-26T01:43:02.340" UserId="919" />
  <row Id="14748" PostId="8767" Text="@dmk I'm just trying to spur you both to have a debate with @Probability, with whom you seem to sharply disagree ;-)." CreationDate="2011-03-26T01:44:24.603" UserId="919" />
  <row Id="14749" PostId="8771" Text="@whuber. That may be the advertising exec's stance I do not disagree, but I guess I fail to see why hypothesis testing and multi-comparisons is not relevant in my question. My response to the exec would be that we could not determine the best one,  between the two or three top performers due to the sample size we had, so we will go back and run more tests with these that were significantly different than the rest but not between each other and determine if we can find a winner... That is exactly what Hsu's MCB gives you..  " CreationDate="2011-03-26T02:03:08.907" UserId="2040" />
  <row Id="14750" PostId="8771" Text="@B If you have the opportunity to do more tests, that changes the picture somewhat.  I doubt that running a series of experiments with multiple comparison hypothesis testing is an efficient use of resources, however.  A good solution would involve balancing the costs of additional testing against the potential gains in discriminating &quot;the best&quot; ad from the second best one.  In other words, this is a problem of decision analysis (with a fairly well characterized loss function), not a problem of hypothesis testing." CreationDate="2011-03-26T02:11:10.477" UserId="919" />
  <row Id="14751" PostId="8767" Score="1" Text="@whuber, @dmk - I think we are destined to disagree not because either of us is wrong, but because each is answering a different question.  The frequentist considers $P(D|H,S,I)$ as &quot;the answer&quot;, and if this is the object, then the stopping rule does matter.  But to what question is it the answer?  To me, this answers the question: &quot;what data are we likely to observe, given the hypothesis is true (or parameter is said value), that we have stopped early, and from our prior information?&quot;  But this is not the question that is actually being asked I think (more later)" CreationDate="2011-03-26T04:03:30.047" UserId="2392" />
  <row Id="14752" PostId="8767" Text="(cont'd) - for why do we stop the experiment? I think that it is because our hypothesis has been confirmed/denied to a satisfactory degree.  So the question in this case is &quot;how plausible is the hypothesis, given the data we have observed, and our prior information?&quot;  People are not usually interested in the data per se, they are usually more interested in good explanations of that data, i.e. in good hypothesis." CreationDate="2011-03-26T04:07:50.450" UserId="2392" />
  <row Id="14753" PostId="8768" Text="@whuber - I would have a small bet that the &quot;best&quot; adjustment of bias (from a frequentist perspective) is equal to $\frac{P(D,S|H,I)}{P(D|H,I)}$." CreationDate="2011-03-26T04:13:52.227" UserId="2392" />
  <row Id="14754" PostId="8785" Text="@Henry: OP wanted to find $D_{\mathrm{KL}}$, right? Why do you find JSD? Sorry if I'm mistaken." CreationDate="2011-03-26T05:44:41.837" UserId="3485" />
  <row Id="14755" PostId="8785" Text="@Ashok: I asked the question as it was at the time. It has since changed. " CreationDate="2011-03-26T07:17:45.093" UserId="2958" />
  <row Id="14756" PostId="8784" Score="2" Text="How do you know one can't just look at the 'size' of the coefficient?" CreationDate="2011-03-26T09:40:35.183" UserId="449" />
  <row Id="14757" PostId="8607" Text="Thanks for the comprehensive answer. I can go out and hunt for resources for details of the second part, where you outline the problematic cases, but if you could provide me a couple of pointers, that'd be fantastic. Books, or even keywords for resources to get a better handling of the problems would be fantastic. Unfortunately, you don't get to see these in most books about Bayesian statistics, networks etc. " CreationDate="2011-03-26T10:15:28.320" UserId="3280" />
  <row Id="14759" PostId="8791" Text="That was just a hypothetical example. I'm not using this algorithm for geolocation." CreationDate="2011-03-26T01:59:46.987" />
  <row Id="14760" PostId="8792" Text="Thanks, I think. :) This is essentially what I was already doing but knowing what it's called will help. If there are no better answers after a while I'll accept this." CreationDate="2011-03-26T02:06:33.913" />
  <row Id="14761" PostId="8792" Score="1" Text="@Mitch Wheat: Was speaking more towards the Weighted Moving Average.  By using the moving average you can get the average speed/direction which will allow one to predict where the next location might be.  The weighting gives him the ability to &quot;phase out&quot; the older reports." CreationDate="2011-03-26T02:07:55.940" />
  <row Id="14762" PostId="8792" Score="2" Text="Particularly for more complex systems, Kalman filtering and related methods should be investigated as well." CreationDate="2011-03-26T02:31:49.500" />
  <row Id="14763" PostId="8792" Text="@user57368 Thanks, that's looks *very* helpful. The &quot;Example application&quot; section on that page is basically a better-worded and more complete version of what I was trying to describe. If you want to post that as an answer I'll accept it. Sorry, Suroot. Your answer was helpful too though. :)" CreationDate="2011-03-26T03:39:03.560" />
  <row Id="14764" PostId="8792" Text="Oops. When I said &quot;that page&quot; I meant the [Kalman filter](http://en.wikipedia.org/wiki/Kalman_filter) page on Wikipedia. I guess I only imagined that you linked to it." CreationDate="2011-03-26T03:49:47.383" />
  <row Id="14765" PostId="8753" Text="Yes it is. There are many variations of the analysis. Some, so-called computer-based (or adaptive, or on-the-fly) conjoint exposes a respondent with not all or random scenarios but with scenarios that will yield most valuable results, based on his/her previous responses." CreationDate="2011-03-26T11:05:06.047" UserId="3277" />
  <row Id="14766" PostId="8687" Text=" Well I just have to up-vote answers linking to my package:)" CreationDate="2011-03-26T11:54:32.940" UserId="3094" />
  <row Id="14767" PostId="8687" Text="@Sacha Thx. Good to hear you may contribute to CV, and congrats for your package (a very nice one)!" CreationDate="2011-03-26T12:25:42.547" UserId="930" />
  <row Id="14768" PostId="8756" Text="@Ejs It is better to update your question (since you don't have enough rep to leave comment on others' answers), and in the future leave comment directly under answerer's post. I'll leave it as is because you already got two answers from @Peter." CreationDate="2011-03-26T12:31:41.153" UserId="930" />
  <row Id="14769" PostId="8784" Text="@Ejs I merged your two accounts. You'll still need to register, as indicated on the [FAQ](http://stats.stackexchange.com/faq). (@onestop Thx for pointing to the duplicate.)" CreationDate="2011-03-26T12:33:42.167" UserId="930" />
  <row Id="14770" PostId="8687" Text="Thanks. I am a lot more active on SO, should really check out CV more :)" CreationDate="2011-03-26T12:34:41.413" UserId="3094" />
  <row Id="14771" PostId="8687" Text="@Sacha Sure. We're looking forward to hearing from you. There's room for R users and psychometrics here :-)" CreationDate="2011-03-26T12:41:53.407" UserId="930" />
  <row Id="14772" PostId="8787" Text="@Muhammad I've added the `Stata` tag, as your question relates to Rabe-Hesketh and coll. approach. Feel free to update if it is misleading." CreationDate="2011-03-26T12:46:41.467" UserId="930" />
  <row Id="14773" PostId="8783" Text="t2 does not add up to one and is thus no valid probability. If you don't fix that, you will get weird results." CreationDate="2011-03-26T12:51:58.560" UserId="2860" />
  <row Id="14774" PostId="8795" Score="2" Text="Clearly the data this graph is based on are not derived from tossing a coin and appear to be continuous, not binary. Could you tell us what the substantive question is you're trying to answer? Putting it in terms of an stereotypical example isn't helping here." CreationDate="2011-03-26T14:22:58.247" UserId="449" />
  <row Id="14775" PostId="8768" Text="@probability Certainly the adjustment would involve that factor, but it has to include more than that: it must account for how large the bias can be at each stage, and that (potentially) depends on the estimator." CreationDate="2011-03-26T14:32:45.383" UserId="919" />
  <row Id="14776" PostId="8767" Score="1" Text="@probability That is one way to look at it.  Another is to dodge the hypothesis altogether and address the question actually being asked; to wit, *what is the size of the treatment effect*?  From this point of view termination can occur once the estimate is known with sufficient accuracy to support decision making.  For example, we might want to have high confidence that the gain in health from prescribing the treatment is likely to exceed the costs (and side effects) of the treatment." CreationDate="2011-03-26T14:38:32.883" UserId="919" />
  <row Id="14777" PostId="8795" Text="The graph is derived from calculating the how much (in percentage terms) today's close is when compared to the close 24 days ago. Option pricing models assume that there is a 50% probability that a stock will be 10% higher or 10% lower in n days. This graph is a distribution of actual prices. Can we accept the hypothesis that there is equal probability that a stock's price will be 10% higher or 10% lower in n days." CreationDate="2011-03-26T14:41:34.113" UserId="3306" />
  <row Id="14778" PostId="8795" Text="@whuber, thanks. I'll remove the chart.ly link." CreationDate="2011-03-26T14:51:21.267" UserId="3306" />
  <row Id="14779" PostId="8797" Text="@Roflcoptr Your [Confidence Interval](http://en.wikipedia.org/wiki/Confidence_interval) (I assume it is for the mean) looks strange because it must be an interval, not a single value. Isn't it [4.28;4.92] instead? You'll also need the sample size to compute it by hand. " CreationDate="2011-03-26T15:58:05.373" UserId="930" />
  <row Id="14780" PostId="8799" Text="@Roflcoptr Are you working with Excel only, as in your related question http://stats.stackexchange.com/questions/8797/i-used-excel-to-calculate-a-confidence-interval-but-what-is-a-result-of-0-32-tel? If you're willing to switch to R or other statistical software, we can provide additional clues for psychometric analysis." CreationDate="2011-03-26T16:01:56.043" UserId="930" />
  <row Id="14781" PostId="8797" Text="@chl: Ah yes it is the interval... So I should use +/- 0.32. For the calculation of the confidence interval I used the standard deviation." CreationDate="2011-03-26T16:02:51.703" UserId="3908" />
  <row Id="14782" PostId="8799" Text="@Chl: I'm working with Excel since it was only a small evaluation and I thought switching to another software is difficult to learn" CreationDate="2011-03-26T16:03:41.157" UserId="3908" />
  <row Id="14783" PostId="8795" Text="@Milktrader, first of all, option models do ***not*** assume that there is an equal probability of a 10% upward return versus an equal percentage downward return. Indeed, option models under a no-arbitrage framework do not even work with the actual distribution of returns. Furthermore, even the *risk-neutral* measure generally assumes that the prices have a higher probability of rising than falling. Finally, your comment makes two very different statements about the returns, even though you seem to be thinking about them as the same. Maybe you can reword and clarify your question." CreationDate="2011-03-26T16:16:57.900" UserId="2970" />
  <row Id="14784" PostId="8799" Text="@chl I just downloaded R and input the first result set of my question. And I see it is not as difficult as I thought..." CreationDate="2011-03-26T16:39:14.073" UserId="3908" />
  <row Id="14785" PostId="8799" Text="@Roflcoptr Welcome to R :-) Seriously, most of the descriptive stuff (that is how responses are distributed) can be done within Excel. But, I was suggesting R, for you could grab a lot of ideas from here (look for [scales](http://stats.stackexchange.com/questions/tagged/scales), [questionnaire](http://stats.stackexchange.com/questions/tagged/questionnaires), and [ordinal](http://stats.stackexchange.com/questions/tagged/ordinal) tags). Your question suggests you just want to summarize the data, which might be done in a purely univariate manner. " CreationDate="2011-03-26T16:43:58.280" UserId="930" />
  <row Id="14786" PostId="8283" Text="Thank you very much for your input. I have used the moment method and I think I am on the right track.&#xD;&#xA;Best Regards,&#xD;&#xA;Sam" CreationDate="2011-03-26T17:07:04.303" UserId="3909" />
  <row Id="14787" PostId="8673" Text="Looks like a great contribution to a topic that had many of us puzzled for a quite a while." CreationDate="2011-03-26T17:58:40.673" UserId="2669" />
  <row Id="14788" PostId="8795" Text="@cardinal I'm actually more interested in probability theory than option pricing models with this question, although the topic of option pricing models is interesting. You likely have a more robust option pricing model, but mine shows there is a 14.81% prob SLV closes &gt; 40.04 and 14.52% prob it closes &lt; 32.75 by APR expiry (20 days). I'm also happy to rephrase my question to clarify it, but I'm not sure how I've made two unique statements about returns. " CreationDate="2011-03-26T18:13:43.090" UserId="3306" />
  <row Id="14789" PostId="8802" Text="@rolando 2, thats good advice, my rustiness with SPSS sure is showing." CreationDate="2011-03-26T18:45:41.050" UserId="656" />
  <row Id="14790" PostId="8803" Text="Thanks for the answer. This helps." CreationDate="2011-03-26T18:51:00.517" UserId="3908" />
  <row Id="14791" PostId="8803" Text="@richiemorrisroe So basically the smaller the interval the better?" CreationDate="2011-03-26T19:01:25.143" UserId="3908" />
  <row Id="14792" PostId="8803" Text="@Roflcoptr essentially, yes. However, its important to assume that the assumptions for calculating the confidence interval are met, or you could run into trouble." CreationDate="2011-03-26T19:06:10.057" UserId="656" />
  <row Id="14793" PostId="8744" Text="Thanks, everyone, for the suggestions so far!" CreationDate="2011-03-26T19:34:32.727" UserId="3891" />
  <row Id="14794" PostId="8803" Text="@Roflcoptr Be aware that reporting only mean $\pm$ SE or a 95% CI for Likert items, whose distribution are likely to be asymmetric (and this is your case), might be misleading. Another solution would be to report mean $\pm$ SD *and* the frequency of observed response in the two upper responses modalities (4+5, or &quot;agree&quot; and &quot;strongly agree&quot;)." CreationDate="2011-03-26T19:40:42.003" UserId="930" />
  <row Id="14795" PostId="8802" Text="@rolando2 I seem to remember that Cronbach's alpha will only equal composite score reliability for essentially tau-equivalent tests. It is very shortly discussed on Garson's SPSS website, http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm; but I will also check for more recent academic references." CreationDate="2011-03-26T19:51:31.120" UserId="930" />
  <row Id="14796" PostId="7318" Text="What about Tukey's median polish?" CreationDate="2011-03-26T20:13:19.357" UserId="1356" />
  <row Id="14797" PostId="8805" Text="@richiemorrisroe (+1) These all are very helpful directions for the OP. I would add that Cronbach's alpha makes *no sense at all* with a multidimensional measurement instrument, but has to be computed for each scale separately, providing they have more than two items. If there're no prior hypotheses, there's no need to try CFA, IMO. It may be worth trying the item clustering approach developed by W. Revelle, also available in the `psych` package." CreationDate="2011-03-26T20:17:09.723" UserId="930" />
  <row Id="14798" PostId="8795" Text="@Milktrader, I'm just trying to figure out what problem you are trying to solve. My reference to option-pricing models was actually meant to refer to even the most basic and &quot;standard&quot; ones. *Currently* they may appear to assume a symmetric distribution, but that would only be because interest rates are near zero. " CreationDate="2011-03-26T20:34:43.563" UserId="2970" />
  <row Id="14799" PostId="8795" Text="@Milktrader, So, is your question about testing whether the distribution of an observed random variable has symmetric tails beyond some cutoff? Is it that the whole distribution itself is symmetric? Is it about testing whether two probabilities are equal in a multinomial distribution with three possible outcomes? Or, something else?" CreationDate="2011-03-26T20:35:24.403" UserId="2970" />
  <row Id="14800" PostId="8795" Text="@Milktrader, finally my reference to the inconsistencies in your original comment were simply that you (a) on one hand state that the probabilities of more than a 10% up or down move are each 50% and (b) on the other hand talk about equal probabilities of an up or down move of that side. Obviously, the former is just a (very) specific case of the latter and doesn't appear to be what you are likely interested in based on the graph you posted. I'm sorry if this all sounds pedantic, I'd just like to clarify the question, as it's likely to be an interesting one! Cheers." CreationDate="2011-03-26T20:37:49.287" UserId="2970" />
  <row Id="14801" PostId="8797" Text="What is the problem you are trying to solve? What meaning does the average have for this problem? Also, is this a confidence interval for a single answer aggregated over people, or is it the mean response of the whole questionnaire by somehow aggregating the responses? Or, is it the analysis of a single questionnaire?" CreationDate="2011-03-26T20:44:16.270" UserId="2970" />
  <row Id="14802" PostId="8795" Text="The word &quot;side&quot; should be &quot;size&quot; above; sorry for the typo." CreationDate="2011-03-26T20:47:39.167" UserId="2970" />
  <row Id="14803" PostId="8805" Text="@chl i agree on cronbach. clustering is also good, i always forget about it for some reason though." CreationDate="2011-03-26T21:22:10.900" UserId="656" />
  <row Id="14804" PostId="8809" Score="1" Text="@onestop: This is very off-the-cuff as I haven't done any calculation. All of what you've written sounds good. The only question that initially comes up to me is that it seems the estimate of variance might be different from if you &quot;threw away&quot; the samples corresponding to the third outcome." CreationDate="2011-03-26T22:33:30.997" UserId="2970" />
  <row Id="14805" PostId="8808" Text="This question needs more information: what exactly do you mean by &quot;working with&quot; a distribution?  Confidence interval for what parameter?  Based on what data?  What prior distribution?" CreationDate="2011-03-26T22:37:03.617" UserId="919" />
  <row Id="14806" PostId="8767" Text="@whuber - good point, but note that this is already contained in the current discussion by re-phrasing $H$ to $\theta\in (\theta_{est},\theta_{est}+d\theta_{est})$.  So in &quot;frequentist&quot; world, the stopping rule should still matter.  One is simply doing a &quot;continuum&quot; of hypothesis tests, the chosen one being the final estimate." CreationDate="2011-03-26T23:08:27.793" UserId="2392" />
  <row Id="14807" PostId="8769" Text="Thank You. This was very helpful." CreationDate="2011-03-26T23:08:59.390" UserId="3897" />
  <row Id="14808" PostId="8734" Text="@chi Saw a Hinton diagram for one once, when there were a lot of classes. Certainly not routine though! @David I agree, good point about esoteric figures; sometimes people get too cute for their own good. I was probably a little too glib in my first comment. Thoughtful presentation should obviously be the priority." CreationDate="2011-03-26T23:12:13.197" UserId="26" />
  <row Id="14809" PostId="8736" Text="@AVB still not clear why you recommend 123.4 instead of 123 (why omit .3 but not .4 in the example?)" CreationDate="2011-03-26T23:24:21.980" UserId="1381" />
  <row Id="14810" PostId="8742" Text="yes $P(D)$ conditional on $B$ and $E$, as T is conditional on $A$, L and B are conditional on $S$, i did not write it correctly." CreationDate="2011-03-27T00:09:05.283" UserId="3681" />
  <row Id="14811" PostId="8795" Text="@cardinal I see your point about my unclear comment. There is not a 50% prob that a stock will be up or down 10%. If a stock moves 10% in 24 days though, is it a fair coin toss on whether that move is to the upside versus the downside? I've included the median to show that whether a move is up or down is a basically a fair coin toss, based on the data. But things get weird out on the tails." CreationDate="2011-03-27T00:16:43.747" UserId="3306" />
  <row Id="14812" PostId="8809" Text="Yes, this is the formal description of my problem. Can a multinomial distribution be reduced to a binomial distribution? What concerns me is the size of the 'middle' outcome. " CreationDate="2011-03-27T00:45:41.507" UserId="3306" />
  <row Id="14813" PostId="8767" Text="Good I didn't take the bait since @prob shows this is more difficult than I appreciated (glad to learn that). Trials are often ended when an early positive response makes denying treatment to placebo group seem morally unacceptable. Someone has to make the decision *whether* the &quot;hypothesis has been confirmed to a satisfactory degree&quot; given the harm to those denied a potentially effective treatment. What's needed, then, is a way to quantify the cost of early termination so an informed assessment of the tradeoff can be made. We are converging on an answer that features a Bayesian framework." CreationDate="2011-03-27T01:25:07.057" UserId="2347" />
  <row Id="14814" PostId="8794" Text="Yes, 'the economy' is shorthand for perceptions of national economic conditions. Does the same advice apply when other predictors (controls) are included in the model?" CreationDate="2011-03-27T03:44:39.720" UserId="3883" />
  <row Id="14815" PostId="8793" Text="Does this have any implications for roland2's suggestion?" CreationDate="2011-03-27T04:12:33.980" UserId="3883" />
  <row Id="14816" PostId="8784" Text="I assumed that I couldn't compare 'effect' of predictors across models by looking at the coefficients from the answers to my previous question. Are things different for my example above?" CreationDate="2011-03-27T04:24:02.837" UserId="3883" />
  <row Id="14817" PostId="7601" Text="You might more detail about what your goals are.  In general, like @rolando2 I encourage emphasizing predictions and confidence intervals rather than hypothesis tests." CreationDate="2011-03-27T04:44:02.047" UserId="3748" />
  <row Id="14818" PostId="8808" Text="X follows a uniform distribution" CreationDate="2011-03-27T05:48:28.523" UserId="3910" />
  <row Id="14819" PostId="8808" Text="find the CI for theta" CreationDate="2011-03-27T05:48:58.630" UserId="3910" />
  <row Id="14820" PostId="8808" Text="the improper prior is used" CreationDate="2011-03-27T05:49:31.440" UserId="3910" />
  <row Id="14821" PostId="8793" Text="@Ejs. Do you refer to the standardisation step? By the way, does my answer help ? Have I misunderstood the question ?" CreationDate="2011-03-27T06:40:32.267" UserId="3019" />
  <row Id="14822" PostId="8819" Score="2" Text="@JMS +1 for NLTK. Olivier Grisel, who is doing Machine Learning research and Natural Language Processing, has some good tutorials and slides around the web. I can think of [Statistical Learning and Text Classification with NLTK and scikit-learn](http://www.pycon.fr/file/2308?vid=download) (PyCON FR 2010), for example." CreationDate="2011-03-27T09:45:47.100" UserId="930" />
  <row Id="14823" PostId="6492" Text="I would try using the Laplace distribution for asset returns, also called the &quot;double exponential&quot; is stats-world, and &quot;variance-gamma&quot; in Finance world." CreationDate="2011-03-27T09:54:12.650" UserId="2392" />
  <row Id="14824" PostId="8821" Score="2" Text="libsvm does not scale well to large numbers of features and samples. Better use liblinear by the same authors. linear models are usually good enough in high dimensional space." CreationDate="2011-03-27T11:22:14.573" UserId="2150" />
  <row Id="14826" PostId="8821" Text="@ogrisel: good to know, thanks!" CreationDate="2011-03-27T12:58:15.197" UserId="364" />
  <row Id="14827" PostId="8831" Text="I just figured this out by myself in Matlab, but thank you anyway, much appreciated!" CreationDate="2011-03-27T14:01:58.870" UserId="3782" />
  <row Id="14830" PostId="8833" Text="@ogrisel : Thanks. This seems like a particularly good way forward. I will try it. " CreationDate="2011-03-27T14:35:49.537" UserId="3301" />
  <row Id="14831" PostId="8817" Text="Thank you so much everyone for your comments. I think for now I will go with NLTK+scikit-learn and see where things go. " CreationDate="2011-03-27T14:37:07.960" UserId="3301" />
  <row Id="14833" PostId="8819" Text="@chi Thanks for that tutorial, I've been poking around a little and it seems he's got a lot of nice material out there. Also displays a confusion matrix with a heatmap in his slides ;)" CreationDate="2011-03-27T15:28:36.677" UserId="26" />
  <row Id="14834" PostId="8827" Text="@probabilityislogic +1 for the additional references. My favorite derivation for the change of variables formula starts with the cdf, like in your other answer." CreationDate="2011-03-27T15:31:51.073" UserId="26" />
  <row Id="14835" PostId="8817" Score="1" Text="@Andy I can't help proposing you to look at [R](http://www.r-project.org/), which has way bigger ML library and (being made especially for this job) provides a much more homogeneous environment. R's text mining package is [presented here](http://www.jstatsoft.org/v25/i05)." CreationDate="2011-03-27T16:13:12.043" UserId="88" />
  <row Id="14836" PostId="8806" Text="@JVM. Do I read you correctly that your suggestion is my last solution option above (in the final update). That you develop models to forecast each Z (TVC) to use in the actual survival model?" CreationDate="2011-03-27T16:13:53.860" UserId="2040" />
  <row Id="14837" PostId="8836" Score="1" Text="This is cross-posted from http://metaoptimize.com/qa/questions/5296/question-on-mle-and-em, and has been closed because it lacks a clear question. Could you clarify your problem in this particular case?" CreationDate="2011-03-27T17:47:06.307" UserId="930" />
  <row Id="14838" PostId="8832" Text="@bgbg: What is the 'classical' way to validate a model on future data?  What do you mean by 'oversampling?'  Thank you!" CreationDate="2011-03-27T18:02:02.930" UserId="2817" />
  <row Id="14839" PostId="8849" Text="Yeah, right... bootstrap is always an option. Thanks!" CreationDate="2011-03-27T19:41:19.763" UserId="1356" />
  <row Id="14840" PostId="8848" Text="@GaBorgulya I think the OP is asking for a graph produced with `ggplot2::qplot`, specifically." CreationDate="2011-03-27T19:44:56.073" UserId="930" />
  <row Id="14841" PostId="8846" Score="1" Text="@aL3xa A permutation test might be appropriate here." CreationDate="2011-03-27T19:52:15.017" UserId="930" />
  <row Id="14842" PostId="8849" Text="@GaBorgulya About your first point, it seems to me that non-parametric tests make the assumption that there's a shift in central tendency, but that overall the distributions that are compared have the same shape. " CreationDate="2011-03-27T19:54:41.937" UserId="930" />
  <row Id="14843" PostId="8849" Text="So what should I use to test basic assumptions? Does that mean that I need to apply a homoscedascity test?" CreationDate="2011-03-27T20:05:05.107" UserId="1356" />
  <row Id="14844" PostId="8794" Text="@Ejs - I'm afraid there's no short answer to your last question.  You're getting into what it means to assess relationships when using statistical control - a fabulously intricate topic worthy of extensive study.  You're also probably getting into the topic of variable selection, which is a big one as well.  Imho the best source for the committed student of these topics is Pedhazur's http://www.amazon.com/Multiple-regression-behavioral-research-Pedhazur/product-reviews/0030728312" CreationDate="2011-03-27T20:08:09.853" UserId="2669" />
  <row Id="14845" PostId="8816" Text="So I cannot claim that P(positive move | 10% move)? Or, if I know there is a 10% move, I can say that such a move has a (268/366) probability of being positive. But I think I can always claim P(10% move | positive move), no? If the move is positive, there is an (268/629) probability that the move will exceed 10%. (I did not print out the total positives on the graph because I didn't think that far ahead)." CreationDate="2011-03-27T20:11:42.253" UserId="3306" />
  <row Id="14846" PostId="8810" Text="`dput()` the contents of `v` for specific help, otherwise - check out http://had.co.nz/ggplot2/scale_datetime.html" CreationDate="2011-03-27T20:13:29.777" UserId="696" />
  <row Id="14847" PostId="8846" Text="I added a little something: &quot;unequal groups&quot;." CreationDate="2011-03-27T21:12:32.203" UserId="1356" />
  <row Id="14849" PostId="8816" Text="@Milktrader: Your original process and numbers are based on a consistent Daily Close. When you get a Close in the future it will also be based on a Daily Close. Neither are based on a &quot;Preferred Close&quot; (which requires KNOWN information after-the-fact). You can represent the process as a multinomial, or one and a half binomials (one binomial process to select the &quot;Preferred&quot; versus &quot;Not Preferred&quot; path, and then another binomial process using your &quot;Preferred Probabilities&quot;). Try it. Is it possible to simulate the overall process with the &quot;Preferred Probabilities&quot; alone?" CreationDate="2011-03-27T21:49:02.297" UserId="2775" />
  <row Id="14850" PostId="8816" Text="If this stock moves 10% in the next 24 days, can I claim that the probability that move will be to the upside is 268/366? I don't mean to mix time frames. (just now sifting through the second part of your comment)" CreationDate="2011-03-27T21:52:59.937" UserId="3306" />
  <row Id="14851" PostId="8849" Text="Unfortunatey boostrapping doesn't work well with very small samples either. @chl's suggestion of a permutation test is a much better idea.  Monte Carlo permutation tests are a good option when exact tests haven't been programmed or would take unfeasibly long." CreationDate="2011-03-27T22:07:09.880" UserId="449" />
  <row Id="14852" PostId="8846" Text="Can you tell us what your three $n$s are?" CreationDate="2011-03-27T22:09:23.933" UserId="449" />
  <row Id="14855" PostId="8849" Text="@chl The $H_0$ of KW is that the multiple samples come from the same distribution of any shape. Looking at the [test statistic](http://goo.gl/gugCT) I agree that the test is more powerful in case of shifts only. But although the [original paper](http://homepages.ucalgary.ca/~jefox/Kruskal%20and%20Wallis%201952.pdf)([permalink](http://www.webcitation.org/5xVUQZjnL)) also states that the $H_A$ &quot;is that the samples come from populations of approximately the same form, but shifted&quot;, I don't see where the test statistic relies on that assumption." CreationDate="2011-03-27T22:28:03.683" UserId="3911" />
  <row Id="14856" PostId="8846" Text="Well, if I do that, then this question can easily be marked as an assignment/homework. =) But, OK, say: `10`, `50`, `100`." CreationDate="2011-03-27T22:37:39.447" UserId="1356" />
  <row Id="14857" PostId="8827" Text="@JMS - the cdf rule is the only one I don't get confused with, I usually struggle to remember if its $\frac{dy}{dx}$ or $\frac{dx}{dy}$ with the jacobian" CreationDate="2011-03-27T22:40:48.757" UserId="2392" />
  <row Id="14858" PostId="8859" Score="1" Text="Using the log-odds is the same as using the probability.  Its just on a different scale.  So I'd say use log-odds if that suits you.  Sounds like you need a decision rule - gets classed as spam if $log(P(spam|D))-log(P(non-spam|D))&gt;C$.  You choose $C$ based on the consequences of false classification. which is worse to get wrong? if spam is worse, then $C&lt;0$, if indifferent $C=0$, else $C&gt;0$.  How big $C$ is is a reflection of &quot;how much worse&quot;.  I'd report the LHS of the equation as the score" CreationDate="2011-03-27T22:48:24.977" UserId="2392" />
  <row Id="14859" PostId="8816" Text="@Milktrader: From the data above, for a 24 day delta, you have 268 Ups, 98 Downs, and 676 Nulls (1042 Total Events). Assuming no structural changes, each trading day in the FUTURE, before the trading day, you face the probabilities of 268/1042 Ups, 98/1042 Downs. The remaining 676/1042 Nulls will show up more often. All of this deals with the future. After the Close, you'll know if it's a &quot;Preferred Day&quot;, but again this is after the close (not the future). Your &quot;Preferred Probabilities&quot; only apply after-the-fact (in the past). I added a diagram in my answer above to help explain." CreationDate="2011-03-27T22:55:01.740" UserId="2775" />
  <row Id="14860" PostId="8849" Text="I crossed my first point as it's useful to know about heteroscedasticity. The [original paper](http://homepages.ucalgary.ca/~jefox/Kruskal%20and%20Wallis%201952.pdf)([p­ermalink](http://www.webcitation.org/5xVUQZjnL)) states: “There is some reason to conjecture … that … the … test may be fairly insensitive to differences in variability, and so may be useful … without assuming equality of variances” (section 4.2, 2nd paragraph)." CreationDate="2011-03-27T23:04:20.330" UserId="3911" />
  <row Id="14861" PostId="8816" Text="@bill: The diagram helps, thanks. " CreationDate="2011-03-27T23:33:06.153" UserId="3306" />
  <row Id="14862" PostId="8861" Text="@probabilityislogic thanks for explaining this to me, but I am still confused - in your notation, is $\mu$ a vector and $I$ an identity matrix?" CreationDate="2011-03-27T23:37:08.480" UserId="1381" />
  <row Id="14863" PostId="8861" Text="@probabilityislogic can you point me to a reference? I have Gelman's Bayesian Data Analysis and Carlin and Louis &#xA;&quot;Bayes and Empirical Bayes Methods&quot; handy but can't relate their explanations to your equations 1 and 2. I understand that how to calculate the PDF from the CDF, but not how this gets you from to eqn 1 or from 1 to 2." CreationDate="2011-03-27T23:50:34.900" UserId="1381" />
  <row Id="14864" PostId="8808" Score="1" Text="@KIRAN: 1. Do you have a sample set of data $\{ X_i \}$?  2. When you say &quot;the improper prior is used&quot;, which one?  Perhaps the one with density proportional to $1/\theta$? Or perhaps something else? 3. Which *credible interval* (not *confidence interval*) do you want?  [Wikipedia offers three](http://en.wikipedia.org/wiki/Credible_interval), though I don't think the third is particularly useful.   " CreationDate="2011-03-28T00:11:06.777" UserId="2958" />
  <row Id="14865" PostId="8849" Text="@onestop aL3xa did not state they has small sample sizes, that was a further question." CreationDate="2011-03-28T00:44:15.483" UserId="3911" />
  <row Id="14866" PostId="8817" Text="@mbq : Yes, I am very aware of R, but since I have to do so much &quot;non-statistics&quot; programming before doing the actual text mining, I think I will stick to Python for now." CreationDate="2011-03-28T00:51:22.310" UserId="3301" />
  <row Id="14867" PostId="8859" Text="When I use log(P(spam|D))−log(P(non−spam|D)) measure, I got values like 32.2273, 1.68025, 109.994.. How can I make it between 0-1?" CreationDate="2011-03-28T01:28:23.167" UserId="3924" />
  <row Id="14868" PostId="8827" Text="@probabilityislogic same for me - Pitman gives a nice geometric explanation, which is why I referenced it in my answer, but I can't ever remember it when it counts :) When I TA'd a probability class we used this text and some students found it very helpful." CreationDate="2011-03-28T01:56:29.650" UserId="26" />
  <row Id="14869" PostId="8855" Text="What do you think is the difference between the standard error of the model and the standard error of the data? As far as I know there is only one type of standard error (that which you get from your data). Please elaborate on why you think these two things are different." CreationDate="2011-03-28T02:08:48.453" UserId="2660" />
  <row Id="14870" PostId="8836" Score="1" Text="What is the question here?" CreationDate="2011-03-28T02:11:36.033" UserId="2660" />
  <row Id="14871" PostId="8861" Score="2" Text="It's the multivariate analog of the usual 'noninformative' prior; a more succint/modern way to write it would be $p(\mu, \sigma)\propto \prod_{i=1}^k 1/\sigma_i$ (with $\mu$ and $\sigma$ k-vectors). Like @probabilityislogic says, the $d$'s are differentials and come from the proper definition of a pdf. I think they ought to be typeset $d\mu_i$. Pop in a one for them and you should be fine :) You're unlikely to see that notation in either of those texts, I think." CreationDate="2011-03-28T02:13:10.930" UserId="26" />
  <row Id="14872" PostId="8448" Text="@chl. Sorry for being late in responding. The function `cut2` is in the `Hmisc` library." CreationDate="2011-03-28T03:40:24.053" UserId="3774" />
  <row Id="14873" PostId="8448" Text="@SabreWolfy, The counts for each of the age classes will depend on how they are binned, thus the differences between Aniko and my answers." CreationDate="2011-03-28T03:44:00.260" UserId="3774" />
  <row Id="14875" PostId="8861" Text="@JMS thanks for the explanation" CreationDate="2011-03-28T04:30:20.910" UserId="1381" />
  <row Id="14876" PostId="8448" Text="@Jim My fault! I didn't notice `cut2` :( (+1, btw)" CreationDate="2011-03-28T06:52:43.210" UserId="930" />
  <row Id="14877" PostId="8783" Text="@bayer: I had read these as equally probable values in my answer below.  I now see they might be probabilities: before the question was changed, they did add up to 1. That would make a difference.  " CreationDate="2011-03-28T07:17:54.553" UserId="2958" />
  <row Id="14878" PostId="8818" Score="1" Text="@Martin08 I personally like this very concise description, by J.K. Vermunt: [Latent Profile Model](http://spitswww.uvt.nl/~vermunt/ermss2004f.pdf)." CreationDate="2011-03-28T07:43:40.657" UserId="930" />
  <row Id="14879" PostId="8787" Text="Thanks chl. Any help regarding my question." CreationDate="2011-03-28T09:31:18.317" UserId="3903" />
  <row Id="14880" PostId="8859" Score="1" Text="If the only two categories are &quot;spam&quot; and &quot;not spam&quot; then their probabilities must add to 1.  So this means you have $p(spam|D)=1-p(non-spam|D)$, now you have $log(\frac{p(spam|D)}{1-p(spam|D)})&gt;C$.  This can be re-arranged to give $p(spam|D)&gt;\frac{exp(C)}{1+exp(C)} $ (assuming your log is natural log, otherwise replace $exp(C)$ with $b^C$ where b is base of the log).  The two big numbers are basically 1 (excel can't tell the difference) - and the $1.68025$ gets a probability of about $84$%." CreationDate="2011-03-28T09:39:53.527" UserId="2392" />
  <row Id="14881" PostId="8660" Text="@whuber - the short answer is that I'm not sure.  But it is one of the drawbacks of this method that it merely goes with the most accurately estimated data, and not the most recent data.  My thinking would be something along the lines of the Kalman Filter/state space models, where the $\beta$ vector follows its own process with some sort of drift.  I don't think the Kalman filter requires extra storage (apart from the additional model equations), or maybe it would be $3p(p+1)$ (need to store 2 covariance matrices I think, one for observation and one for state)." CreationDate="2011-03-28T10:01:37.743" UserId="2392" />
  <row Id="14882" PostId="8868" Score="5" Text="Why do you want the p-value? Significance tests for the coefficients of an AR model are not particularly helpful as significance is not a good way to select the model order. Use the AIC instead." CreationDate="2011-03-28T10:08:27.527" UserId="159" />
  <row Id="14883" PostId="8870" Text="Many thanks! I wrote it like this... But to my surprise that almost all the parameters are insignificant... But in SAS it says that they are significant... So I doubt if there is any error in my programming words...." CreationDate="2011-03-28T10:28:45.863" UserId="3926" />
  <row Id="14884" PostId="8870" Text="what I wrote:    t = rep(0,5)&#xA;std = rep(0,5)&#xA;pvalue = rep(0,5)&#xA;nobs = 369&#xA;npara = 5&#xA;for (i in 1:5){&#xA;std[i] = sqrt (fit$var.coef[i,i])&#xA;t[i] = fit$coef[i]/std[i]&#xA;pvalue[i] = 1 - pt(t[i],nobs-npara)&#xA;}   " CreationDate="2011-03-28T10:31:24.253" UserId="3926" />
  <row Id="14886" PostId="8651" Text="there are lots of literature that imply this, for example, http://www.cesma.usb.ve/~lbravo/co6111/ModellingCopulas.pdf, check section 3. but i will really really appreciate if you can help me with suggestion on how to form a joint weibull distribution that accounts for the correlation between the marginals. right now, I am looking at the possibility of using copulas. thanks so much for looking." CreationDate="2011-03-28T11:24:49.520" UserId="3854" />
  <row Id="14887" PostId="8871" Text="@Jeffrey04 pls let me know if pLSA and pLSI are different." CreationDate="2011-03-28T11:25:36.197" UserId="1307" />
  <row Id="14888" PostId="8871" Score="1" Text="@suncoolsu, I edited the latex code for better readability. If it is not ok, please feel free to rollback." CreationDate="2011-03-28T11:53:55.183" UserId="2116" />
  <row Id="14889" PostId="8852" Text="@bayer, it seems that the OP is doing exactly this, i.e. fitting the MLE, only in excel." CreationDate="2011-03-28T11:56:56.910" UserId="2116" />
  <row Id="14890" PostId="8855" Text="@Felipe, indeed it is not obvious what are you doing. You wrote about linear _regression_ (??) model first. Therefore you get standard deviation that is related to the estimated parameter, but is actually based on the properties of your explanatory variable(s). However, from the next sentence it seems you are just looking for differences in the weekly means, isn't it? It is another statistical problem, but not directly related to linear regression. So, please, clarify your question.  " CreationDate="2011-03-28T11:58:28.907" UserId="2645" />
  <row Id="14891" PostId="8871" Text="@mpiktas. Thank you; I did not know how to format LaTeX code here. You are a great moderator!" CreationDate="2011-03-28T12:11:35.283" UserId="1307" />
  <row Id="14892" PostId="8871" Text="@suncoolsu, it is actually not different from the usual latex code. The only difference that slashes need to be doubled sometimes. You can view the source of any latex formula by right clicking on it by the way." CreationDate="2011-03-28T12:14:33.480" UserId="2116" />
  <row Id="14893" PostId="8863" Text="Thank you so much for this thorough post (and accent on R)." CreationDate="2011-03-28T12:38:34.527" UserId="1356" />
  <row Id="14894" PostId="8867" Text="I had a try at it myself, but it is beyond my abilities.  Absolute value functions ruin everything!  You are basically stuck with numerical methods." CreationDate="2011-03-28T12:40:47.200" UserId="2392" />
  <row Id="14895" PostId="8774" Text="@chl, my question is this, in books, when they explain Cochrane and Armitage test for independency (for trend) they say, for recessive model x=0.0 for codominant (additive)=1.0 and for dominant x=0.5... It is variable in the input of the independence.test function. Why do we assume these values? How does it affect the outcome ? Which value to assume if I don't know yet what kind of interaction there is between alleles. Yes it was a binomial phenotype." CreationDate="2011-03-28T12:56:21.157" UserId="3870" />
  <row Id="14896" PostId="8780" Text="Thank you very much. Naturally more questions follows then: 1) What happens if I ignore the PCA and stratify my GWAS sample only by GENDER / RACE / AGE and ignore PCA. How will it reflect the my association analysis and its result ? 2) If I in fact want to use PCA how many SNPS i have to have genotyped at least to have truthful PCA ? Is 200 enough ? Do they have to be evenly dispersed over all chromosomes ? 3) Which SNPs are used in PCA ? Is this predefined set or any ?" CreationDate="2011-03-28T12:58:32.183" UserId="3870" />
  <row Id="14897" PostId="8777" Text="@onestop, what if only stratify by gender / race ? can you elaborate on your answer please ?" CreationDate="2011-03-28T13:01:55.503" UserId="3870" />
  <row Id="14898" PostId="8774" Text="@suprvisr Ah, I think I understand now. So, x={0,1/2,1} would be the numerical scores corresponding to {AA,AB,BB} (with B=minor allele) for an additive model. I will provide an example later." CreationDate="2011-03-28T13:19:32.347" UserId="930" />
  <row Id="14899" PostId="8780" Text="@suprvisr I can answer right there or update my response, but I think it is better to ask a new question (sth along the idea of &quot;pros and cons of adjusting with PCA vs. stratifying&quot;) and link to this one so that people can clearly make the necessary connections. " CreationDate="2011-03-28T13:22:28.727" UserId="930" />
  <row Id="14900" PostId="8847" Text="@B_Miner This pretty much hits the nail on the head.  I'm trying to figure out to apply Machine Learning techniques to time series analysis." CreationDate="2011-03-28T13:30:40.433" UserId="2817" />
  <row Id="14902" PostId="8873" Score="1" Text="I assume GWAS is a genome-wide association study, but what is MAF?" CreationDate="2011-03-28T13:36:54.040" UserId="495" />
  <row Id="14903" PostId="8809" Text="I'm accepting this as &quot;Yes, you can, provided that your conditional likelihood is the same as the binomial distribution likelihood&quot;. I'm not sure how you would set up that test, but that's reaching beyond the scope of my original question." CreationDate="2011-03-28T13:40:49.833" UserId="3306" />
  <row Id="14905" PostId="8873" Text="@Simon MAF stands for minor allele frequency." CreationDate="2011-03-28T14:10:28.783" UserId="930" />
  <row Id="14906" PostId="8659" Text="Thanks for this informative answer. But I don't have sufficient information to plug in those numbers to make use of the equation." CreationDate="2011-03-28T14:17:46.170" UserId="3847" />
  <row Id="14908" PostId="8810" Text="@andresmh please take the time to provide a dataset to work with in the future, as per Chase's suggestion" CreationDate="2011-03-28T14:37:18.020" UserId="1381" />
  <row Id="14910" PostId="8660" Text="@probability I agree; a Kalman filter is a good idea to consider.  We don't know enough about this particular problem to go any further in this direction, though." CreationDate="2011-03-28T15:00:53.857" UserId="919" />
  <row Id="14912" PostId="8848" Text="@GaBorgulya, I disagree but appreciate the help on the calculation. That is why I originally just added qplot to your answer and deleted my own." CreationDate="2011-03-28T15:10:30.857" UserId="1381" />
  <row Id="14913" PostId="8424" Text="Can this also be done for pairwise.t.test? The structure doesn't accept any subset command. I had to try a roundabout method of re-defining the variables to include only data from &quot;low&quot; speed but I'm not sure I believe the results from the t-test. " CreationDate="2011-03-28T16:27:22.330" UserId="3761" />
  <row Id="14914" PostId="8878" Text="Well... AFAICT homoscedascity is not so problematic, but normality is. I used `shapiro.test`, `lillie.test` and `ad.test`, and $p &lt; 0.01$. Density plot shows clear bi/multimodality and positive skewness." CreationDate="2011-03-28T16:36:35.787" UserId="1356" />
  <row Id="14916" PostId="8867" Text="@probability You can obtain an expression for the integrand simply by splitting the integral into regions $\mu \ge 0$ and $\mu \lt 0$; no absolute values are needed.  But the result is a messy rational function of $x$, $exp(-x^2)$, and error functions, and so is unlikely to be integrable in closed form." CreationDate="2011-03-28T17:23:59.320" UserId="919" />
  <row Id="14918" PostId="8879" Score="1" Text="These questions look like standard questions from random matrix theory, surely it is possible to find reference with the answers? " CreationDate="2011-03-28T18:22:12.107" UserId="2116" />
  <row Id="14919" PostId="8809" Text="Although the explanation of answer involved conditional likelihood, i intended my answer to your question &quot;would the typical fair coin tests still be valid?&quot; to be an *un*conditional **yes**!" CreationDate="2011-03-28T19:08:33.423" UserId="449" />
  <row Id="14921" PostId="8777" Text="Not entirely sure what style of elaboration you're after but to deal with categorical covariates, $t$-tests generalize to ANOVA and chi-squared tests generalize to log-linear modelling of contingency tables or logistic regression." CreationDate="2011-03-28T19:17:10.827" UserId="449" />
  <row Id="14922" PostId="8881" Score="2" Text="Please provide more details. What have you tried, what are you trying to model? Also define best." CreationDate="2011-03-28T19:35:18.020" UserId="2116" />
  <row Id="14923" PostId="8881" Score="2" Text="Edwin: Some more information would help people answer your question.  What is the level of measurement of your dependent variable?  What are the level(s) of measurement of your independent variable(s).  How many of each I and D vars are there?  What software are you using?" CreationDate="2011-03-28T19:35:51.623" UserId="485" />
  <row Id="14924" PostId="8881" Text="Also include what the actual IVs and DVs _are_." CreationDate="2011-03-28T19:49:54.527" UserId="2660" />
  <row Id="14925" PostId="8884" Text="Could you give more details for your second question? How do you define selection of distribution at random with probability determined by measure $\mu$?" CreationDate="2011-03-28T20:20:50.710" UserId="2116" />
  <row Id="14926" PostId="8747" Text="@charles - Can you supply some real-type examples of the columns?" CreationDate="2011-03-28T20:21:45.993" UserId="3489" />
  <row Id="14927" PostId="8884" Text="@John, I've introduced latex and changed some sentences, please check if this is ok." CreationDate="2011-03-28T20:26:06.523" UserId="2116" />
  <row Id="14928" PostId="8884" Text="@John, what constitutes &quot;pretty strong&quot; convergence? Are you asking about bounds?" CreationDate="2011-03-28T20:27:12.373" UserId="2116" />
  <row Id="14929" PostId="8885" Text="@Tomek, in microeconomics you can show that if ordering satisfies certain conditions there exists a real convex function $p$ which for $u \le v$ gives $p(u)\le p(v)$. Using this result then the question of ordering the unknown set given the known set is the question of fitting appropriate function for the known set. Did you pursue this line of attack? " CreationDate="2011-03-28T20:32:51.013" UserId="2116" />
  <row Id="14930" PostId="8886" Text="@onestop (+1) I also have this one in my list, [Power analysis for genome-wide association studies](http://www.biomedcentral.com/1471-2156/8/58)." CreationDate="2011-03-28T21:12:54.607" UserId="930" />
  <row Id="14931" PostId="8885" Score="1" Text="@mpikat: I'm not aware of this theorem, could You provide some more details about those conditions? I was thinking about a linear transformation $f:R^{n}\to R$, but it is not obvious for me how to estimate parameters of such transformation." CreationDate="2011-03-28T21:32:41.843" UserId="1643" />
  <row Id="14933" PostId="8867" Text="@whuber - that is what I meant by &quot;hopeless&quot;.  Not that the integral is impossible, but the fisher information is impossible.  Because you have to take the expected value over $X$ of a ratio of two of these types of integral" CreationDate="2011-03-28T22:02:24.903" UserId="2392" />
  <row Id="14935" PostId="8885" Text="@Tomek If the ordering information is error-free and you seek a linear transformation, then you have a linear program.  Solving it (e.g., use the Simplex Method) will typically give an entire polytope of possible solutions (or else prove the data are inconsistent).  To obtain a unique solution, then within that polytope you could, say, easily minimize the variance of the coefficients of the linear transformation or otherwise optimize some convex function of them.  BTW, when such a transformation exists, then the order must be total." CreationDate="2011-03-28T22:52:33.497" UserId="919" />
  <row Id="14936" PostId="8877" Score="1" Text="John Tukey describes median polish, with many examples and exercises, in his EDA book.  It's easily carried out by hand (although a spreadsheet helps for doing the subtractions correctly :-).  It's important that there not be too many missing data (empty cells) in the array, for then median polish can fail to converge and can be pretty biased.  Otherwise, there are no other requirements.  The proof of the pudding is in the analysis of the residuals.  Tukey describes several very clever ways to extract additional information (e.g., interactions, transformations) from them." CreationDate="2011-03-28T22:57:23.757" UserId="919" />
  <row Id="14937" PostId="8878" Score="4" Text="@aL3 Normality of the data is irrelevant: normality of the *residuals* is what matters in ANOVA." CreationDate="2011-03-28T22:58:35.100" UserId="919" />
  <row Id="14938" PostId="8877" Text="You should provide this as an answer instead of a comment." CreationDate="2011-03-28T23:02:42.070" UserId="1356" />
  <row Id="14939" PostId="8893" Text="My answer assumes that you're doing typical cognitive science work where you have repeated measurement within each of multiple participants and therefore need to model participants as a random effect. If you somehow don't need to specify random effects, then use gam() from the mgcv package (http://cran.r-project.org/web/packages/mgcv/index.html)." CreationDate="2011-03-28T23:14:49.280" UserId="364" />
  <row Id="14940" PostId="8888" Text="Maybe the original poster is interested in quantifying the phenomenon." CreationDate="2011-03-28T23:15:58.813" UserId="364" />
  <row Id="14942" PostId="8883" Text="What you're describing above in a round-about way is a Poisson process with rate $\theta$." CreationDate="2011-03-28T23:44:13.047" UserId="2970" />
  <row Id="14943" PostId="8826" Text="It's worth adding that this prior is improper." CreationDate="2011-03-29T00:09:09.000" UserId="2970" />
  <row Id="14944" PostId="8875" Text="erm... i meant knowing bayes theorem and basic probability rules do not seem enough to be able to derive the formula... sorry for the confusion" CreationDate="2011-03-29T01:39:36.733" UserId="3837" />
  <row Id="14945" PostId="8871" Text="@suncoolsu pLSA/pLSI should be similar, it is just that IR community prefers the term pLSI iirc" CreationDate="2011-03-29T01:40:28.433" UserId="3837" />
  <row Id="14946" PostId="8809" Text="@onestop, thank you. " CreationDate="2011-03-29T01:40:32.277" UserId="3306" />
  <row Id="14947" PostId="8875" Text="@Jeffrey04 No worries; hopefully between @suncoolsu's answer and mine it's clearer." CreationDate="2011-03-29T02:07:28.920" UserId="26" />
  <row Id="14948" PostId="8882" Text="This is not my research. The authors kept the first data point in and I was wondering why instead of doing a post-hoc analyses using a Tukey HSD they used some other technique that took into account the outlier." CreationDate="2011-03-29T02:48:02.547" UserId="2660" />
  <row Id="14949" PostId="8871" Text="erm... why is there a summation sign when you derive from P(w,d) to P(w,z,d)?" CreationDate="2011-03-29T03:18:43.297" UserId="3837" />
  <row Id="14950" PostId="8887" Text="What do you mean by interesting and non-interesting cases?" CreationDate="2011-03-29T05:10:41.113" UserId="3485" />
  <row Id="14951" PostId="8845" Text="More than half the problem is that you have not correctly transcribed this expression!  The right hand side is $d\mu_1 d\mu_2 \cdots d\mu_k \frac{d\sigma_1 d\sigma_2 \cdots d\sigma_k}{\sigma_1 \sigma_2 \cdots \sigma_k}$: the $d$'s are differentials, not $d$'s with subscripts. See formula (2.1) at http://www.jstor.org/pss/2285344 ." CreationDate="2011-03-29T06:07:43.420" UserId="919" />
  <row Id="14952" PostId="8895" Score="1" Text="Hi! Thanks for the hint! I am trying to predict the prices for different objects based on the properties they have (e.g. size, location etc.) independent of the time the sale took place, so it's no time series." CreationDate="2011-03-29T06:13:32.367" UserId="3933" />
  <row Id="14953" PostId="8883" Text="Well, that's true. There is a close relationship the gamma distribution and Poisson processes. But I don't see how that helps me." CreationDate="2011-03-29T06:15:54.177" UserId="3932" />
  <row Id="14954" PostId="8249" Text="because there is no linear combination of the dimensions but one that depends on a kernel function. Is it in this setting possible to determine the weight one dimension has for a (non-linear) principal component?" CreationDate="2011-03-29T06:24:59.813" UserId="3683" />
  <row Id="14955" PostId="8800" Score="1" Text="@berkay: I have some questions (just to be sure). 1. You have 5000 records, 13 features (12 predictors + 1 response) and the response variable is binary with values &quot;flower&quot; (15 times) and &quot;not flower&quot;(5000-15 times) =&gt; is this correct ?. 2. I do not understand experiments number 2. Can you try to explain it again ? 3. Do you mean that 1 and 2 did not differ for C4.5 and NB *respectively* or do you mean that the results of NB and C4.5 did not differ for 1 and 2 *respectively* ?" CreationDate="2011-03-29T06:29:10.763" UserId="264" />
  <row Id="14956" PostId="8435" Text="@Mav Here are two review papers that I hope will help you sort out the terminology and possible applications for GLMM: [Generalized Linear Mixed Models](http://www.uic.edu/classes/bstt/bstt513/GLMM.pdf), [Statistical inference in generalized linear mixed models: A review](http://ppw.kuleuven.be/okp/_pdf/Tuerlickx2006SIIGL.pdf) -- the second one is a bit more technical as it mainly deals with estimation methods." CreationDate="2011-03-29T06:30:14.367" UserId="930" />
  <row Id="14957" PostId="8862" Text="... and that you have soft responsibilities instead of hard assignments." CreationDate="2011-03-29T06:54:05.110" UserId="2860" />
  <row Id="14958" PostId="8899" Text="if $P(B)=1$ then $P(B|A)$ should be $1$ also (except, maybe if $P(A)=0$) ? " CreationDate="2011-03-29T07:22:58.910" UserId="223" />
  <row Id="14959" PostId="8899" Text="$P(B)=1$,  $P( \lnot B)=0$, $P(B|A)=1$, $P(B|\lnot A )$ ? $P(A)$?" CreationDate="2011-03-29T07:28:44.563" UserId="3681" />
  <row Id="14960" PostId="8894" Text="@I'm satisfied with the solution for the example problem, but the question is how to find such ordering in $R^{1000}$? It is not said that there will exist one attribute that You can use to order vectors. It might be some a linear combination of all attributes or such transformation might even not exist. The question is how to find a transformation that will minimize inversions on the training data?" CreationDate="2011-03-29T07:40:36.893" UserId="1643" />
  <row Id="14961" PostId="8885" Text="@whuber: Your solution seems to be good. Is it some well known method or You just figured it out ad hoc? I'm asking in context of some references." CreationDate="2011-03-29T07:43:08.063" UserId="1643" />
  <row Id="14962" PostId="8826" Text="But this is only the prior, the posterior is always proper, as $X\geq 1$" CreationDate="2011-03-29T08:05:12.440" UserId="2392" />
  <row Id="14963" PostId="8887" Text="@Ashok, interesting cases are where condition of infinite smallness is satisfied. Naturally you can consider that the interesting cases are where this condition is not satisfied." CreationDate="2011-03-29T08:07:22.097" UserId="2116" />
  <row Id="14965" PostId="8899" Score="1" Text="somehow the question feels incomplete, have you posted all the probabilities?" CreationDate="2011-03-29T08:40:29.427" UserId="3837" />
  <row Id="14966" PostId="8871" Text="@jeffrey. think about it as integrating it out. as $z$ is discrete so you want to sum it out." CreationDate="2011-03-29T09:24:27.943" UserId="1307" />
  <row Id="14967" PostId="8887" Text="Thanks mpiktas. Good answer." CreationDate="2011-03-29T10:49:14.157" UserId="3485" />
  <row Id="14968" PostId="8898" Score="2" Text="Good first question! I'm not convinced each person's three measurements on the same specimen can really be treated as independent though. Did they know (or were they likely to realise) that the specimen was the same? Did they know the 'true' results when they made the measurements? Also do you know how many of the 'true' results were positive and how many were negative?" CreationDate="2011-03-29T12:06:35.557" UserId="449" />
  <row Id="14969" PostId="8911" Score="2" Text="Almost any environment that includes a decent random number generator, I would have thought?! A little more info on the intended application might help elicit a more helpful answer: MC simulations of what?" CreationDate="2011-03-29T12:09:17.830" UserId="449" />
  <row Id="14970" PostId="8424" Text="I was able to get around it by using &#xA;i&lt;-factor==&quot;low&quot;&#xA;pairwise.t.test(animals$feature[i], anotherfactor[i], p.adjust.method = &quot;holm&quot;, pool.sd = TRUE, paired = FALSE, data=animals)&#xA;&#xA;but if you have a better suggestion I'd appreciate it. Thanks!" CreationDate="2011-03-29T12:20:14.570" UserId="3761" />
  <row Id="14971" PostId="8911" Score="3" Text="You can look at my answers, whenever they contain Monte Carlo simulations, they were done on Mac OS X using [R](http://www.r-project.org)." CreationDate="2011-03-29T12:28:03.817" UserId="2116" />
  <row Id="14973" PostId="8888" Text="@John: Your answer is useful from a conceptual point of view, but I want to know about statistical tests for phenomena like the data shown." CreationDate="2011-03-29T12:57:57.727" UserId="2660" />
  <row Id="14974" PostId="8424" Text="@Alexis You can subset a data.frame at any time, for instance: `df &lt;- data.frame(y=rnorm(20), g1=gl(2, 10, labels=letters[1:2]), g2=gl(5, 2, 20)); with(subset(df, g1==&quot;a&quot;), pairwise.t.test(y, g2))`." CreationDate="2011-03-29T13:12:32.997" UserId="930" />
  <row Id="14975" PostId="8915" Text="@GaBorgulya You can use [JAGS](http://www-ice.iarc.fr/~martyn/software/jags/) on OS X (it works with R too)." CreationDate="2011-03-29T13:15:53.760" UserId="930" />
  <row Id="14976" PostId="8908" Score="1" Text="[Cross-posting](http://www.reddit.com/r/MachineLearning/comments/gdtv3/critique_this_visual_taxonomy_of_pattern/) is not encouraged." CreationDate="2011-03-29T13:18:44.177" UserId="930" />
  <row Id="14977" PostId="8871" Text="@suncoolsu thanks" CreationDate="2011-03-29T13:23:24.723" UserId="3837" />
  <row Id="14978" PostId="8895" Text="Thanks for the nice writeup, I'm reading it now. I'm currently working on electricity usage based on outdoor temperature and think I have a nice linear model (two parts: outdoor temp below 55F and outdoor temp above 55F)... That said, you may want to summarize a point or two in your answer: CrossValidated policy discourages answers that mainly consist of links to sites other than your basic reference sites like Wikipedia." CreationDate="2011-03-29T13:24:35.177" UserId="1764" />
  <row Id="14979" PostId="8911" Text="CWized, since I thought it will become a dictionary of MC software." CreationDate="2011-03-29T13:29:05.000" UserId="88" />
  <row Id="14980" PostId="8566" Text="@ocram You were right, I have tried and results with test data indicates &quot;overfitting&quot;." CreationDate="2011-03-29T13:40:16.100" UserId="2170" />
  <row Id="14981" PostId="8918" Text="It is not clear what is your question. If you have set of regression models ranking them according to some criteria is trivial. The problem is producing the set of regressions. To do that you need some kind of data. No method can create regressions out of thin air." CreationDate="2011-03-29T13:59:04.597" UserId="2116" />
  <row Id="14982" PostId="8845" Text="@whuber thanks for pointing that out. The paper was pre-$\LaTeX$ and the Greek letters are just a bit smaller than the Latin ones.  " CreationDate="2011-03-29T14:10:56.107" UserId="1381" />
  <row Id="14983" PostId="8845" Text="fixed. This makes me really appreciate Donald Knuth." CreationDate="2011-03-29T14:16:32.640" UserId="1381" />
  <row Id="14984" PostId="8897" Score="1" Text="I have to disagree.  For someone who has already said s/he's feeling overwhelmed by the information out there, Harrell with his level of detail will surely make things worse." CreationDate="2011-03-29T14:17:39.437" UserId="2669" />
  <row Id="14985" PostId="8918" Score="2" Text="@mpiktas - I don't think your comment is on-point.  levon9 never said he wants to create regressions without data.  He's looking for macros or code that constitute shortcuts to doing the work of ranking according to some criterion.  This may be &quot;trivial&quot; in the sense of straightforward, but as I see it he's looking for a way to shorten the process." CreationDate="2011-03-29T14:22:40.790" UserId="2669" />
  <row Id="14986" PostId="8910" Text="Great! Thanks a lot. That got me going." CreationDate="2011-03-29T14:28:14.663" UserId="3943" />
  <row Id="14987" PostId="8918" Text="@rolando2, ok my comment about thin air is not appropriate. Still I think the question is too vague." CreationDate="2011-03-29T14:31:06.317" UserId="2116" />
  <row Id="14991" PostId="5442" Text="This is an old thread, but in case you read this: OLS with dummy variables and interactions does not borrow strength as the other techniques you mention, right? I have some data where I've broken my analysis into two parts and used two lm (R linear model) commands to model the two parts. I introduced a dummy variable to indicate the two parts, then used lm again on this &quot;unified&quot; model and the answers are close, but not the same. My question would be: is that answer &quot;better&quot;, or simply different because of the algorithm? " CreationDate="2011-03-29T15:13:00.760" UserId="1764" />
  <row Id="14992" PostId="8917" Text="Why are you mixing $n$ and $N$? The OP appears to be using $N$ as a fixed, known integer. Also neither $N X_i$ nor $X_i$ are uniformly distributed on $[0,M]$ conditional on the given sum. " CreationDate="2011-03-29T15:13:13.597" UserId="2970" />
  <row Id="14993" PostId="8923" Text="@inwit - you will get infinitely better answers if you provide a reproducible example, i.e. `dput(mydata)` into the question." CreationDate="2011-03-29T15:22:21.987" UserId="696" />
  <row Id="14994" PostId="8906" Text="So lets say  instead of $P(B)=1$ I observe $P( \lnot B)=1$, so this changes the whole thing as you say, How do you apply Bayes in this case, I will use $Pr(A | \lnot B)$ right?" CreationDate="2011-03-29T15:30:11.943" UserId="3681" />
  <row Id="14995" PostId="8906" Text="@darkcminor: You might previously have calculated $Pr(A| \lnot B) = \frac{Pr(\lnot B|A)Pr(A)}{Pr(\lnot B)} $ in much the same way." CreationDate="2011-03-29T15:33:50.513" UserId="2958" />
  <row Id="14996" PostId="8910" Text="no problem, I have found GPML a great peice of kit, I thoroughly reccomend it to anyone wanting to investigate GPs." CreationDate="2011-03-29T15:37:03.943" UserId="887" />
  <row Id="14997" PostId="8906" Text="So I will have something like:&#xA;when $P(\lnot B)=$1, I have  $P(\lnot A| \lnot T)= \frac{0.99*0.99}{1-0.0104} \approx  0.99040016$, and doing the same $P(\lnot A)=  \approx  0.99040016$ is this correct?" CreationDate="2011-03-29T15:41:50.263" UserId="3681" />
  <row Id="14998" PostId="8920" Score="1" Text="I was going to write a separate answer, but I like yours. I'd expand #6 to say to bring ALL data into R and don't be in a hurry to aggregate, bin, etc. I think one error us non-experts make is to force data into a form that we think we'll need, or that we've seen others do. Let the actual data and the real-world factors behind that data guide you instead of naive assumptions related to processing the data." CreationDate="2011-03-29T15:47:00.093" UserId="1764" />
  <row Id="14999" PostId="8885" Text="@Tomek It is merely one possible interpretation of your question where I have supplied some missing assumptions.  Linear programming and the Simplex Method are extremely well known, of course; Googling will answer any followup questions you might have about them." CreationDate="2011-03-29T15:54:23.067" UserId="919" />
  <row Id="15000" PostId="8905" Text="This answer seems to use unconventional notation.  What do these matrices mean?" CreationDate="2011-03-29T16:01:41.343" UserId="919" />
  <row Id="15001" PostId="8924" Text="Update. So far, I've found this paper addressing a problem similar to mine: http://www.maths.otago.ac.nz/home/downloads/david_fletcher/Fletcher_et_al_2005.pdf" CreationDate="2011-03-29T16:12:56.057" UserId="3945" />
  <row Id="15002" PostId="8904" Score="1" Text="It's in [the manual](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_model_sect035.htm): look below Figure 18.21." CreationDate="2011-03-29T16:13:14.297" UserId="919" />
  <row Id="15003" PostId="8917" Text="Woops. I wasn't used to typing $N$. I'll change that. However I do believe $X_i$ conditional on the sum is uniform $[0,M/N]$ where $M$ is the sum." CreationDate="2011-03-29T16:26:32.757" UserId="3834" />
  <row Id="15004" PostId="8921" Text="Aha! That's a good tip for the conjugate prior. I didn't realise that's what it could be used for." CreationDate="2011-03-29T16:28:52.897" UserId="3932" />
  <row Id="15006" PostId="8924" Text="I'm wondering whether this extremely simplified approximation would make sense, given that zeros form the absolute majority:&#xA;1) find the proportion of zeros in each subset. 2) assume that in the subset with the smallest number of zeros all zeros are true. 3) from each subset, remove the proportion of zeros equal to the proportion of zeros in the most &quot;zero-rich&quot; dataset. 4) run standard non-parametric stats on this modified dataset. " CreationDate="2011-03-29T17:10:56.600" UserId="3945" />
  <row Id="15007" PostId="8917" Text="No, the distribution is not uniform. I think you've likely misunderstood the order-statistic property of the Poisson process. A quick way to check is that $X_i$ definitely has positive probability outside that stated interval for each $i$ conditional on the total sum. You can also check the conditional mean via linearity and see that they don't match.  " CreationDate="2011-03-29T17:28:58.277" UserId="2970" />
  <row Id="15008" PostId="8800" Text="@steffen, i edited thanks for the great questions." CreationDate="2011-03-29T17:29:23.207" UserId="3270" />
  <row Id="15009" PostId="8895" Text="@Wayne: Point taken. Since you read the article I wonder if you can help me pick out the &quot;point or two&quot; that would have communicated my ideas on the misuse of Regression for the analysis of time series data. In my opinion this was one of those times that more than one sentence or a snippet was needed to communicate the ideas. Would you have me excerpt the entire article and paste it into my response so as to not reflect on non-Wiki sources." CreationDate="2011-03-29T17:38:37.350" UserId="3382" />
  <row Id="15010" PostId="8931" Text="@B_Miner, thanks very much for your answer, sorry don't have enough rating to vote you up... I'll have a look at the links! My only worry about conditional models is that they postulate that zeros cannot belong to the second (continuous) component, am I right? Does my setting not feel a bit more like a mixture model?  What do you think?" CreationDate="2011-03-29T17:42:18.820" UserId="3945" />
  <row Id="15011" PostId="8905" Text="Two by two tables of prior and posterior probabilities.&#xA;$\begin{smallmatrix}&#xA;&amp; \mathbf{A} &amp; \mathbf{\lnot A} \\&#xA;\mathbf{B} &amp; P(A \cdot B) &amp; P(\lnot A \cdot B) \\&#xA;\mathbf{\lnot B} &amp; P(A \cdot \lnot B) &amp; P(\lnot A \cdot \lnot B)&#xA;\end{smallmatrix}$&#xA;Sum of the elements is 1.&#xA;I'm not sure about the conventional notation, feel free to rewrite." CreationDate="2011-03-29T18:00:32.890" UserId="3911" />
  <row Id="15013" PostId="8918" Text="@mpiktas - Fair enough." CreationDate="2011-03-29T18:15:51.457" UserId="2669" />
  <row Id="15015" PostId="8934" Text="Why shouldn't it be a normal distribution? You are right about the scale. I added some clarification to my original question, maybe that helps." CreationDate="2011-03-29T18:32:35.997" UserId="1205" />
  <row Id="15016" PostId="8925" Text="Keep in mind that spreadsheet software have poor numerical accuracy, see [www.jstatsoft.org/v34/i04/paper](www.jstatsoft.org/v34/i04/paper) ." CreationDate="2011-03-29T18:46:55.190" UserId="3911" />
  <row Id="15017" PostId="8934" Score="1" Text="I you are sampling from a normal distribution the probability of getting two equal values is zero. If you use reasonable rounding, you may get a few equal values in a large sample." CreationDate="2011-03-29T18:55:49.423" UserId="3911" />
  <row Id="15018" PostId="3645" Text="do you have a function command for doing that?" CreationDate="2011-03-29T18:58:37.287" UserId="1050" />
  <row Id="15019" PostId="8931" Text="I've now replicated the two-stage approach proposed in the Gelman book. If subset_factor (with 25 levels) serves as subset label, the first step is fit1=glm(response~subset_factor, family=binomial); and the second step is fit2=lm(response~subset_factor, subset=response&gt;0). I can then run simulations as they describe to obtain the distribution of fitted response values for each factor level. However, I'm still unsure how to translate this to what I need, which is (a) the probability that coefficients aren't zero and (b) significance of difference between coefficients at different factor levels." CreationDate="2011-03-29T19:02:25.680" UserId="3945" />
  <row Id="15022" PostId="8925" Text="@GaBorgulya This is absolutely true, but if Ionuț needs a quick answer he might be willing to make the tradeoff in accuracy." CreationDate="2011-03-29T19:18:39.860" UserId="2817" />
  <row Id="15023" PostId="8917" Text="Indeed, I don't understand how it can be uniform. The distribution of $X_i$ is indeed exponential. But if you condition it on the sum it has to change, but I don't see how the result can be uniform." CreationDate="2011-03-29T19:20:21.463" UserId="3932" />
  <row Id="15024" PostId="8931" Text="The two stage approach (Gelman method of two seperate models) assumes two populations, those at zero and those above. " CreationDate="2011-03-29T19:25:10.930" UserId="2040" />
  <row Id="15025" PostId="8931" Text="...so would it be appropriate to simply say that if the impact of some factor level is significant (and significantly different from that of some other factor level) in either of the two models in Gelman's method then it's significant overall?  " CreationDate="2011-03-29T19:28:07.560" UserId="3945" />
  <row Id="15026" PostId="8917" Text=" The conditional distribution of each $X_i$ conditioned on the total sum is the same as the distribution of the *minimum* of $N-1$ i.i.d. uniform $[0,M]$ random variables and so has distribution function $1-(1-x/M)^{N-1}$ on $[0,M]$. Note that this distribution has mean $M/N$, as you should expect. There is a close connection to the beta distribution. " CreationDate="2011-03-29T19:29:17.623" UserId="2970" />
  <row Id="15027" PostId="8931" Score="1" Text="Yes, the two stage approach (Gelman method of two seperate models) assumes two populations, those at zero and those &gt; 0. Regarding the hypothesis tests, can you frame them in terms of the predicted values for varying levels of the inputs and construct empirical confidence intervals related to the simulations for each? For hypothesis tests for the coefficient != 0, you need to test this seperately for both models." CreationDate="2011-03-29T19:32:00.097" UserId="2040" />
  <row Id="15028" PostId="8939" Text="Thing is, I can't assume anybody rounded. The calculation of MOS is standardized (see above) and therefore I must take the CI using the data I have: &quot;The average of the opinion scores should be calculated for each test condition. Confidence limits should be evaluated.&quot;" CreationDate="2011-03-29T19:36:33.020" UserId="1205" />
  <row Id="15029" PostId="8940" Text="I was guessing that there's no other solution. As the ITU Rec says, I should just take the mean and compute CI: &quot;The average of the opinion scores should be calculated for each test condition. Confidence limits should be evaluated.&quot; - so I'll go with that. Thanks!" CreationDate="2011-03-29T19:38:00.053" UserId="1205" />
  <row Id="15030" PostId="8931" Text="thanks, this is helpful!" CreationDate="2011-03-29T19:38:45.443" UserId="3945" />
  <row Id="15031" PostId="8939" Score="1" Text="If an authority tells you what to do then you have little choice. People, however, very rarely think exactly the same. It's more that the 1-2-3-4-5 scale did not have the resolution to capture the variation among the opinions. You will get a confidence interval of zero width, understating the inaccuracy. If I had to follow the standardized calculation I would put a remark into my report about this methodical weakness." CreationDate="2011-03-29T20:01:26.060" UserId="3911" />
  <row Id="15032" PostId="8940" Text="@slhck There are other solutions.  For instance, you could construe the responses as rounded versions of continuous values.  If the rounding is to the nearest integer, use $\sqrt{1/12}$ in place of the standard deviation when all the values are the same.  (This also has its problems, but it at least acknowledges the potential for sample variation in the mean response.)" CreationDate="2011-03-29T20:28:19.423" UserId="919" />
  <row Id="15033" PostId="8890" Text="Thanks GaBorgulya. You have some great points. I read the documents of ACE and AVAS and they are quite interesting and useful. However, I don't see how some of your recommendations work when we have a model selectoin problem to solve before even knowing what regression problem we have at hand. Using the biology of the problem does not eliminate the model selection in this case since we have already done that and that's one reason many of the predictors are correlated. " CreationDate="2011-03-29T20:39:26.730" UserId="2020" />
  <row Id="15034" PostId="8880" Score="1" Text="@henry - you forgot to multiply the likelihood by the prior (or you multiplied by $\frac{1}{y}$ instead), and your likelihood has $n\frac{y^{n-2}}{\theta^n}$, when you have just written the density as $n\frac{y^{n-1}}{\theta^{n}}$.  your CI is based on $n-1$ observations, should replace $n-1\rightarrow n$ in your answer. (posterior should be $n\frac{y^n}{\theta^{n+1}}$ )" CreationDate="2011-03-29T20:40:18.997" UserId="2392" />
  <row Id="15035" PostId="8929" Text="@Chase I was about to suggest to choose a convenient common No. bins to pass to `stat_binhex()` (`bins` or `binwidth`), but facetting is definitively the way to go, so my +1!" CreationDate="2011-03-29T20:42:31.817" UserId="930" />
  <row Id="15036" PostId="6596" Text="@rob this is coming in handy. fyi: it appears that, although bw = &quot;SJ&quot; affects the density in untransformed space, the logdensity is the same using &quot;SJ&quot; and the default &quot;nrd0&quot;... I'm about to read the SJ reference: &quot;Sheather and Jones (1991) A reliable data-based bandwidth selection method for kernel density estimation.&quot; http://www.jstor.org/stable/2345597" CreationDate="2011-03-29T20:54:37.117" UserId="2750" />
  <row Id="15037" PostId="8917" Text="That's interesting, even if it's not the answer I was hoping for. But why the minimum? Do you have a reference?" CreationDate="2011-03-29T20:56:05.390" UserId="3932" />
  <row Id="15038" PostId="8904" Text="@whuber (+1) That is *the* response, isn't it?" CreationDate="2011-03-29T20:57:56.983" UserId="930" />
  <row Id="15039" PostId="8904" Text="@chl A *good* response would explain what the Bartlett kernel is and perhaps address the final question about adding more commands.  I put this in as a comment really hoping it would result in an improved, more specific question." CreationDate="2011-03-29T21:01:06.010" UserId="919" />
  <row Id="15040" PostId="8929" Text="@chl - thanks. I really take advantage of the `scales` argument in the facets frequently to display information that is consistent across one axis. For example when reviewing diagnostic plots from CBC HB models, I find it helpful to check convergence by reviewing the percent certainty, RLH, average variance, and parameter RMS values. All of these have different scales on the y-axis, but should converge and flatten out if you include enough iterations in your model run ( x-axis). It's pretty trivial to set up an R script to parse the log files from HB and generate these plots as a pdf or jpeg" CreationDate="2011-03-29T21:02:50.937" UserId="696" />
  <row Id="15041" PostId="8880" Text="@probabilityislogic: thank you - I had indeed used $\frac{1}{y}$ rather than $\frac{1}{\theta}$.  Better now I hope. " CreationDate="2011-03-29T21:04:17.150" UserId="2958" />
  <row Id="15043" PostId="8904" Text="@whuber That's a fair point, indeed." CreationDate="2011-03-29T21:10:28.797" UserId="930" />
  <row Id="15044" PostId="8931" Text="@msp: If you can fit the model shown in the link I supplied (SAS) in either SAS or R (I am sure there is an NLMIXED analog in R, especially since this model uses only fixed effects, it's just that SAS does not have any other procedure to write your own log-likelihood) you can accommodate a hypothesis test using P(Y|X) instead of separate hypothesis using the two models in the two-stage approach." CreationDate="2011-03-29T21:31:05.000" UserId="2040" />
  <row Id="15045" PostId="8931" Text="For example, I work in direct marketing and so want to use a model to test the differences between two direct mail versions. I am interested in the response rate difference (P(Y=0|X)) and the average revenue of those who order (P(Y|X; Y&gt;0)) yes, but at the end of the day I am concerned about the expected revenue value of those that received version A versus B and can not use a simple ANOVA because of the (large amount = those that did not order) zeroes. Again, don't know your domain but hope this helps." CreationDate="2011-03-29T21:31:57.013" UserId="2040" />
  <row Id="15046" PostId="8836" Text="@Subhabrata Please identify one specific question at a time and communicate it in a succinct and clear manner so we can understand and appreciate it.  You will then likely find there are many people who can provide useful responses.  In the meantime, it has to be closed as not being answerable in its current form." CreationDate="2011-03-29T21:45:04.287" UserId="919" />
  <row Id="15047" PostId="8895" Text="@Wayne: To the point you made about having two nice linear models, one for below 55F and one for over 55F. Anheuser-Busch used Transfer Function models to deal with the fact that beer consumption was not effected by temperatures under 65F but was effected by temperatures over 65F. If you want to discuss this one model which incorporated this structure, you can feel free to contact me." CreationDate="2011-03-29T21:47:09.260" UserId="3382" />
  <row Id="15048" PostId="8814" Score="2" Text="**Better yet**: for more on time series, check out the [top users for the &quot;time-series&quot; tag](http://stats.stackexchange.com/tags/time-series/topusers).  IMHO, it's best to pay attention to users who have relatively high votes per question answered.  (Currently an average of 4.0 or more votes per question would be pretty good.)" CreationDate="2011-03-29T21:59:04.073" UserId="919" />
  <row Id="15049" PostId="8646" Text="(This would be more appropriate as a comment than a reply.)" CreationDate="2011-03-29T22:00:47.253" UserId="919" />
  <row Id="15050" PostId="8599" Text="-1 This reply does not appear to respond to the question, which is looking for a &quot;well-explained ...*example*.&quot;" CreationDate="2011-03-29T22:02:32.430" UserId="919" />
  <row Id="15051" PostId="8597" Text="Doesn't &quot;predictable&quot; have to do with the relationship between a forecast and *reality* rather than with the coefficient of variation?  In the context of the question, a highly variable stock (high beta) that nevertheless trends upward rapidly (high alpha) would be considered &quot;valuable&quot; but a low-beta (&quot;predictable&quot;) stock that trends downwards consistently would be considered the opposite of &quot;valuable&quot;." CreationDate="2011-03-29T22:07:15.220" UserId="919" />
  <row Id="15052" PostId="8595" Text="-1 This sounds more like a self advertisement than a helpful reply." CreationDate="2011-03-29T22:08:18.120" UserId="919" />
  <row Id="15053" PostId="8923" Score="1" Text="You can specify the limits of the fill scale: `scale_fill_continuous(limits = c(0, 0.05))`" CreationDate="2011-03-29T22:26:21.040" UserId="46" />
  <row Id="15054" PostId="8947" Score="1" Text="This question might answer itself once you elucidate what it means to you, in this application, for a sample to &quot;represent&quot; a population.  For instance, in some applications you want to estimate the population mean from the sample; in others you want to estimate an extreme (such the chance of a flood or nuclear disaster); in others you want the sample to reproduce yet other aspects of the population.  In this instance, perhaps (hypothetically) you want the *eigenvalues* of the sample matrix to reflect those of the population matrix.  If so, that indicates a completely different analysis." CreationDate="2011-03-29T22:49:43.560" UserId="919" />
  <row Id="15055" PostId="8595" Text="@whuber: Not intentional. Just a reflection on a similar &quot;sticky problem&quot; that I ran into ." CreationDate="2011-03-29T22:50:43.903" UserId="3382" />
  <row Id="15056" PostId="8595" Text="@Irish That would make an interesting comment, then.  Comments are good ways to inject some of this valuable experience that we are so glad you share with us.  Replies should be reserved for actual responses to the question: they will be voted on, archived, made searchable, etc., and therefore need to be more directly relevant, of permanent value, and be able to withstand criticism better.  (Of course this is an idealization, but it's what we strive for. :-)" CreationDate="2011-03-29T22:55:39.997" UserId="919" />
  <row Id="15057" PostId="8599" Text="@whuber: The OP asked for an answer that &quot;uses minimal math&quot;. My response detailed minimal math and was motivated to explain ARIMA models in common everyday words. This is never done as the math theory guys focus on the &quot;high-end explanation&quot; using polynomial , differencing operators , non-linear optimization etc." CreationDate="2011-03-29T22:59:56.203" UserId="3382" />
  <row Id="15058" PostId="8898" Text="You're right to worry about the independence of the measurements, but I think they did a pretty good job designing the experiment to minimize any memory between measurements (they spread out the diagnoses over several sessions and would only run a randomized subset of specimens on a single instrument during a session).  They did not know the true results at the time.  I can find out what the positive result rate was, but I think it's roughly 50%." CreationDate="2011-03-29T23:00:13.707" UserId="3939" />
  <row Id="15059" PostId="8599" Text="@Irish I agree with the motivation to keep the math down, especially when requested by the user.  But this reply seems to answer a different question: &quot;what is ARIMA&quot;.  The specific nature of the original question also indicates the OP has a good idea of what ARIMA is and what it's good for; they want to *see it in action.*  I bet you could easily contribute such a case study :-)." CreationDate="2011-03-29T23:02:39.913" UserId="919" />
  <row Id="15060" PostId="8906" Text="Apart from the $T$ instead of $B$, this looks sensible.  You can also write this as $\frac{0.99\times 0.99}{0.99\times 0.99 + 0.01\times 0.95}$" CreationDate="2011-03-29T23:09:27.353" UserId="2958" />
  <row Id="15061" PostId="8595" Text="@whuber: You teach ! I learn ! I will reserve comments for the &quot;comment area&quot;. To reiterate there was absolutely no intention of promoting anything or any particular piece of software/consultancy  just a well-intentioned willingness to share experiences to other people of the village. I am quite sure that the OP thought my comments helpful. What say you Brett ?" CreationDate="2011-03-29T23:12:28.110" UserId="3382" />
  <row Id="15062" PostId="8599" Text=":whuber: That would have been very easy for me to do and I might just do that. " CreationDate="2011-03-29T23:15:10.730" UserId="3382" />
  <row Id="15063" PostId="8599" Text="@Irish I look forward to seeing it.  Moreover--this issue did not come up here, but it has come up in other places--such contributions are potentially more powerful, and more appreciated, ways of letting people know what you can do than many more overt forms of marketing are." CreationDate="2011-03-29T23:20:52.253" UserId="919" />
  <row Id="15064" PostId="8599" Text="&quot;whuber&quot; agreed !" CreationDate="2011-03-29T23:28:10.533" UserId="3382" />
  <row Id="15065" PostId="8939" Text="+1 Nice conceptual approach.  You don't need a simulation, though: you can directly compute the chance that 8 independent draws from a normal(0, $\sigma$) distribution all lie within $1/2$ of 0 and then find $\sigma$ for which that chance is fairly high; 95%, say (solution: .183351)." CreationDate="2011-03-29T23:29:23.493" UserId="919" />
  <row Id="15066" PostId="8918" Text="@rolando2 - thanks for your comment to mpiktas, you got it exactly right. I'll be the first to admit I am a newbie when it comes to statistics, but I didn't think you could do regression without data so I didn't specifically mention it. If one can, then I guess mpiktas has a point and I should have mentioned it." CreationDate="2011-03-29T23:35:16.723" UserId="3311" />
  <row Id="15067" PostId="8927" Text="@Chase Thanks! great pointers .. much appreciated." CreationDate="2011-03-29T23:36:34.023" UserId="3311" />
  <row Id="15068" PostId="8932" Text="@bill_080 Thank you, that looks very much along the lines of what I was trying to do. I'll have to dig into this some more." CreationDate="2011-03-29T23:38:50.400" UserId="3311" />
  <row Id="15069" PostId="8947" Score="1" Text="Could you be more specific about what you mean by 'data is from the entire population'? ANOVA is a method than can be used on samples from large populations. When you have the data of an entire population the methods to use are different." CreationDate="2011-03-30T00:45:59.857" UserId="3911" />
  <row Id="15070" PostId="8888" Text="Then I guess I'd have to ask what you mean by &quot;account for&quot; and &quot;significant&quot; in your question.  Perhaps you should edit it to make it more clear what conclusion might be drawn from the data that you don't want and what you would like to achieve.  My answer has general applications as well.  If a point is really far away from everything else it might be stochastic... or it might just be something that's very different." CreationDate="2011-03-30T01:17:07.743" UserId="601" />
  <row Id="15071" PostId="8917" Text="@kleptog, the result, though maybe not explicitly stated as such, is well-known via the characterizations of a Poisson process. Section 2.6 of Durrett, 3rd ed. quickly covers the Poisson process, but leaves this result as an exercise at the end. Resnick's book on stochastic processes has a more complete and general treatment. You can also just derive it directly by transformation of variables since $X_i$ and $\sum_{j\geq 2} X_j$ are independent and have known distributions. If you want, I can post the details in a separate answer because it won't fit here." CreationDate="2011-03-30T02:17:33.370" UserId="2970" />
  <row Id="15072" PostId="8597" Text="@whuber, yes that's something that I'm trying to create a model for..." CreationDate="2011-03-30T02:23:44.470" UserId="2664" />
  <row Id="15073" PostId="8942" Text="What does your model look like? (can you specify the canonical equations at least?) Is it discrete or continuous? Is your noise additive?" CreationDate="2011-03-30T04:09:17.213" UserId="2833" />
  <row Id="15074" PostId="8917" Text="Ah, thank you cardinal! I got confused with another similar formulation... [Here](http://robotics.caltech.edu/~zoran/Research/poisson/node1.html) , slightly before the half of the page was the theorem I thought I was referring to." CreationDate="2011-03-30T04:21:26.397" UserId="3834" />
  <row Id="15075" PostId="8952" Text="The 95% comes from the desired confidence level, so it has already been specified in the problem.  The burden on anyone suggesting an alternative confidence interval procedure is to demonstrate that their proposal guarantees 95% (or whatever is desired) coverage of the true mean--no matter what the true mean might happen to be--and that it does so efficiently: that is, its minimum coverage really is 95% or close to it.  Although your proposal might have this property, it's not apparent that it does." CreationDate="2011-03-30T04:35:53.483" UserId="919" />
  <row Id="15076" PostId="8917" Text="Thanks both of you for the explanations and references. @cardinal: I'm going to see if I can work it out. Unfortunately I can't upvote a comment." CreationDate="2011-03-30T07:00:06.353" UserId="3932" />
  <row Id="15077" PostId="8931" Text="I work in biology, but yes - mine is a very similar problem. Will try to find an r implementation of the SAS code you mentioned. I'm also wondering if the r package cozigam is doing a similar thing. Will investigate." CreationDate="2011-03-30T07:06:10.593" UserId="3945" />
  <row Id="15078" PostId="8950" Text="Thanks! I'm wondering if you're aware of an existing r implementation of this?" CreationDate="2011-03-30T07:07:26.660" UserId="3945" />
  <row Id="15079" PostId="8927" Text="(+1) I would suggest the same myself." CreationDate="2011-03-30T07:21:41.733" UserId="2645" />
  <row Id="15080" PostId="8938" Text="(+1). Additional remark: Even the parameters of the weak learners to not differ, the behavior (i.e. what they predict) changes when the structure of the dataset (as done in boosting) changes." CreationDate="2011-03-30T09:09:55.900" UserId="264" />
  <row Id="15081" PostId="8744" Text="[k-means-algorithm-variation-with-equal-cluster-size](http://stackoverflow.com/questions/5452576/k-means-algorithm-variation-with-equal-cluster-size) is the same question on stackoverflow, with an answer using a greedy assignment algorithm." CreationDate="2011-03-30T11:04:06.120" UserId="557" />
  <row Id="15082" PostId="8955" Score="1" Text="@user3136, please clarify is your main concern heterogeneity (different species with unique attributes, probably taken from known distribution) or heteroscedasticity (the property of random process to have time varying variance), as these two concepts are clearly different. Both problems are interesting though, the former leads to mixed effects models or random coefficient models, the latter has many cures to deal with (but is less important, since $OLS$ just is not efficient, but estimates are unbiased)." CreationDate="2011-03-30T11:04:18.007" UserId="2645" />
  <row Id="15083" PostId="8960" Score="1" Text="@klonq - when you say you have the histogram - do you mean also have the information about the observations or do you ONLY know the bin width and height?" CreationDate="2011-03-30T11:10:08.590" UserId="1307" />
  <row Id="15084" PostId="8960" Text="@suncoolsu I have all data points. Datasets ranging from 5,000 to 50,000 records." CreationDate="2011-03-30T11:12:38.147" UserId="3579" />
  <row Id="15085" PostId="8959" Text="@lucy can you pls be bit more detailed? are you using `plot`?" CreationDate="2011-03-30T11:13:02.597" UserId="1307" />
  <row Id="15086" PostId="259" Text="There isn't anything *really* special about the SVM, other than it forces the user to think about regularisation.  For most practical problems [kernel] ridge regression works just as well." CreationDate="2011-03-30T11:36:42.810" UserId="887" />
  <row Id="15087" PostId="8959" Text="@lucy Just set 'axes=FALSE' when you call '(box)plot' and then set your custom axes with 'axis' (look around the 'at' and 'labels' params)." CreationDate="2011-03-30T11:38:33.387" UserId="930" />
  <row Id="15089" PostId="8959" Text="i am using:&#xA;boxplot(ucenie~hral,names=c(&quot;hral&quot;,&quot;nehral&quot;))    &#xA;i get http://www.freeimagehosting.net/image.php?7dc443a69f.jpg&#xA;i need to replace -2,-1,0,1,2 with 0hodin,1hodina,etc&#xA;then I type:&#xA;boxplot(ucenie~hral,names=c(&quot;hral&quot;,&quot;nehral&quot;),axes=FALSE)&#xA;&gt; lablist.x=as.vector(c(&quot;nehral&quot;,&quot;hral&quot;))&#xA;&gt;lablist.y=as.vector(c(&quot;0hodin&quot;,&quot;0.5=1hodina&quot;,&quot;1.5-2hodiny&quot;,&quot;2-3hodiny&quot;,&quot;nad 3hodiny&quot;))&#xA;&gt; axis(1,at=seq(1,2,by=1),labels=FALSE)&#xA;&gt; text(x=seq(1,2,by=1),labels=lablist.x,pos=1)&#xA;&gt; axis(2,at=seq(1,100,by=25),labels=FALSE)&#xA;&gt; text(y=seq(1,100,by=25),lables=lablist.y,pos=2)&#xA;but it doen´t work correctly" CreationDate="2011-03-30T12:05:04.093" UserId="3959" />
  <row Id="15090" PostId="8961" Score="5" Text="@schenectady: With all due respect, pls post one liners like these as comments instead of answers. May be I am an exception, IMHO, it increases noise on this forum." CreationDate="2011-03-30T12:09:59.313" UserId="1307" />
  <row Id="15091" PostId="8967" Text="@GaBorgulya (+1) Obviously, this will also work for the y-axis, as requested by the OP :-)" CreationDate="2011-03-30T12:17:24.680" UserId="930" />
  <row Id="15092" PostId="259" Text="@dikran i think SVM is a great classifier because it is sparse and robust to outliers -- this is not true for Logistic Regression! and thats why SVM is state-of-the-art classifier. Only issue which may be a problem is -- time complexity -- but i think its ok." CreationDate="2011-03-30T12:17:46.793" UserId="1307" />
  <row Id="15093" PostId="259" Text="@suncoolsu If you want sparsity, you get more sparsity from regularised logistic regression with LASSO than you do with the SVM.  The sparsity of the SVM is a by-product of the loss function, so you don't get as much as you do with an algorithm where sparsity is a design goal.  Also often with the optimal value of the hyper-parameter (e.g. chosen via cross-validation) most of the sparsity of the SVM dissapears.  SVM is no more robust to outliers than regularised logistic regression - it is mostly the regularisation that matters, not the hinge loss." CreationDate="2011-03-30T12:22:49.263" UserId="887" />
  <row Id="15094" PostId="259" Text="@Dikran -- my point exactly -- some kind of penalization is important. You can get that using Priors, adding Penalty, etc." CreationDate="2011-03-30T12:29:22.917" UserId="1307" />
  <row Id="15095" PostId="259" Text="@suncoolsu In which case, the SVM isn't a great classifier, it is just one amongst many regularised classifiers, such as ridge regression, regularised logistic regression, Gaussian Processes.  The main benefir of the SVM is its appeal from computational learning theory.  In practice, other considerations are more important, such as whether you need probabilistic classifier, where other loss functions are likely to be superior.  IMHO, there is too much attention given to the SVM, rather than the wider family of kernel methods." CreationDate="2011-03-30T12:37:14.233" UserId="887" />
  <row Id="15096" PostId="8967" Text="@chl Thanks, but unfortunately this does not work for the y axis :-(" CreationDate="2011-03-30T13:03:28.213" UserId="3911" />
  <row Id="15097" PostId="8965" Text="One thing I would be cautious of is that it sounds like we have time-series data here. If the data are sampled over a short time frame, assuming independence could be hazardous. That said, (+1). " CreationDate="2011-03-30T13:09:29.213" UserId="2970" />
  <row Id="15098" PostId="8938" Text="MaxEnt is logistic regression. Visualizing how decision boundaries evolve would actually be quite useful. How can one go about doing that? Also, I still don't understand how AdaBoost still works when it's only given one feature set and one classifier type (as in MALLET). To me, this seems completely counter-intuitive." CreationDate="2011-03-30T13:25:25.753" UserId="3948" />
  <row Id="15099" PostId="8931" Text="I think I've found it: package gamlss provides model fitting based on several zero-inflated and adjusted continuous distributions. Thanks for your advice, B_Miner!" CreationDate="2011-03-30T13:38:55.790" UserId="3945" />
  <row Id="15100" PostId="8965" Text="@cardinal Please explain. The data ranges over the course of a month or up to a year, but sampled regularly (10 minutes). What might this imply?" CreationDate="2011-03-30T13:48:30.477" UserId="3579" />
  <row Id="15101" PostId="8965" Text="@cardinal Thanks for pointing it out. I wasn't sure either if independence assumption is appropriate. " CreationDate="2011-03-30T13:58:21.170" UserId="1307" />
  <row Id="15102" PostId="8965" Text="@klonq, how is the sample taken? Is it the average speed over the ten minutes between recordings? Over one minute prior to recording? The instantaneous speed at the time of recording? Mostly I'd be looking for serial correlations, which could reduce your effective sample size considerably. Using an ML estimate based on an assumption of independent samples may or may not still give you a good estimate in that context, and special care should be taken regarding any *inference* based on the estimate. Suncoolsu's approach definitely provides a first line of attack, though. " CreationDate="2011-03-30T14:06:38.557" UserId="2970" />
  <row Id="15103" PostId="8964" Text="Thanks, that sounds like a great example." CreationDate="2011-03-30T14:08:08.383" UserId="3136" />
  <row Id="15104" PostId="8955" Text="Hi, sorry about that. My concern really is about heteroscedasticty. One problem I have had is that these two terms (heteroscedasticity and heterogeneity) are used almost interchangeably. In this context , both are supposed to refer to the situation when the error in the residuals is not constant" CreationDate="2011-03-30T14:08:59.087" UserId="3136" />
  <row Id="15105" PostId="8963" Score="1" Text="HI, thanks for the advice. I am currently working on some simulations so I hope they work out. As far as I know heteroscedasticty does not affect estimation of regression coefficinets, but it can over- or under-estimate the standard errors of these estimates. " CreationDate="2011-03-30T14:11:49.223" UserId="3136" />
  <row Id="15106" PostId="8965" Text="@klonq -- If possible, can you please describe how was your sample collected? What does the data look like?" CreationDate="2011-03-30T14:13:05.917" UserId="1307" />
  <row Id="15107" PostId="8965" Text="@cadinal @suncoolsu see edited post." CreationDate="2011-03-30T14:23:58.700" UserId="3579" />
  <row Id="15108" PostId="8969" Text="@bill_080 Thanks, but I am trying to find a solution in Java." CreationDate="2011-03-30T14:40:52.757" UserId="3579" />
  <row Id="15109" PostId="8938" Text="Just plot your dataset. Additionaly, plot a grid where you pick a color for each point how it's being classified by your system. This works only good for 2D, I guess. You could still plot the most important axes or something.&#xA;&#xA;If you only have one feature set and one classifier, it reduces to only one classifier.&#xA;&#xA;I was under the impression, that maximum entropy was a paradigm and not a concrete classifier. Anyway." CreationDate="2011-03-30T15:15:02.320" UserId="2860" />
  <row Id="15110" PostId="8971" Text="This question looks incomplete or corrupted.  Feel free to re-post it once you have fully formulated it." CreationDate="2011-03-30T15:20:48.807" UserId="919" />
  <row Id="15111" PostId="8967" Text="@GaBorgulya Indeed, I wrote too quickly! Maybe you could add a solution where you remove numerical labs on the y-axis and use 'mtext' to add text labels?" CreationDate="2011-03-30T15:22:02.923" UserId="930" />
  <row Id="15112" PostId="8968" Text="How do you do a regression when M is a single &quot;observation&quot; and N is a single observation?  Sure, it's mathematically possible, but it doesn't make sense to do so, so I wonder whether your data might actually differ from what this description seems to say." CreationDate="2011-03-30T15:25:08.270" UserId="919" />
  <row Id="15113" PostId="8965" Text="+1 For estimating the distribution parameters, the sample sizes are large enough to overcome @Cardinal's (good) objections.  (You do need to account for the serial correlation in deriving standard errors or confidence intervals for the parameters, however.)  From the look of the illustration, though, a Weibull is a bad fit." CreationDate="2011-03-30T15:28:54.163" UserId="919" />
  <row Id="15114" PostId="8968" Score="1" Text="If you have data for M, N, and R, and if err is small, don't you have a solvable situation of 3 equations and 3 unknowns (a, b, c)?" CreationDate="2011-03-30T15:31:20.577" UserId="2775" />
  <row Id="15115" PostId="8955" Score="2" Text="Not really so, sources of heterogeneity are many: differences in parameters (random parameters for instance), variables (usual regression thing), residuals (idiosyncratic part that may belong to different distributions, or distribution parameters may be different, heteroscedasticity belongs here, thus it is a separate case of heterogeneity), differences in functional form. So I would leave just the more particular term - heteroscedasticity." CreationDate="2011-03-30T15:38:16.423" UserId="2645" />
  <row Id="15116" PostId="8955" Text="Thanks Dimitrij, one question I had meant to ask was about the correct terminology in this area. " CreationDate="2011-03-30T16:10:54.410" UserId="3136" />
  <row Id="15117" PostId="8965" Text="@whuber, I wouldn't call them objections as much as notes of caution. I've also seen and worked with data sets where the effective sample size was reduced by two or three *orders of magnitude* due to dependence structure. Without having any domain knowledge myself for this problem or seeing the raw data, it's hard to know how warranted my notes of caution might be. It is, of course, entirely possible that they could safely be ignored. " CreationDate="2011-03-30T16:55:47.163" UserId="2970" />
  <row Id="15118" PostId="8972" Text="Just a side comment, but it might be relevant. Peter McCullagh has a paper in *Bernoulli* where he shows that *no* bootstrap correctly estimates the variance in a random effects model. " CreationDate="2011-03-30T17:02:19.280" UserId="2970" />
  <row Id="15119" PostId="8968" Score="1" Text="I think that OP has three latent factors and three observables which are linear combinations of these factors plus noise. The OP knows that noise is present only in the one of the observables, i.e the covariance matrix of the noise is singular. This makes it a factor analysis problem, although I think the problem is underdetermined. " CreationDate="2011-03-30T17:08:50.617" UserId="2116" />
  <row Id="15120" PostId="8965" Text="@Cardinal Sorry about the mischaracterization; &quot;caution&quot; is much better than &quot;objection.&quot;  Everybody knows *something* about wind!  It varies diurnally and seasonally.  If you want an adequate speed distribution for power generation you *must* have a representative sample of both forms of variation.  50K records = 500K minutes = one year definitely captures them both and so will be great to characterize *that year* and perhaps adequate for near-term forecasts *assuming* there is no secular trend or longer cycle (e.g., climate change, sunspot cycles)." CreationDate="2011-03-30T17:14:02.950" UserId="919" />
  <row Id="15121" PostId="8965" Text="@Cardinal I should also re-emphasize that effective sample size affects estimates of error, not estimates of the parameters themselves.  Of course, estimating the error is of paramount importance for forecasting.  We haven't been told the purpose of this distribution estimate, though, so we are left merely to speculate and issue words of caution :-)." CreationDate="2011-03-30T17:16:27.817" UserId="919" />
  <row Id="15122" PostId="8961" Text="@Sun agreed, but apart from length, I think another important distinction between comments and replies lies in their degree of speculation.  An authoritative, well-supported, correct one-liner often makes an excellent reply.  (And certain extended replies are really just comments...)  In this case it's the tentativeness of the response that indicates it ought to be a comment, IMHO." CreationDate="2011-03-30T17:19:02.917" UserId="919" />
  <row Id="15123" PostId="8960" Score="1" Text="@klonq What's the purpose of the estimation?  To retrospectively characterize past conditions?  To predict future power generation at one location?  To predict power generation within a grid of turbines?  To calibrate a meteorological model?  Etc.  For this question, determining an appropriate solution depends critically on how it will be used." CreationDate="2011-03-30T17:22:45.830" UserId="919" />
  <row Id="15124" PostId="8965" Text="@whuber, my previous comment was not meant to suggest that I thought you were mischaracterizing. I only meant it as clarification. I also haven't done a good job separating out my comments. I agree that effective sample size is not immediately relevant to the parameter estimates themselves and didn't mean to appear to suggest otherwise." CreationDate="2011-03-30T17:25:20.883" UserId="2970" />
  <row Id="15125" PostId="8965" Text="@whuber, however, the dependence between measurements *does* affect the estimates through the assumptions made on the likelihood. Assuming independence in such a circumstance can result in optimizing the &quot;wrong&quot; function which may have relative maxima farth(er) away from the true value. I have seen some work related to examining such situations. Lindsey is one of the contributors I am aware of and *composite likelihood* is the name of the game. In some cases, using the &quot;wrong&quot; likelihood does not result in a big penalty in terms of estimation efficiency. " CreationDate="2011-03-30T17:29:48.010" UserId="2970" />
  <row Id="15126" PostId="8975" Text="But isn't it more natural to use a pie chart for depicting market share rather than a bar chart? Despite area is perceived with less power..." CreationDate="2011-03-30T17:31:36.790" UserId="2902" />
  <row Id="15127" PostId="8965" Text="@Cardinal It all depends on what you're trying to accomplish. Taking the OP at face value, *if all you want is a parametric characterization of the numbers in your dataset,* then ML will do a great job.  If you're going to make inferences, estimates, or predictions--which is a more interesting and valuable exercise, to be sure--then the result will be only as good as your model.  In particular, if you don't model strong serial correlation and it's there, watch out!  I believe this is what you're saying and I'm agreeing with you." CreationDate="2011-03-30T17:40:04.833" UserId="919" />
  <row Id="15128" PostId="8975" Score="2" Text="@deps You are right, but it seems likely that people are comparing angles, not areas, in circular and near-circular pie charts, so the situation is not quite as bad as with area perception. Pie-like charts and &quot;spinners&quot; have been found to be good ways to help people think correctly about probabilities and proportions, too, so there is some possible merit in the use of pie charts. (See [Decision making from a cognitive perspective](http://books.google.com/books?id=PIOidpRPz18C&amp;lpg=PA307&amp;ots=W2ibugZrUe&amp;dq=spinner%20probability%20cognitive&amp;pg=PA307#v=onepage&amp;q&amp;f=false) (Busemeyer et al), p. 307)" CreationDate="2011-03-30T17:48:24.340" UserId="919" />
  <row Id="15129" PostId="8975" Text="@whuber (+1) Good that you mentioned the distinction between angle and area (I didn't notice your comment while I was writing my response)." CreationDate="2011-03-30T18:21:56.967" UserId="930" />
  <row Id="15130" PostId="8972" Text="@Mike (+1) That is one very well written question!" CreationDate="2011-03-30T18:30:23.277" UserId="930" />
  <row Id="15131" PostId="8780" Text="@chl but how to do it ;) ? i mean linking ?" CreationDate="2011-03-30T18:33:49.857" UserId="3870" />
  <row Id="15132" PostId="8780" Text="@suprvisr In your new question, just refer to this thread with an hyperlink; in Markdown syntax, it can be done as `[In genome wide association studies, what are principal components?](http://stats.stackexchange.com/questions/8777/in-genome-wide-association-studies-what-are-principal-components)`." CreationDate="2011-03-30T18:35:28.687" UserId="930" />
  <row Id="15133" PostId="8975" Score="1" Text="@deps_stats - Yes typically it is used for market share.  But based upon what you already know, don't you think it's easier to manipulate the display to your own tastes with a pie chart?" CreationDate="2011-03-30T19:07:36.863" UserId="3489" />
  <row Id="15134" PostId="8972" Text="Why wouldn't you resample participants with replacement and then resample their data as well?  That seems to be more in keeping of the spirit of a multilevel model with one distribution nested within another.  Another point is that there is a potential problem with binomial data because the extreme ends of the samples will be less likely to converge." CreationDate="2011-03-30T19:09:33.320" UserId="601" />
  <row Id="15136" PostId="8972" Text="@John: I prefer resampling residuals because (1) it is faster for when the original model is laborious to estimate, and (2) it yields CIs that have removed variability attributable to variability between participant means. #2 means that you don't have to create multiple plots when you want to show raw data and a repeated-measures effect; you can just plot these between-variance-removed CIs on the raw data and they will be appropriate for comparison of conditions repeated within individuals. Granted there may be confusion on the meaning of such CIs, but that's what figure captions are for." CreationDate="2011-03-30T19:25:37.830" UserId="364" />
  <row Id="15137" PostId="8980" Text="Your quotation marks are ambiguous: I'm pretty sure the two concepts you want to distinguish are &quot;data distribution&quot; and &quot;sampling distribution of the sample mean.&quot;" CreationDate="2011-03-30T19:25:42.557" UserId="919" />
  <row Id="15138" PostId="8972" Text="@John: Could you elaborate on your fears regarding the applicability of this approach to binomial data?" CreationDate="2011-03-30T19:26:11.090" UserId="364" />
  <row Id="15139" PostId="8980" Text="@mazatlan If you remember your last course, then maybe useful indicators for characterizing the empirical distribution, such as *central tendency*, *variance* or *scale*, and *shape*, might be a good start. The second question about sampling distribution of the mean has already been addressed elsewhere on this site." CreationDate="2011-03-30T19:27:22.090" UserId="930" />
  <row Id="15140" PostId="8693" Text="@bill thanks, it makes sense... can I simply add new variable neww being b*c for each case in the database (medical) and then treat it as linear regression ?" CreationDate="2011-03-30T20:05:55.597" UserId="3870" />
  <row Id="15141" PostId="8689" Text="@chl, yeap, thanks, I wasnt clear. My questions is really this:  If I write LM in R does R understands it as linear always or triesd to fit any model, not necessarily linear regression but any regression ?" CreationDate="2011-03-30T20:06:18.527" UserId="3870" />
  <row Id="15142" PostId="8713" Text="@bayer, so what would be an example of non linear regression in R ?" CreationDate="2011-03-30T20:08:43.800" UserId="3870" />
  <row Id="15143" PostId="8689" Text="@suprvisr No, `lm()` stands for a linear regression. Your model includes three parameters (minus the intercept) for `b`, `c`, and their interaction `b:c`, which stands for `b + c + b:c` or `b*c` for short (R follows Wilkinson's notation for statistical models). Fitting a Generalized Linear Model (i.e., where the link function is not identity, as is the case for the linear model expressed above) is requested through `glm()`." CreationDate="2011-03-30T20:16:23.420" UserId="930" />
  <row Id="15144" PostId="8972" Text="Well, I guess I'd need to clarify a bit.  I thought you resampled the participants and then that participant's data was fixed.  Are you saying you then resample the residuals from the participant? That's what I was suggesting be done. (I get that it's all residuals)" CreationDate="2011-03-30T20:17:43.043" UserId="601" />
  <row Id="15145" PostId="8884" Text="I suppose I was really just trying to avoid a theorem that gives convergence in such a weak sense that, while technically true, it has little application, e.g. pointwise converence almost everywhere." CreationDate="2011-03-30T20:42:34.020" UserId="2912" />
  <row Id="15146" PostId="8802" Text="@chl:  nice citation.  Garson sure has a lot to offer, although [side comment] I believe I have found a chink in his armor when it comes to centering in regression (he advocates it unnecessarily)." CreationDate="2011-03-30T20:53:32.043" UserId="2669" />
  <row Id="15147" PostId="8887" Text="Very interesting. Is the theorem you cite effectively the theorem loosely summarized in the [wikipedia article](http://en.wikipedia.org/wiki/Infinite_divisibility_%28probability%29#Limit_theorem) you cited. I don't have access to that book, especially at a retail of $250. :) &#xA;" CreationDate="2011-03-30T20:54:42.443" UserId="2912" />
  <row Id="15148" PostId="8887" Text="The wikipedia summary is quite loose, as there must clearly be more required or there are requirements on the members of the triangular array that aren't stated there, otherwise you could take two such triangular arrays with different limits and interchange every other row with the result that now neither one converges (even though the wikipedia article _sounds_ like it says they should)" CreationDate="2011-03-30T20:57:53.523" UserId="2912" />
  <row Id="15149" PostId="8884" Text="@John, if you are talking about convergence of distribution functions, then convergence pointwise almost everywhere (on the continuity points to be more precise) is called weak convergence or convergence in distribution and is the most applied type of convergence, since all the assymptotic statistics are based on it. " CreationDate="2011-03-30T20:59:51.080" UserId="2116" />
  <row Id="15150" PostId="8802" Text="@rolando2 Good to know. I came across his website because some of my colleagues use SPSS for psychometrics, esp. for factor analysis or reliability studies. Centering remains useful when one wants to avoid spurious collinearity effects or to interpret the effect of a predictor as deviation from its means (e.g., age)." CreationDate="2011-03-30T21:02:17.293" UserId="930" />
  <row Id="15151" PostId="8887" Text="@John, the access in the google books is not enough? You might get access to the book via library you know, it is old-fashioned way, but it works :)" CreationDate="2011-03-30T21:02:51.893" UserId="2116" />
  <row Id="15152" PostId="8887" Text="@John, the theorem 3.1 is also stated for triangular arrays, only infinite smallness is needed. If you want the conditions when the convergence happens, they are stated in [theorem 3.3](http://books.google.com/books?id=4LkdSaI4xXMC&amp;lpg=PP1&amp;dq=inauthor%3A%22Valentin%20Vladimirovich%20Petrov%22&amp;hl=fr&amp;pg=PA99#v=onepage&amp;q&amp;f=false)." CreationDate="2011-03-30T21:14:37.647" UserId="2116" />
  <row Id="15153" PostId="8887" Text="So is infinite divisibility and convergence is the most you can get? Infinite divisibility sounds like a pretty weak restraint e.g. take the fourier transform, take an nth (complex) root, now take the reverse fourier transform and call it h, then h convoluted with itself n times gives your original - so the only restriction is that h needs to be only non-negatively valued (should automatically have area 1). Or am I missing something? I could well have made a mistake there." CreationDate="2011-03-30T21:16:51.203" UserId="2912" />
  <row Id="15154" PostId="8887" Text="@John, your question was about the limiting distribution of certain sum of random variables. It can be understood in two ways: a) you are asking what types of limiting distributions there are _when limit exists_ b) you are asking when does the sequence converge. My answer is for a) from the comments I gather you are interested in b) also. Then my previous comment applies." CreationDate="2011-03-30T21:19:34.430" UserId="2116" />
  <row Id="15155" PostId="8887" Text="@John, I am not sure that I follow. If the sum of random variables satisfying infinite smallness converges, then the limiting distribution is infinitely divisible. That is very strong result, since infinitely divisible distributions have unique characterisation." CreationDate="2011-03-30T21:24:38.387" UserId="2116" />
  <row Id="15156" PostId="2575" Text="@ars that is a nice link, as are the papers discussed there. But, does this approach hold for simulations, in which case we are actually trying to estimate a density?" CreationDate="2011-03-30T21:41:24.793" UserId="1381" />
  <row Id="15157" PostId="8036" Text="One question is, what are these documents? Scanned images, PDFs, Excel spreadsheets? And a second question is, do you know which documents will be marked as accept and which ones as reject? Depending on this, the problem may be either solvable or extremely difficult. E.g. if 'reject' means blank form and accept means actual filled out form, this could be doable. If 'accept' means the return is actually valid, this could be extremely difficult." CreationDate="2011-03-30T21:47:04.047" UserId="3369" />
  <row Id="15158" PostId="8951" Text="@zbicyclist Thanks. Just curious, and only if you know of the top of your head, do you know if it would be possible to specify a different statistic as criteria? I.e., rank according to Adjusted $R^2$ or AIC?" CreationDate="2011-03-30T21:49:26.847" UserId="3311" />
  <row Id="15159" PostId="8975" Text="@Ralph Winters - I think my problem was more related to &quot;good taste&quot; or &quot;page design&quot; ie How to create a visually appealing report without sacrificing clarity? I need to add lots of tables and charts to a client, and I face the decision of using charts for this and tables for that, too many tables or too many charts. What I really need is a balance between visual appeal and good data depiction. Waffle charts seem like a very interesting alternative to pie-charts!" CreationDate="2011-03-30T21:52:01.680" UserId="2902" />
  <row Id="15160" PostId="8984" Score="1" Text="just to make sure, you calculate cv for the sample only from the two measurements?" CreationDate="2011-03-30T21:53:12.497" UserId="2116" />
  <row Id="15161" PostId="8984" Score="1" Text="Picking the two closest to each other and averaging them seems reasonable. But do you know what exactly goes wrong and why? If you do, this could influence how you do this. For example, if you know the measurement sometimes gets stuck at 0, then you may want to just ignore any zeros." CreationDate="2011-03-30T21:55:11.357" UserId="3369" />
  <row Id="15162" PostId="8972" Text="Your first impression was correct, I don't resample within participants. I seem to have therefore mis-remembered your previous suggestion (http://stats.stackexchange.com/questions/1399/obtaining-and-interpreting-bootstrapped-confidence-intervals-from-hierarchical-da) that bootstrapping hierarchical data should apply resampling both at the participant level as well as within participants." CreationDate="2011-03-30T23:00:02.627" UserId="364" />
  <row Id="15163" PostId="8972" Text="OK, I'm glad you see that.  My concern with binomial data is that you'd resample and can come up with samples without enough variability if there are a high rate of successes or failures.  The new model may not converge then.  It's theoretically possible with any data... but would be more likely with binomial where the means were large or small values." CreationDate="2011-03-30T23:17:01.250" UserId="601" />
  <row Id="15164" PostId="8976" Text="@chl Do you know any software that implements waffle charts?" CreationDate="2011-03-30T23:30:23.203" UserId="2902" />
  <row Id="15165" PostId="8984" Text="Or you could go for the median of three (in a sense you already choose the median of two)" CreationDate="2011-03-30T23:52:18.443" UserId="2958" />
  <row Id="15166" PostId="8971" Text="please don't downvote... this seems like an error." CreationDate="2011-03-31T00:20:05.320" UserId="2902" />
  <row Id="15167" PostId="8984" Text="@mpiktas: yes, we calculate cv from only the two measurements. @SheldonCooper: we usually do not know what goes wrong (but the stuck at zero idea is a good one to explore)." CreationDate="2011-03-31T02:24:50.900" UserId="3968" />
  <row Id="15168" PostId="7026" Text="I used kruskal-wallis test to know if there are gender differences, income differences and so on. Now i have a trouble on how to interpret the results.if the there are no significant differences for each group, is it possible to use the calculated mean rank to see if there is a increasing or decreasing trend?" CreationDate="2011-03-31T02:31:40.417" UserId="3118" />
  <row Id="15169" PostId="8961" Text="@whuber. Well put - thank you. I feel good about objecting now. I think one liner terse comments are the hardest to think about, at least for me. And none of us (I hope so) is not competing for any reputation points. Most of us (at least I am) are here to learn by solving problems and learn from others. I really hope others understand the point." CreationDate="2011-03-31T02:33:42.553" UserId="1307" />
  <row Id="15170" PostId="8963" Text="It actually does both (over- and under-estimate) if there is heteroscedasticity - in a similar way that &quot;averaging&quot; of ordinary data will both over and under-estimate the actual values.  For a slope, you will have different degrees of accuracy at different points on the line." CreationDate="2011-03-31T03:32:51.277" UserId="2392" />
  <row Id="15171" PostId="8969" Text="@bill_080 - (+1) Thx for the R-code." CreationDate="2011-03-31T04:06:15.890" UserId="1307" />
  <row Id="15172" PostId="8996" Text="See http://stats.stackexchange.com/q/6498/159" CreationDate="2011-03-31T05:39:36.303" UserId="159" />
  <row Id="15173" PostId="8998" Text="i computed my expectation of lognormal variable incorrectly. thanks." CreationDate="2011-03-31T06:02:58.230" UserId="862" />
  <row Id="15174" PostId="8800" Text="@berkay: Sorry, I am not able to understand this part: _In experiment 2 we change the only one attribute of the flower class out of 12 attributes (15 flower labeled class stays same but only one attribute is changed.)_. What is the &quot;attribute of flower class&quot; ? Maybe you can provide some example data (i.e. showing one or two exemplary rows before and after the change). What is &quot;15 flower labeled class&quot; ? 15 rows with label = flower ? Maybe can one of those who have upvoted this question explains what it means." CreationDate="2011-03-31T06:36:20.900" UserId="264" />
  <row Id="15175" PostId="8960" Text="@whuber at present the idea is to summarise wind data sets in a form allowing comparison from period to period and/or site to site. Later the goal will be compare trends and as you say to form judgements as to future production, etc. I am very much a newbie to stats but I have a mountain of data (which I can't share) and would like to extract as much information from it as possible. If you can point me to any reading on this subject it would be much appreciated." CreationDate="2011-03-31T07:02:42.390" UserId="3579" />
  <row Id="15176" PostId="8965" Text="@suncoolsu @cardinal @ whuber thanks for your comments. The mathematical solution by @suncoolsu really makes my head spin. I will have to investigate &quot;standard iterative procedures such as Newton-Raphson&quot; and may require help on this in the future. Much appreciated" CreationDate="2011-03-31T07:05:34.633" UserId="3579" />
  <row Id="15177" PostId="8732" Text="@TD3 I would have suggested something along the [point processes](http://en.wikipedia.org/wiki/Point_process). But I don't know how to do it exactly, you can look at this [thesis](ftp://ftp.dina.kvl.dk/pub/Staff/Morten.Larsen/papers/thesis.ps) for some ideas... " CreationDate="2011-03-31T08:16:24.183" UserId="1443" />
  <row Id="15178" PostId="8967" Text="Thank you both, but unfornately it doesnt work in my case" CreationDate="2011-03-31T08:48:03.097" UserId="3959" />
  <row Id="15179" PostId="8976" Score="1" Text="@deps_stats Nope. If you're interested in implementing it in R, you might find some ideas in [calendar heatmap](http://blog.revolutionanalytics.com/2009/11/charting-time-series-as-calendar-heat-maps-in-r.html)." CreationDate="2011-03-31T09:46:56.227" UserId="930" />
  <row Id="15180" PostId="8963" Text="In my particular field there is also a large reliance on significance testing and therefore p-values as well . So I think that fact that the SE can be over- and under-estimated may cause some problems if you base all inference on your p-values. " CreationDate="2011-03-31T09:57:22.257" UserId="3136" />
  <row Id="15181" PostId="9003" Text="Could you please tell more about the dependent variable? How did the experts express their rankings? Did each expert rate all the products? If not how was it decided which product would be assessed by which expert?" CreationDate="2011-03-31T10:34:47.637" UserId="3911" />
  <row Id="15183" PostId="9003" Text="4 experts sat down around the table, discussed each products and rank ordered each product (in excel). Then they took first 1% of products and said &quot;these products are excelent&quot;, then they took following 10% and said &quot;these products are very good&quot;... etc" CreationDate="2011-03-31T10:37:55.803" UserId="333" />
  <row Id="15184" PostId="9004" Text="How can I measure how good the model is? Don't have much experience using ordinal logistic regression?" CreationDate="2011-03-31T10:51:50.260" UserId="333" />
  <row Id="15185" PostId="9005" Score="1" Text="@Peter (+1) Your first sentence reminds me of [World’s Greatest Pie Chart](http://zoomdoggle.com/2008/09/worlds-greatest-pie-chart/) :-)" CreationDate="2011-03-31T10:53:39.683" UserId="930" />
  <row Id="15186" PostId="9004" Text="I know I can use ROC, Gini or Accuracy Ratio for binary logistic regression, what are the model performance metrics for ordinal logistic regression model?" CreationDate="2011-03-31T11:20:58.250" UserId="333" />
  <row Id="15187" PostId="8963" Text="I'd say you have bigger problems than standard errors if all inference is based on p-values and significance testing.  These kinds of things encourage &quot;mindless statistics&quot;." CreationDate="2011-03-31T12:10:17.210" UserId="2392" />
  <row Id="15189" PostId="8866" Text="Got `table` to work using `cut2`. Couldn't get `xtabs` to work. Thanks for the ideas." CreationDate="2011-03-31T13:35:07.093" UserId="2824" />
  <row Id="15190" PostId="9001" Score="2" Text="If you use R, see the **ppoints** function. " CreationDate="2011-03-31T13:41:34.603" UserId="2970" />
  <row Id="15192" PostId="8983" Text="Thanks An, that will give me a good start." CreationDate="2011-03-31T14:31:15.810" UserId="3965" />
  <row Id="15193" PostId="9009" Text="There are a couple of good questions on the site already talking about transforming data like this, see http://stats.stackexchange.com/q/1444/1036 or http://stats.stackexchange.com/q/298/1036" CreationDate="2011-03-31T14:35:59.467" UserId="1036" />
  <row Id="15194" PostId="9010" Text="&quot;Maximum likelihood estimator of a *random variable*&quot;? Not sure what that is, but I think you've (almost) calculated the ***mode***." CreationDate="2011-03-31T14:39:44.460" UserId="2970" />
  <row Id="15195" PostId="8876" Text="I apologize for taking a long time to answer. I put some example model types. Let me know if you want more clarification. I was thinking of discrete distributions as more difficult to sample from because it seems like they would be more prone to multimodal-like behavior. Is explicit normalization applicable when you have a mix of discrete and continuous variables?" CreationDate="2011-03-31T15:05:30.187" UserId="1146" />
  <row Id="15196" PostId="9010" Text="Something mysterious happens about two-thirds of the way through when suddenly $\mu$ and $\sigma$ appear without warning or definition." CreationDate="2011-03-31T15:17:09.323" UserId="919" />
  <row Id="15197" PostId="8998" Score="1" Text="+1 Good answer.  One might also observe that $\mathbb{E}[e^{t X}]$ is, by definition, the [mgf of the Standard Normal distribution](http://en.wikipedia.org/wiki/Normal_distribution) and therefore equals $e^{t^2/2}$.  Set $t=-\mu$, QED." CreationDate="2011-03-31T15:26:42.597" UserId="919" />
  <row Id="15198" PostId="9005" Text="@Peter (+1) Your blog contains excellent examples of dotplot vs pie-chart" CreationDate="2011-03-31T15:32:56.490" UserId="2902" />
  <row Id="15199" PostId="9010" Score="1" Text="I don't mean to &quot;pile on&quot;, but it's also hard for me to see how the quantity in brackets can be approximated by a negative number. " CreationDate="2011-03-31T15:34:17.817" UserId="2970" />
  <row Id="15200" PostId="8800" Text="@steffen, we have a class called flower and it has 15 row vectors consisting 12 features, we change only one of the features out of 12 in flower class and again make experiments to evaluate the results of 1 and 2." CreationDate="2011-03-31T17:13:05.320" UserId="3270" />
  <row Id="15202" PostId="8713" Text="I don't know R." CreationDate="2011-03-31T17:59:47.027" UserId="2860" />
  <row Id="15203" PostId="9013" Text="What kind of questions can I ask here?&#xA;&#xA;    * statistical and data-driven computing (e.g., questions about R, SAS, SPSS, Stata and Minitab)&#xA;" CreationDate="2011-03-31T18:01:30.150" UserId="62" />
  <row Id="15204" PostId="8895" Text="I see your point. It's a complicated subject, and without an answer to your first question it's not clear how to answer the OP. Perhaps just a bit more info (I'm not the local expert, just someone who reads CV a lot). Something like: If you try doing time series analysis using common regression tools, they may give you a false sense of confidence due to issues with auto-correlated residuals, or biased answers that can't be corrected even with large amounts of data. I explore the issues in depth in a paper I wrote: http:___." CreationDate="2011-03-31T18:18:08.580" UserId="1764" />
  <row Id="15205" PostId="9015" Text="Yes, a colleague came by and I got him to help. It was also an issue putting the name in quotes in the assignment statement. " CreationDate="2011-03-31T18:19:57.923" UserId="62" />
  <row Id="15206" PostId="8895" Text="P.S. In a blinding flash of light -- blinding to me, probably dim to most -- I realized that I simply had to create a factor, weather, indicating when the temperature was above or below 55F, and change my R model from lm (daily ~ temp) to lm (daily ~ weather / temp). Poof, it magically works, though slightly harder for me to interpret the coefficients." CreationDate="2011-03-31T18:27:07.527" UserId="1764" />
  <row Id="15207" PostId="9013" Text="@Baltimark, see the [FAQ](http://stats.stackexchange.com/faq). Questions about statistical packages are usually on tricky ground, since sometimes they do not have any statistical content, but community usually answers them out of good spirits." CreationDate="2011-03-31T19:08:14.823" UserId="2116" />
  <row Id="15208" PostId="6713" Text="No, with 50 coin tosses you do get to 2^50≃10^15 possible configurations, so that's right. But what is the question? You said you set an initial value for the sequence you want to observe, but you didn't say why or how." CreationDate="2011-03-31T19:11:15.820" UserId="2456" />
  <row Id="15209" PostId="8998" Text="@whuber, good comment :) I myself always turn to either chf or mgf if I see exponential transformation, it is like reflex :)" CreationDate="2011-03-31T19:12:46.253" UserId="2116" />
  <row Id="15210" PostId="9016" Score="1" Text="@berkay How do you define an error rate for this unsupervised method? (or do you mean the within SS?)" CreationDate="2011-03-31T19:17:26.147" UserId="930" />
  <row Id="15211" PostId="6713" Text="If you have an evolution rule you want to study, you might have to look at the stationarity analytically, since you can't simulate with all one quadrillion starting conditions. That said, you can certainly try a few to possibly prove yourself wrong. Oh, and halfway to an entropy of 50 bits is an entropy of 49 bits, depending on what it is you're looking at." CreationDate="2011-03-31T19:20:24.827" UserId="2456" />
  <row Id="15212" PostId="9008" Text="@ProbabilityIsLogic derived this integral for all order statistics in the first half of his reply." CreationDate="2011-03-31T19:42:58.910" UserId="919" />
  <row Id="15213" PostId="9013" Text="@mpiktas That's correct, but SAS seems special IMHO, because it is *solely* for statistics (and, AFAIK, it is not ever used by non-statisticians: who could afford to?).  Where else on SE would one reasonably pose such a question and expect an answer?" CreationDate="2011-03-31T19:54:40.413" UserId="919" />
  <row Id="15214" PostId="9016" Text="@chl, i can use sum of squared errors for all clusters or overall accuracy (in this case i know the class labels.)" CreationDate="2011-03-31T20:10:25.883" UserId="3270" />
  <row Id="15215" PostId="9013" Text="@whuber, note that original comment asked about all statistical packages. I agree that SAS is different, that is why I usually upvote all the questions about it. What I wanted to say in the comment, that our community is nice, so even if question is not strictly on topic, it usually gets answered." CreationDate="2011-03-31T20:13:32.317" UserId="2116" />
  <row Id="15216" PostId="8036" Text="This [answer](http://stats.stackexchange.com/questions/7467/quality-assurance-and-quality-control-qa-qc-guidelines-for-a-database/7472) might be of interest." CreationDate="2011-03-31T20:17:46.777" UserId="2116" />
  <row Id="15217" PostId="9016" Text="@berkay A simple algorithm for finding the No. clusters is to compute the average WSS for 20 runs of k-means on an increasing number of clusters (starting with 2, and ending with say 9 or 10), and keep the solution that has minimal WSS over this clusters set. Another method is the [Gap statistic](http://gremlin1.gdcb.iastate.edu/MIP/gene/MicroarrayData/gapstatistics.pdf). But if you already have labelled instances, then why are you trying an unsupervised method?" CreationDate="2011-03-31T20:21:19.073" UserId="930" />
  <row Id="15218" PostId="9016" Text="@chl thanks, good question, we can guess the clusters depending features of the intances, i'm analyzing the new intrusion characteristics, mimicry of legal applications." CreationDate="2011-03-31T20:30:07.210" UserId="3270" />
  <row Id="15219" PostId="9016" Text="@berkay Sorry but I'm not very familiar with that field. Do you mean the No. features will evolve with time?" CreationDate="2011-03-31T21:01:41.317" UserId="930" />
  <row Id="15220" PostId="8823" Text=" I'm a little confused by the index on $\bar H_i$; is $\bar H_i = \arg\max_{h\subset H_i} P(D_1, \dots D_m | h)$? Or is it $\bar H_i = \arg\max_{h\in \{H_1, \dots, H_n\} } P(D_1, \dots D_m | h)$? Seems like it ought to be the latter, but then I'm not sure why the subscript. Or maybe I'm missing something else entirely :)" CreationDate="2011-03-31T21:06:50.820" UserId="26" />
  <row Id="15221" PostId="3645" Text="@EDi Lookup `adonis` in the on-line help for vegan. (And [Jari Oksanen](http://cc.oulu.fi/~jarioksa/softhelp/vegan.html) for more info on the package itself.) BTW, this is a pretty elegant answer, you can vote it up like I did a long ago :-)" CreationDate="2011-03-31T21:10:55.270" UserId="930" />
  <row Id="15222" PostId="9020" Text="@Btibert3 How many respondents and items do you have? Can we assume there are different blocks of items, or do you put all items into the same box, so to speak? (This can help to figure out the best clustering solution.)" CreationDate="2011-03-31T21:18:11.490" UserId="930" />
  <row Id="15223" PostId="9016" Text="@chl, exactly.." CreationDate="2011-03-31T21:20:59.567" UserId="3270" />
  <row Id="15224" PostId="9007" Text="Thanks, this is exactly what I was looking for." CreationDate="2011-03-31T21:25:07.863" UserId="2425" />
  <row Id="15226" PostId="9020" Text="@Btibert3 I'm not sure I understand how your dataset looks like. But you may want to transpose your data matrix (turning variables into cases and cases into variables) and run clustering. Would you find it useful to see how to do this in R?" CreationDate="2011-03-31T21:45:02.083" UserId="3911" />
  <row Id="15227" PostId="9010" Text="`mean(qnorm(qbeta(((1:100000) - 0.5 ) / 100000, 1, 200)))` in R will give an approximate numerical integration, and in this case gives `-2.746042`.  Just change the `1, 200` to get other quantiles." CreationDate="2011-03-31T21:58:15.683" UserId="2958" />
  <row Id="15228" PostId="8645" Text="@whuber I have edited the &quot;overt marketing&quot; . I am not sure how to refer to tools that are in the marketplace that can help a poster with a problem/question. I see other references by responders pointing to Minitab , SPSS , R , SAS. " CreationDate="2011-03-31T21:59:47.513" UserId="3382" />
  <row Id="15229" PostId="9020" Text="Does this relate to [biclustering](http://en.wikipedia.org/wiki/Biclustering)? That's a method for clustering both variables and cases simultaneously. See also [this previous Q](http://stats.stackexchange.com/questions/7419/getting-started-with-biclustering)." CreationDate="2011-03-31T22:01:16.310" UserId="449" />
  <row Id="15230" PostId="9023" Text="Is this for a homework or a test?  &quot;A5&quot;, &quot;A6&quot;, and &quot;A7&quot; look like they are placeholders for values that were produced but are being hidden from you on purpose." CreationDate="2011-03-31T22:02:32.177" UserId="919" />
  <row Id="15231" PostId="8645" Text="@Irish Thank you.  The &quot;marketing&quot; refers to services, not software.  One purpose of SE is to create a lasting record of great solutions to questions.  Offering consulting services--whether professional or amateur, for free or not--explicitly circumvents this.  If you need to elicit additional information from the OP in order to answer the question effectively, do so in comments, until you have enough to be able to formulate a clear and helpful reply, and then please share that reply with us!" CreationDate="2011-03-31T22:06:20.503" UserId="919" />
  <row Id="15232" PostId="8645" Text="@Irish re: mentioning software.  Software, both commercial and free, is an essential part of statistics.  It is understood that software recommendations are necessary but might not be completely objective, because they depend partly on the respondent's experience and access to the SW.  However, when a respondent has a connection with software--as a developer, marketer, official proponent, or whatever--then recommending it can sometimes appear like a conflict of interest.  Such recommendations always need to be accompanied with disclosures of interest and handled as circumspectly as possible." CreationDate="2011-03-31T22:09:56.070" UserId="919" />
  <row Id="15233" PostId="9024" Text="Can you check your equation editing? It's somehow not showing up clear. Thanks!" CreationDate="2011-03-31T22:11:04.430" UserId="1102" />
  <row Id="15234" PostId="8645" Text="@whuber I see people referring to their textbook that they have written. Software is no different from textbooks , both deliver revenue to their authors. I guess if I mention a piece of software that I have been involved with , I should say so. " CreationDate="2011-03-31T22:21:17.807" UserId="3382" />
  <row Id="15235" PostId="9023" Text="It's a homework problem. I can do A6 and A7 by myself, I just need some tips on A5." CreationDate="2011-03-31T22:28:23.520" UserId="3062" />
  <row Id="15236" PostId="8645" Text="@Irish I think you're right about the textbooks (although the revenue typically is so paltry it's of no consequence: people usually write textbooks for other reasons).  One difference is that the textbook always reveals its authors' names, whereas a lot of software does not explicitly acknowledge its developers by name.  Yes, if you've been involved with the SW that you mention, you should say so." CreationDate="2011-03-31T22:35:03.023" UserId="919" />
  <row Id="15237" PostId="9023" Text="Consider the relationships between SD, variance, and total sum of squares about the mean." CreationDate="2011-03-31T22:36:57.543" UserId="919" />
  <row Id="15238" PostId="9022" Score="1" Text="You might like to review replies to [What is the most surprising characterization of the Gaussian (normal) distribution?](http://stats.stackexchange.com/q/4364/919).  For instance, the accepted answer suggests that if you standardize the marginals to $X$ and $Y$, you need only test that (a) $X$ and $Y$ have the same distribution and (b) $X-Y$ is independent of $X+Y$.  Another reply by @Robin Girard suggests yet another test." CreationDate="2011-03-31T22:47:40.110" UserId="919" />
  <row Id="15239" PostId="9022" Text="It is difficult to see how the variogram applies.  It summarizes a *single* multivariate observation and requires strong additional assumptions in order to estimate variances and covariances." CreationDate="2011-03-31T22:49:57.753" UserId="919" />
  <row Id="15240" PostId="9025" Text="In that case it's not the &quot;sample standard deviation,&quot; but the residual standard deviation. :-)" CreationDate="2011-03-31T22:51:21.120" UserId="919" />
  <row Id="15241" PostId="9022" Text="@whuber: Thanks for the link. However, that question talks about univariate gaussianity, and nothing about bivariate gaussianity." CreationDate="2011-03-31T22:53:56.110" UserId="1102" />
  <row Id="15242" PostId="9025" Text="I see, thanks for your help :)" CreationDate="2011-03-31T23:42:32.237" UserId="3062" />
  <row Id="15243" PostId="9022" Text="Good point.  But it seems like those replies generalize in some ways.  For instance, they suggest that one way to detect bivariate normality is to check various linear combinations of the two variables separately for normality.  This reduces it to a set of univariate tests." CreationDate="2011-03-31T23:50:08.120" UserId="919" />
  <row Id="15244" PostId="9022" Text="@whuber: I guess what you are suggesting is regression, and I have stated in my question that *without the use of regression*" CreationDate="2011-03-31T23:53:35.300" UserId="1102" />
  <row Id="15246" PostId="9022" Text="No, that's the whole point: univariate checks are manifestly *not* regression methods.  Neither is taking a linear combination of variables." CreationDate="2011-03-31T23:54:36.307" UserId="919" />
  <row Id="15249" PostId="9021" Text="Could you elaborate a little more on what a `Local Time Trend` intervention variable looks like?  I am familiar with the other three." CreationDate="2011-04-01T00:05:04.297" UserId="179" />
  <row Id="15251" PostId="9021" Text="Also, can you point me towards an R package that might be able to do intervention detection?" CreationDate="2011-04-01T00:12:11.363" UserId="179" />
  <row Id="15252" PostId="9021" Score="1" Text="If you have a series that exhibits 1,2,3,4,5,7,9,11,... there has been a change in trend at period 5. The main question in time series is the detection of level shifts e.g. 1,2,3,4,5,8,9,10,..or another example of a level shift 1,1,1,1,2,2,2,2, and or the detection of time trend breaks." CreationDate="2011-04-01T00:44:45.563" UserId="3382" />
  <row Id="15253" PostId="9028" Text="@Hong Ooi:  Thanks!  This code is really useful, and will do the trick, but what I was mainly aiming for was a function which would give a new R model which I could then put into predict() later on in the workflow.  However, I think your function is pretty elegant, and like the way you can set it to give probabilities." CreationDate="2011-04-01T00:54:33.090" UserId="1991" />
  <row Id="15254" PostId="9028" Text="@Andrew you can take advantage of R's object-oriented programming for this. Put your two models into a list, and give it a class, say `glm_2`. Call the above function `predict.glm_2` and you can then use `predict()` on your object as required." CreationDate="2011-04-01T01:04:42.423" UserId="1569" />
  <row Id="15255" PostId="8988" Text="@GaBorgulya: Very helpful. Thanks!" CreationDate="2011-04-01T01:46:43.673" UserId="3968" />
  <row Id="15256" PostId="9022" Text="@H_S, please don't *simultaneously* [cross-post](http://math.stackexchange.com/questions/30220/how-to-check-for-bivariate-gaussianity-without-the-use-of-regression). It's impolite." CreationDate="2011-04-01T02:25:10.397" UserId="2970" />
  <row Id="15258" PostId="9027" Text="You might find some useful information in the related thread at http://stats.stackexchange.com/q/8502/919 ." CreationDate="2011-04-01T03:04:21.073" UserId="919" />
  <row Id="15259" PostId="9028" Score="2" Text="Why is averaging the coefficients appropriate?  If the two datasets have different sizes, then surely at least some form of weighted average should be used." CreationDate="2011-04-01T03:05:57.077" UserId="919" />
  <row Id="15260" PostId="3187" Text="@Roman could you point out where the book discusses time series? I just searched through it and could not find the topic covered." CreationDate="2011-04-01T03:42:21.400" UserId="1381" />
  <row Id="15261" PostId="9012" Score="2" Text="It appears this would force year-on-year comparisons to have equal means and variances, artificially eliminating most information about temporal change." CreationDate="2011-04-01T04:44:12.127" UserId="919" />
  <row Id="15262" PostId="9029" Score="1" Text="@paul, in multivariate regression if you estimate each equation separately you lose out only in efficiency, the estimates of the coefficients and their standard errors are estimated correctly. I suspect that the same should hold for multivariate multinomial logit. " CreationDate="2011-04-01T06:05:50.143" UserId="2116" />
  <row Id="15263" PostId="9029" Score="1" Text="@paul Mpiktas' hunch is right. There exists multivariate multinomial logit models and I have seen it in Agresti 2002. I will get back with the exact page numbers as I dont have the book on me. But I have seen the theory being developed in the same chapter in which they introduce Mutivariate logit; I think immediately after the horse-shoe crabs or crocodile examples." CreationDate="2011-04-01T07:17:42.510" UserId="1307" />
  <row Id="15264" PostId="8963" Text="I couldn't agree more, I think most people I speak to know that such an approach is suspect, but its difficult to stop them from focusing solely on p-values, often at the expense of everything else." CreationDate="2011-04-01T08:19:58.727" UserId="3136" />
  <row Id="15266" PostId="8549" Text="@stefanos: Can you please tell me the name of the application as I am always interested in pursuing software solutions dealing with this issue. You can email me at my contact info." CreationDate="2011-04-01T09:12:36.247" UserId="3382" />
  <row Id="15267" PostId="8823" Text="@JMS - $\overline{H}_{i}$ stands for the logical statement &quot;$H_{i}$ is false&quot;, or that one of the other hypothesis is true.  So in &quot;Boolean algebra&quot; we have $\overline{H}_{i}\equiv H_1+H_2+\dots+H_{i-1}+H_{i+1}+\dots+H_{n}$ (because the hypothesis are exclusive and exhaustive)" CreationDate="2011-04-01T09:35:26.703" UserId="2392" />
  <row Id="15268" PostId="9010" Text="@cardinal - my mistake, this should be the inverse CDF $-\Phi^{-1}(\frac{1}{N+1})$ as defined in the question.  And I have calculated the mode of the PDF, which is (but for an idealogical restriction) a maximum likelihood estimator - all I have to do is replace the &quot;random variable&quot; with a &quot;greek letter&quot; and then magically - its an MLE!  But clearly my answer is in need of repair" CreationDate="2011-04-01T09:48:13.703" UserId="2392" />
  <row Id="15269" PostId="9010" Text="@whuber - they do not appear out of nowhere, for they are defined in the question. but I will update my answer to &quot;introduce&quot; them better." CreationDate="2011-04-01T09:49:32.563" UserId="2392" />
  <row Id="15270" PostId="8963" Text="@user3136 - and I think the main reason is that &quot;mindless statistics&quot; such as the use of p-values, is easy.  It may be wrong, but easy often wins.  And software is another killer - just contributes to the difficulty of doing anything else.  you need to have something to put in their place." CreationDate="2011-04-01T10:01:57.597" UserId="2392" />
  <row Id="15271" PostId="9033" Text="@Pedro, could you please check if the formula is ok? I tried to reflow your formula, since he original version did not fit in the page, but had some problems with MathJax, so I ended up splitting it. " CreationDate="2011-04-01T10:04:45.500" UserId="2116" />
  <row Id="15272" PostId="9028" Text="@ Hong - thanks, I'll try this out. @whuber - thanks for pointing this out.  If my two datasets are the same size, is weighting an issue (I hadn't thought of this before).  If they are different do I just weight by the ratio of the sizes (e.g. the if one is twice as big, do I give it twice the weight)?" CreationDate="2011-04-01T10:05:32.097" UserId="1991" />
  <row Id="15273" PostId="9033" Text="Yeah I think so @mpiktas; thanks for splitting, and apologies for the inconvenience." CreationDate="2011-04-01T10:09:24.503" UserId="3987" />
  <row Id="15274" PostId="9037" Text="Thanks!  That's what I needed." CreationDate="2011-04-01T11:47:23.320" UserId="1138" />
  <row Id="15276" PostId="9010" Text="@probabilityislogic, while at the level of calculus, you might say that *in this case* we're considering a bivariate function and simply maximizing over one variable instead of another, I think there are reasons mathematical, statistical, and pedagogical **not** to call what you've done &quot;maximum likelihood estimation&quot;. They are too numerous to enumerate in this space, but a simple one that I think is compelling enough is that we use a particular, arcane vocabulary in statistics for a reason. Changing that on a whim for a single problem can lead to misunderstanding(s).../..." CreationDate="2011-04-01T11:52:01.973" UserId="2970" />
  <row Id="15277" PostId="9010" Text=".../...Besides, we already have a specific, arcane term in our vocabulary to describe what you are doing. So, &quot;renaming&quot; it to use another specific, arcane term that does **not** (normally) describe what you are doing seems unwarranted (at least to me)." CreationDate="2011-04-01T11:54:00.117" UserId="2970" />
  <row Id="15279" PostId="9040" Score="1" Text="For any even moderate sized graph, an edge-vertex graph visualization is simply going to tell you much more about the layout algorithm than anything about the data itself. " CreationDate="2011-04-01T12:45:46.490" UserId="2970" />
  <row Id="15281" PostId="9010" Text="@cardinal - point taken, I have removed this from my answer, and acknowledged its inadequacy." CreationDate="2011-04-01T12:59:52.783" UserId="2392" />
  <row Id="15282" PostId="9043" Text="There are a lot of terms which could be what you are after, but without more details or at least more context (how is the estimate calculated?) I don't see how anybody could help you." CreationDate="2011-04-01T09:05:21.340" />
  <row Id="15283" PostId="9043" Score="1" Text="@Vilx-, can you provide the latvian term in the comments? I am Lithuanian, so I might guess better. Also in what context that term is used? Maybe there is some data on which it is calculated? It would be easier to  guess precisely what it is." CreationDate="2011-04-01T09:08:05.740" UserId="2116" />
  <row Id="15284" PostId="9043" Text="Give a few examples, without seeing some numbers and the process we can't help" CreationDate="2011-04-01T09:08:59.497" UserId="2818" />
  <row Id="15285" PostId="9043" Text="The Latvian term is &quot;vidējo kvadrātisko kļūdu novērtējums&quot;. I'll try to get some formulas, though I doubt my father will know them. It's just something that the device does. :P" CreationDate="2011-04-01T09:14:45.407" UserId="2590" />
  <row Id="15286" PostId="9043" Text="@Vilx-, I suspect this is root mean square error, if we assume that latvian statistical terminology is based on russian, as lithuanian is. Unfortunately without formulas or data examples it will be impossible to tell for non-Latvian.  There is a strong possibility that this term was translated from English by non-statistician with liberal interpretation, so exact meaning might be lost." CreationDate="2011-04-01T09:34:02.327" UserId="2116" />
  <row Id="15287" PostId="9043" Text="@Vilx-, try asking at gis.SE, if you form the question what GPS devices usually report as an error, you might get the correct English terminology  and come to the translated term without actually translating." CreationDate="2011-04-01T09:36:31.140" UserId="2116" />
  <row Id="15288" PostId="9044" Text="I played with lithuanian language and it seems that google translator has a rule to translate _mean square_ to _rms_ no matter what comes afterwards. I suspect strongly that this holds for latvian also." CreationDate="2011-04-01T10:22:47.620" UserId="2116" />
  <row Id="15289" PostId="9044" Score="1" Text="@mpiktas: It probably means that most of the times that Google has come across the Latvian phrase, it has also come across *rms* in parallel English texts.  The obvious environment is European Union documentation. " CreationDate="2011-04-01T10:45:59.370" UserId="2958" />
  <row Id="15290" PostId="9044" Text="Turns out this was it." CreationDate="2011-04-01T13:13:10.530" UserId="2590" />
  <row Id="15295" PostId="9048" Text="Very interesting idea, I assume you mean something akin to http://en.wikipedia.org/wiki/James_Cameron#Casting? My only worry is having a 1000x2500 column grid but for a subset it could definitely work. Thanks for the suggestion!" CreationDate="2011-04-01T14:00:16.853" UserId="3988" />
  <row Id="15296" PostId="9041" Text="I've made a quick attempt with Graphviz using the example you suggested and it works perfectly apart from one little issue. It's currently very short and wide as it puts the movies on top of the actors and not spread about. Is there anyway to get Graphviz to create a more square arrangement? Cheers for the recommendation though!" CreationDate="2011-04-01T14:02:32.623" UserId="3988" />
  <row Id="15297" PostId="9048" Score="2" Text="The idea is similar, though I think you could make the visualization much nicer. It looks like there the ordering is chronological, which also brings out some interesting features. " CreationDate="2011-04-01T14:07:14.527" UserId="2970" />
  <row Id="15298" PostId="9048" Text="Ordering by release year vs actor DOB could provide a nice visual timeline flow. Thanks again :)" CreationDate="2011-04-01T14:08:40.517" UserId="3988" />
  <row Id="15299" PostId="9033" Text="I'm pretty sure some normalizing constants are missing from the integrals: you need to divide them by $\sqrt{2\pi}s$." CreationDate="2011-04-01T14:28:10.013" UserId="919" />
  <row Id="15300" PostId="9048" Text="I think it would be hard to make this visualization take up *more* real estate than any (readable) edge-vertex graph visualization, for example. Comparatively, it would seem that a heatmap would be quite compact. " CreationDate="2011-04-01T14:34:01.267" UserId="2970" />
  <row Id="15302" PostId="9049" Text="Actors on the inner ring and films on the outer perhaps? Could be an interesting mix, cheers." CreationDate="2011-04-01T14:37:51.063" UserId="3988" />
  <row Id="15305" PostId="8887" Text="It sounds like your answer is both correct and about as good as is possible. Thanks for all your excellent comments. " CreationDate="2011-04-01T15:07:57.140" UserId="2912" />
  <row Id="15308" PostId="8706" Text="Yes, because $x$ is not the quantity of interest (the one you optimize for) but instead $\beta$ is. Thus, it is linear in $\beta$." CreationDate="2011-04-01T15:34:38.257" UserId="2860" />
  <row Id="15309" PostId="8989" Text="I agree with the database suggestion and the plan is that the data will reside in a database eventually but the method of data collection must be in Excel. In terms of database management this is the easiest method and I will write a short script to transpose the data from storage method #2 to #1. Thanks" CreationDate="2011-04-01T15:37:47.743" UserId="3484" />
  <row Id="15312" PostId="9029" Text="@paul May not be what you're looking for, but it might be worth considering mixed effects multinomial logit models. Yet another approach would be to look at package `drm` in R, which claims to do regression for clustered variables using &quot;dependence ratios&quot; to  model intra-unit association (http://www.helsinki.fi/~jtjokine/drm/)." CreationDate="2011-04-01T15:57:54.120" UserId="3432" />
  <row Id="15313" PostId="9052" Score="2" Text="the `reshape` package can be useful as well. Once you have `e`, consider something like `library(reshape) cast(melt(e), name1 ~ name2)`" CreationDate="2011-04-01T16:08:11.393" UserId="696" />
  <row Id="15314" PostId="9010" Text="@probability +1 for a nice exposition of working with order statistics.  However, this approximation cannot be expected to work well in the tails (extreme values of i), because the inverse CDF has asymptotes at 0 and 1 and its Taylor series approximation is poor there.  E.g., with N=200 and i=1 and the standard normal distribution, the approximation gives -2.57755 compared to -2.74604--a fairly big difference--whereas for n=100, the approximation is -0.00376 compared to -0.00626, a small difference.  A better approach is to substitute $p_i=F_X(u)$ in the integral and use a saddlepoint approx." CreationDate="2011-04-01T16:27:49.147" UserId="919" />
  <row Id="15315" PostId="9054" Score="2" Text="+1 for `ggfluctuation`, hadn't seen that before! This post has other useful code to visualize this type of dater: http://stackoverflow.com/questions/5453336/r-plot-correlation-matrix-into-a-graph/5453398#5453398" CreationDate="2011-04-01T16:42:45.297" UserId="696" />
  <row Id="15316" PostId="9054" Text="@Chase (+1) Thx. BTW, it seems I had some problem with my color scheme for negative correlation values." CreationDate="2011-04-01T16:59:11.520" UserId="930" />
  <row Id="15317" PostId="9053" Score="3" Text="Look at the works of [Alain Celisse](http://math.univ-lille1.fr/~celisse/). His work as far as I read (too little alas) is about merits of cross-validation. " CreationDate="2011-04-01T17:10:52.870" UserId="2116" />
  <row Id="15318" PostId="9054" Score="1" Text="+1 for nice visualisation" CreationDate="2011-04-01T17:12:32.570" UserId="2116" />
  <row Id="15319" PostId="9054" Text="Reordering the rows and columns by (`hclust(…)$order`)[http://stat.ethz.ch/R-manual/R-devel/library/stats/html/hclust.html] the visualization will be often easier to overview." CreationDate="2011-04-01T17:23:54.290" UserId="3911" />
  <row Id="15321" PostId="9028" Text="@Andrew It is correct to weight by data count only if the values of the independent variables are the same in both subsets.  For a better approach (which is also theoretically valid), see the thread I referenced in a comment to your question." CreationDate="2011-04-01T18:57:48.563" UserId="919" />
  <row Id="15322" PostId="9054" Text="@GaBorgulya Good point. I use this when I'm doing exploratory data analysis *and* the variables have no particular order (as would be the case for spatial or temporal data, or structured data that you want to see as is). The `mixOmics::cim` function is very good for that. A related issue was discussed here, http://stats.stackexchange.com/questions/8370/how-to-visualize-summarize-a-matrix-with-number-of-rows-gg-number-of-columns/8372#8372." CreationDate="2011-04-01T19:29:44.733" UserId="930" />
  <row Id="15323" PostId="9053" Text="@mpiktas Indeed, and one of his paper was already proposed for the CVJC, http://www.mendeley.com/groups/999241/crossvalidated-journal-club/papers/." CreationDate="2011-04-01T19:57:51.027" UserId="930" />
  <row Id="15324" PostId="9001" Text="@Chris, @probabilityislogic has given some good intuition for the approximations you list. Would it be helpful at all if I gave some more from an alternative viewpoint, or have you satisfied your curiosity on this matter?" CreationDate="2011-04-01T19:59:21.460" UserId="2970" />
  <row Id="15325" PostId="9051" Text="+1 This excellent analysis demonstrates why the question is appropriate on this site." CreationDate="2011-04-01T20:14:43.797" UserId="919" />
  <row Id="15326" PostId="9010" Text="@probabilityislogic (+1) for the revised answer. One suggestion, maybe $\Rightarrow$ is better than $\to$ to mean &quot;implies&quot;. It took staring at a couple lines for a few seconds to realize you weren't making some convergence claim. " CreationDate="2011-04-01T20:25:05.653" UserId="2970" />
  <row Id="15327" PostId="9034" Text="+1 Interesting examples.  Are any of them accessible on the Web?" CreationDate="2011-04-01T20:32:01.737" UserId="919" />
  <row Id="15328" PostId="9049" Text="I think actors in the middle and films outer is more logical. I'm glad you say film and not movie :) Films could either be arranged chronologically or by gross profit/budget. Possibly actors in most films (well connected) nearer the centre but there are several ways to present different versions linking actors to directors etc. In theory you could leave the countries on the outside and represent the film industry by country but I suspect more than 2 dimensions are then needed?" CreationDate="2011-04-01T20:54:19.383" UserId="3484" />
  <row Id="15329" PostId="9029" Text="Thanks for the responses, this is helpful." CreationDate="2011-04-01T21:29:36.560" UserId="3984" />
  <row Id="15330" PostId="9057" Text="Ok, thanks. It was the terminology that got me - I didn't realize clustered/repeated response data meant basically the same thing as mutlivariate in regression, but it seems that's the case.&#xA;&#xA;In Agresti 2002 I think it's Ch. 11 &amp; 12. 2007 and 2002 cover most of the same stuff, but '02 is a little more in depth. &#xA;&#xA;Time to read those chapters." CreationDate="2011-04-01T21:30:43.783" UserId="3984" />
  <row Id="15331" PostId="6539" Text="@bill_080: Your U.S. Seasonal Component is very close to what I'm seeing in the microcosm of our condominium association's usage. Which I find a bit strange. (Though our summer peak is much larger.) And your underlying trend looks familiar, too, which is even weirder  to me." CreationDate="2011-04-01T21:48:19.637" UserId="1764" />
  <row Id="15332" PostId="9063" Text="Thanks for suggestions. I gave up from WMW because of ties." CreationDate="2011-04-01T22:51:00.243" UserId="1356" />
  <row Id="15335" PostId="9067" Score="1" Text="In SAS or R you need a few lines of code to remove empty columns, but how come it did not work in Excel? Have you tried right-clicking the header of the empty column and selecting delete?" CreationDate="2011-04-02T00:13:35.810" UserId="3911" />
  <row Id="15336" PostId="9012" Text="true. I was not thinking of comparing means of the two samples, in which case converting to z-scores is self-deating. Had covariance in mind-- e.g., assessing how one or more predictors relate to the likert item score in either yr or both combined. I tend to think covariance matrices are the only thing one should try to learn from when using likert items (people tend to invest  the points on the measure with too much meaning--&quot;but mine goes to 11&quot;). Hope I did not throw anyone off track." CreationDate="2011-04-02T00:29:35.080" UserId="2347" />
  <row Id="15337" PostId="9069" Text="@probabilityislogic Very interesting. Do you propose the last formula as the “measure of association”?" CreationDate="2011-04-02T00:58:54.363" UserId="3911" />
  <row Id="15339" PostId="9069" Text="yep, although the expected value should really be adjusted to $\frac{1+\sum_{k=1}^{j}n_{\bullet k}}{3+n_{\bullet\bullet}}$.  But this only matters if the counts are small." CreationDate="2011-04-02T01:23:29.110" UserId="2392" />
  <row Id="15340" PostId="8974" Text="I find your waffle chart unclear, because of the excessive whitespace. " CreationDate="2011-04-02T01:57:35.767" UserId="2456" />
  <row Id="15341" PostId="6539" Text="@Wayne: If your summer electrical peak is much larger than the U.S. average, then I would assume that you live in an area of higher than U.S. average air conditioning usage.   The blue underlying U.S. trend reflects a general drop in long term usage due to the 2001 recession and the 2008/2009 recession, which probably affected your area also." CreationDate="2011-04-02T02:37:21.007" UserId="2775" />
  <row Id="15342" PostId="9012" Text="Agreeably some comparisons are lost. However, much of my analysis focuses on net promoter scores rather than means and variance. So, I'll try it out and see what it gets me. Cheers for the response." CreationDate="2011-04-02T03:12:54.080" UserId="776" />
  <row Id="15343" PostId="9067" Text="SAS uses [keep and drop](http://www.ats.ucla.edu/stat/sas/faq/efficient_keep_drop.htm) which are about as simple as it gets.  What exactly &quot;doesn't work&quot;?" CreationDate="2011-04-02T04:39:28.413" UserId="919" />
  <row Id="15344" PostId="9068" Text="There is no &quot;data relationship&quot; in this example because y does not vary.  Assigning *any* numerical value to *r* would be a mistake." CreationDate="2011-04-02T04:42:11.840" UserId="919" />
  <row Id="15345" PostId="9070" Score="1" Text="That's a useful relevant *test*, but the question asked for a *measure of association*. Is there one that goes with this test?" CreationDate="2011-04-02T06:34:50.333" UserId="449" />
  <row Id="15346" PostId="9068" Text="@whuber - it is true that the $r$ is undefined, but not necessarily that the &quot;true&quot; unknown correlation $\rho$ cannot be estimated.  Just have to use something different to estimate it." CreationDate="2011-04-02T07:26:56.837" UserId="2392" />
  <row Id="15347" PostId="9081" Text="@richiemorrisroe In this particular case, I think it would be best to further clarify the R syntax and meaning of `summary` and `plot`, or to give external links that help to understand how to interpret LM output in R. Maybe John Maindonald or Peter Dalgaard's tutorials, or directly the CRAN contrib docs?" CreationDate="2011-04-02T10:41:00.177" UserId="930" />
  <row Id="15348" PostId="9073" Text="That is very impressive, and rather more complete than I had been asking for.  I would have been satisfied with the diagram and a statement that the tangent to the ellipse does not pass through the centre of the ellipse, so the tangent red point must take disproportionately more from the random variable with a higher standard deviation. " CreationDate="2011-04-02T10:45:27.487" UserId="2958" />
  <row Id="15349" PostId="9069" Text="Neat... I should replicate this in R, if not already availlable." CreationDate="2011-04-02T11:49:57.783" UserId="1356" />
  <row Id="15351" PostId="9067" Text="@GaBorgulya: It is a huge dataset and it is not possible to manually identify columns which are empty. I was thinking of a way in which we can automatically remove empty columns and also generate a list of these columns for ananlysis. I am not sure how to do it automatically in excel and SAS." CreationDate="2011-04-02T13:21:33.633" UserId="3897" />
  <row Id="15352" PostId="9067" Text="@whuber: I have to do it automatically since it is a huge dataset. SAS should go over the data automatically and remove the empty columns and also generate a list of those empty columns for analysis" CreationDate="2011-04-02T13:22:48.483" UserId="3897" />
  <row Id="15353" PostId="9083" Text="Even though you link to the help page, I'd probably put a `library(ggplot2)` before qplot as that code will not run in a fresh R session w/o loading the appropriate package." CreationDate="2011-04-02T14:05:11.533" UserId="696" />
  <row Id="15354" PostId="9089" Text="Thanks for a swift answer, not a homework though. Would have tagged it as such if it really was." CreationDate="2011-04-02T14:18:59.477" UserId="1371" />
  <row Id="15355" PostId="9085" Score="3" Text="@gsk3 - interesting question.  I would have a guess that this &quot;programaphobe&quot; does not have to deal with the complicated spreadsheets?  perhaps he/she would like to live up to their ideology and volunteer to do the hard work with the spreadsheets themselves? :)" CreationDate="2011-04-02T14:37:31.273" UserId="2392" />
  <row Id="15356" PostId="9085" Text="@probabilityislogic: The &quot;programaphobe&quot; does deal with them once they cause problems, but has a whole bag of tricks for dealing with the results.  I've tried the classic anti-spreadsheet arguments to little avail (well, some avail, since they're willing to consider this style of solution!)." CreationDate="2011-04-02T14:46:45.807" UserId="3488" />
  <row Id="15357" PostId="9073" Text="That was not well worded.  What I meant was the line from the centre to the red point is not perpendicular to the tangent. " CreationDate="2011-04-02T15:13:21.877" UserId="2958" />
  <row Id="15358" PostId="9067" Text="@user In other words, your question is how to *identify* empty columns, not how to remove them." CreationDate="2011-04-02T15:52:19.903" UserId="919" />
  <row Id="15359" PostId="9068" Text="@probability You presuppose this is a problem of estimation and not simply one of characterization.  But accepting that, what estimator would you propose in the example?  No answer can be universally correct because it depends on how the estimator will be used (a loss function, in effect).  In many applications, such as PCA, it seems likely that using *any* procedure that imputes a value to $\rho$ may be worse than other procedures that recognize $\rho$ cannot be identified." CreationDate="2011-04-02T15:57:11.240" UserId="919" />
  <row Id="15360" PostId="9091" Text="(+1) not so straightforward algebra, nice alternative solution." CreationDate="2011-04-02T16:14:00.723" UserId="2645" />
  <row Id="15361" PostId="9082" Text="+1 Thorough and nicely illustrated." CreationDate="2011-04-02T16:15:08.900" UserId="919" />
  <row Id="15362" PostId="9040" Text="@cardinal That is true for general graphs, but this is a special type: it is bipartite.  That allows for some potentially effective and revealing layouts." CreationDate="2011-04-02T16:17:30.693" UserId="919" />
  <row Id="15363" PostId="9085" Text="@gsk3 - ah ha, so its not the program that is the issue per se, but this person will no longer be &quot;the expert&quot; and will have to start from the bottom?" CreationDate="2011-04-02T16:51:35.930" UserId="2392" />
  <row Id="15364" PostId="9082" Text="Thanks chl for thorough explanation. I was aware of biclust but mixOmics is new to me. I tried both R packages. As mixOmics do clustering for rows and rows independently so I guess it is not as technical as biclust. I'm wondering if is it possible to get the two-sided dendrograms in biclust as in mixOmics with cim() function. Thanks again for nice answer." CreationDate="2011-04-02T16:55:18.043" UserId="3903" />
  <row Id="15365" PostId="9068" Text="@whuber - estimate is a bad choice of words for me (you may have noticed I'm not the best wordsmith), what I meant was that although $\rho$ may not be uniquely identified, this does not mean that the data are useless in telling us about $\rho$.  My answer gives an (ugly) demonstration of this from an algebraic point of view." CreationDate="2011-04-02T16:59:21.620" UserId="2392" />
  <row Id="15366" PostId="9068" Text="@Probability It seems your analysis is contradictory: if indeed y is modeled with a normal distribution, then a sample of five 2's shows this model is inappropriate.  Ultimately, you don't get something for nothing: your results depend strongly on the assumptions made about the priors.  The original problems in identifying $\rho$ are still there but have been hidden by all these additional assumptions.  That seems IMHO just to obscure the issues rather than clarify them." CreationDate="2011-04-02T17:30:54.190" UserId="919" />
  <row Id="15367" PostId="9096" Text="+1 for mentioning R! =)" CreationDate="2011-04-02T17:51:31.053" UserId="1356" />
  <row Id="15368" PostId="9068" Text="@whuber - I'd say it actually &quot;brings out&quot; all the assumptions that one actually makes when using pearson correlation (normality).  Its not that you get something for nothing, its that it highlights the &quot;something&quot; that was already there, left unspecified or thrown out with the passage to infinite limits in the parameter space and/or sample space (which is not a reflection of &quot;reality&quot; but a mathematical convenience which enables closed form solutions to many problems)." CreationDate="2011-04-02T17:51:59.353" UserId="2392" />
  <row Id="15369" PostId="9085" Text="@probabilityislog - I think sunk costs are a large part of the issue, yes.  Some of those costs are doubtless social, and some of them are time and efficiency losses while they learn the new way of doing things." CreationDate="2011-04-02T17:53:51.597" UserId="3488" />
  <row Id="15370" PostId="9100" Score="1" Text="The model mispecification issue is the key.  Bayesian methods (especially the &quot;poor-mans&quot; Bayes of evidence maximisation) can perform very poorly under model misspecification, whereas cross-validation seems to work pretty well almost all the time.  The gain when the assumptions (priors) are &quot;right&quot; is generally much smaller than the penalty when they are &quot;wrong&quot;, so cross-validation wins on average (as it makes almost no assumptions).  It isn't nearly as intellectually satisfying though! ;o)" CreationDate="2011-04-02T17:54:40.337" UserId="887" />
  <row Id="15371" PostId="9068" Text="@whuber - the five 2's are highly implausible, but so is any five numbers with continuous distributions.  All the 5 two's mean is that the data do not rule out arbitrarily small variances, so the results do depend on what is known about small variances apart from the data.  the fact that @andree wants a correlation suggests that he has information other than data which indicates non-zero variation.  So why not use that information in the most conservative way, by specifying a lower bound?" CreationDate="2011-04-02T17:58:29.973" UserId="2392" />
  <row Id="15372" PostId="9068" Text="@probability I agree your analysis makes some assumptions explicit, but I don't think they are ones that most people &quot;usually&quot; make.  For instance, one does not normally assume all correlations are equally likely (your uniform prior on $\rho$.)  There are similar unsupported assumptions in the other priors.  Some unstated assumptions are that it even makes sense to use priors and that &quot;uninformative&quot; priors are not appreciably influencing the results.  Only the OP can settle the former question, but the latter question bothers me." CreationDate="2011-04-02T17:58:40.670" UserId="919" />
  <row Id="15373" PostId="9068" Text="@prob An alternative model for five 2's is that measurements are discrete.  That leads to a potentially fruitful investigation of measurement process and measurement error and possibly incorporating it in the model.  *Then*, if you use Bayesian techniques, that's fine, because *the priors can be justified.*  What bothers me is proposing priors without any such analysis.  One ill effect of that approach is that it can inhibit people from thinking constructively about the data." CreationDate="2011-04-02T18:01:10.513" UserId="919" />
  <row Id="15374" PostId="9092" Text="There's no evidence of *lack* of correlation either, so why not plug in 1?  Or -1?  Or anything in between?  They all lead to re-usable code!" CreationDate="2011-04-02T18:02:55.503" UserId="919" />
  <row Id="15375" PostId="9079" Text="+1 That's a creative idea.  It sounds like the &quot;Gower Similarity&quot; is a scaled [Hamming distance](http://en.wikipedia.org/wiki/Hamming_distance)." CreationDate="2011-04-02T18:04:26.673" UserId="919" />
  <row Id="15376" PostId="9097" Text="(+1) Very interesting. Your post also made me insert a comment in my answer." CreationDate="2011-04-02T18:08:21.157" UserId="3911" />
  <row Id="15377" PostId="9040" Text="@whuber, I agree that it helps that this graph is bipartite. On the other hand, I haven't seen compelling graph visualizations for them even of the rather small size the OP is considering. When you bump things up to a bipartite graph from a modern data set with millions (or hundreds of millions) of vertices in each part, such methods really go out the window, I think. Though, I'm willing to be proved wrong. " CreationDate="2011-04-02T19:29:30.923" UserId="2970" />
  <row Id="15378" PostId="9103" Text="In terms of missing values, the simplest approach is to see how many complete cases you have, and if there's any particular bias likely to be introduced by dropping all cases which are not complete (listwise deletion in some statistics packages).  Beyond that you really should repost; missing data is a big issue and a lot more complicated than editing columns." CreationDate="2011-04-02T20:38:44.453" UserId="3919" />
  <row Id="15379" PostId="9097" Text="Perhaps I have missed it, but where is $M_1$ defined?" CreationDate="2011-04-02T21:06:46.650" UserId="2970" />
  <row Id="15380" PostId="9111" Score="1" Text="`dnorm(x)` gives the density function $\varphi(x) = \frac{1}{\sqrt{2\pi}} e^{-x^2/2}$. If you want the density for mean $\mu$ and *standard deviation* $\sigma$, then you can pass these as the second and third parameters, respectively. Use `pnorm` in the same way to get the cumulative distribution function. `qnorm` gives the quantile function (inverse cdf). Typing `?pnorm` will give some more informatio on all three. " CreationDate="2011-04-02T20:42:32.450" UserId="2970" />
  <row Id="15381" PostId="9112" Text="(-1) Sorry. Those are the correct functions, but your description of them is, sadly, incorrect. I'll remove the downvote if you correct this." CreationDate="2011-04-02T20:38:46.263" UserId="2970" />
  <row Id="15382" PostId="9109" Text="How do you mean difficult? Difficult to prepare the program? Difficult to find appropriate initial parameters? Difficult in terms of number of EM steps to reach the result?" CreationDate="2011-04-02T23:32:19.827" UserId="3911" />
  <row Id="15383" PostId="9097" Text="@cardinal, my apologies - its a typo.  will be removed" CreationDate="2011-04-02T23:44:16.060" UserId="2392" />
  <row Id="15384" PostId="9068" Text="@whuber - if the &quot;true&quot; standard deviation of these measurements was say $0.1$ with &quot;true&quot; mean $2$, this is consistent with the observed data. A discrete model would also be useful here.  I think we are debating over certain ambiguities in the question though.  I took pearson correlation to mean that normal distribution was implied - you feel that this is not justified, but do not give an alternative solution and declare this an &quot;ill-posed&quot; problem (implicitly).  And the uniform prior for $p(\rho)$ has nothing to do with the point of my answer." CreationDate="2011-04-02T23:56:59.030" UserId="2392" />
  <row Id="15385" PostId="9068" Text="(cont'd) this is why I left $p(\rho)$ arbitrary, with a possible recommendation for uniform.  the point of my answer was to indicate that even with the most non-informative priors for the other parameters can give some information about the true correlation.  The problem just needs to be specified in a little more detail, to extract what is actually known about the data $x$ and $y$.  The priors I use for the means and standard deviations can be justified on conservative grounds.  If more is known, then the analysis will be improved." CreationDate="2011-04-03T00:03:16.073" UserId="2392" />
  <row Id="15386" PostId="9092" Text="@whuber - you plug in zero because the data is &quot;less constrained&quot; when it is independent - this is why maxent distributions are independent unless you explicitly specify correlations in the constraints.  Independence can be viewed as a conservative assumption when you know of no such correlations - effectively you are averaging over *all possible correlations*." CreationDate="2011-04-03T00:07:04.423" UserId="2392" />
  <row Id="15387" PostId="9100" Text="@dikran - interesting.  I'm not so sure I agree with what you say though.  So you say if the model is mispecified, then cross validation with that same model is better than using Bayes theorem?  I would like to see an example of this." CreationDate="2011-04-03T00:16:30.580" UserId="2392" />
  <row Id="15388" PostId="9097" Text="@probabilityislogic, no apology necessary. Just wasn't sure if I had skipped over it while reading or was simply missing something obvious. " CreationDate="2011-04-03T00:43:55.483" UserId="2970" />
  <row Id="15389" PostId="9098" Text="Good advice about visualising the data. (+1)" CreationDate="2011-04-03T00:56:58.767" UserId="2392" />
  <row Id="15390" PostId="9098" Text="@probabilityislogic Thanks! I also welcome criticism, particularly if constructive." CreationDate="2011-04-03T01:16:37.127" UserId="3911" />
  <row Id="15391" PostId="9109" Text="@GaBorgula: I mean the learning curve in general." CreationDate="2011-04-03T01:19:54.817" UserId="2729" />
  <row Id="15392" PostId="1772" Text="@user956 - I would disagree with this.  Neither theory nor experiment are &quot;in charge&quot; of the other - they work together.  Sometimes the theory leads to an experiment, we have an untested hypothesis we want to confirm or deny with some data." CreationDate="2011-04-03T01:20:16.917" UserId="2392" />
  <row Id="15393" PostId="2569" Text="nice quote - and you can see the inherent &quot;pessimism&quot; in stats folk, for she speaks of &quot;false discoveries&quot; and not of &quot;missed discoveries&quot; (which are just as important)." CreationDate="2011-04-03T01:24:27.943" UserId="2392" />
  <row Id="15394" PostId="9098" Text="@Gaborgulya - the only criticism I have is the use of p-values, but this is more a &quot;chip on my shoulder&quot; than anything in your answer - seems like if you are going to use p-values, then what you recommend is good.  I just don't think using p-values is good.  [see here](http://stats.stackexchange.com/questions/6601/what-is-the-hardest-statistical-concept-to-grasp) for my exchange with @eduardo in the comments about p-values." CreationDate="2011-04-03T01:32:41.137" UserId="2392" />
  <row Id="15395" PostId="9106" Score="3" Text="It's not mathematically possible for &quot;all three skewness measures to be unbiased,&quot; because (obviously) their expectations all differ.  Perhaps you mean *asymptotically* unbiased?" CreationDate="2011-04-03T02:10:21.683" UserId="919" />
  <row Id="15396" PostId="9106" Text="@whuber - I'm going to defer to Friedrich.Leisch@R-project.org who maintains the `e1071` package for clarification on what he meant specifically there. If my post wasn't clear, that comes from the [help page for `skewness()`](http://rgm2.lab.nig.ac.jp/RGM2/func.php?rd_id=e1071:skewness)" CreationDate="2011-04-03T02:15:07.607" UserId="696" />
  <row Id="15397" PostId="9112" Text="Downvote removed following revision. " CreationDate="2011-04-03T02:46:50.800" UserId="2970" />
  <row Id="15398" PostId="9085" Text="I think you need to provide a few more bits of information: must it run under a particular OS, and what do you mean by &quot;data exploration&quot;? If they're using Excel pivot tables and graphing and that's it, what advantage is there to, say, gretl?" CreationDate="2011-04-03T03:14:33.960" UserId="1764" />
  <row Id="15399" PostId="9118" Score="1" Text="@goober - good answer, it could be improved by writing out the steps to get each number, e.g. $6.67\text{%}=\frac{10}{10+100+40}\times 100\text{%}$.  similarly for second step $4.67\text{%}=6.67\text{%}\times 70\text{%}$.  If you have the equations as well as the words, people are less likely to get confused" CreationDate="2011-04-03T03:38:18.910" UserId="2392" />
  <row Id="15400" PostId="9060" Text="Thanks a lot for this reference." CreationDate="2011-04-03T04:05:41.330" UserId="3840" />
  <row Id="15402" PostId="9106" Score="1" Text="@whuber I thought the same so I upvoted your comment then started reading the Joanes &amp; Gill paper (start of section 3, p185) and realised we're wrong : a normal distribution has zero skewness, so any estimator that is a multiple of $g_1$ is unbiased *under normality*. Unfortunately [there's no way to undo a comment upvote](http://meta.stats.stackexchange.com/questions/764/how-to-unvote-on-comments/765#765)." CreationDate="2011-04-03T06:23:48.990" UserId="449" />
  <row Id="15403" PostId="9121" Score="1" Text="Could you add a couple of links to the couple of articles in question?" CreationDate="2011-04-03T06:33:17.313" UserId="449" />
  <row Id="15405" PostId="9079" Text="@whuber: Indeed it is !" CreationDate="2011-04-03T08:30:27.883" UserId="264" />
  <row Id="15406" PostId="9110" Text="@Neil Why is it so? The OP specified &quot;but ideally which would extend to programming...&quot;. R has been plugged in almost all commercial (e.g., SPSS, JMP, Statistica) or free (Knime, Rapidminer) statistical software and Rserve can be used as a backend to communicate with R (e.g., with the plink software for genetic studies), and it is used for this purpose. The Free Trial version of JMP doesn't let you work with external datasets, am I wrong? (Interestingly, all of us who answered R, *and an alternative software*, were downvoted.)" CreationDate="2011-04-03T09:02:52.720" UserId="930" />
  <row Id="15407" PostId="9121" Text="Judging by the limited description, I think you are right @MYaseen208." CreationDate="2011-04-03T09:37:35.527" UserId="1307" />
  <row Id="15408" PostId="9116" Text="@goober This question was tagged with &quot;pie-chart&quot; but it seems to involve some kind of intermediate computation, and your own response didn't address the data visualization issue. Could you update it accordingly?" CreationDate="2011-04-03T10:48:19.990" UserId="930" />
  <row Id="15409" PostId="9070" Text="Well, there is a JT test statistic, and a z equivalent; but it's not something like an odds ratio.  So, I guess it depends on exactly what you mean by &quot;measure of association&quot;" CreationDate="2011-04-03T11:09:34.153" UserId="686" />
  <row Id="15410" PostId="9106" Score="1" Text="The point is that $g_1 = m_3 / m_2^{3/2}$, where $m_2$ and $m_3$ are the second and third moments about the mean, is the population skewness.  As a sample statistic, it then raises similar issues to unbiased estimates of standard deviation, leading to corrections based on $n$ which have some justification but still do not produce unbiased estimates.  But I think it is not very helpful to say that these are unbiased estimates of skewness for symmetric distributions; so too is 0, which has a lower variance but is inconsistent and is useless for estimating skewness of asymmetric distributions. " CreationDate="2011-04-03T12:44:02.560" UserId="2958" />
  <row Id="15411" PostId="8823" Text="I feel like there has to be a more intuitive solution than the algebra given in Sanders' partial solution.  If the data are independent given each of the hypotheses then this continues to hold when the priors of the hypothesis are varied.  And somehow, the result is that the same must apply for the conclusion..." CreationDate="2011-04-03T12:58:45.740" UserId="3567" />
  <row Id="15412" PostId="8823" Text="@charles - I know exactly how you feel.  I thought I could derive it using some qualitative inconsistency (Reductio ad absurdum), but I couldn't do it.  I Could extend Sander's maths though.  And it is Condition 2 which is &quot;the dodgy one&quot; in terms of what the result means." CreationDate="2011-04-03T13:19:54.653" UserId="2392" />
  <row Id="15413" PostId="9100" Text="@probabiltyislogic I don't think it is a particularly new observation, Rasmussen and Williams mention it on page 118 of their excellent Gaussian Process book (although it is essentially a reference to a similar comment in Grace Wahba's monograph on splines).  Essentially the marginal likelihood is the probability of the data given the assumptions of the model, whereas the XVAL likelihood is an estimate of the probability of the data, regardless of the model assumptions, hence more reliable when the assumptions are not valid.  A proper empirical study would be useful." CreationDate="2011-04-03T14:06:20.800" UserId="887" />
  <row Id="15414" PostId="9100" Text="@probabilityislogic I should add that I like the Bayesian approach to model selection, but I almost always used cross-validation in practice simply because it generally gives results that are (statistically) as good as, or better than Bayesian approaches." CreationDate="2011-04-03T14:08:46.480" UserId="887" />
  <row Id="15415" PostId="9118" Text="Here are a couple things that may make you trust that you've done it properly. These may also make better information displays: (1) Scale the pie charts according to the number of contracts per company pie; (2) Draw stacked bar charts (same as pie charts but in a line instead as a circle), cut them out and arrange the capture and non-capture cut-outs in two bars." CreationDate="2011-04-03T15:04:12.530" UserId="3874" />
  <row Id="15417" PostId="9110" Text="i like R and am in favour of it, but people that want to learn stats/datamining and don't know how to program really can't use it. and i've played with all of its guis and they are just horrible compared to jmp. i am just waiting for the day when R gets a gui like jmp, then all will be well w the universe. " CreationDate="2011-04-03T16:52:41.567" UserId="74" />
  <row Id="15418" PostId="9110" Text="@Neil Thanks for the clarification. " CreationDate="2011-04-03T17:23:27.910" UserId="930" />
  <row Id="15419" PostId="9106" Text="@onestop @Henry I agree with you." CreationDate="2011-04-03T17:45:28.413" UserId="919" />
  <row Id="15420" PostId="9130" Text="Thanks. Actually I already do that&#xA;(https://github.com/barrycarter/bcapps/blob/master/bc-get-cycle-files.pl),&#xA;but that data only goes back 24 or so hours and isn't archived as far &#xA;as I can tell. I am looking for international data. As an example, &#xA;http://ns1.data.barrycarter.info/HOURLY/ is what I have so far. I took &#xA;a look at the berkeleyearth.org site. The dataset sounds exciting, &#xA;though I wonder if &quot;4 daily and 6 monthly&quot; means they have no hourly &#xA;data? &#xA;" CreationDate="2011-04-03T17:52:49.243" UserId="1566" />
  <row Id="15421" PostId="9131" Text="The *Confidence Interval on New Observations* section of http://www.weibull.com/DOEWeb/confidence_intervals_in_multiple_linear_regression.htm may help." CreationDate="2011-04-03T18:32:46.263" UserId="3911" />
  <row Id="15422" PostId="9114" Text="The OP states they're interested in multivariate mixture modeling. The linked paper appears to treat only the univariate case. Is there an obvious extension to multivariate observations that remains clean, accessible, and for which the computation grows (reasonably) slowly with the dimensionality?" CreationDate="2011-04-03T19:10:25.777" UserId="2970" />
  <row Id="15423" PostId="9129" Score="1" Text="What is your ultimate goal for the clustering? Are you familiar with basic preprocessing and modeling in IR like TF-IDF weights and vector-space modeling?" CreationDate="2011-04-03T19:13:34.867" UserId="2970" />
  <row Id="15424" PostId="9104" Text="Is your `size=100` a typo? I don't believe you're getting the right upper and lower bounds since the implicit sample size appears to be 1000 when you calculate your CIs in the loop (since you use `sqrt.n` in the calculation). Also, why are you comparing to `mu` and not 0 directly (the latter being the true mean)?" CreationDate="2011-04-03T19:24:53.340" UserId="2970" />
  <row Id="15425" PostId="9104" Text="Also, `smpl = x[sample(1:n, size = 100, replace = TRUE)];&#xA;` can be simplified to `smpl = sample(x, size=100, replace=TRUE)`." CreationDate="2011-04-03T19:25:29.610" UserId="2970" />
  <row Id="15426" PostId="9131" Text="@Tal Sorry, but it's not really clear to me what you actually mean by &quot;predict the lower and upper predictions of y&quot;. Does it have something to do with prediction or tolerance bands?" CreationDate="2011-04-03T19:43:19.120" UserId="930" />
  <row Id="15427" PostId="9131" Text="@Tal - a couple of queries. When you say &quot; .. y based on x1 and x2, using a OLS regression.&quot; , you mean your create a linear model and _estimate parameters using OLS_. Am I right? and @chl's question -- do you want to predict the lower and upper bounds for the prediction interval? " CreationDate="2011-04-03T19:55:39.703" UserId="1307" />
  <row Id="15428" PostId="9131" Text="@chl, sorry for not being more clear.  I am looking for two formulas that will give an interval for that will &quot;catch&quot; the &quot;real&quot; value of y 95% of the time.  I feel how I'm using definitions for the CI for the mean, when there is probably some other term I should be using, sorry about that..." CreationDate="2011-04-03T20:02:11.967" UserId="253" />
  <row Id="15429" PostId="9131" Text="@suncoolsu - yes and yes.  " CreationDate="2011-04-03T20:03:17.770" UserId="253" />
  <row Id="15430" PostId="9132" Score="1" Text="Can you edit and add more context? It's hard for me to tell from your current description what you are doing and what your concerns are." CreationDate="2011-04-03T20:21:03.053" UserId="2970" />
  <row Id="15431" PostId="9132" Text="Edited question." CreationDate="2011-04-03T20:30:19.983" UserId="4010" />
  <row Id="15432" PostId="9129" Text="Essentially this is a standard text mining task, thus achievable in many ways; did your teacher suggest any tools? Also, 1k records is still quite a small data (ok, small enough that in this case the computational performance is not a major software selection factor), so will you need to scale up?" CreationDate="2011-04-03T20:56:33.057" UserId="88" />
  <row Id="15433" PostId="9104" Text="@cardinal - Yes it was a typo and same with `mu` being 0. The normal CI works fine, it's the basic bootstrap CI that I've having difficulty with." CreationDate="2011-04-03T21:47:13.393" UserId="1894" />
  <row Id="15434" PostId="9120" Text="When you started to integrate, you introduced a term $s^2$.  It persists through the final answer.  What is it?" CreationDate="2011-04-03T22:13:39.943" UserId="919" />
  <row Id="15435" PostId="9134" Text="Hi Dirk, that is indeed what I wish to find, but I want the upper and lower bonds to be in the form of a *formula* (so to later implement in some low form of statistical software, for example, excel...)" CreationDate="2011-04-03T22:16:28.493" UserId="253" />
  <row Id="15436" PostId="9134" Text="p.s: I now see that there was an edit to the title of my question that might had led you to think I was asking about predict.lm interval parameter (which I am not)  :)" CreationDate="2011-04-03T22:18:48.320" UserId="253" />
  <row Id="15437" PostId="9134" Score="3" Text="You are abusing terminology here.  Excel is not statistical software.  " CreationDate="2011-04-03T22:23:23.470" UserId="334" />
  <row Id="15438" PostId="2146" Text="Are you trying to decide if a pair of voice recordings is stereo recording of the same sound source or coming from two independent sources?" CreationDate="2011-04-03T22:32:46.813" UserId="3911" />
  <row Id="15439" PostId="2146" Text="@GaBorgulya: Independent sources. The other questions are technical and not statistical." CreationDate="2011-04-03T22:35:51.800" UserId="1098" />
  <row Id="15440" PostId="9120" Text="@whuber - $s$ is the standard deviation MLE, $s^2=N^{-1}\sum_{i=1}^{N}(Y_i-\overline{Y})^2$.  I thought it was the usual notation for standard deviation, just as $\overline{Y}$ is usual for average - which I have incorrectly written as $\overline{x}$, will edit accordingly" CreationDate="2011-04-03T22:39:06.077" UserId="2392" />
  <row Id="15441" PostId="9134" Text="You're right, my bid, how about &quot;a spreadsheet application&quot; ?" CreationDate="2011-04-03T22:42:16.630" UserId="253" />
  <row Id="15442" PostId="2146" Text="In what sense do your audio streams form pairs then?" CreationDate="2011-04-03T22:43:53.890" UserId="3911" />
  <row Id="15443" PostId="9114" Text="There is, but you would have to do the maths - I've had a go, and you get a mixture of multivariate student densities, but I can't exactly remember the formula, the digamma functions are a bit more tricky to approximate in the multivariate case." CreationDate="2011-04-03T22:46:56.127" UserId="2392" />
  <row Id="15444" PostId="9134" Score="2" Text="I can live with that; it calls the devil by its name ;-)" CreationDate="2011-04-03T22:52:46.923" UserId="334" />
  <row Id="15445" PostId="2294" Text="I'm trying to imagine a situation that is matching your model. Maybe a physician checks their patients weekly to see if they are (1) still sick, (2) dead, (3) cured or (4) lost to follow-up. Are you trying to estimate the dead/cured ratio assuming that the transition probabilities of LFU patients are the same?" CreationDate="2011-04-03T23:24:03.900" UserId="3911" />
  <row Id="15446" PostId="9137" Score="1" Text="You may want to look at http://en.wikipedia.org/wiki/Pareto_efficiency#Pareto_frontier" CreationDate="2011-04-03T23:30:47.047" UserId="3911" />
  <row Id="15447" PostId="8867" Text="A lower bound for the Fisher information in this case is $1/(1+2c^2)$. Is it possible to get a tighter uppper bound on the Fisher information than the general $1 + 1/c^2$? " CreationDate="2011-04-03T23:56:44.590" UserId="530" />
  <row Id="15448" PostId="9137" Text="Assuming higher quality and lower price are better, you can in this particular case reject B and C since they both have a lower quality and a higher price than A.  There are other combinations of values where you need a more sophisticated approach,  " CreationDate="2011-04-04T00:33:31.973" UserId="2958" />
  <row Id="15449" PostId="9100" Text="Cross validation selects models based solely on predictive performance; marginal likelihoods don't - they &quot;account&quot; for every dimension. In very high dimensional settings this matters - sometimes a lot. Say you've got a big predictor vector $X_i$ and a 1 dimensional response $y_i$. You need a model for $X_i$ to do dimension reduction in a fully Bayesian way. So you specify a joint model as $p(y_i|X_i, \theta_y)p(X_i|\theta_X)$. The second term has a much bigger contribution to the likelihood, so if a model does well there and bites it on the prediction the marginal likelihood won't care." CreationDate="2011-04-04T00:34:48.053" UserId="26" />
  <row Id="15450" PostId="9100" Text="... Here's a paper describing just this situation, for a particular model: http://ftp.isds.duke.edu/WorkingPapers/10-14.html. But the problem exists in big joint models regardless - the question is whether there is disagreement between the two &quot;pieces&quot; of the likelihood." CreationDate="2011-04-04T00:37:13.493" UserId="26" />
  <row Id="15454" PostId="9093" Text="@John - Good points, but there some caveats here: Excel has tool tips that pop up when entering a formula in a cell and so it makes it much easier to &quot;program&quot; simple tasks. Also, chances are the person next to you can help if you get stuck since there are lots of Excel power users. And, Excel comes installed. Convincing someone to try R involves helping them set it up and learn how to leverage open source software (and which forums to search, which of the umpteen bootstrap packages should be used, etc). The programming is in some sense the easy part. Its the ecosystem that is the hard part." CreationDate="2011-04-04T02:18:22.040" UserId="1080" />
  <row Id="15456" PostId="9114" Text="Hmmm. If that's true, the approach of the paper sounds more appealing and appropriate as an alternative to univariate kernel density estimation than it does as an alternative to estimation using the EM algorithm for multivariate mixture modeling. " CreationDate="2011-04-04T02:34:40.370" UserId="2970" />
  <row Id="15457" PostId="2294" Text="That's exactly analogous to my situation. It is about apprentices, who can be continuing, quit, completed or unknown outcome." CreationDate="2011-04-04T03:08:55.933" UserId="1144" />
  <row Id="15458" PostId="9144" Score="1" Text="+1, excellent answer. I should note though, that regression model always estimates conditional expectation, so it is as good as its regressors are. So the last comment although is very good, it is not strictly necessary, since if you build regression model you must trust the regressors." CreationDate="2011-04-04T03:51:50.160" UserId="2116" />
  <row Id="15459" PostId="9099" Text="a link to a reference paper on &quot;Griffing and Hayman diallel analysis&quot; would be very helpful for non geneticists" CreationDate="2011-04-04T04:32:51.460" UserId="1709" />
  <row Id="15460" PostId="9150" Text="I more or less agree with the sentiment, but I don't think this high-minded view is really possible in many settings." CreationDate="2011-04-04T05:19:47.833" UserId="3488" />
  <row Id="15461" PostId="9093" Text="Also there is no real formatting issues in Excel... In R sometimes one package uses xts, another one data.frames... it's a mess for a newbie, that's true" CreationDate="2011-04-04T06:18:38.510" UserId="1709" />
  <row Id="15462" PostId="9132" Text="@cardinal: does this answer your question." CreationDate="2011-04-04T07:09:59.513" UserId="4010" />
  <row Id="15463" PostId="9114" Text="@cardinal - good point, but the multivariate generalisation is not polemic, just follows from using a multivariate guassian mixture model, and doing the same &quot;splitting&quot; of the data into &quot;location learning&quot; and &quot;scale learning&quot;.  Same with the choice of components.  All the necessary principles of what is being done are there - Jose Bernardo just hasn't done the maths." CreationDate="2011-04-04T07:51:20.637" UserId="2392" />
  <row Id="15467" PostId="9129" Text="Thanks for the quick response guys.&#xA;My end goal is to create a spam filter for the strings. Basically I have the list of strings (as mentioned), I have manually input a number of what I consider spam strings. I want to be able to cluster the strings based on term freq and from this see that spam has clustered together. Once I get my head around the problem I will probably scale to maybe 10,000 records for my testing but that is still quite a small data set.&#xA;I have not been recommended any tools so I am searching the web. " CreationDate="2011-04-04T08:26:45.480" UserId="4020" />
  <row Id="15468" PostId="9110" Text="-1 for suggestion that people do not read questions before answering and for not providing any real arguments why R is worse than JMP or RapidMiner. Note that in general I agree that R is not an answer to everything, but this topic is a bit contraversial with the potential to get really nasty. Hence the downvote." CreationDate="2011-04-04T10:42:04.783" UserId="2116" />
  <row Id="15469" PostId="9085" Score="1" Text="I think that this [comic](http://ars.userfriendly.org/cartoons/?id=20031106&amp;mode=classic) might be appropriate representation of the situation. If the person you are refering is dead set against programming, this may be too strong a preference to go against. A better strategy might be to point out the deficiencies of spreadsheet based analysis. Try for example in new Excel file to enter values $1\frac{1}{6}$ and $1\frac{1}{4}$ and sum them (this works on Excel 2007). Then explain that no self-respecting statistical package will provide similar mistake without any warning and work from that." CreationDate="2011-04-04T10:53:15.440" UserId="2116" />
  <row Id="15470" PostId="9115" Text="Thanks for your great answer. It would be very helpful if you could you detail how you reach this conclusion? &#xA;&quot;The block 380-399 with 42246 people should be about 1.6 times the density of the block 400-499 with 132485 people&quot;.&#xA;&#xA;Would it be incorrect to divide the 132485 people with 5 (because the interval is 5 times bigger than the normal interval)? Thanks." CreationDate="2011-04-04T11:34:09.070" UserId="4003" />
  <row Id="15471" PostId="9157" Text="Ok, but I think nested cross-validation doesn't validate a particular parameters choise. It might validate this classification technique but not a particular parameters choise. I should find out new data, or optimize with c-v in a subset and validate with remaining records." CreationDate="2011-04-04T12:07:52.490" UserId="2719" />
  <row Id="15472" PostId="9115" Score="1" Text="@user4003: If you base your graph on bands of width 20 then yes, you should divide the 132485 by 5, as it will be 5 bands wide. The density ratio of about 1.6 comes from $\frac{42246}{400-380} \div \frac{132485}{500-400}$" CreationDate="2011-04-04T12:46:19.240" UserId="2958" />
  <row Id="15473" PostId="9157" Text="@Simone, yes, either of those approaches would be fine.  However if the model is &quot;very variable&quot;, that suggests that there isn't really enough data to fit the model properly, so any single training/test split is likely to give a highly variable plot.  It might be worth adding error bars to the plot, by bootstrapping when computing the observed relative frequencies?" CreationDate="2011-04-04T13:22:18.100" UserId="887" />
  <row Id="15474" PostId="9157" Text="@Simone alternatively, you could just overlay the plots for each fold of the cross-validation procedure, shich would give an indication of the &quot;reliability&quot; of the reliability plot." CreationDate="2011-04-04T13:23:38.083" UserId="887" />
  <row Id="15475" PostId="9148" Text="What about open access scientific publications? One of the large collections is http://www.biomedcentral.com/ ." CreationDate="2011-04-04T14:02:39.577" UserId="3911" />
  <row Id="15476" PostId="9159" Score="1" Text="I think it will depend a bit on the field - can you give us some more context?" CreationDate="2011-04-04T14:31:50.380" UserId="71" />
  <row Id="15477" PostId="9155" Text="What do the arrows mean in the table headings?" CreationDate="2011-04-04T14:36:37.363" UserId="919" />
  <row Id="15478" PostId="9158" Text="(+1) Censoring is the key idea; time is just a particular (and the most common) application.  For instance, [Dennis Helsel](http://www.practicalstats.com/nada/) has applied survival analysis to chemical measurements having detection limits (left censoring).  There's no time involved, but the model and the techniques are still useful." CreationDate="2011-04-04T14:40:13.447" UserId="919" />
  <row Id="15479" PostId="9142" Text="(+1) for making the distinction.  However, I believe the OP is asking for (1), not (2) (and I have edited the question's title accordingly).  Also note that your formula appears to assume the regression depends only on one variable." CreationDate="2011-04-04T14:48:18.977" UserId="919" />
  <row Id="15480" PostId="9161" Text="Wow! - Awesome explanation on what I was doing wrong. Also - thanks for the code tips! This works perfectly!" CreationDate="2011-04-04T15:36:58.110" UserId="1894" />
  <row Id="15481" PostId="9161" Text="@caracal - Ok one last question: when I try to replicate this information, I created a function `computeCIs` and called `results = replicate(500, computeCIs());`. At the end of `computeCIs` it returns `c(ciBasic, ciPerc)`. To test the coverage probabilities, shouldn't I then test for `mean(results[1, ] &lt; 0 &amp; results[2, ] &gt; 0)` to test for all Basic CI's containing the true mean (the coverage probability)?  When I run this, I get `1` when I think I should get `0.95`." CreationDate="2011-04-04T16:02:42.547" UserId="1894" />
  <row Id="15482" PostId="9148" Text="I'm curious what kind your goal is, how do you want to use the data?" CreationDate="2011-04-04T16:03:49.780" UserId="1146" />
  <row Id="15483" PostId="9165" Score="1" Text="I'm not sure if something like the &quot;gene set enrichment analysis&quot; can be used for DNA methylation as well, but if yes that may help you to aggregate collections of genes into metagenes." CreationDate="2011-04-04T16:31:32.880" UserId="3911" />
  <row Id="15485" PostId="9093" Text="fRed, I agree that there can be issues in data formatting but keep in mind that my response was qualified as analysis that already could be done in a spreadsheet.  That's not a lot of varied analysis and would generally be under one data format." CreationDate="2011-04-04T16:55:42.460" UserId="601" />
  <row Id="15486" PostId="9093" Text="@John - There is somewhat more overhead in &quot;programming&quot; with R than with Excel, and these are the things that drive users crazy. If R had a top notch GUI akin to a Pivot Table, maybe.  But until then..." CreationDate="2011-04-04T17:23:16.530" UserId="3489" />
  <row Id="15487" PostId="9165" Score="1" Text="@EdS Not sure whether you know her work, but [Anne-Laure Boulesteix](http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/eng.html) have some papers/reports on survival coupled to high-dimensional genetic data." CreationDate="2011-04-04T18:00:39.847" UserId="930" />
  <row Id="15488" PostId="9159" Score="1" Text="The phrase &quot;gold standard&quot; comes from when you could exchange a currency for gold, guaranteeing its value. Thus any &quot;gold standard&quot; item is a benchmark and of the highest quality... which is a matter of consensus or opinion, not some kind of statistical property." CreationDate="2011-04-04T18:09:30.797" UserId="1764" />
  <row Id="15489" PostId="9157" Text="Every plot of each cross-validation uses different attributes and find out different cut-off, ovelay them could be misunderstood. How could I get error bars? It seems that you are familiar with Reliability Plot. For each leaf I will find out the corrisponding test records, then I'll get different probability estimation bootstrapping in order to let change only the dot ordinate position. Does it work?" CreationDate="2011-04-04T18:13:54.810" UserId="2719" />
  <row Id="15490" PostId="8704" Text="@Dmitrij: What you are suggesting is what I thought at first. However, I^2 and R are not given to be independent. So how would you compute the cdf and pdf of I^2*R? Also, I didn't find the solution to this question on the book by H.Hsu you suggested? Would be helpful if you could point out the chapter or page number. Thanks!" CreationDate="2011-04-04T18:15:13.000" UserId="1102" />
  <row Id="15491" PostId="9165" Text="@GaBorgulya +1, though from a quick Google and a v brief skim i like the look of [this &quot;simple alternative&quot;](http://smm.sagepub.com/content/18/6/565) ([poss. more freely accessible preprint](http://www.bepress.com/jhubiostat/paper185/))" CreationDate="2011-04-04T18:19:34.813" UserId="449" />
  <row Id="15492" PostId="9155" Score="1" Text="You might get more helpful responses if you have a particular application in mind.  The term &quot;Granger causality,&quot; is unfortunate, given that strong evidence of &quot;Granger causality,&quot; may be weak evidence of &quot;causality.&quot;" CreationDate="2011-04-04T18:36:17.530" UserId="3748" />
  <row Id="15493" PostId="1041" Score="2" Text="I'll nitpick your answer I basically agree with.  @svadali said: &quot;If you know that the model is incorrect then obviously you would not use it to compute the confidence interval.&quot;  But to paraphrase George Box: all models are incorrect, sometimes computing confidence intervals is useful anyway." CreationDate="2011-04-04T18:49:38.007" UserId="3748" />
  <row Id="15494" PostId="9171" Text="Could you tell us the difference between (1) and (2)?  Clearly something changed, because Model 3 in (1) and Model 2 in (2) are nominally identical but the p-values and AIC differ." CreationDate="2011-04-04T18:52:28.670" UserId="919" />
  <row Id="15495" PostId="1040" Text="There is no general interpretation for parameters in misspecified models.  It depends on the details of the misspecification.  A confidence interval maybe useful anyway if the misspecification isn't too bad." CreationDate="2011-04-04T18:54:12.063" UserId="3748" />
  <row Id="15496" PostId="9166" Text="ehh .. missed the article somehow. Thanks for the explanation (and the link)." CreationDate="2011-04-04T18:55:08.250" UserId="2192" />
  <row Id="15497" PostId="9132" Text="Why use only one predictor variable? When you say &quot;b-valued&quot; do you mean dichotomous?  " CreationDate="2011-04-04T19:00:01.187" UserId="3748" />
  <row Id="15498" PostId="9165" Text="@chl, kudos for the Anne-Laure Boulesteix link - I was unaware of her work, but it seems to include answering the precise question that I have asked here." CreationDate="2011-04-04T19:02:07.410" UserId="3429" />
  <row Id="15499" PostId="9169" Text="@GaBorgulya, thanks for the comments. The bimodality criterion is purely to ensure that any selected features are clinically relevant, since measuring DNA methylation is still fairly crude, and in a routine lab setting, the most sensitive you can hope for DNA methylation is unmethylated, hemi-methylated or methylated, so these extreme bimodal values are most useful" CreationDate="2011-04-04T19:03:56.707" UserId="3429" />
  <row Id="15500" PostId="9148" Text="@John Salvatier the goal is redacted :D ... actually I want to do some analysis on what kind of information generally gets redacted in free form text documents. In other words what sorts of information do people consider sensitive - and if its possible to use some machine learning algorithms to aid automatic redaction. I read sometime back Xerox had something like this - but there's no more info on how that can be done. I have some idea I want to try out, but no data ..." CreationDate="2011-04-04T19:05:45.880" UserId="2192" />
  <row Id="15512" PostId="9110" Text="R is great, but it is not easy. That's what he asked for in the title. " CreationDate="2011-04-04T20:01:03.643" UserId="74" />
  <row Id="15515" PostId="9161" Text="@TheCloudlessSky For the complete function and full simulation with the expected results in terms of coverage frequencies, see http://pastebin.com/qKpNKK0D" CreationDate="2011-04-04T20:33:20.517" UserId="1909" />
  <row Id="15517" PostId="9171" Text="@MEL This question has been reposted two times, which means that not only we have to close them, but also the associated response(s) that were already provided to you. Could you please register your account (see the [FAQ](http://stats.stackexchange.com/faq)), and pay attention to StackExchange posting policy in the future? Thanks." CreationDate="2011-04-04T20:56:25.493" UserId="930" />
  <row Id="15518" PostId="9175" Text="@Bogdan Sorry if this seems like a stupid question, but what do you call &quot;bounds provided by software&quot;?" CreationDate="2011-04-04T21:01:23.460" UserId="930" />
  <row Id="15519" PostId="9166" Text="@nico @Tathagata The real question is: What do we do when there's no gold standard, which is quite frequent in diagnostic medicine (except through autopsy!) or clinical assessment/screening :-) Then comes in various flavour the concept of *convergent validity*, etc." CreationDate="2011-04-04T21:22:05.023" UserId="930" />
  <row Id="15520" PostId="9168" Score="1" Text="@GaBorgulya This will depend on the number of predictors, though. I'd favour L1/L2 penalties (or the elasticnet criteria). Furthermore, I think the issue here is that overfitting will result from selecting genes (or meta-genes) first, and then testing a model on this particular feature subspace. What do you think?" CreationDate="2011-04-04T21:26:40.057" UserId="930" />
  <row Id="15523" PostId="9161" Text="@caracal - Yup, I'm an idiot :)... I made a typo when copying the code in R... thanks for all your help! :)" CreationDate="2011-04-04T22:28:26.080" UserId="1894" />
  <row Id="15524" PostId="9155" Text="I agree with Michael. I have seen it used for explorative purposes by economists and political scientists working with time series, but it really all depends on the data, as it is used mostly to think more about causality and further methods." CreationDate="2011-04-04T23:33:58.917" UserId="3582" />
  <row Id="15525" PostId="9175" Text="Something to do with computational limits, as with the max. number of obs/vars dealt by different versions of Stata SE/IC/MP perhaps?" CreationDate="2011-04-04T23:37:12.157" UserId="3582" />
  <row Id="15526" PostId="9174" Text="If your goal is prediction accuracy, you want to use AIC (as it minimises the expected KL divergence between the fitted model and the truth). If you want a consistent model selection procedure (fixed p, growing n), you may use, say, BIC instead. Using p-values in stepwise regression to select hypotheses is definitelly not recommended." CreationDate="2011-04-04T23:54:45.397" UserId="530" />
  <row Id="15527" PostId="9182" Score="1" Text="I don't mean to sound negative, but in your paragraph you've progressed from what sound like interesting and useful questions (do the groups show differences in this or that respect) to a much less interesting one (can they be grouped together--yes or no).  What makes you dissatisfied with the nuanced approach and steers you toward that seemingly sterile, either-or question?" CreationDate="2011-04-05T00:01:16.780" UserId="2669" />
  <row Id="15528" PostId="8589" Text="I had thought there were 3 or 4 tried and true methods, such as Levene's, for testing the difference between variances. Why would one want to turn to these intricate and improvised approaches?" CreationDate="2011-04-05T00:43:54.250" UserId="2669" />
  <row Id="15530" PostId="9155" Text="@Michael Bishop: I agree that the term causality is not the best. I use it alongside with Granger. I am actually not looking for causality (Indeed Z may cause both X and Y) but maybe something like &quot;X leads Y&quot;, and how it is different from doing the lagged correlation, for instance" CreationDate="2011-04-05T01:48:05.110" UserId="1709" />
  <row Id="15531" PostId="9155" Text="This is used all the time in fMRI analysis as well" CreationDate="2011-04-05T05:26:37.033" UserId="2660" />
  <row Id="15532" PostId="9166" Text="@chl: in effect, when thinking about *gold standard* for whatever reasons my mind thinks more of chemistry than medicine ;)" CreationDate="2011-04-05T06:36:26.300" UserId="582" />
  <row Id="15534" PostId="9175" Text="@chl @Fr. [&quot;Some normal probability plots include bounds that depict the natural variability one could see in data even when it comes from a normal distribution&quot;.](http://www.public.iastate.edu/~wrstephe/stat403/probplot.html)" CreationDate="2011-04-05T07:25:27.110" UserId="449" />
  <row Id="15536" PostId="9175" Text="Those curved bounds are just a formalization of &quot;how linear&quot;: if your data were completely linear on the quantile-quantile plot, it would match the expectation exactly, which is extremely unlikely and suggests manipulation. So you should allow for a slightly ragged line, and those bounds are the result of one way to define the &quot;fat marker&quot; that you can smooth the line with. If the data were naturally curved but within the bounds, a larger sample would push the bounds closer to each other until the curve no longer fit inside." CreationDate="2011-04-05T09:28:35.290" UserId="2456" />
  <row Id="15537" PostId="8704" Text="@H_S, if $I$ and $R$ are independent so are the functions of $I$ and $R$, therefore $I^2$ and $R$ are independent too (hope you do not mix it with correlation). So you have that $f_{IR}=f_I\cdot f_R$ from the original independence of $I$ and $R$. What you need to add is a specific region of integration for the event $Z=I^2R$. Alternatively you may go for two steps, calculate $I^2$ from example 4.6 in Hsu for the 1st one, and after go for 4.21 (and look for 4.29 for a more general example of 3 independent r.v.)." CreationDate="2011-04-05T11:34:32.810" UserId="2645" />
  <row Id="15538" PostId="9157" Text="@Simone I think the distinction is whether you are evaluating the particular model, or the process used to generate the model.  In this case the former is likely to be very difficult as it sounds as if there isn't really enough data to generate a stable model (and hence the test data will be too small to give a stable estimate of generalisation).  The overlay method (or bootstrap estimates of the error bars) are more estimates of the performance of the method used to generate the model, at least the performance of the particular model should be within those error bars." CreationDate="2011-04-05T11:51:28.200" UserId="887" />
  <row Id="15539" PostId="9168" Text="Thanks to the links and thoughtful commentary provided, I've been led to [Coxboost](http://cran.r-project.org/web/packages/CoxBoost/index.html), a technique that enables me to include established covariates and add molecular information if it usefully extends the current model. It seems to do the trick, although I need to study it more to understand more fully what is going on with that approach." CreationDate="2011-04-05T12:23:55.720" UserId="3429" />
  <row Id="15540" PostId="9168" Text="@chl Thanks for your comment, I have to read and think about it!" CreationDate="2011-04-05T12:29:42.800" UserId="3911" />
  <row Id="15541" PostId="9190" Score="3" Text="Make sure to check out the questions with the tags on this forum about visualization, http://stats.stackexchange.com/questions/tagged/data-visualization and http://stats.stackexchange.com/questions/tagged/interactive-visualization ." CreationDate="2011-04-05T12:36:04.873" UserId="1036" />
  <row Id="15542" PostId="9168" Score="1" Text="@EdS I think this was what A-L Boulesteix used in a benchmark study (should find the notes I took during an informal talk she gave to my colleagues and me, though -- so let's take it for granted)" CreationDate="2011-04-05T12:38:48.803" UserId="930" />
  <row Id="15544" PostId="9184" Text="MANOVA will tell if all the variables are significantly different among all species:    &#xA;&#xA;### MANOVA&#xA;manovaData &lt;- as.matrix(myData[,-1])&#xA;summary(manova(manovaData ~ Fruit, data=myData),test=&quot;Wilks&quot;)&#xA;&#xA;          Df      Wilks approx F num Df den Df   Pr(&gt;F)   &#xA;Fruit      3 0.00029932   9.5251     12  5.583 0.007467 **&#xA;Residuals  5                                                      &#xA;&#xA;At a 0.05 confidence level, there is at least one fruit different from others (p-value=0.007467). The question is: after that, is there a way to test if groups can be created?&#xA;&#xA;Regards." CreationDate="2011-04-05T12:52:39.810" UserId="4028" />
  <row Id="15545" PostId="9110" Text="@Neil, yes, but you can argue that there is no such thing as easy-yet-robust data exploration." CreationDate="2011-04-05T13:03:50.750" UserId="2116" />
  <row Id="15547" PostId="9184" Text="@dernier It will also tell you if there are significant differences between pairs of species: `mod &lt;- lm(cbind(Weight, horDiam, Price, Dummy) ~ Fruit, data=myData); summary(mod, test=&quot;Pillai&quot;)`. If your test for differences will be post hoc, you will have to figure out how to implement a multiple comparisons procedure yourself, because neither `TukeyHSD` nor `multcomp` seem to work on multivariate models. It is not standard practice, in my field at least, to create groups for further analysis in this way." CreationDate="2011-04-05T14:45:01.950" UserId="3432" />
  <row Id="15548" PostId="9169" Text="@Ga Please post comments like this as comments, not replies." CreationDate="2011-04-05T15:51:06.103" UserId="919" />
  <row Id="15549" PostId="9201" Text="cross posted: http://stackoverflow.com/questions/5554037/is-it-wrong-to-jitter-before-performing-wilcox-test" CreationDate="2011-04-05T15:54:48.767" UserId="696" />
  <row Id="15550" PostId="9201" Text="@weblover When I suggested (on tex.SE) to ask your stats question here, I wasn't aware you already asked on SO. Please, delete one of your two posts (probably on SO, since this is a stats question)." CreationDate="2011-04-05T16:15:48.603" UserId="930" />
  <row Id="15551" PostId="9110" Text="@mpiktas, please tell us about your experience with JMP" CreationDate="2011-04-05T16:20:45.637" UserId="74" />
  <row Id="15554" PostId="9201" Score="1" Text="You don't tell us what your data is, but I'd be very suspicious of any procedure that adds random noise to data and then runs a statistical test on it.  In addition, are you sure your code does what you think it does?  My reading is that despite your loop, it will return a single p.value - that of x=y={lastvaluof}(uni)" CreationDate="2011-04-05T15:12:47.340" UserId="3556" />
  <row Id="15555" PostId="9201" Text="@Andrie: thanks for your help, but that's not the complete code, that's a part of the function that calculate the wilcox test , and the code producing different values of p-value for each (xp,yp) ,that's a link of a part of my data , you can view it and check if i can do this, thanks in advance .                    http://www.mediafire.com/?mnj26kp4bv5lcr5" CreationDate="2011-04-05T15:29:29.017" />
  <row Id="15556" PostId="9201" Score="1" Text="I would be very suspicious of links to data that require redirecting to other links, especially when R provides the very useful `dput()` function that eliminates any need for doing that. Please provide a reproducible example to receive assistance." CreationDate="2011-04-05T15:34:20.267" UserId="696" />
  <row Id="15557" PostId="9201" Text="hello , i think someone did migrate the question from SO to here , but now there is 2 question with the same title , you will have to delete one of them , and btw when using dput(t,&quot;test.txt&quot;) on the data i have, the result was not readble nicely , is there any option to use with it?" CreationDate="2011-04-05T16:34:30.527" UserId="4038" />
  <row Id="15558" PostId="9201" Text="@weblover - use `dput()` on the R object in your working environment which looks like `t` above. the output isn't intended for human consumption - it is intended for the R interpreter to recreate the object `t` exactly as it exists in your working environment." CreationDate="2011-04-05T17:02:51.157" UserId="696" />
  <row Id="15559" PostId="9201" Text="@Chase: i edited the question and added the output of dput(head(t)) , thanks" CreationDate="2011-04-05T17:04:09.207" UserId="4038" />
  <row Id="15560" PostId="9169" Text="@whuber I did not want my original answer to stay, but neither wanted to delete it because that would have deleted the useful comment of EdS as well. If you suggest a better solution I'm happy to follow." CreationDate="2011-04-05T17:46:42.680" UserId="3911" />
  <row Id="15561" PostId="9169" Text="@Ga Oh, I see, you had an answer and then edited it.  I'm always sorry when interesting answers are deleted, even when it turns out upon further clarification that they might be less relevant, but it sounds like you have hit on a reasonable solution here." CreationDate="2011-04-05T17:55:37.227" UserId="919" />
  <row Id="15562" PostId="9213" Text="It appears that problems of this type were addressed in your previous question at http://stats.stackexchange.com/questions/8742/bayes-network-computing-conditional-probabilities" CreationDate="2011-04-05T18:00:41.577" UserId="919" />
  <row Id="15563" PostId="9213" Text="Did you want to write P(E|L,¬T)= 1.0 instead of the second P(E|L,T)= 1.0?" CreationDate="2011-04-05T18:03:56.910" UserId="3911" />
  <row Id="15564" PostId="9132" Text="@Michael: zeros and ones" CreationDate="2011-04-05T18:53:42.077" UserId="4010" />
  <row Id="15565" PostId="9213" Text="Yes, you are right, sorry" CreationDate="2011-04-05T18:58:54.563" UserId="3681" />
  <row Id="15566" PostId="9219" Text="well, you are right, assuming there are not any numbers at all, how would the bayes formula look like? when I Observe $Pr(E)=1$, How $L$ and $T$ are modified? What probabilities are necessary?" CreationDate="2011-04-05T19:11:25.610" UserId="3681" />
  <row Id="15567" PostId="9219" Text="As a first step calculate the ($T \land L \land E$, $T \land L \land \lnot E$, $T \land \lnot L \land E$, $T \land \lnot L \land \lnot E$, $\lnot T \land L \land E$, $\lnot T \land L \land \lnot E$, $\lnot T \land \lnot L \land E$, $\lnot T \land \lnot L \land \lnot E$) probabilities (I suggest using a spreadsheet), their sum must be 1. Then write zeros into the cells containing $? \land ? \land \lnot E$ as $Pr(\lnot E) = 0$. Then multiply each value with the same number to make the sum of the posterior probabilities 1." CreationDate="2011-04-05T19:20:16.657" UserId="3911" />
  <row Id="15569" PostId="9211" Score="1" Text="Under Linux Window emulation is not necessary, the shared library can be directly used. Only the GUI is not working this way, but that shouldn't be a problem for DanB as they want to invoke it from R." CreationDate="2011-04-05T19:36:08.763" UserId="3911" />
  <row Id="15570" PostId="9211" Text="Thanks for the correction." CreationDate="2011-04-05T19:45:39.317" UserId="334" />
  <row Id="15571" PostId="9225" Text="Sounds exactly like my intuitive understanding of this term." CreationDate="2011-04-05T19:59:30.423" UserId="442" />
  <row Id="15573" PostId="9223" Score="1" Text="I did search statistics website but it is very generic:                  213 Info &amp; Communication Technology&#xA;&#xA;2131 IT strategy and planning prfsnals&#xA;2132 Software professionals&#xA;" CreationDate="2011-04-05T20:10:18.083" UserId="4043" />
  <row Id="15574" PostId="9190" Text="@Andy W: ty for your valuable tips :)" CreationDate="2011-04-05T20:26:43.483" UserId="3856" />
  <row Id="15575" PostId="9212" Text="@robin Of note, the OP mentioned &quot;decision theory&quot; (that you addressed in your last sentence), not &quot;statistical decision theory&quot; (that you're actually using in your first sentence). I do not intend to be punctilious, but we human beings are known to have (and express) limited rationality: In short, we are able to take good (not necessarily optimal) decisions when faced with a limited amount of information, often thanks to [abduction](http://en.wikipedia.org/wiki/Abduction_(logic))--this is not a criticism, just some overt reflexions. (...)" CreationDate="2011-04-05T20:31:52.037" UserId="930" />
  <row Id="15576" PostId="9212" Text="(...) Long ago, Kahneman &amp; Tversky showed that human decisions can depart from rational decisions, or those outcomes predicted by Game Theory, to a great (and somewhat unexpected) extent." CreationDate="2011-04-05T20:43:22.507" UserId="930" />
  <row Id="15577" PostId="9220" Score="3" Text="You should obtain a multiple of a non-central chi-squared distribution provided all four coordinates are uncorrelated.  Otherwise, the result looks much more complicated." CreationDate="2011-04-05T20:45:44.653" UserId="919" />
  <row Id="15578" PostId="9224" Text="@Henrik : thnx for you answer , but what should i do in this case ??" CreationDate="2011-04-05T21:14:55.063" UserId="4038" />
  <row Id="15579" PostId="9222" Text="@Aaron : thanks for your reply , but i did not got the point , what should i do now ? change the test , or adjust the data based on something , and what's this something???" CreationDate="2011-04-05T21:15:55.330" UserId="4038" />
  <row Id="15580" PostId="9229" Text="This does not seem to have any data about number of people who work as either Interaction Designers or User Experience Designers" CreationDate="2011-04-05T21:36:24.033" UserId="4043" />
  <row Id="15582" PostId="9201" Text="@Chase : what do i have like alternative to this test?? or how should prevent ties without adding jitter???" CreationDate="2011-04-05T21:54:20.427" UserId="4038" />
  <row Id="15584" PostId="9220" Text="@whuber any details/pointers you could provide as to how the parameters of the resulting non-central chi-squared distribution relate to those of the objects a, b would be fantastic" CreationDate="2011-04-05T22:09:21.603" UserId="1913" />
  <row Id="15586" PostId="9220" Score="4" Text="@Nick the first few paragraphs of the [Wikipedia article](http://en.wikipedia.org/wiki/Noncentral_chi-square_distribution) provide the details.  By looking at the characteristic functions you can establish that a similar result is *not* available when not all variances are the same or there are some correlations." CreationDate="2011-04-05T22:13:20.427" UserId="919" />
  <row Id="15587" PostId="9226" Text="Sorry, &quot;big deal&quot; is too much of an editorial comment. The question of whether it's a &quot;bigger deal&quot; than I'm making it out to be is basically a question of whether there are additional considerations of which I may be ignorant." CreationDate="2011-04-05T22:21:02.480" UserId="3836" />
  <row Id="15588" PostId="9233" Text="Have you tried plotting the data? I would say the strength of correlation between your environmental variables and your response variable will affect the answer. " CreationDate="2011-04-05T23:03:15.813" UserId="2965" />
  <row Id="15589" PostId="9223" Score="1" Text="I have little experience with contacting the ONS, but if getting the data were important to me I would ask them." CreationDate="2011-04-05T23:10:20.690" UserId="3911" />
  <row Id="15590" PostId="9228" Score="1" Text="Is this a case-control study? Or cohort study? Why is there an age difference (sampling scheme? pathomechanism?)? Is age age at diagnosis? Or is this a chronic disease and age is current age at taking tissue sample for gene expression analysis? Is age known to be related to the disease? Is the age effect on gene expression more the effect of time since birth or since diagnosis?  --- I would need the answers to these questions to see your question if &quot;the classification analysis results are or are not influenced by age?&quot; in proper perspective." CreationDate="2011-04-06T00:07:27.490" UserId="3911" />
  <row Id="15591" PostId="9237" Text="Do the nominator and denominator and denominator add up to 100% in your formula? If yes, your formula is the odds of an event caused by the minority. Is the number of events caused by either group small (less then ~10)?" CreationDate="2011-04-06T00:14:50.527" UserId="3911" />
  <row Id="15592" PostId="9237" Text="No, they do not add up to 100%, as my denominator is % events of one racial group over the % of another, and not the total number of events. I also have more than 1 racial group." CreationDate="2011-04-06T00:18:34.653" UserId="3309" />
  <row Id="15594" PostId="9237" Score="2" Text="The reason why I asked the second question was to establish whether normal or Poisson approximations were justified. It would be important to approximately know the number of events and the total number of events for both the nominator and denominator." CreationDate="2011-04-06T00:33:52.287" UserId="3911" />
  <row Id="15596" PostId="9205" Text="Thanks, you helped me a lot. I discovered some great tools while discovering _betadisper()_. Namely, discriminant analysis using package _ape4_ and principal component analysis with ellipses drawn using function _clusplot()_ from package _cluster_." CreationDate="2011-04-06T01:52:56.823" UserId="4028" />
  <row Id="15597" PostId="9240" Text="What you want to estimate is not a sample size, but the size of a population (total number of unique items returned by the web sercice)." CreationDate="2011-04-06T02:34:32.137" UserId="3911" />
  <row Id="15598" PostId="9244" Score="1" Text="Be cautious with observed power: http://nottinghamtrent.academia.edu/ThomBaguley/Papers/212458/Understanding_Statistical_Power_In_the_Context_of_Applied_Research" CreationDate="2011-04-06T02:47:12.657" UserId="3911" />
  <row Id="15599" PostId="9245" Text="Yes, it is mean that the treatments are ineffective. But the question arises how this F ratio can be smaller than 1." CreationDate="2011-04-06T03:20:49.257" UserId="3903" />
  <row Id="15600" PostId="9241" Text="It seems some justification for the use of capture-recapture models is needed, because this is not a standard capture-recapture experiment.  (Possibly it can be viewed as 300 capture events, but the call to closedp does not seem to indicate that.)" CreationDate="2011-04-06T03:23:16.230" UserId="919" />
  <row Id="15601" PostId="9228" Text="This is a retrospective study on blood microRNA expression and lung cancer. The cases have lung cancer. The controls do not and were chosen from patient population appearing at a lung cancer screening clinic usually because of a history of smoking. Matching for age, gender, etc., was not done when selecting cases and controls. Lung cancer typically is diagnosed after 45-50 y of age. It is not known if blood microRNA expression is affected by lung cancer, but some other diseases are known to affect expression. " CreationDate="2011-04-06T03:50:55.937" UserId="4045" />
  <row Id="15602" PostId="9245" Text="It means that the data provides insufficient evidence to reject the null hypothesis. It is possible that the null hypothesis is false, but that the group means, just by chance varied less than you would expect by chance. As population effect sizes and sample sizes get larger, the probability of getting an F ratio less than one gets smaller." CreationDate="2011-04-06T04:00:22.127" UserId="183" />
  <row Id="15603" PostId="9228" Text="The effect of age on blood microRNA expression is unknown. The mean (and std. deviation) age of cases and controls of the study are 71 (7) and 60 (9) y, respectively." CreationDate="2011-04-06T04:01:41.320" UserId="4045" />
  <row Id="15605" PostId="9245" Score="2" Text="Additionally it means that $\frac{r\cdot\sum_{i=1}^{t}\tau_{i}^{2}}{t-1}\leq{0}$, which would mean that $r$ would have to be negative or 0, since squaring $t_{i}$s gives you all positive numbers." CreationDate="2011-04-06T04:27:28.903" UserId="2660" />
  <row Id="15606" PostId="9245" Text="As r is the number of replications which always greater than 1 in this case." CreationDate="2011-04-06T04:41:31.373" UserId="3903" />
  <row Id="15607" PostId="9212" Text="@chl I added &quot;statistical&quot; before decision to make clear that there is a part of statistic that is dedicated to decision theory. Do you think it is false ? (...)" CreationDate="2011-04-06T06:02:49.740" UserId="223" />
  <row Id="15608" PostId="9217" Text="Is your question, simply, how many people are employed as an interaction designer in the United Kingdom? or are you asking something more about sampling procedures?" CreationDate="2011-04-06T06:03:44.620" UserId="183" />
  <row Id="15609" PostId="9212" Text="@chl I'd love to read Kahneman &amp; Tversky. I'd be glad to see how they define &quot;rational&quot; decision (and departure from it) and more importantly what they call a &quot;great&quot; extent. " CreationDate="2011-04-06T06:06:27.847" UserId="223" />
  <row Id="15610" PostId="9212" Text="@chl if the question is about how to make a good decision, the main discussion may arise from the need to caracterise what is a **good** decision. " CreationDate="2011-04-06T06:07:59.267" UserId="223" />
  <row Id="15611" PostId="9237" Text="The number of events is not that small; definitely not less than 10 on average. For example, one variable is major disciplinary referrals at a school on a given year; the range is something like 1-2000ish." CreationDate="2011-04-06T06:09:00.857" UserId="3309" />
  <row Id="15612" PostId="9110" Text="@Neil, I do not have any." CreationDate="2011-04-06T06:46:17.323" UserId="2116" />
  <row Id="15613" PostId="9212" Score="1" Text="@robin No, on the contrary. I like the idea of making a distinction between statistical and human decision. My remark was mainly about this, and your last two points are clearly relevant: what is a good decision, and how to characterize any departure from it. (K&amp;T highlighted a lot of judgment biases, but this was really a sidenote.) I'll try to develop this view later." CreationDate="2011-04-06T07:15:24.327" UserId="930" />
  <row Id="15614" PostId="9231" Text="hello , is it rational to accept this ?? because all groups and subgroups when compared together , gives the same p-value , that's not logical because in each case i'm testing 1 variable e.g : test(&quot;Ph&quot;,V1) , soo p-value should not be the same for all . any ideas ??" CreationDate="2011-04-06T08:01:09.387" UserId="4038" />
  <row Id="15615" PostId="9241" Text="@whuber Yes, I did view the example as 300 capture events. How do you mean that &quot;the call to closedp does not seem to indicate that&quot;? I appreciate (constructive) criticism and I'm happy to correct (or delete if necessary) my answer if it turns out to be wrong." CreationDate="2011-04-06T09:45:25.850" UserId="3911" />
  <row Id="15616" PostId="9242" Score="2" Text="The formula is probably incorrect, since if $r$ is positive (and it always is judging from your comments below), the quantity in the formula is always greater than 1. Then it cannot be distributed as Fischer distribution, since random variable with Fischer distribution can get values less than 1." CreationDate="2011-04-06T10:23:41.577" UserId="2116" />
  <row Id="15617" PostId="5710" Text="Not churlish.  Petulant." CreationDate="2011-04-06T10:28:27.827" UserId="2669" />
  <row Id="15618" PostId="9241" Text="this seems a reasonable approach. However I won't be using R so need to understand the maths behind it. The wiki page covers a 2 event situation - how do I apply it to this case?" CreationDate="2011-04-06T10:57:05.250" UserId="4049" />
  <row Id="15619" PostId="9205" Text="I have just finished a draft for a second answer based on the idea that you might want also a LDA, so I'll post it in a while using your data as the example." CreationDate="2011-04-06T11:25:22.443" UserId="3467" />
  <row Id="15620" PostId="9259" Text="Thanks, Christophe, for formatting my post!" CreationDate="2011-04-06T11:46:50.553" UserId="3945" />
  <row Id="15621" PostId="9100" Text="@JMS - this paper does make a seemingly good point with the &quot;10 dimensional Gaussian&quot; - but I think if you were only interested in prediction of $X_{10}$ then the marginal distribution of $X_{10}$ should be used, not the joint distribution of all X's.  And it appears as though the factor loadings and idiosyncratic variances were just plugged in, rather than being estimated from the data - for we never know the &quot;true model&quot; or its closest approximation.  Interesting read though - I may have to test out some simulations of my own." CreationDate="2011-04-06T12:07:08.197" UserId="2392" />
  <row Id="15622" PostId="9100" Text="@JMS For one &quot;model&quot; that isn't considered for prediction is to keep the second component, and not the first.  This is sensible because the goal is prediction of $X_{10}$ not summarising the variation in $X_i$, its choosing the linear components which best predict $X_{10}$.  So the example is incomplete in that only one 1-D model is considered.  By using PCA regression one is implicitly asserting the *prior information* that higher variance components of $X_i$ predict $X_{10}$ better." CreationDate="2011-04-06T12:14:10.953" UserId="2392" />
  <row Id="15623" PostId="9100" Text="@JMS - perhaps this is one reason why PCA regression is not always computationally advantageous unless you actually know this - for then you just have a regular model selection - which 1-D component do I use?" CreationDate="2011-04-06T12:20:05.303" UserId="2392" />
  <row Id="15624" PostId="9100" Text="...continuing my ramblings... I would speculate if the &quot;third model&quot; was included which specified fitting a 2-factor model, but only taking the second factor for prediction would be favored much more by the log-likelihood." CreationDate="2011-04-06T12:26:19.090" UserId="2392" />
  <row Id="15625" PostId="9220" Text="@Nick, just to clarify, both $a$ and $b$ are random vectors with values in $\mathbb{R}^2$?" CreationDate="2011-04-06T13:01:26.200" UserId="2116" />
  <row Id="15626" PostId="8422" Text="That is the point I have inverted the ROC AUC for values below 0.5, so the graphics should be remade. Thanks a lot!" CreationDate="2011-04-06T13:06:01.687" UserId="3345" />
  <row Id="15627" PostId="9220" Score="1" Text="@Nick, if $a$ and $b$ are jointly normal, then the difference is $a-b$ is normal too. Then your problem is to find distribution of random normal vector. Googling I found this [link](http://www.springerlink.com/content/n8l9151t5816r0x2/). The paper describes much more complex problem which in very particular case coincides with yours. This gives some hope that there is a definite answer to your question. References might give you further ideas where to search. " CreationDate="2011-04-06T13:15:23.627" UserId="2116" />
  <row Id="15628" PostId="9260" Score="1" Text="This question needs more information.  What do you mean by &quot;20% of the time&quot;?  is it 20% of the batches given by 1 supplier are bad, or that 20% of suppliers give batches which are all bad?  And does 1 supplier give only 1 batch - or can they give multiple batches?  And what is the random sample of?  is it of 15 batches from 1 shipment, or a random selection of 15 batches from multiple shipments?" CreationDate="2011-04-06T13:28:12.103" UserId="2392" />
  <row Id="15629" PostId="9231" Text="Sure, it could be rational.  A p-value of 1 means that the two groups are as close to the same as possible.  If you've got almost all ties, this could be true." CreationDate="2011-04-06T14:27:09.427" UserId="3601" />
  <row Id="15630" PostId="9201" Text="There are no ties in the subset of the data you posted here, and as the data goes to six decimal places, it seems odd that you would have any ties at all.  How much data do you have and how many ties are there? Something like this `sapply(d, function(x) sum(duplicated(x)))` will give you the number of ties in each column of a data set `d`." CreationDate="2011-04-06T14:30:11.923" UserId="3601" />
  <row Id="15631" PostId="9222" Text="I'll summarize the page I linked to." CreationDate="2011-04-06T14:31:08.880" UserId="3601" />
  <row Id="15632" PostId="8855" Text="@Felipe This question was closed because you did not respond to requests for clarification.  If you are still interested in having it answered, then please reformulate it and post it again." CreationDate="2011-04-06T15:08:34.153" UserId="919" />
  <row Id="15633" PostId="9271" Score="1" Text="Thanks that helps!. BTW people should know that all.equal does NOT return FALSE when false. So to do the comparison you need to wrap isTRUE around all.equal. A long way to go just to compare 2 numbers. " CreationDate="2011-04-06T15:10:36.480" UserId="4056" />
  <row Id="15634" PostId="9241" Score="1" Text="@Ga I see: You created a 300 x 300 matrix for the data! The inefficiency of this code fooled me: it would be simpler and more direct to use `closedp.0(Y, dfreq=TRUE, dtype=&quot;nbcap&quot;, t=300)' where Y is the frequency matrix {{1,234}, {2,27}, {3,4}} (which you computed twice and actually displayed!).  More to the point, the convergence failures are alarming, suggesting there are problems with the underlying code or models.  (An exhaustive search of the [docs](http://cran.r-project.org/web/packages/Rcapture/Rcapture.pdf) for &quot;M0&quot; turns up no references or description for this method...)" CreationDate="2011-04-06T15:32:05.507" UserId="919" />
  <row Id="15635" PostId="9261" Text="This appears to lose some information because it does not account for the *frequencies* with which items were observed 2, 3, 4, ... times." CreationDate="2011-04-06T15:33:53.320" UserId="919" />
  <row Id="15636" PostId="9266" Text="In brief, this question in its current form appears to ask &quot;how can I cluster a [poset](http://en.wikipedia.org/wiki/Partially_ordered_set)&quot;. As your point #1 indicates, this question is not answerable (nor is it of any statistical interest) until you can quantify similarity or dissimilarity of subsets.  You have provided a nice example of what you're looking for but all you reveal about the basis of the clustering is that you &quot;imagined it&quot;!  So: can you specify what would constitute a &quot;cluster&quot;?" CreationDate="2011-04-06T15:42:01.270" UserId="919" />
  <row Id="15637" PostId="9220" Text="@mpiktas, yes, $a$ and $b$ are both random vectors with values in $\mathbb{R}^2$. I'm not sure what you mean by 'jointly' normal. I'm assuming that $a$ and $b$ are independent r.v.'s each with known parameters. I'm looking for the distribution of the squared euclidean distance: $dist(a,b) = \sum_i (a_i - b_i)^2 = (a_1 - b_1)^2 + (a_2 - b_2)^2$. I know that $c = (a - b)$ is normally distributed, but how about $\sum c^2$? &#xA;Hope that helps clarify - thanks for the link I'll have a look." CreationDate="2011-04-06T15:54:25.817" UserId="1913" />
  <row Id="15638" PostId="9242" Text="@mpiktas Good point. @Bogdan explains the discrepancy in a reply.  The orthography suggests $\sigma$, $r$, and $\tau$ are *parameters,* not statistics (as @Jeromy points out), further indicating that this expression cannot be the usual F-statistic." CreationDate="2011-04-06T16:04:02.513" UserId="919" />
  <row Id="15639" PostId="9275" Score="1" Text="How does one even apply the $\chi^2$ test to smoothed data?  Regardless, the smoothing is going to change the sampling distribution and introduce *dependencies,* thereby ruining the $\chi^2$ test." CreationDate="2011-04-06T16:07:40.513" UserId="919" />
  <row Id="15640" PostId="9182" Text="The quest for a statistical procedure seems to sweep under the rug what appears to be the *key* question: how does one compare incommensurate attributes?  Specifically, how should a difference of (say) 10 gm in weight be balanced against a difference of 1 in price?  If you just throw your data into a PCA/LDA/clustering/MANOVA/whatever grinder, you'll get sophisticated and perhaps interesting answers, but *they won't answer this question* and each (implicitly and silently) will make its own assumptions about what the answer is--for good or ill." CreationDate="2011-04-06T16:14:18.870" UserId="919" />
  <row Id="15641" PostId="9246" Text="Thanks, this is very informative. I understand that the p-value definition doesn't change. Certainly from a statistical standpoint, the rate of type I errors does not increase." CreationDate="2011-04-06T16:15:22.877" UserId="3836" />
  <row Id="15642" PostId="9258" Text="+1 I love the [&quot;Best Pie Chart Ever&quot;](http://www.datavis.ca/gallery/images/fox-news-piechart.jpg)!" CreationDate="2011-04-06T16:29:58.327" UserId="919" />
  <row Id="15643" PostId="9246" Text="By definition, we are fixing the type I error rate in setting the p-value threshold. However, it seems like the difference between &quot;statistical&quot; and &quot;practical&quot; significance is the issue here. When the sample size is able to detect differences much finer than the expected effect size, a difference that is correctly statistically distinct is not practically meaningful (and from the perspective of the &quot;end-user&quot; this is effectively a &quot;false positive&quot; even if it's not a statistical one). However, as you say this starts to get outside the realm of statistics." CreationDate="2011-04-06T16:33:00.320" UserId="3836" />
  <row Id="15644" PostId="9261" Text="@whuber: It may appear not to use the information, but if you investigate further you should find that the number of unique items is a sufficient statistic. For example, if you take a sample with replacement of 4 items from a population of $n$, the probability of getting 3 of one item and 1 of another is $\frac{4}{3}$ that of getting 2 each of two items, no matter what $n$ is, so knowing the detailed frequencies gives no more useful information about the population than simply knowing there were two unique items found in the sample." CreationDate="2011-04-06T16:44:05.943" UserId="2958" />
  <row Id="15646" PostId="9266" Text="@whuber: thanks for the link, I was not familiar with the concept, I'll read further into it. Regarding your comments; I realize that the question is perhaps not well formulated, though I think your critique is slightly harsh; because it's not completely irrelevant to want to find underlying hierarchies in the data, is it? Furthermore the reasoning behind the question is to minimize the effects of redundancy in databases from which the data comes. I'll try and clarify the question, giving more details about the underlying problem; and if it's still no good, you can close the question..." CreationDate="2011-04-06T17:46:57.707" UserId="3014" />
  <row Id="15647" PostId="9274" Text="Ah, perfect! I didn't realise that such a method was possible. Thanks :)" CreationDate="2011-04-06T17:52:08.777" UserId="4055" />
  <row Id="15648" PostId="9266" Text="@posdef I apologize for sounding harsh; that was not intended.  The point was that the current formulation isn't a statistical problem at all, because essential information is missing, which suggests there's more going on that you haven't yet shared with us.  (The poset *already is* a set of &quot;underlying hierarchies&quot;.)" CreationDate="2011-04-06T18:00:48.560" UserId="919" />
  <row Id="15649" PostId="9261" Text="@Henry Interesting point about the sufficiency of the number of unique items.  So the frequencies can serve as a check on the assumptions (of independence and equal probability), but otherwise are unnecessary." CreationDate="2011-04-06T18:06:01.300" UserId="919" />
  <row Id="15650" PostId="9228" Text="When you say &quot;influenced by age&quot;, what exactly do you mean? Here are two possibilities. One possibility is that your microarrays contain no disease markers whatsoever. But, they do contain information about age, and since in your case the sick and control populations are of different age, you get the illusion of good classification performance. Another possibility is that the microarrays do contain disease markers, and, moreover, these markers is exactly what SVM focuses on. However, since in your data the ages are different, there is still correlation between age and category." CreationDate="2011-04-06T18:37:09.890" UserId="3369" />
  <row Id="15651" PostId="9266" Text="@whuber: no need to apologize, I recall having a similar &quot;mini-discussion&quot; with you on another question of mine. Such misses in communication seem to happen often in cross-disciplinary projects and people that work in different fields. Please see the edits, and let me know if further clarifications are necessary. If the question is still not a suitable one for this forum, I'd be happy to see referrals or suggestions on how to proceed with the problem. " CreationDate="2011-04-06T18:40:14.393" UserId="3014" />
  <row Id="15652" PostId="9241" Text="@whuber The matrix is 265x300, there were 256 unique values observed in this example." CreationDate="2011-04-06T18:48:46.583" UserId="3911" />
  <row Id="15654" PostId="9241" Text="@whuber It was the Mh Chao model that converged, the classical one. I haven't read the original paper, but it may be this one: http://www.jstor.org/pss/2531532 ." CreationDate="2011-04-06T19:02:22.407" UserId="3911" />
  <row Id="15655" PostId="8422" Text="@Vladimir +1 for coming back, I'm waiting for your update. If you're working with different classifiers, you might have a look at [ROCR](http://rocr.bioinf.mpi-sb.mpg.de/)." CreationDate="2011-04-06T19:10:41.937" UserId="930" />
  <row Id="15656" PostId="9241" Score="1" Text="@whuber I simplified the code following your suggestion (dfreq=TRUE, dtype=&quot;nbcap&quot;, t=300). Thanks again." CreationDate="2011-04-06T19:18:03.813" UserId="3911" />
  <row Id="15657" PostId="9228" Text="@SheldonCooper: Right, and I want to know if we can or cannot figure out which of the two possibilities it is. If not, can we roughly estimate the extra value the gene markers provide over age? The SVM classifier has good performance characteristics (accuracy in internal cross-validations &gt;90%, and AUC &gt;0.95). AUC in ROC analysis of age is 0.82." CreationDate="2011-04-06T19:20:22.377" UserId="4045" />
  <row Id="15658" PostId="9276" Text="Just out of curiosity: is anything in this model conditional on whether the accused is actually *guilty*?" CreationDate="2011-04-06T20:51:50.720" UserId="919" />
  <row Id="15659" PostId="9275" Text="Thank you for the clarification.  I still don't follow, because according to the edit you appear to be smoothing both the estimators of the marginal distributions *and* you seem to be changing the data!  The former is ok, subject to some conditions (although it does nothing to change the &quot;data support&quot;), but--again--if you smooth the data, it's hard even to know what is being tested here." CreationDate="2011-04-06T20:56:30.093" UserId="919" />
  <row Id="15660" PostId="9280" Text="Not quite sure why my formulas aren't rendering on the question page as they render just fine in the preview. If someone has enough reputation and wants to edit so that they display correctly it would be much appreciated." CreationDate="2011-04-06T21:04:15.623" UserId="3713" />
  <row Id="15661" PostId="9028" Text="@whuber - thanks! very helpful thread" CreationDate="2011-04-06T21:41:07.037" UserId="1991" />
  <row Id="15662" PostId="9100" Text="@probabilityislogic I don't necessarily disagree about using the marginal likelihood of X10. But it doesn't get around the problem of a good joint model - the marginal distribution of X10 is obtained by integrating out X1-X9! CV would consider instead $X10|X1, ..., X9, \Theta$. Your objections about the example taking the PC with the largest eigenvalue is correct; PCR is often done this way and the example is to demonstrate that this is not necessarily a great idea. They augment the West 2003 factor regression to reflect this" CreationDate="2011-04-06T22:11:07.923" UserId="26" />
  <row Id="15663" PostId="9100" Text="@probabilityislogic In terms of your last comment, I'm skeptical in general (although you might be right about this example) Suppose that the data is given as $D = (D_1, D_2)$ with $D_1$ 1 dimensional and $D_2$ much higher dimensional. Then write the BF as $P(D_1|D_2, M_1)P(D_2|M_1) / [P(D_1|D_2, M_2)P(D_2|M_2)]$. I claim in this case that without some seriously informative priors, or maybe fantastic amounts of data, $P(D_1|D_2, M_1)P(D_2|M_1) / [P(D_1|D_2, M_2)P(D_2|M_2)] \approx P(D_2|M_1) / P(D_2|M_2)$" CreationDate="2011-04-06T22:20:19.973" UserId="26" />
  <row Id="15664" PostId="9100" Text="In 10 dimensions, maybe not such a big deal. In 100, or 10,000 it *could* be as the likelihood is dominated by the &quot;irrelevant&quot; pieces. Think of partial least squares vs PCA. I'm very much a Bayesian but I haven't seen a satisfactory answer to the problem yet. " CreationDate="2011-04-06T22:27:56.147" UserId="26" />
  <row Id="15665" PostId="9281" Score="1" Text="Bray-Curtis dissimilarity may do the job... A great ressource is the book 'Numerical Ecology' by Legendre &amp; Legendre (1998)." CreationDate="2011-04-06T22:31:04.317" UserId="1050" />
  <row Id="15666" PostId="9100" Text="Also, in my haste I think I misstated what $D_1$ and $D_2$ are. I'm thinking of something like $D_1 = \{y_{i1}\}_{i=1}^n$ and $D_2 = \{(y_{i2}, \dots, y_{ip})\}_{i=1}^n$. So $D_1$ is a collection of 1d observations, not 1d itself." CreationDate="2011-04-06T22:33:06.343" UserId="26" />
  <row Id="15667" PostId="9276" Text="The model is based off of survey responses to a single fact pattern, so actual guilt doesn't vary. I'm predicting how different juries would come out in a single contested case." CreationDate="2011-04-06T22:36:23.267" UserId="4059" />
  <row Id="15668" PostId="9287" Text="Thank you for the various suggestions. I think you are right that checking for correlation of age with principal components doesn't provide an answer. I did do that analysis and there are good correlations (Spearman r &gt; 0.5) for each of the first three PCs (they together contribute to ~55% of variance). There is also a good correlation of age with the probability values from the SVM analysis. For the first two options you suggest, I have to check if there are enough samples and how to go about it (I use LOOCV and 1000-iteration Monte Carlo CV with 4:1 split for training and testing)." CreationDate="2011-04-06T22:38:25.327" UserId="4045" />
  <row Id="15669" PostId="9287" Text="Regarding ROC using both age and microarray data, I will try it. An increase in AUC from 0.95 (microarray data alone) will suggest that the expression data has disease-specific information that is independent of age. An absence of an increase, however, will mean nothing since the expression data is affected by age. Right?" CreationDate="2011-04-06T22:45:02.713" UserId="4045" />
  <row Id="15670" PostId="9287" Text="You already have an increase in AUC, from 0.82 for age only to 0.95 for microarray. This is what's important I think. If you get further increase, great. If you don't get further increase, you are right that it doesn't mean anything. The important part is that you have the increase from 0.82 to 0.95." CreationDate="2011-04-06T23:03:16.093" UserId="3369" />
  <row Id="15671" PostId="9283" Score="4" Text="not R, but I use biogeme to estimate discrete choice models for transportation and private sector research: http://biogeme.epfl.ch/. Be glad to give you some tips to get up and running if this is the type of analysis you need to do." CreationDate="2011-04-06T23:03:21.290" UserId="696" />
  <row Id="15672" PostId="9100" Text="@JMS - once again, this is using the joint model to assess the marginal model as your equation is $\frac{P(D_1,D_2|M_1)}{P(D_1,D_2|M_2)}$.  Bayes theorem is doing exactly what you ask of it, which is why the 1-D model is &quot;swamped&quot;.  You should be asking for $\frac{P(D_1|D_2,M_1)}{P(D_1|D_2,M_2)}$ if predicting $D_1$ with $D_2$ is the target of inference." CreationDate="2011-04-06T23:03:45.750" UserId="2392" />
  <row Id="15673" PostId="9100" Text="And those &quot;irrelevant&quot; pieces are only irrelevant to summarising variation in $D_2$, they may be quite relevant for predicting $D_1$." CreationDate="2011-04-06T23:07:44.170" UserId="2392" />
  <row Id="15674" PostId="9100" Text="@JMS - point taken about using the condition distribution $X_{10}|X_{k},\Theta$, but this is not a &quot;CV&quot; thing, this is a model selection thing - there is nothing in CV which forces you to make this choice as &quot;the model&quot;." CreationDate="2011-04-06T23:09:42.887" UserId="2392" />
  <row Id="15675" PostId="9276" Text="OK, kidding aside, you report three estimates: the mean and 2.5 and 97.5 percentiles of PG.  For which one(s) do you need an &quot;accurate&quot; determination and how accurate must it be?" CreationDate="2011-04-06T23:10:57.960" UserId="919" />
  <row Id="15676" PostId="9100" Text="(cont'd) and further, this procedure assumes that you know $X_{k}$ and that you know $\Theta$ when it comes to actually predicting - hardly likely in practice (especially for $\Theta$).  These should be integrated out of the model if they are unknown, so that you are conditioning only on the information you actually have." CreationDate="2011-04-06T23:14:44.537" UserId="2392" />
  <row Id="15677" PostId="9276" Text="Also, step (6) is mysterious.  Could you explain what it is intended to do?  Is there a different &quot;randomly selected value&quot; for each juror(5), each jury(4), each model(3), or some combination thereof?" CreationDate="2011-04-06T23:16:47.137" UserId="919" />
  <row Id="15679" PostId="9275" Text="This seems vaguely like a test of independence for BB($\alpha,\alpha,N_i$)-distributed variables, but I wouldn't stake much on it. You'd probably be better off using Fisher's exact test - I'm not sure what your worry is with support." CreationDate="2011-04-07T00:14:59.267" UserId="2456" />
  <row Id="15680" PostId="9275" Text="@sesqu I don't think Fisher's exact test scales up, so I would have to use one technique for &quot;small&quot; datasets and another for &quot;large&quot; datasets, which has a number of drawbacks for my purposes.  As for data support, I'm concerned because there is a big difference between 25% and 0%, but not so much between 1 out of 4 and 0 out of 4.  I'm not sure that smoothing will solve this issue, but I thought it was worth a try." CreationDate="2011-04-07T00:22:58.863" UserId="2485" />
  <row Id="15681" PostId="9276" Text="Thanks, whuber. I want to be able to report the mean -- the overall probability that a jury drawn from _C_ will convict -- and give a 95% confidence interval around that estimate. Are there conventions I don't know about that say how precise you need to be about the 95% confidence interval? I would think necessary precision would depend on what this interval turns out to be: if it's 61%-63% I might want three or four decimal places. (.612 or something), but if it's 40%-84%, I might just need two decimal places. Does that sound right?" CreationDate="2011-04-07T00:41:55.763" UserId="4059" />
  <row Id="15682" PostId="9276" Text="For step (6): The model _M_ only gives a probability that each juror will convict. But since I'm simulating juries, I need to assign an actual first ballot vote to each juror, not just a probability. For example, say that according to _Mi_, juror _n_ has a 70% probability of voting &quot;guilty&quot; on the first ballot. I draw a random number between 0 and 1. If that number is below 0.7, I assign _n_ a first ballot vote of &quot;guilty.&quot; If that number is above 0.7, I assign _n_ a first ballot vote of &quot;not guilty.&quot; I determine the jury's final vote based on the jurors' initial votes. " CreationDate="2011-04-07T00:48:30.827" UserId="4059" />
  <row Id="15683" PostId="9275" Text="This smoothing scheme really only does much when some counts are very low, in which case Fisher's test is fast. If you do have datasets with large and small counts, Yates's continuity corrected chi-square might be sufficient." CreationDate="2011-04-07T00:56:59.353" UserId="2456" />
  <row Id="15684" PostId="9282" Text="So really the spatial element of this problem is more of a nuisance rather than the story. If I were interested in the value at a specific location I agree that the kriging prediction would be error would be the ticket. If one were to view the problem as more of a weighted least-squares problem though, how would the prediction interval formula change?" CreationDate="2011-04-07T00:58:54.407" UserId="3713" />
  <row Id="15685" PostId="9287" Text="In a new analysis, with age added as a variable to the expression data-set, the AUC increases ~0.04. I guess one cannot conclude anything from this." CreationDate="2011-04-07T01:51:43.370" UserId="4045" />
  <row Id="15686" PostId="9100" Text="@probabilityislogic I think you mistook what I meant by irrelevant bits (or maybe I misstated), although I think we agree: What's relevant is the marginal fit to $D_1$. If a model fits well to that one dimension then we should tolerate some misfit in the others, since we're interested in predicting $D^*_1|D^*_2$ given new data. In practice (ie as in Bayesian factor regression but many other areas too) people fit big joint models to do prediction on one margin. Generally we will infer models that (asymptotically) minimize the KLD from the true model..." CreationDate="2011-04-07T01:52:16.623" UserId="26" />
  <row Id="15687" PostId="9275" Text="I thought Yates' continuity correction didn't work for general `n x m` contingency tables.  In case my post wasn't clear, I'm not guaranteed to always have `2x2`, even though the example was." CreationDate="2011-04-07T01:54:48.413" UserId="2485" />
  <row Id="15688" PostId="9246" Text="i.e. i think i agree - &quot;the difference in interpretation that is needed is that there is a different relationship between statistical significance and scientific significance&quot;" CreationDate="2011-04-07T02:04:17.253" UserId="3836" />
  <row Id="15689" PostId="9100" Text="...but there is no reason in general that this model should also minimize any other risk function we care about, e.g., predictive MSE or similar. Really the model/estimator we want to pick is something like $\hat\theta = \arg\min_{t\in\Theta} \int\int R(D^*, t)p(D^*|\theta)p(\theta | D) d\theta dD$, the minimizer of our expected predictive risk (expectation wrt posterior pred.). In principle we could of course obtain this through a complete Bayesian analysis, but practically it'll range from difficult to impossible. I think Bernando &amp; Smith discusses this problem further..." CreationDate="2011-04-07T02:06:15.550" UserId="26" />
  <row Id="15690" PostId="9100" Text="...So where does CV come in? Leave k out CV approximates that integral with empirical averages, and rather effectively with large data. In that sense it is eminently convenient as it can be applied to any risk function with a decent empirical estimator with no trouble. The same can't be said of Bayesian procedures." CreationDate="2011-04-07T02:09:31.943" UserId="26" />
  <row Id="15691" PostId="9100" Text="I forgot to mention that if this is at all interesting, you may want to have a look at some of the work using Gibbs posteriors. A nice paper (with references to the more foundational work) is &quot;Gibbs posterior for variable selection in high-dimensional classification and data mining&quot; by Jiang &amp; Tanner, available on the arXiv: http://arxiv.org/abs/0810.5655. I haven't decided quite how to think about them, but it's a really interesting idea to me." CreationDate="2011-04-07T02:14:55.043" UserId="26" />
  <row Id="15692" PostId="9295" Text="@Ganesh: I've tried to edit your post to (a) make the math slightly more readable and (b) change what I believe were some typos in the math. Please make sure I have not messed things up in the process." CreationDate="2011-04-07T02:16:21.577" UserId="2970" />
  <row Id="15693" PostId="9295" Text="Also, what do you mean by &quot;ensure&quot;? Strictly speaking, Monte Carlo can't (quite) provide such guarantees." CreationDate="2011-04-07T02:16:58.630" UserId="2970" />
  <row Id="15694" PostId="9295" Text="@cardinal: Thanks for your editing. I think I have fixed the guarantees part." CreationDate="2011-04-07T02:24:15.630" UserId="4011" />
  <row Id="15695" PostId="9295" Score="1" Text="The integrals do not appear to be &quot;intractable&quot;; indeed, they appear to be solvable with elementary functions.  The [mean distance from a point to a line segment](http://gis.stackexchange.com/questions/7440/transform-polygon-to-equivalent-circle-qgis) involves linear expressions times quadratic expressions and quadratic expressions times logarithms of linear plus quadratic expressions.  Integrating the latter by parts gives rational functions of quadratic algebraic expressions.  Those all can be done by completing the square and trig substitutions." CreationDate="2011-04-07T05:18:24.183" UserId="919" />
  <row Id="15696" PostId="9295" Text="E.g., 48 times the mean distance between two non-parallel, non-adjacent sides of a regular hexagon of side 2 equals $32-48 \sqrt{3}-22 \sqrt{7}+46 \sqrt{13}+27 \text{ArcCsch}(3 \sqrt{3})-108 \text{ArcSinh}(\frac{1}{3 \sqrt{3}})+81 \text{ArcSinh}(\frac{5}{3 \sqrt{3}})$ +  $66 \text{ArcSinh}(\sqrt{3})-27 \text{Log}(3)-18 \text{Log}(27)+36 \text{Log}(432)+36 \text{Log}(7-4 \sqrt{3})+6 \text{Log}(-135+78 \sqrt{3})$ +  $48 \text{Log}(2+\sqrt{7})-72 \text{Log}(-\frac{2}{3} (2+\sqrt{7}) (-7+\sqrt{13}))-24 \text{Log}(-1+\sqrt{13})+24 \text{Log}(1+\sqrt{13})$ (yielding a mean of about 2.8331)." CreationDate="2011-04-07T05:25:50.480" UserId="919" />
  <row Id="15697" PostId="9231" Text="@Aaron @Web These data (V1 through V7) are not tied.  An error must have been made somewhere in obtaining p-values of 1.  I get a lot of p-values of 0.036 when comparing them with the Wilcoxon signed-rank test." CreationDate="2011-04-07T05:41:16.817" UserId="919" />
  <row Id="15698" PostId="9275" Text="If you want to &quot;smooth&quot; (use a prior) then why not take it all the way home and do a formal Bayesian hypothesis test for independence? If you're dealing with straight contingency tables (no correlated observations or other complications) then I think you ought to be able to compute marginal likelihoods and Bayes factors analytically using Dirichlet (or just uniform) priors. " CreationDate="2011-04-07T06:03:24.690" UserId="26" />
  <row Id="15700" PostId="9305" Text="You can use $\LaTeX$ by putting `$...$` or `$$...$$` around your code" CreationDate="2011-04-07T08:07:02.123" UserId="2958" />
  <row Id="15701" PostId="9302" Text="@fRed, the wikipedia page is a bit strange. The condition for $I(0)$ presented is satisfied for all processes which admit MA representation as in MA link. For all practical purposes $I(0)$ is synonim for stationarity. " CreationDate="2011-04-07T08:20:52.857" UserId="2116" />
  <row Id="15702" PostId="9220" Text="@Nick, jointly normal means that $(a,b)\in \mathbb{R}^4$ is normal. Otherwise you have only marginal densities and since $dist(a,b)$ is a function of $(a,b)$ you will not be able to calculate its distribution without knowing the distribution of $(a,b)$." CreationDate="2011-04-07T08:23:10.867" UserId="2116" />
  <row Id="15703" PostId="9256" Text="Let me clarify your answer.  So you suggest to use gradient descent like methods for SVM loss function minimization (that depends on $\nu, \alpha_1, \dots, \alpha_n$), instead of SMO method that optimizes loss function for fixed $\nu$. Right? To use gradient descent, the loss function should be unimodal. So does the loss function unimodal with respect to $\nu$ variable?" CreationDate="2011-04-07T08:29:16.037" UserId="4051" />
  <row Id="15704" PostId="9305" Text="@vatna, since the conditional density is defined in the open subset $Y$, this means that it should be zero outside. Since the $Y$ is open, its boundary does not belong to it, hence it should be zero on it. " CreationDate="2011-04-07T08:29:54.053" UserId="2116" />
  <row Id="15705" PostId="9256" Text="@TheBug, no, the SVM itself is still trained using SMO, but the hyper-parameters ($\nu$ and any kernel parameters) are optimised by gradient descent (or Nelder Mead simplex) minimisation of some suitable model selection criterion (e.g. cross-validation error).  At each step you have to train several SVMs with fixed $\nu$ in order to define the simplex or to compute the gradient (using finite differences if it can't be done analytically).  The cross-validation error is not generally unimodal, but it is normally smooth and such methods work well in practice." CreationDate="2011-04-07T09:17:04.240" UserId="887" />
  <row Id="15706" PostId="9256" Text="Thank you, now it's clear." CreationDate="2011-04-07T09:58:27.557" UserId="4051" />
  <row Id="15708" PostId="9307" Score="3" Text="@user333 Could you work on improving your question, please? As it stands, it is difficult to determine exactly what question is being asked, and what would be the purpose of such a function *in a statistical context* (ok I guess it might be related to your previous questions, e.g. on random walk)." CreationDate="2011-04-07T11:19:11.370" UserId="930" />
  <row Id="15710" PostId="9306" Text="What about those quantities you have mentioned: (mean) power during the period; power in a high frequency band? This is a planned experiment, so you surely have assumptions how the periods will differ. A good measure will be sensitive to the kind of difference you hypothesise." CreationDate="2011-04-07T11:36:59.593" UserId="3911" />
  <row Id="15711" PostId="9233" Text="&quot;_Within each year, there is data from over 100k locations._&quot; Do you actually observe all locations or just the average value based on them? If yes then yo can go for panel data models as @crayola suggested in linear context. Though some special ecological models as @GaBorgulya mentioned may require much less information for the parameters to calibrate rather than estimate. " CreationDate="2011-04-07T11:46:56.933" UserId="2645" />
  <row Id="15712" PostId="9306" Text="@GaBorgulya: well, I thought of doing some kind of histogram of the frequency distribution in the 3 blocks, but I'm not sure how to integrate the power information in that..." CreationDate="2011-04-07T11:47:05.713" UserId="582" />
  <row Id="15713" PostId="9231" Text="The OP only posted the first few rows of the data; apparently there must be ties further on, though that does seem unlikely as the data goes to six decimal places.  So yes, it does seem that there might be an error in the analysis, though not exactly the one you point out." CreationDate="2011-04-07T12:00:47.190" UserId="3601" />
  <row Id="15714" PostId="9299" Score="1" Text="What do you mean by a Gaussian with a variance *and* scale parameter? The variance parameter *is* the scale parameter of a Gaussian! I'm also a little unsure about the model you've set up so far. Can you describe the problem you're actually trying to solve in more detail? Using a Gaussian to model low granularity integer-valued observations seems fishy. " CreationDate="2011-04-07T12:19:09.947" UserId="2970" />
  <row Id="15715" PostId="9274" Text="@whuber assuming the sample of 10 has some variance, isn't the homogeneity of variance assumption of an ANOVA violated? Because error is pooled across the cells, the cell with only 1 person will pull down the MSE making a significant result more likely, no?" CreationDate="2011-04-07T12:21:29.757" UserId="2322" />
  <row Id="15716" PostId="9299" Text="(+1) For an interesting question. Looking forward to understanding what you're trying to solve a bit better. " CreationDate="2011-04-07T12:22:41.550" UserId="2970" />
  <row Id="15717" PostId="9299" Text="Here are several observations: 1. If your values are integer, using Gaussian does not seem appropriate. 2. It is not clear what is the purpose of your model, do you want to identify the clusters of strong belief for example? What will be interpretation of your parameters if you had them? 3. Since you have a grid, why not try to fit mixture of bivariate distributions? Then the grid will be the support of the distribution (say unit square) and the intensities will correspond to high probability regions." CreationDate="2011-04-07T12:28:32.577" UserId="2116" />
  <row Id="15722" PostId="9276" Text="I think you edited your step (6) question: in that step I draw a random value only for the juror. Each juror (and therefore each jury) is simulated from _C_, so they're randomly drawn from a set of demographic distributions. The models, too, are simulated, but I run through each of the 1,000 simulated models in sequence." CreationDate="2011-04-07T12:40:51.707" UserId="4059" />
  <row Id="15728" PostId="8422" Text="I did ROC curve with ROCR" CreationDate="2011-04-07T12:56:32.353" UserId="3345" />
  <row Id="15733" PostId="9268" Text="(+1) It's interesting to note that if the ratio $s/n$ is very small then essentially no information about $n$ can be present and so one can't expect to do very well estimating $n$. If $s/n$ is very large then $U$ is a good estimator. So, we need something that works in a mid-range." CreationDate="2011-04-07T13:21:51.503" UserId="2970" />
  <row Id="15734" PostId="9268" Text="Also $1 - (1-1/n)^s \geq (1-f_k(s/n)) / f_k(s/n)$ where $f_k(x) = \sum_{i=0}^k x^i/i!$ is the $k$th-order Taylor series approximation to $e^x$. Using $k=1$ gives an estimator $\tilde{n} = \frac{s}{s-U} U$. A continuity correction for small $s$ can be obtained by adding a constant (like 1) in the denominator. This estimator doesn't do as well for the example as numerically solving for $\hat{n}$ as you've done, though." CreationDate="2011-04-07T13:22:56.230" UserId="2970" />
  <row Id="15742" PostId="9144" Text="@Rob, why the 1 comes up in the formula? We have $\hat{y}=X^*\beta+X^*(X'X)^{-1}X'e$. Then $var \hat{y}=var X^*(X'X)^{-1}X'e=\sigma^2X^*(X'X)^{-1}(X^*)'$?" CreationDate="2011-04-07T14:25:44.460" UserId="2116" />
  <row Id="15743" PostId="9312" Text="You seem to be calculating the [coefficient of variation](http://en.wikipedia.org/wiki/Coefficient_of_variation) of the measurement, not the standard deviation or standard error." CreationDate="2011-04-07T14:47:32.450" UserId="3911" />
  <row Id="15744" PostId="9281" Text="All the answers so far have been very useful, and I really appreciate all the reference material you provided. I wish I could mark more than one answer! Thanks for everyone's help." CreationDate="2011-04-07T14:52:13.990" UserId="2830" />
  <row Id="15746" PostId="9090" Text="@probabilityislogic: Wow. Simply wow. After seen some of your answers I really wonder: what should a doofus like me do to reach such a flexible bayesian state of mind ? " CreationDate="2011-04-07T15:02:18.497" UserId="264" />
  <row Id="15747" PostId="9303" Text="Thank you for that explanation.  So what I gather is that when the software returns the pilot parameters, they can be plugged directly into the journal formulas, except for C.  The software returns gamma = exp(C).  To translate into the journal article's equation, take log(gamma) to get C.  ** You've helped a lot, thanks again **" CreationDate="2011-04-07T15:06:25.500" UserId="2473" />
  <row Id="15748" PostId="9090" Text="@steffen - lol.  Its not that difficult, you just need to practice.  And always always always remember that the product and sum rules of probability are *the only rules you will ever need*.  They will extract whatever information is there - whether you see it or not.  So you apply product and sum rules, then just do the maths.  That is all I have done here." CreationDate="2011-04-07T15:07:11.037" UserId="2392" />
  <row Id="15749" PostId="9090" Text="@steffen - and the other rule - more a mathematical one than stats one - don't pass to an infinite limit too early in your calculations, your results may become arbitrary, or little details may get thrown out.  Measurement error models are a perfect example of this (as is this question)." CreationDate="2011-04-07T15:11:41.920" UserId="2392" />
  <row Id="15751" PostId="9274" Text="@whuber - nice answer (+1) from me. Although I do find it rather amusing that people think the normal distribution is an &quot;agressive&quot; assumption for variation - quite the opposite for only 1 aspect of the variation will influence the results $s^2$.  Choosing any other distribution will involve more detailed statistics getting involved in the test.  I'd rephrase your italicised part as *unless you have a good reason to assume something else, then a normal distribution is appropriate for this problem*." CreationDate="2011-04-07T15:27:49.170" UserId="2392" />
  <row Id="15752" PostId="9267" Text="One clarifying question, is the same data for the single patient being used in each comparison, or is it 10 different data from the same patient?" CreationDate="2011-04-07T15:31:46.110" UserId="2392" />
  <row Id="15754" PostId="9090" Text="@probabilityislogic: Thank you, I'll keep this in mind... as soon as I am done working through my &quot;Bayesian Analysis&quot;-copy ;)." CreationDate="2011-04-07T15:33:45.707" UserId="264" />
  <row Id="15764" PostId="9293" Text="(+1) for the Anderson paper" CreationDate="2011-04-07T16:08:22.717" UserId="1050" />
  <row Id="15765" PostId="9318" Text="what is a pseudo code?" CreationDate="2011-04-07T16:10:32.577" UserId="1050" />
  <row Id="15766" PostId="9100" Text="Seems like this paper takes the stance that in decision theory, the product of loss function and posterior matters, not each piece separately.  &quot;Why draw the arbitrary distinction?&quot;  Seems to be the idea here. sounds interesting" CreationDate="2011-04-07T16:13:08.100" UserId="2392" />
  <row Id="15767" PostId="9274" Text="@probability Thank you.  Those of us who frequently find outliers and large amounts of skew in good data will be less emphatic about this and say only that the assumption of normality depends on the nature of the data and the importance of the decision(s) made with the test." CreationDate="2011-04-07T16:19:17.463" UserId="919" />
  <row Id="15768" PostId="9217" Text="Well this question is designed to help me develop my sampling method. But prior to that I want to have an idea about how many people are there in total with that job title in a given territory." CreationDate="2011-04-07T16:26:25.587" UserId="4043" />
  <row Id="15769" PostId="9313" Text="Are you specifically looking for a review of algorithms or a more general look at the methods and design of studies? Is this in an experimental or observation context?" CreationDate="2011-04-07T16:49:21.700" UserId="3265" />
  <row Id="15770" PostId="9318" Text="@EDi http://en.wikipedia.org/wiki/Pseudocode" CreationDate="2011-04-07T16:50:16.907" UserId="919" />
  <row Id="15772" PostId="9319" Text="@rm There are 366 possible birthdays.  The &quot;253 pairs chosen independently&quot; are assumed, in contrast to the 253 pairs available from the 23 selected people, to be chosen from all possible pairs of birthdays.  The point is that the number of birthday matchings available among 23 people (253) is of the same order of magnitude as the number of possible birthdays (366), making it plausible that at least one matching will involve the same two birthdays." CreationDate="2011-04-07T17:10:37.747" UserId="919" />
  <row Id="15773" PostId="9275" Text="@JMS Basically my audience wants to see p-values, and the Bayesian inside of me is doing what it can while still giving the audience what they want to see.  So this post was my compromise between somewhat conflicting philosophies." CreationDate="2011-04-07T17:11:51.600" UserId="2485" />
  <row Id="15774" PostId="9292" Text="Check out a related question here: http://stats.stackexchange.com/q/8570/2660. All of the books listed discuss joint distributions." CreationDate="2011-04-07T17:12:29.647" UserId="2660" />
  <row Id="15775" PostId="9319" Text="@whuber sorry I deleted my comment because I realized what you meant. To everyone reading it, I was basically asking why he was calculating 366*365/2. " CreationDate="2011-04-07T17:13:51.017" UserId="2965" />
  <row Id="15776" PostId="9319" Text="@rm I genuinely appreciated your comment; it's always good to get such suggestions to clarify a reply." CreationDate="2011-04-07T17:26:21.503" UserId="919" />
  <row Id="15777" PostId="9320" Text="@whuber , Many thanks for the extremely detailed response." CreationDate="2011-04-07T17:40:03.230" UserId="4011" />
  <row Id="15778" PostId="9287" Text="Is the new AUC (for age + microarray) 0.99, or is it 0.86?" CreationDate="2011-04-07T17:58:48.973" UserId="3369" />
  <row Id="15779" PostId="9322" Score="1" Text="Maybe this will help: http://www2.econ.iastate.edu/classes/econ674/falk/Lecture%2016_conditional_heterosked_I.pdf" CreationDate="2011-04-07T18:48:53.267" UserId="919" />
  <row Id="15780" PostId="9318" Score="1" Text="Try [Wikipedia page about MDS](http://en.wikipedia.org/wiki/Multidimensional_scaling), especially the &quot;External links&quot; section." CreationDate="2011-04-07T18:51:45.417" UserId="88" />
  <row Id="15781" PostId="9319" Text="In the [Wikipedia analysis](http://en.wikipedia.org/wiki/Birthday_problem#Calculating_the_probability) there seem to be 365 equally probable birthdays." CreationDate="2011-04-07T19:39:02.547" UserId="2958" />
  <row Id="15782" PostId="7095" Text="Incur a fee for the call.  (a 900-number in the US...)" CreationDate="2011-04-07T19:50:39.660" UserId="4070" />
  <row Id="15783" PostId="9319" Text="@Henry In light of that discrepancy I guess I had better retract my answer, then :-)." CreationDate="2011-04-07T19:50:50.653" UserId="919" />
  <row Id="15784" PostId="9323" Score="1" Text="&quot;Want to assume spatial stationarity&quot; appears directly to contradict the OP's assumption that &quot;the intensities taper off radially in all directions.&quot;" CreationDate="2011-04-07T19:52:12.760" UserId="919" />
  <row Id="15785" PostId="9323" Text="How so?  Such a pattern would occur with the autocorrelation structure I proposed." CreationDate="2011-04-07T19:57:07.377" UserId="3567" />
  <row Id="15786" PostId="9323" Score="1" Text="@charles  It's an important point: if indeed this apparent trend is to be attributed to autocorrelation, then in principle another independent realization of the process could appear to have a dramatically different trend, such as an increase in value away from a central point.  Because the OP has clearly articulated and distinguished some deterministic elements to the trend (the &quot;radial tapering&quot;) and correlational elements (&quot;have influence over multiple grid cells&quot;), a reply that respects this would likely be viewed more positively than one that asserts the OP &quot;will want&quot; to change his mind." CreationDate="2011-04-07T20:04:39.800" UserId="919" />
  <row Id="15787" PostId="9287" Text="Sorry, the increase is 0.004 and not 0.04. For microarray alone to microarray + age. Age alone is 0.82 " CreationDate="2011-04-07T20:55:36.170" UserId="4045" />
  <row Id="15788" PostId="9324" Text="@MEL Just a quick check: `FEMALE` is supposed to be factor with several levels (actually considered as your random effect), but I would have expected something like `lmer(size ~ (1|id), data=Data, subset=gender==&quot;female&quot;)`, where `id` is subject number, `gender` codes for the sex (with two levels, `male`/`female`, the analysis being restricted here on females only). Could you explain a little bit more the structure of your data, or just put the results of `head(Data)`? " CreationDate="2011-04-07T20:58:07.690" UserId="930" />
  <row Id="15789" PostId="8746" Text="+1: Thank you for your answer. It was helpful to my search, but I can't mark two answers as accepted." CreationDate="2011-04-07T21:02:13.280" UserId="1583" />
  <row Id="15790" PostId="1602" Score="1" Text="As a non-expert, I think that the key to the entire debate is that people actually reason like Bayesians. You have to be trained to think like a frequentist, and even then it's easy to slip up and either reason or present your reasoning as if it were Bayesian. &quot;There's a 95% chance that the value is within this confidence interval.&quot; Enough said." CreationDate="2011-04-07T21:08:03.193" UserId="1764" />
  <row Id="15791" PostId="9288" Text="@probabilityislogic +1. Thank you for your answer; it is a little over my head, but I will be looking into it as a possible alternative. I have moved forward with these models using a Gaussian link as the simplest method available that I can reasonably explain. :)" CreationDate="2011-04-07T21:56:41.063" UserId="3309" />
  <row Id="15792" PostId="9299" Text="Thanks all for the interesting points. Let me attempt to clarify. The choice of &quot;Gaussian&quot;, in the light of comments, might be a red herring that causes more confusion than it helps. The key feature of the data is the high intensity values at point of highest belief in the location of the person, and the tapering off &quot;radially&quot; around it (which I've observed empirically). The intensity values come from  the solution to a (linear) inverse problem, and so actually don't necessarily need to be integral - that's just the data we happen to have. " CreationDate="2011-04-07T22:00:48.050" UserId="139" />
  <row Id="15793" PostId="9323" Text="I'm not sure I understand the spatial stationarity condition. On the surface, it seems to be at odds with the idea of having a &quot;peak that tapers out&quot; at a specific location, but I'm clearly not understanding something. " CreationDate="2011-04-07T22:02:22.460" UserId="139" />
  <row Id="15794" PostId="9299" Text="btw I appreciate the attempts to make the question more well defined and better modelled. I'll do my best to explain the actual data setting so as to converge on the right modelling assumptions. " CreationDate="2011-04-07T22:03:15.860" UserId="139" />
  <row Id="15795" PostId="9319" Text="@whuber: 23 seems to be the correct answer for 341 through to 372 equally probable birthdays, so I doubt the details matter" CreationDate="2011-04-07T22:25:02.463" UserId="2958" />
  <row Id="15796" PostId="9327" Text="I'm not aware of any question that covers &quot;exactly the same ground.&quot; While these criteria are sometimes negative, my question is, have I done something wrong?" CreationDate="2011-04-07T22:52:51.213" UserId="2993" />
  <row Id="15797" PostId="9322" Text="There a slight issue in that the lecture says &quot;Therefore, conditional homoskedasticity implies unconditional homoskedasticity&quot; in contradiction to the Econometrics book.  They seem to be conditioning on different things." CreationDate="2011-04-07T23:02:06.787" UserId="2958" />
  <row Id="15798" PostId="9287" Text="OK, so you went from 0.82 for age alone, to 0.954 for age + microarray. So it seems that there is a reasonable amount of new information in the microarray data which wasn't present in age alone." CreationDate="2011-04-07T23:09:40.640" UserId="3369" />
  <row Id="15799" PostId="9329" Text="Some complexity... You could imagine counting the number of times the Legs variable was part of the decision path. But would you just do that for trees that predicted the majority answer, or all of them? Or the difference?" CreationDate="2011-04-07T23:53:12.100" UserId="6" />
  <row Id="15801" PostId="9322" Score="1" Text="@Henry It is difficult to tell from the present question which definitions are accurate and which ones not--some of them do not seem to make sense out of the context of the textbook.  Some clarification would be welcome." CreationDate="2011-04-08T00:25:16.300" UserId="919" />
  <row Id="15802" PostId="9287" Text="But, is the comparison appropriate? My ROC analysis for age alone was 'simple', directly using values of the age variable. But for microarray, or microarray + age, I am using class-membership probability values for an SVM classifier, and not the variable values directly." CreationDate="2011-04-08T00:30:38.143" UserId="4045" />
  <row Id="15803" PostId="9275" Text="@Michael McGowan If they want p-values so badly then it's hard to justify any kind of smoothing IMO. @whuber's comments are on the money; you've induced dependency *and* muddied up the sampling distribution. " CreationDate="2011-04-08T00:52:26.760" UserId="26" />
  <row Id="15804" PostId="9171" Text="@whuber,&#xD;&#xA;&#xD;&#xA;I'm afraid I don't understand your question fully. It's probably my lack of statistical understanding. But to try to clarify. Model 1 has 4 variables, Model 2 has 3 variables and Model 3 has 2 variables. The variables are in the same order in every model (meaning variable one = temp in each model). &#xD;&#xA;&#xD;&#xA;I think @GaBorgulya and @djma answered my question perfectly. Variable 4 IS correlated with variable 3. AH-HA! Makes sense.&#xD;&#xA;&#xD;&#xA;thanks oodles!" CreationDate="2011-04-07T11:23:39.880" UserId="4027" />
  <row Id="15805" PostId="9171" Text="@MEL I've converted your response to the above comment. If you feel one of the current responses helped you or answered your question, don't forget to accept it, as kindly reminded by @richiemorrisroe. BTW, good to see you registered your account." CreationDate="2011-04-08T01:01:38.773" UserId="930" />
  <row Id="15806" PostId="9100" Text="@probabilityislogic Intuitively, the idea that fixing a likelihood function should be about specifying how the data inform rather than just trying to match the sampling distribution appeals to me. It's also somewhat of a guard against model misspecification. Too many Bayesians wring their hands over priors while their likelihood is running the show! The question becomes how to make that operative, when can we get valid inferences, etc. Gibbs posteriors are the only work I'm familiar with which make a serious attempt to do this in applications." CreationDate="2011-04-08T01:02:05.037" UserId="26" />
  <row Id="15807" PostId="9329" Text="And changing all of the `m` predictors one by one and looking to see how the forest predicts differently seems sorta expensive. There's gotta be a better way." CreationDate="2011-04-08T01:03:54.973" UserId="6" />
  <row Id="15808" PostId="9312" Text="@GaBorgulya Usually, [SEM](http://www.ncme.org/pubs/items/16.pdf) is computed in a different way; contrary to SD or SE, it is supposed to account for scores reliability, specific to the measurement instrument." CreationDate="2011-04-08T01:10:00.560" UserId="930" />
  <row Id="15809" PostId="9329" Text="@Harlan, my first thought was to wonder how what you are wanting to do differs from the variable importance of the training data? Are you looking to say that given the other values as they were, what was the sensitivity of the prediction on legs =4 versus legs=2 or legs=0? Have you looked at the partial plot function in the randomforest package?" CreationDate="2011-04-08T01:22:43.397" UserId="2040" />
  <row Id="15810" PostId="7567" Text="@Maurizio, see my updated answer. I've done a simulation study on two different chi-square statistics, including the one that you show above using the same values for the scale parameter, sample size, and truncation point. Using my code to generate the truncated Zipf, I get that the chi-square tests give very sensible results, with a Type I error very close to 0.05. So, I think you may be doing something wrong. Also, you might consider accepting an answer if you think one of them provides enough useful information." CreationDate="2011-04-08T01:53:33.407" UserId="2970" />
  <row Id="15811" PostId="9329" Text="The variable importance is usually defined conditional on the whole training set (or the assumed population, or something). But what I want is the variable importance for a *single* predicted item. Imagine a case where the forest consists of very lopsided decision trees. Test Instance 1 could be explained by 1 or very few decision nodes, while Test Instance 2 could be explained by a much larger set of decision nodes. I want a very simple human-interpretable version of that, like a ranked set of decisions, the top 5 of which I can provide. For a single decision tree, I'd just read them off." CreationDate="2011-04-08T01:59:57.747" UserId="6" />
  <row Id="15812" PostId="9331" Text="try `?splinefun`" CreationDate="2011-04-08T02:20:38.993" UserId="1381" />
  <row Id="15813" PostId="9287" Text="The appropriate comparison is to train SVM on the age feature and use its output, just like you do for microarray + age. For linear SVM, if you use the raw SVM output, the ROC curve would be the same as for using just the age itself. If the probabilities are monotonic in the output (which is probably the case), then again the curve will be the same. The simplest thing is probably to run SVM and see." CreationDate="2011-04-08T02:45:29.660" UserId="3369" />
  <row Id="15814" PostId="9333" Text="perhaps a duplicate of http://stats.stackexchange.com/questions/8586/how-can-i-test-h-0-sigma2-1-sigma2-2" CreationDate="2011-04-08T02:45:29.940" UserId="1381" />
  <row Id="15815" PostId="9333" Text="@David - cheers for the link" CreationDate="2011-04-08T02:46:13.547" UserId="4078" />
  <row Id="15816" PostId="9333" Text="Thanks for the awesome edits Jeromy." CreationDate="2011-04-08T02:53:07.427" UserId="4078" />
  <row Id="15817" PostId="9324" Text="Here is the head(data)" CreationDate="2011-04-08T03:22:35.913" UserId="4027" />
  <row Id="15818" PostId="9332" Text="It is true I have two sites. I have done the plots it's pretty messy but no real visual differences that I can tell." CreationDate="2011-04-08T03:26:35.687" UserId="4027" />
  <row Id="15819" PostId="9336" Score="1" Text="Whether quadratic fits better, will depend on how you have operationalised good fit. In particular, if you use a measure of fit that penalises models with more parameters (e.g., AIC), then, for example, fit can be worse for quadratic versus linear." CreationDate="2011-04-08T03:28:52.967" UserId="183" />
  <row Id="15820" PostId="9324" Text=" I am only concerned with females because I am doing a nesting study.head(data)&#xA;   FEMALE YEAR  SITE FSCLmm FWTg FAGE NOEGGS NumH NumNOTHatch     TEMP     ONETRI     TWOTRI THREETRI HUMIDITY PercHatch perchatch AVHWt&#xA;1 WDW1039 2010 SPSWA    155  537    7      4    0           4 18.83456 0.01372995        NaN      NaN      NaN   0.00000 0.0000000   NaN&#xA;" CreationDate="2011-04-08T03:29:02.127" UserId="4027" />
  <row Id="15821" PostId="9313" Text="@JWM Thanks for the question. I'm particularly interested in descriptions of database handling procedures and computer programmes for the situation when a database of rare cases is available and it is easy and cheap to get appropriate controls. But an overview of matching algorithms would be nicer." CreationDate="2011-04-08T03:55:32.327" UserId="3911" />
  <row Id="15822" PostId="9266" Text="I think whuber's point still hasn't been answered: you need some notion of similarity or distance to complete the specification of the problem. " CreationDate="2011-04-08T04:10:04.693" UserId="139" />
  <row Id="15823" PostId="9266" Text="btw, an alternate formulation that is well formed is: given a set of points in a poset, partition the set into the minimum number of chains. Would that make sense ? " CreationDate="2011-04-08T04:10:40.153" UserId="139" />
  <row Id="15824" PostId="9241" Text="@whuber Do you still find the convergence failures alarming? The Mh Chao model converged in 10000 of 10000 simulations, see the edit." CreationDate="2011-04-08T04:20:44.340" UserId="3911" />
  <row Id="15825" PostId="9274" Text="@whuber - but you see, the assumption of normality is not about &quot;the data&quot;, it is about &quot;the noise&quot;.  If you see &quot;outliers&quot; or &quot;skewness&quot; (which only exist relative to a model in this context), this is telling you something about the model - i.e. that something is missing from it, and it gives you a clue as to how to improve it.  The normal distribution is very good a picking these kind of things up.  And when there is outliers, the residual variance becomes large, and so you get appropriately wide prediction/confidence intervals, as you should." CreationDate="2011-04-08T06:57:48.607" UserId="2392" />
  <row Id="15828" PostId="9343" Score="3" Text="@fRed, the usual answer to non-linearity in context of ARMA models is ARCH/GARCH models. For your first question, it is probably possible to construct PACF using rank correlation concepts, it will boil down probably to independent component analysis. As for the second the answer is probably no, but for this [article](http://arxiv.org/pdf/0712.0679) might be of interest, since it involves non-linear specification and ARMA is a subclass of it." CreationDate="2011-04-08T11:15:33.980" UserId="2116" />
  <row Id="15829" PostId="9323" Score="1" Text="@charles, the pattern you describe will be present for *each* of grid points, due to spatial stationarity assumption. Stationarity is basically saying that all my points behave similarly. This is not the case described by OP. The answer still is very good, but not appropriate in this case. " CreationDate="2011-04-08T11:28:51.780" UserId="2116" />
  <row Id="15832" PostId="7314" Text="you are right whuber, I've underestimated the abilities of Winbugs. " CreationDate="2011-04-08T11:46:54.420" UserId="3280" />
  <row Id="15833" PostId="9299" Text="@Suresh, does each individual get the same amount of influence on neighboring cells (i.e., same variance in Gaussian setting)? What's the density of individuals with respect to grid points? Is influence on neighbors assumed additive?" CreationDate="2011-04-08T11:58:32.527" UserId="2970" />
  <row Id="15834" PostId="9323" Text="Ah, in that case I would suggest applying &quot;detrending&quot; first and then proceeding with the assumption of stationarity.  Apologies for misunderstanding the problem." CreationDate="2011-04-08T12:06:47.233" UserId="3567" />
  <row Id="15835" PostId="3876" Text="I don't understand how for repeated measures designs, `lmer` is suggested rather than the good old `lme`. In such desgins crossed random effects, the main strenght for `lmer`, are rare but quite often you want to model the correlation structure of the residuals. As far as I understand `lmer` does not support that but `lme` does. Am I wrong to assume in such cases `lmer` is an inferior tool compared to `lme`?" CreationDate="2011-04-08T12:32:35.967" UserId="2020" />
  <row Id="15838" PostId="9336" Score="2" Text="@rolando, perhaps I am misunderstanding, but, frankly this sort of (unqualified) advice is precisely the kind of thing that, as statisticians, we spend so much time &quot;fighting&quot; against. Particularly, if the OP is interested in anything beyond simple curve fitting, e.g., prediction or inference, it is *very* important to understand the implications of the &quot;just try whatever you can think of&quot; approach to statistics. " CreationDate="2011-04-08T12:46:00.290" UserId="2970" />
  <row Id="15839" PostId="9339" Text="Do these show how to calculate the new variables from new data (i.e what predict() is doing)?" CreationDate="2011-04-08T13:07:39.323" UserId="2040" />
  <row Id="15840" PostId="9339" Text="@B_Miner: yes, the SO link does." CreationDate="2011-04-08T13:09:10.013" UserId="582" />
  <row Id="15841" PostId="9321" Text="thanks for your helpful feedback" CreationDate="2011-04-08T13:58:52.460" UserId="1417" />
  <row Id="15842" PostId="9339" Text="Why the downvote?" CreationDate="2011-04-08T14:25:30.213" UserId="582" />
  <row Id="15843" PostId="9336" Text="@cardinal, I agree with you. The choice of the model should reflect some prior knowledge of the system and terms should generally not be added simply to improve the fit.  I do encourage my students to use plots, correlations, etc... as tools to explore their data initially - but like you said with the goal of understanding the results and thus the implications of what they have done. " CreationDate="2011-04-08T15:29:12.817" UserId="4048" />
  <row Id="15844" PostId="7982" Text="Have you tried treating the value &quot;missing&quot; as just another label?" CreationDate="2011-03-09T02:10:17.380" UserId="3567" />
  <row Id="15845" PostId="9330" Text="Firth's penalized likelihood methods could be used to address your separation problems, but as a non-SAS user, I don't know whether it is available to GLIMMIX (see http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_logistic_sect063.htm and references). For technical details, see http://www.meduniwien.ac.at/user/georg.heinze/techreps/tr2_2004.pdf" CreationDate="2011-04-08T15:44:21.643" UserId="3432" />
  <row Id="15846" PostId="9361" Text="RE: 1. I know for sure that &quot;higher the number, higher the quality&quot; for all four variables&#xA;&#xA;RE: 2. Good point. What can I do to make two datasets comparable" CreationDate="2011-04-08T16:02:12.117" UserId="333" />
  <row Id="15847" PostId="9319" Text="But what about February 29th?" CreationDate="2011-04-08T16:04:54.253" UserId="1084" />
  <row Id="15848" PostId="9361" Text="@user My recommendations are in the last paragraph: preferably, find a quantitative expression of &quot;quality&quot; in the scientific literature.  Barring that, apply MVAT.  Both produce a fixed formula independent of the dataset.  That assures comparability." CreationDate="2011-04-08T16:06:01.010" UserId="919" />
  <row Id="15849" PostId="9319" Text="@Adam People have done the computations for actual birth frequencies and to account for the relative scarcity of 2/29.  It makes little difference.  After all, the chance that one of your 23 people has a 2/29 birthday is roughly only 23/365/4 = 1.6%, too small to appreciably change the chance of a birthday collision." CreationDate="2011-04-08T16:07:42.523" UserId="919" />
  <row Id="15850" PostId="9361" Text="RE: 1. I know for sure that &quot;higher the number, higher the quality&quot; for all four variables RE: 2. Good point. What can I do to make two datasets comparable RE: 3. Agree. But not relevant. RE 4. Ad advise farmers. RE 5. yes they will be disapointed... they put a lot of effort to grow cultures on something which is not high quality land. RE 6. I will apply some sort of weighing scheme 0.2 0.2 0.3 0.3 or so" CreationDate="2011-04-08T16:09:49.990" UserId="333" />
  <row Id="15851" PostId="9361" Text="If I can assess quality manually... then I can fit logistic regression model. But it would be too comlicated... not very intuitive (for farmers). Simpler the better!" CreationDate="2011-04-08T16:12:44.030" UserId="333" />
  <row Id="15852" PostId="9361" Text="Book seems very good. Thanks..." CreationDate="2011-04-08T16:14:08.100" UserId="333" />
  <row Id="15853" PostId="9361" Text="@user The way to handle complication is to simplify *afterwards.*  When you avoid appropriate analyses in the name of simplification, you lose the opportunity to get superior results.  (Anyway, it is difficult to see how logistic regression would apply to a concept like &quot;quality,&quot; which typically is a continuous variable, not a binary one.)" CreationDate="2011-04-08T16:16:03.543" UserId="919" />
  <row Id="15854" PostId="9362" Text="How can I use correlation analysis to determine weigh?" CreationDate="2011-04-08T16:36:43.293" UserId="333" />
  <row Id="15855" PostId="9287" Text="You are right: the curve (ROC for age using SVM) is the same. Thanks." CreationDate="2011-04-08T16:50:37.567" UserId="4045" />
  <row Id="15856" PostId="9362" Text="If you already have a pre-existing overall measure of quality, e.g expert opinions, (or are willing to accept other variables as a proxy for this), you could choose the highest correlated variables and give it the highest weighting.  " CreationDate="2011-04-08T17:07:14.077" UserId="3489" />
  <row Id="15857" PostId="9339" Text="Sorry Nico, I went to click on the other button! It wont let me change it. I will try to edit your answer ever so slightly and then vote it up. I am still trying to figure out how exactly to use compute the variables, seems there must be a simple formula that I am just not seeing in the SO post." CreationDate="2011-04-08T17:07:31.613" UserId="2040" />
  <row Id="15858" PostId="9361" Text="@whuber, Couldn't one view this as a problem of making a formative measure based on the available information, in which case summing the Z-scores is not as bad as you make it sound? " CreationDate="2011-04-08T17:38:21.367" UserId="1036" />
  <row Id="15859" PostId="9361" Text="@Andy Could you explain what you mean by &quot;formative measure&quot; and &quot;available information&quot;? // I should point out that *many* measures of soil suitability for agriculture are not even monotonic, much less linear: for instance, a plant might flourish within a range of pH but suffer with pH's beyond this range *in either direction*.  It would be a special circumstance indeed--maybe one involving a narrow range of values--if a simple linear combination of soil characteristics had any objective relationship to agricultural qualities." CreationDate="2011-04-08T17:52:42.120" UserId="919" />
  <row Id="15860" PostId="9361" Text="@whuber, by available information I mean solely the attributes of soil that the original poster has access to. By formative measure I mean a construct (unobserved) that is theoretically formed by the observed indicators (in this instance &quot;soil quality&quot;). Both the comments about monotonic and non-linear relationships to &quot;quality&quot; suggest specific types of data need to be transformed (in ways other than Z-score) to be appropriately combined. Unfortunately one can not appropriately transform without having some sort of measure of quality or knowledge of the expected relationship." CreationDate="2011-04-08T18:34:40.337" UserId="1036" />
  <row Id="15861" PostId="9363" Score="2" Text="I disagree. While one would likely be interested in the inter-item correlations for curiosity, all of the variables could be orthogonal yet still contribute to quality. For a silly example the soil in Antarctica may have optimal nitrogen content, but I doubt it would suffice as a suitable climate." CreationDate="2011-04-08T18:45:45.443" UserId="1036" />
  <row Id="15862" PostId="9361" Text="It just seems strange to me (although entirely reasonable given the context) to approach the problem via making decisions given certain criteria. What if the problem could not be formulated as discrete decisions? For instance what if one wanted to make a composite to predict crop yields (but you didn't have data on crop yields to fit a model)? Although it would be difficult (to the point that it would be a silly endeavor) the goal in the above scenario doesn't strike me as all that different." CreationDate="2011-04-08T19:03:40.123" UserId="1036" />
  <row Id="15863" PostId="9361" Text="@Andy Assuming &quot;quality&quot; is a numeric value to be used for ranking soil samples, then definitely the problem is one of discrete decisions: given a pair of attributes $(y_1, \ldots, y_k)$ and $(x_1, \ldots, x_k)$, which has better quality?  You are correct that you need to know something about what quality is in order to create the desired combination of the attributes.  The approach I have taken supposes you do not have an independent assessment of quality (that would put us into a regression or response surface modeling situation), but you can make these comparisons with reasonable accuracy." CreationDate="2011-04-08T19:09:36.047" UserId="919" />
  <row Id="15864" PostId="9299" Text="Do you know in advance how many people there are? Have you tried looking at k-means?" CreationDate="2011-04-08T19:13:34.907" UserId="3369" />
  <row Id="15865" PostId="9327" Score="2" Text="@J. Winchester I'm afraid your question still appears to focus on AIC, and responses to the related question on AIC apply as well here. However, I feel you are also wondering whether your `lmer` syntax allows you to test the hypothesis formulated in your first paragraph. If this is the case, you can simply *edit the title and your question* and focus on this issue, rather than AIC. You might also consider adding the output of `summary(lmera.1)`. Finally, the outcome you consider is a ratio of counts (% `Sp_A`): Is it usual in your field to treat this as a continuous response variable?" CreationDate="2011-04-08T19:30:14.873" UserId="930" />
  <row Id="15866" PostId="9290" Text="I really appreciate the answer (though I'm not quite sure how to plug in my two numbers). I think I may have introduced confusion with my &quot;wrinkle&quot;. See my update above. " CreationDate="2011-04-08T19:34:46.587" UserId="1531" />
  <row Id="15867" PostId="9336" Text="@cardinal - I am thinking carefully about your comments.  If you care to elaborate I'd like to hear more." CreationDate="2011-04-08T19:57:30.390" UserId="2669" />
  <row Id="15868" PostId="7816" Score="1" Text="@raegtin - I'm most familiar with CHAID, which comes up with so-called &quot;interactions&quot; that are often main effects masquerading as interactions because that is the only way the procedure will &quot;let them in.&quot;  (In CHAID there can be only 1 main effect identified as such, so all other main effects get squeezed into &quot;interaction&quot; cells.)  But CHAID has the advantage of being able to check many many interactions. So once a few promising ones are identified, they can be incorporated into a regression or anova, with all of their lower-order components, and one can test for which ones are truly useful." CreationDate="2011-04-08T20:09:52.967" UserId="2669" />
  <row Id="15869" PostId="9118" Text="Thank you both for the comments! &#xA;&#xA;@probabilityislogic, you're absolutely right from a technical perspective. This is being done for a business proposal though, and so unfortunately we need to conserve space. I was mostly looking for a way to ensure our visualizations would be mathematically correct.&#xA;&#xA;@Thomaas, I do have plans to scale the pie charts, I just wanted to make sure the numbers were technically correct. " CreationDate="2011-04-08T20:51:33.263" UserId="4004" />
  <row Id="15870" PostId="9363" Text="@Andy W: In that case, all the variables should be weighted equally, and PCA will tell you that. It would also tell you that the leading component only accounts for a relatively small fraction of the overall variability in the scores matrix." CreationDate="2011-04-08T20:52:32.143" UserId="4062" />
  <row Id="15871" PostId="9299" Text="@cardinal: it depends. as a first approximation one can assume that each individual influences neighboring cells the same way. the catch comes in at corners, where the effect can be a little skewed. The density is small: a typical image might have 20x20 grid cells and 5 individuals, each having a &quot;footprint&quot; that's significant only over 5x5. Influence can be assumed additive for now. @Sheldon we don't know how many people are there, which is why we are trying to stay away from k-means" CreationDate="2011-04-08T21:46:11.220" UserId="139" />
  <row Id="15872" PostId="9323" Text="what is &quot;detrending&quot; ? and please don't apologize :)" CreationDate="2011-04-08T21:47:04.167" UserId="139" />
  <row Id="15873" PostId="9276" Text="Much of this will depend on what the distributions are. E.g. if the distribution of M is very peaked (has a very sharp mode), then 1000 samples may be too much. If it is very wide, then 1000 samples may not be enough. So one question is, do you know (or can you assume something about) how these distributions look?" CreationDate="2011-04-08T22:57:06.503" UserId="3369" />
  <row Id="15877" PostId="9088" Text="@GaBorgulya Thank You. This works." CreationDate="2011-04-09T00:11:09.523" UserId="3897" />
  <row Id="15878" PostId="9278" Text="@sprugman - you say you have 10,000 samples - I take it this means you have *one* sample, and it has 10,000 items.  Or is it that you have *ten thousand samples* of unspecified number of units each?  I'll update my answer with some more details about how the equations work." CreationDate="2011-04-09T01:18:04.030" UserId="2392" />
  <row Id="15879" PostId="9365" Score="3" Text="Do you have some general interests that you'd like to list? That might help guide suggestions. Applications of statistics have become pretty pervasive in a remarkably broad array of fields." CreationDate="2011-04-09T01:58:29.253" UserId="2970" />
  <row Id="15880" PostId="9338" Score="1" Text="Another use of KL-divergence worth mentioning is in hypothesis testing. Assume $X_1, X_2, \ldots$ are iid from measures with density either $p_0$ or $p_1$. Let $T_n = n^{-1} \sum_{i=1}^n \log( p_1(X_i) / p_0(X_i) )$. By Neyman--Pearson, an optimal test rejects when $T_n$ is large. Now, under $p_0$, $T_n \to -D(p_0 \,\vert\vert\, p_1)$ in probability and under $p_1$, $T_n \to D(p_1 \,\vert\vert\, p_0)$. Since $D(\cdot \,\vert\vert\, \cdot)$ is nonnegative, the implication is that using the rule $T_n &gt; 0$ to reject $p_0$ is asymptotically perfect." CreationDate="2011-04-09T02:06:10.990" UserId="2970" />
  <row Id="15881" PostId="9276" Text="I'm not so much interested in how many samples I need to run a Monte Carlo simulation, per se. Instead, I'm interested in the relationship between (a) the number of juries I need, and (b) the number of simulated models I need. For (a), 1,000 juries will sufficiently reduce sampling error. For (b), 1,000 models will fill out the probability distribution of the coefficients. But this setup results in 1000x1000 juries (12 million jurors!) total, which takes too long to run. So I want to know if I can reduce either (a) or (b). I have an idea, which I'll post in the next comment..." CreationDate="2011-04-09T02:33:52.307" UserId="4059" />
  <row Id="15882" PostId="9276" Score="1" Text="(See above comment) I think I can reduce (a), the number of juries. Sampling error is a function of the number of samples. With 1,000 juries per model, I have a million samples total. Sampling error associated with 10^6 samples is ~0.1%. If I use only 35 juries per model, I'd  have 3.5*10^4 samples, and ~0.5% sampling error. This sampling error is much smaller than my measurement error of ~5.0%. Therefore, I should be able to use 35 juries per model and just use the measurement error to estimate my confidence interval." CreationDate="2011-04-09T02:34:54.210" UserId="4059" />
  <row Id="15883" PostId="9363" Score="2" Text="@Hans, I still disagree. It does not tell you if the scores should be weighted equally. Two items could have a positive correlation yet each has opposite relationships to &quot;quality&quot;. The inter-item correlations do not necessarily say anything about the unobserved measure in the given context. If quality were a latent variable and the variables were &quot;reflective&quot; of that latent construct that may be true, but that is not the case in this given example." CreationDate="2011-04-09T03:48:14.483" UserId="1036" />
  <row Id="15884" PostId="9348" Text="Thanks for your response.  I have clarified my question a little to highlight that the 'DirectAgglomerate[...]' algorithm takes a distance matrix only.  Given this, would the modified implementation of Ward's linkage be based on the assumption that the distance Matrix is Euclidean?  Matlab's implementation of Ward's linkage, for example, notes that it is suitable for Euclidean distances only (http://www.mathworks.com/help/toolbox/stats/linkage.html)." CreationDate="2011-04-09T05:44:10.147" UserId="4079" />
  <row Id="15885" PostId="9339" Text="@B_Miner: don't worry, no problem. PS: depending on the type/amount of data you have, you may want to have a look at LOESS regression." CreationDate="2011-04-09T06:20:54.337" UserId="582" />
  <row Id="15886" PostId="8699" Text="No this isn't a homework problem :) but on a serious note let me explain it with my actual problem, it might clear things up. I am looking at different aspects of a product with consumer satisfaction as the indicator. Sadly, I don't have the overall consumer satisfaction but I have the satisfaction ratings for individual characteristics of the product - intrinsic factors like features &amp; I also have the extrinsic satisfaction factors like satisfying needs, expectations, etc. Each of the extrinsic factors are independent to each other but dependent on the intrinsic factors." CreationDate="2011-04-09T07:39:02.320" UserId="3859" />
  <row Id="15887" PostId="8670" Text="@schenectady: Canonical correlation was perfect! I have completed my analysis and got appropriate results as well. Thanks a lot!" CreationDate="2011-04-09T07:41:38.480" UserId="3859" />
  <row Id="15889" PostId="9342" Text="I have recently analyzed a fairly big set of data using Ward method. In my specific case Manatthan distance gave essentially the same clustering as Euclidean distance. I cannot give you any mathematical prove in favour of any combination of methods, but -at least in my case- the clustering was not affected by the distance method" CreationDate="2011-04-09T10:52:20.760" UserId="582" />
  <row Id="15890" PostId="9378" Text="@Gurzo Instead of &quot;data subset&quot;, I think the precise term for what you're trying to do is &quot;stratified sampling&quot;. Maybe, this is a solution: http://rapid-i.com/api/rapidminer-5.1/com/rapidminer/operator/preprocessing/sampling/AbsoluteStratifiedSampling.html." CreationDate="2011-04-09T12:54:12.383" UserId="930" />
  <row Id="15891" PostId="9378" Text="as much I'd like to see more people use rapidminer, I think that this question is way more appropriate for the rapidminer forum (http://forum.rapid-i.com/). Beside: Naive Bayes can handle unevenly distributed discrete labels/targets." CreationDate="2011-04-09T12:54:25.800" UserId="264" />
  <row Id="15892" PostId="9378" Text="@chl: Stratified Sampling works best if the classes are equally distributed, which they aren't. :(" CreationDate="2011-04-09T12:59:54.837" UserId="1522" />
  <row Id="15893" PostId="9342" Text="@Rachel All R functions do not necessarily wait for a distance matrix. See e.g., the on-line help for `agnes` in the [cluster](http://cran.r-project.org/web/packages/cluster/index.html) package." CreationDate="2011-04-09T13:07:22.617" UserId="930" />
  <row Id="15894" PostId="9378" Score="1" Text="@Gurzo Ok, I mean you can impose to keep a certain number of cases from each class." CreationDate="2011-04-09T13:08:46.123" UserId="930" />
  <row Id="15895" PostId="9369" Text="@ chl (+1) those Borsboom papers were wonderful, they really broadened my thinking about measurement" CreationDate="2011-04-09T13:13:16.247" UserId="656" />
  <row Id="15896" PostId="9348" Text="@Rachel: aaah, I see. Any ward implementation has to calculate the distance between cluster members and the centroid. Intuitively it is clear the metric used for this should be equivalent to the metric used to calculate the distances between observations... hence matlab requires an euclidean distmatrix. But now the question arises why implementations do not request a function instead of distance matrix ? How much damage is done when one uses different metrices for both tasks ? I admit, I don't know it right know." CreationDate="2011-04-09T13:25:09.100" UserId="264" />
  <row Id="15897" PostId="9378" Score="1" Text="@Gurzo: What ? Stratified Sampling is exactly the way to go if the classes are unevenly distributed. If they are evenly distributed (given such a huge number of observations), the result of simple random sampling will be equivalent to stratified. btw: vote for close !" CreationDate="2011-04-09T13:29:15.973" UserId="264" />
  <row Id="15898" PostId="9032" Text="@mpiktas: I think you should write that $Q$ is defined on $\sigma(\cup \mathcal{F}_t)$, since $\cup \mathcal{F}_t$ is not necessarily (and indeed, usually is *not*) a $\sigma$-algebra." CreationDate="2011-04-09T13:49:18.213" UserId="2970" />
  <row Id="15899" PostId="9378" Text="@steffen: Stratified Sampling alone boosts accuracy by 2%, but also completely &quot;ignores&quot; the 2 values with the least entries... That's what I was trying to avoid." CreationDate="2011-04-09T13:50:22.297" UserId="1522" />
  <row Id="15900" PostId="9378" Text="Why have you removed your answer? Was it wrong?" CreationDate="2011-04-09T14:23:54.327" UserId="88" />
  <row Id="15901" PostId="9336" Text="@cardinal, #rolando2: I'd like some elaboration for both. What are the pitfalls or benefits of doing three separate regressions, 1 linear, and 1 each with exponential and logarithmic transformed data as opposed to, say, 1 model with 6 params ($a x + b + c \exp(d x) + log(f x + g)$, or something else entirely." CreationDate="2011-04-09T14:49:24.063" UserId="3186" />
  <row Id="15902" PostId="9131" Text="Than you need the formula I suggested above. I will re-post it as an answer." CreationDate="2011-04-09T14:52:57.673" UserId="3911" />
  <row Id="15903" PostId="9375" Text="Thank you very much Henry!  this is most helpful.  I had already written an excel maro that gave me all of the combinations without replacement (essentially, the table you suggest) but I didn't know what to do with the data.  Thanks again for your time and help!" CreationDate="2011-04-09T15:21:26.550" UserId="4085" />
  <row Id="15904" PostId="9374" Text="Thanks MBQ!  As I mentioned to Henry, I can already table out all of the different possibilities using a macro, I should be able to create the tree that you suggest. " CreationDate="2011-04-09T15:23:53.303" UserId="4085" />
  <row Id="15905" PostId="9378" Text="@mbq: I couldn't make it work for my problem... I probably should have waited to post it as answer... :/" CreationDate="2011-04-09T15:48:39.337" UserId="1522" />
  <row Id="15908" PostId="9392" Text="@yO2gO What would/should your confidence score reflect? What is the purpose of the analysis? (You added the `feature-selection` tag, but what are those features?)" CreationDate="2011-04-09T18:57:25.753" UserId="930" />
  <row Id="15909" PostId="9392" Text="@chl this is from the biology domain. For each ID, how important each feature is, is the score (tf-idf) given to it in the sample table above. The confidences reflects, statistically how significant the combination of features are in the whole dataset and not just for one ID." CreationDate="2011-04-09T19:11:21.223" UserId="3111" />
  <row Id="15910" PostId="9392" Text="@yO2gO (1) What do you mean by tf and idf? (2) I checked your example for no pair of F5 and F3 values holds that `F5-F3 = 0.667`, could you explain this?" CreationDate="2011-04-09T19:21:43.190" UserId="3911" />
  <row Id="15911" PostId="9392" Text="@GaBorgulya The values in the table are calculated using tf-idf (term frequency - inverse document frequency). When I say F5-F3=0.667, I mean the two features (F5 and F3) together are 0.667 significant" CreationDate="2011-04-09T19:40:15.317" UserId="3111" />
  <row Id="15912" PostId="9323" Text="I borrow my terminology from time series analysis.  A simple way to estimate (or remove) a global &quot;trend&quot; in the data is to take local averages, for example: the average value within the radius $d$ neighborhood of each cell." CreationDate="2011-04-09T19:59:29.047" UserId="3567" />
  <row Id="15913" PostId="9392" Text="@yO2gO So this is text mining (or information retrieval). And you use the `-` sign to group features into combinations of features, not subtraction. What do you mean by “0.667 significant”, is this a p value of a significance test? What kind of test?" CreationDate="2011-04-09T20:00:41.113" UserId="3911" />
  <row Id="15915" PostId="9392" Text="@GaBorgulya This is post the text mining step. And `-` is not subtraction, it is to attach. What kind of test is my question? I want a test that will give me a ranked list. Say from the 40 features I need a combination of best features. I want a score for the &quot;best&quot;. I am looking for a test for this." CreationDate="2011-04-09T20:29:49.987" UserId="3111" />
  <row Id="15917" PostId="9363" Text="@Andy, I agree with your point if nothing were known about the association of the observed variables with &quot;quality&quot;. But the OP wrote &quot;All of [the variables] are measures of soil quality. Higher the variable, higher the quality&quot; implying a positive association throughout. To be more precise: Let $A$ be the $m \times n$ matrix of observations. Consider the first term $\sigma_1 uv^T$ in the singular value decomposition of $A$. If all $n$ variables have the same association with &quot;quality&quot;, one expects all $v_j$ to have the same sign. In that case, use multiples of these $v_j$ as weights." CreationDate="2011-04-09T21:56:01.400" UserId="4062" />
  <row Id="15918" PostId="9381" Text="Interesting, why is the downvote? I wish I could give you some reputation for your explanation." CreationDate="2011-04-09T21:56:21.487" UserId="1005" />
  <row Id="15919" PostId="9380" Text="Oh I just got confused with the substitution as I got caught up with how did they set $\kappa=1$ to get that, which is not case at all and it is a simple substitution from previous equation. I did understand the derivation of variance of mean reverting process using ito-isometry. Thanks for a nice description." CreationDate="2011-04-09T22:21:08.127" UserId="862" />
  <row Id="15920" PostId="9392" Text="@yO2gO So 0.667 is not related to [significance tests](http://en.wikipedia.org/wiki/Significance_test), but you want some kind of score that measures how good a combination of features is? What do you mean by good and “best”? In what case do you want the combination {F1, F4, F9} be assigned a higher score than {F5, F3}? What's the meaning of a higher score?" CreationDate="2011-04-09T22:33:49.060" UserId="3911" />
  <row Id="15921" PostId="9380" Text="@user862: I figured it might just be a simple situation like this. But, I thought the background of the question might be interesting enough to others that I'd put together a quick exposition. That way the question and answer are more self-contained, especially in case that link ever goes down." CreationDate="2011-04-09T23:06:01.060" UserId="2970" />
  <row Id="15922" PostId="9380" Text="Completely agree. Thanks again." CreationDate="2011-04-09T23:07:03.407" UserId="862" />
  <row Id="15923" PostId="9390" Text="Sounds like homework" CreationDate="2011-04-09T23:16:51.293" UserId="46" />
  <row Id="15924" PostId="9377" Score="2" Text="I posted another question on the same paper - http://stats.stackexchange.com/questions/9396/why-are-cumulative-residuals-from-regression-on-stock-and-index-returns-mean-reve. Please look at it if possible. Thanks!" CreationDate="2011-04-09T23:26:11.783" UserId="862" />
  <row Id="15925" PostId="9373" Text="not sure anyone has been fully responsive to question. It has two parts: (1) Does the modeling strategy described supply a defensible solution to the problem she wants to solve—namely, what is the likelihood that a jury drawn randomly from a community, *C*, with specified demographic characteristics will vote to find a defendant guilty? And (2) If the modeling strategy is reasonable, how many “juries” must she select, and how many “verdicts” must she simulate for each, to report a defensible estimate of the prob of conviction &amp; 0.95 CI? She wants to economize on computing. See her last comment" CreationDate="2011-04-09T23:29:58.703" UserId="2347" />
  <row Id="15926" PostId="9393" Text="Thanks charles. Any reference or reason why this approach is better ?  " CreationDate="2011-04-09T23:31:53.947" UserId="4091" />
  <row Id="15927" PostId="9385" Score="1" Text="Please note that [the numerical accuracy of spreadsheets is very limited](http://www.jstatsoft.org/v34/i04/paper)." CreationDate="2011-04-10T00:43:32.000" UserId="3911" />
  <row Id="15928" PostId="9399" Text="Thanks Ugo - agreed, there definitely needs be both matches and non-matches in the training data. The question is about how many of the (N-1)(N-1) non-matches are needed. For part (2), I would definitely sample w/ equal weight over all observations. " CreationDate="2011-04-10T00:52:53.463" UserId="4095" />
  <row Id="15929" PostId="9399" Text="Well if you don't have apriori on your data there no reasonable way to sample the data. So you have to do uniform sampling, and in this case, the more you take, the better it is. You could however estimate the error introduced by the sampling, but we are missing information here to help you on this point." CreationDate="2011-04-10T00:58:10.167" UserId="3896" />
  <row Id="15930" PostId="9399" Text="It seems to me that the error will depend on the type of classifier used. Anyway you can always try to predict at different sample rate and fix a threshold where you think the error introduced is satisfactory." CreationDate="2011-04-10T01:05:31.913" UserId="3896" />
  <row Id="15931" PostId="9338" Text="Indeed. that's an excellent example. And in fact most general versions of the Chernoff-Hoeffding tail bounds use the KL-divergence. " CreationDate="2011-04-10T02:07:17.000" UserId="139" />
  <row Id="15932" PostId="9391" Text="@Parbury - This may be true, but mostly irrelevant, because the question asks for the expected return of the gambler, not the expected return of the bookie" CreationDate="2011-04-10T02:26:16.607" UserId="2392" />
  <row Id="15933" PostId="9403" Text="Based on my fleeting understanding of dependent errors, not using an AR term is going to throw off the confidence intervals for my estimations, isn't it? Let's just say I want to build a bunch of 95% confidence intervals for each parameter and say &quot;are they different?&quot; (allowing for Bonferonni correction). Aren't the s.e.'s of k (as calculated by R or Matlab) way too conservative if my errors have dependency?" CreationDate="2011-04-10T02:46:12.347" UserId="4096" />
  <row Id="15934" PostId="9400" Text="Wouldn't you need a $b$ parameter in the function like $y = a + b \cdot e^{-kt}$? Where $b=70° - a$." CreationDate="2011-04-10T02:56:58.473" UserId="3911" />
  <row Id="15935" PostId="9403" Text="If the $y = a + e^{-(k_i + \alpha_j)t} + \epsilon$ model correctly describes the cooling the $\epsilon_{j,t}$ values should be independent." CreationDate="2011-04-10T02:58:14.200" UserId="3911" />
  <row Id="15936" PostId="9397" Text="Thanks for your response. Though, I found neither circles plot nor colored scatterplot was easy to interpret. However, I suddenly realize that my question is about the interaction between the two continuous variable X1 and X2. So I Googled and found this [link](http://www.ats.ucla.edu/stat/sas/faq/spplot/reg_int_cont.htm) seemed to be helpful to me." CreationDate="2011-04-10T03:19:37.670" UserId="400" />
  <row Id="15937" PostId="2719" Text="Your item 1 is incorrect: the P value is the probability of obtaining data as extreme, or more extreme, given the null hypothesis. As far as I know that means that P _is_ a probability--conditional but a probability nonetheless. Your statement is correct in the circumstances that one is working within the Neyman-Pearson paradigm of errors, but not is one is working within the Fisherian paradigm where P values are idices of evidence against the null hypothesis. It is true that the paradigms are regularly mixed into an incoherent mish-mash, but both are 'correct' when used alone and intact." CreationDate="2011-04-10T09:37:37.420" UserId="1679" />
  <row Id="15938" PostId="2719" Text="For confidence intervals you are, again, correct only within the confines of Neymanian confidence intervals. Fisher (and others before him) also devised and used things that one would interpret as confidence intervals, and there is a perfectly valid interpretation of such intervals refering to the particular experiment yielding the interval. In my opinion, they far preferable to Neyman's. See my answer to the question Discrete functions: Confidence interval coverage? for more detail: http://stats.stackexchange.com/questions/8844/discrete-functions-confidence-interval-coverage/8860#8860" CreationDate="2011-04-10T09:39:40.927" UserId="1679" />
  <row Id="15939" PostId="9405" Text="@Adam By &quot;value&quot;, I guess you mean &quot;parameter&quot;?" CreationDate="2011-04-10T10:09:06.913" UserId="930" />
  <row Id="15940" PostId="9401" Text="@Neil (+1 for the rapidminer tips) Agree about R questions and stats software. However, questions about statistical software that are barely related to statistical analysis per se will be better served on SO (which is clearly stated on the [FAQ](http://stats.stackexchange.com/faq), see the *programming* subject). Now, the ongoing debate about stratified sampling in the comments makes it relevant (to a certain extent) on here. I'm sure @steffen will agree with that, and his first comment might also be temperated by the ensuing ones." CreationDate="2011-04-10T10:18:42.503" UserId="930" />
  <row Id="15941" PostId="9392" Score="1" Text="I'm not sure this question deserves *so many downvotes*. Although it lacks some information, some clarifications are given as comments, upon request -- @yO2gO, maybe you could now update your question and just add a rough definition of what you think of or expect from such a *confidence score*?" CreationDate="2011-04-10T10:26:47.857" UserId="930" />
  <row Id="15942" PostId="9388" Text="Thanks. I know the papers your mention. Their setting is however different from the one I describe." CreationDate="2011-04-10T11:29:42.443" UserId="30" />
  <row Id="15943" PostId="9385" Text="@GaBorgulya - MS reviewed a few of their functions in response to papers such as this for Office 2010.  [Function Improvements in Excel 2010 Blog](http://blogs.office.com/b/microsoft-excel/archive/2009/09/10/function-improvements-in-excel-2010.aspx) and [Function Improvements in Excel 2010 PDF](http://blogs.office.com/cfs-file.ashx/__key/CommunityServer-Components-PostAttachments/00-00-00-49-53/ExcelFunctionImprovements-10_2D00_05_2D00_09-_2800_1_2900_.pdf)" CreationDate="2011-04-10T11:30:54.600" UserId="3484" />
  <row Id="15944" PostId="9363" Score="2" Text="@Hans, I still disagree. Even if the association is expected to be in the same direction this does not mean the indicators should be inherently given any weight based on their inter-item correlation. The shared variance can only say something about the relationship between the indicators. Think of a regression model in which we predict a known measure of quality from these indicators. The inter-item correlations between the indicators do not tell you what the expected slopes will be." CreationDate="2011-04-10T12:38:29.797" UserId="1036" />
  <row Id="15945" PostId="9369" Text="+1, I enjoy Borsboom as well. For those interested in The Attack article I think would also be interested in &quot;The Concept of Validity&quot;, http://rhowell.ba.ttu.edu/borsboomValidity2004.pdf . Although it is a little more verbose so it is not as easy to follow as the Attack article." CreationDate="2011-04-10T13:08:16.693" UserId="1036" />
  <row Id="15946" PostId="9409" Text="Why the new account?" CreationDate="2011-04-10T14:52:10.977" UserId="2970" />
  <row Id="15947" PostId="9381" Text="Yes, why the downvote? If you're willing to downvote you should have the common decency and confidence to explain why." CreationDate="2011-04-10T14:53:37.697" UserId="2970" />
  <row Id="15948" PostId="9311" Text="A related question: http://stats.stackexchange.com/questions/4/assessing-the-significance-of-differences-in-distributions" CreationDate="2011-04-10T15:07:34.720" UserId="3911" />
  <row Id="15949" PostId="4" Text="A related question: http://stats.stackexchange.com/questions/9311/kullbackleibler-vs-kolmogorov-smirnov-distance" CreationDate="2011-04-10T15:09:11.863" UserId="3911" />
  <row Id="15950" PostId="9401" Text="@chl: I agree. However, I estimate that 99% of the questions about rapidminer will be related to machine learning (it's in the nature of the tool). Take a look at the rm-forum. It is overflowed with questions similar to this one, which is a good representative example. In depth coverage of machine learning in questions like &quot;how to do X in tool Y&quot; will only emerge if the goal and the way of the OP are questioned. I am just afraid that stats* will mirror the rm-forum. Ceterum censeo R is special and deserves a special treatment ;)" CreationDate="2011-04-10T15:15:39.120" UserId="264" />
  <row Id="15951" PostId="9416" Text="n/w = network? Isn't packet-transmission data usually long-tailed?" CreationDate="2011-04-10T15:31:47.260" UserId="2970" />
  <row Id="15952" PostId="2719" Text="@Michael you are correct, but lets see: How many times is the Null correct? Or better: Can anyone prove if the null is correct? We can also have deep philosophical debates about this but that is not the point. In quality control repetitions make sense, but in science any good _decision rule_ _must_ condition data." CreationDate="2011-04-10T16:51:33.077" UserId="1307" />
  <row Id="15954" PostId="2719" Text="Fisher knew this (conditioning on the observed data and the remark about quality control is based on that). He produced many counter examples based on this. Bayesian have been fighting about this, lets say, for more than half-a-century." CreationDate="2011-04-10T16:58:19.140" UserId="1307" />
  <row Id="15956" PostId="9391" Text="@probability So what *is* the gambler's expected return?  I couldn't find it in your reply :-)." CreationDate="2011-04-10T17:06:10.397" UserId="919" />
  <row Id="15960" PostId="9409" Text="@Mike You currently hold three unregistered accounts. I've merged them all onto your first account (user id=1920), but please register once and for all (use an OpenID or see the [FAQ](http://stats.stackexchange.com/faq)). Thanks @cardinal!" CreationDate="2011-04-10T19:48:10.320" UserId="930" />
  <row Id="15961" PostId="9401" Text="@steffen Be sure I've heard you, and I'm certainly sharing the same concern about technical or basic questions. But please note that a lot of users here (including myself) answered similar questions about R in the past, even if those questions would have been rejected/downvoted on R-help or SO. It seems @Neil's response proved to be helpful, the OP is registered, replied to (somewhat fruitful) comments, and, as I hope, may come again with some good stats questions. Sorry, these are my Sunday evening lucubrations..." CreationDate="2011-04-10T20:05:12.887" UserId="930" />
  <row Id="15962" PostId="9418" Text="Thank you! Although not entirely clear, this does give me some idea. Could you please recommend some reading where these kind of distributions are tackled? I have been only exposed to Gaussians and the other standard distributions. Another question, does this have something to do with the radial distribution functions of the atoms etc? " CreationDate="2011-04-10T23:24:57.747" UserId="4101" />
  <row Id="15963" PostId="2719" Text="@suncoolsu I'm sorry, but I don't understand your response. Either the P value is a probability of it isn't, and either P values and confidence intervals reflect on the data from which they were generated, or they don't. The choice of paradigm determines the correct option." CreationDate="2011-04-10T23:25:36.250" UserId="1679" />
  <row Id="15964" PostId="3402" Text="Why do we always pick on the gingers? :) " CreationDate="2011-04-11T00:18:57.937" UserId="776" />
  <row Id="15965" PostId="9418" Score="1" Text="@Infinity Let me know what part is not clear so I can try to fix it up.  I don't know where such distributions are discussed, but related analysis can be found in the literature on &quot;circular distributions.&quot;  The underlying mathematical ideas are indeed somewhat tenuously related to atomic orbital theory.  The relevant concepts include separability of the Schrodinger Equation in spherical coordinates, constructing Haar measure of a compact Lie group by averaging, and comparing orbitals by means of overlap integrals." CreationDate="2011-04-11T01:43:16.630" UserId="919" />
  <row Id="15966" PostId="9423" Text="I just noticed that this question was asked 2 months ago! How did it get bumped up to the top of the list?" CreationDate="2011-04-11T02:57:21.760" UserId="1569" />
  <row Id="15967" PostId="9408" Text="In my case the &quot;population&quot; size is typically around 100-2000, so at least at the low end, is it wrong to use tests that assume the sample is drawn from an infinite population?" CreationDate="2011-04-11T03:01:34.987" UserId="4099" />
  <row Id="15968" PostId="9032" Text="@cardinal, you are right, although in the book I cited, only union is used. I will have to recheck." CreationDate="2011-04-11T03:24:56.987" UserId="2116" />
  <row Id="15969" PostId="9384" Score="1" Text="Thanks for your comment; I think you are correct.  However, in practise it seems that Ward's linkage is often used with non Euclidean distances.  I am still not sure what the implications of this might be." CreationDate="2011-04-11T04:35:08.010" UserId="4079" />
  <row Id="15970" PostId="9417" Score="3" Text="Could you expand on how this concept provides an answer to the question?" CreationDate="2011-04-11T07:02:42.437" UserId="449" />
  <row Id="15971" PostId="9279" Text="well it's actually the other way around; I have scores for all sets, independent of the hierarchy. The problem is that, the results given in a table form, is rather large and unintuitive. What I would like to do, is to group the sets, by checking out their relationships with one another, and thus re-creating the hierarchy. In this fashion, one would easily see if a whole branch of the hierarchy has scored high, or if small subsets (leaves of the tree) have scored high. &#xA;That said, I don't quite follow how regression could be of use here." CreationDate="2011-04-11T07:41:58.317" UserId="3014" />
  <row Id="15972" PostId="9266" Text="@Suresh: the problem with coming up with a measure of similarity is that there could (and probably will)  two seemingly identical sets which belong to completely different branches of the hierarchy. So looking at the elements only, one would say that they have very little distance to one another and yet in the &quot;right answer&quot; they should be far apart from one another. I am starting to feel like what I want to do is rather impossible with the amount of information available at this point. :S" CreationDate="2011-04-11T07:46:10.183" UserId="3014" />
  <row Id="15973" PostId="9406" Score="1" Text="@Dikran: (+1), however I think one has to dinguish between the goal of ranking (measure:AUC) and separating the two classes (measure: Accuracy). In the former case, given a probabilistic classfier like Naive Bayes, imbalance playes a lesser role, I suppose. Or should one be worried in this case, too ? Another question: What do you mean by &quot;post-process the output&quot; ? Converting scores to actual probabilities ?" CreationDate="2011-04-11T08:01:11.933" UserId="264" />
  <row Id="15974" PostId="9396" Score="1" Text="@user862, The sum of residuals equal to zero is a standard feature of regression with intercept. Differentiate sum of squares with respect to intercept to see why." CreationDate="2011-04-11T08:23:24.620" UserId="2116" />
  <row Id="15975" PostId="9406" Text="@Steffen My intuition is that the class imbalance problem is less of an issue for ranking, but that it won't go away completely (I am working on a paper on this problem, so that is something worth resolving).  By post-processing, I meant multiplyng the outputs by the ratio of the operational and training set class frequencies and then re-normalising so the probabilities of all possible outcomes sum to one.  However in practice the actual optimal scaling factor is likely to be somewhat different - hence optimise with XVAL (but still re-normalise)." CreationDate="2011-04-11T08:26:14.187" UserId="887" />
  <row Id="15976" PostId="9385" Text="@osknows, try adding $1\frac{1}{6}$ and $1\frac{1}{4}$ in Excel 2010 as fractions. They fixed some long-standing problems, but still lots of them remain. " CreationDate="2011-04-11T08:26:17.377" UserId="2116" />
  <row Id="15978" PostId="9385" Text="@Elizabeth, could you provide a data so we can test on other statistical packages to see whether the problem is in Excel or in data or general applicability of exponential smoothing to this particular data set?" CreationDate="2011-04-11T08:32:27.620" UserId="2116" />
  <row Id="15979" PostId="9400" Text="Since your data has time trend it is natural that it will be autocorrelated. This does not mean however that you really need to correct for AR. Try fitting model without any special error variance assumption and then check for signs of autocorrelation." CreationDate="2011-04-11T08:35:21.657" UserId="2116" />
  <row Id="15980" PostId="9418" Text="Thanks. I am very new to probability and stats so it was probably because of that. I don't really understand what you mean by &quot;average the distribution circularly around its center&quot;, I think it means to average out all the circles so that there is only one circle left with center at $(\mu_X, \mu_Y)$ and radius $\rho$ kinda like a linear regression line fit. Is that correct?" CreationDate="2011-04-11T08:42:02.927" UserId="4101" />
  <row Id="15981" PostId="9418" Text="The other doubt I have is that the distribution function $F(\rho)$ seems to describe a disc but the figure (and what I had in mind)  is a ring. The random variable  $(\Xi, H)$ describes the average circle in polar form. I am sorry I do not clearly get what happens next. I understand we compare the two distributions using some distance metric, but why is the $(\Xi, H)$ special and how it helps I am unable to reason. I am sorry if the questions seem too stupid." CreationDate="2011-04-11T08:55:02.323" UserId="4101" />
  <row Id="15983" PostId="9385" Text="@mpiktas, the links were just for info. I'm not suggesting at all Excel is either fixed, works or can be used reliably." CreationDate="2011-04-11T10:36:24.453" UserId="3484" />
  <row Id="15984" PostId="9408" Score="1" Text="Google search found a [JSTOR paper](http://www.jstor.org/pss/2286664) for linear models in finite populations. Substituting your figures should decide if the difference between the finite and infinite population setting is negligible or not." CreationDate="2011-04-11T11:05:07.110" UserId="3911" />
  <row Id="15985" PostId="9393" Text="@ablimit Your problem will just likely average out =)" CreationDate="2011-04-11T11:05:08.750" UserId="88" />
  <row Id="15986" PostId="4009" Text="@Lisanne I see you've lost your account -- I have merged them, but please use your original Facebook login." CreationDate="2011-04-11T11:06:34.520" UserId="88" />
  <row Id="15987" PostId="9425" Score="4" Text="At the bottom of the following Web page from NIST/Sematech, http://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm,it is recommended that both tests should be preformed and the smaller of the two intervals be taken. I have found similar comments in Johnson and Wichern on doing MANOVA. " CreationDate="2011-04-11T12:31:34.543" UserId="3805" />
  <row Id="15988" PostId="9426" Score="1" Text="@walkytalky You mention “other things that don't properly apply”, but let me give a counter example. The [phi coefficient](http://en.wikipedia.org/wiki/Phi_coefficient) and [Cramér's V](http://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V) rely on the determinant of such a contingency table:&#xA;$$\phi=\frac{n_{11}n_{00}-n_{10}n_{01}}{\sqrt{n_{1\bullet}n_{0\bullet}n_{\bullet0}n_{\bullet1}}}=&#xA;\frac{\text{det}\begin{bmatrix}&#xA;n_{11} &amp; n_{10}\\n_{01} &amp; n_{00}&#xA;\end{bmatrix}}{\sqrt{n_{1\bullet}n_{0\bullet}n_{\bullet0}n_{\bullet1}}}$$" CreationDate="2011-04-11T12:57:51.067" UserId="3911" />
  <row Id="15989" PostId="9426" Text="@walkytalky I divided through by the total count to [estimate probabilities with relative frequencies](http://en.wikipedia.org/wiki/Frequency_probability#Definition)." CreationDate="2011-04-11T13:04:11.407" UserId="3911" />
  <row Id="15990" PostId="9385" Text="@mpiktas, I have added some example data to my question. I am new to this forum, so if there is a better way of uploading my data, please let me know. " CreationDate="2011-04-11T13:14:36.700" UserId="4092" />
  <row Id="15991" PostId="9426" Text="@GaBorgulya Those stats work from count tables, though. And estimating the probabilities is not what I'm objecting to, it's presenting a single set of probabilities (summing to one) as a 2x2 matrix. These are two different kinds of information, and this presentation risks confusing them. I am not saying it is actively wrong, just that it is unclear." CreationDate="2011-04-11T13:16:48.327" UserId="174" />
  <row Id="15992" PostId="9429" Text="@George, it would help a lot if you wrote precisely what model are you trying to fit in mathematical notation." CreationDate="2011-04-11T13:49:05.730" UserId="2116" />
  <row Id="15993" PostId="9418" Score="1" Text="@Infinity I added some clarifying remarks.  You don't average out circles; rather, you average out (or &quot;smear&quot;) all the probability *across* each circle so that no matter what you started with, it ends up looking like my picture (with circular contours).  If the original distribution was truly circular, this averaging doesn't change it.  Thus, comparing the distribution to its averaged version tells you how far it is from being circular in the first place." CreationDate="2011-04-11T14:55:53.417" UserId="919" />
  <row Id="15995" PostId="9405" Text="@Adam Please do not post &quot;thank-you&quot; answers, use upvotes and accept instead. And please register your account by going here: http://stats.stackexchange.com/login" CreationDate="2011-04-11T14:57:57.050" UserId="88" />
  <row Id="15996" PostId="9435" Text="The thread at http://stats.stackexchange.com/questions/9040/visualize-movie-actor-relationships is closely related: although the application is different, it concerns visualizing bipartite graphs." CreationDate="2011-04-11T14:59:51.510" UserId="919" />
  <row Id="15997" PostId="9434" Score="1" Text="According to the abstract, this paper seems to deal with an &quot;error-prone binary covariate&quot;: that is, with misclassified *independent* variables only." CreationDate="2011-04-11T15:03:10.540" UserId="919" />
  <row Id="15998" PostId="9425" Score="1" Text="@schenectady Good answer!  Why don't you paste it into a reply?  BTW, the link is corrupted in your comment; the correct one is http://itl.nist.gov/div898/handbook/prc/section4/prc473.htm ." CreationDate="2011-04-11T15:07:11.440" UserId="919" />
  <row Id="15999" PostId="9032" Text="@mpiktas: interesting, as that is a well-known and highly regarded reference. You might be interested in [this question](http://math.stackexchange.com/questions/26888/the-union-of-a-strictly-increasing-sequence-of-sigma-algebras-is-not-a-sigma) on math.SE. " CreationDate="2011-04-11T15:27:48.157" UserId="2970" />
  <row Id="16001" PostId="2719" Text="@Michael Sorry if I wasn't clear enough. All I wanted to say: P-value is a probability _ONLY_ when the null is true, but most of the times null is _NOT_ true (as in: we never expect $\mu=0$ to be true; we assume it to be true, but our assumption is practically incorrect.) In case you are interested, I can point out some literature discussing this idea in greater detail." CreationDate="2011-04-11T15:36:14.787" UserId="1307" />
  <row Id="16002" PostId="9429" Text="Hmm, I think that part of the problem is that i am not sure what exactly what the model would be in notation.&#xA; &#xA;For an observation level predictor X, a repeated measure group level predictor Z, &#xA;and for i observation, j individuals, and g groups, &#xA;Although i am not sure, I think that what i need is something like this:&#xA;&#xA;$\ Y_{ijg}= \beta_{0j}+\beta_{1g}+\beta_{2j}time_{ij}+ \beta_{3g}time_{ig}+ \beta_{4}X_{ij}+e_{ij} $ &#xA;&#xA;$\ \beta_{0j} = \gamma_{00}+u_{0j} $   &#xA;&#xA;$\ \beta_{1g} = \gamma_{10}+\gamma_{11}Z_{g}+u_{0g} $ &#xA;&#xA;$\ \beta_{2j} = \gamma_{20}+u_{2j} $ &#xA;&#xA;$\ \beta_{3g} = \gamma_{30}+u_{3g} $ &#xA;" CreationDate="2011-04-11T15:38:06.310" UserId="1871" />
  <row Id="16004" PostId="9434" Score="1" Text="Actually the abstract deals with both: &quot;For outcome misclassification, we argue that a likelihood-based analysis is the cleanest and the most preferable approach. In the case of covariate misclassification, we combine [....]" CreationDate="2011-04-11T16:00:50.433" UserId="2669" />
  <row Id="16005" PostId="9426" Text="@walky &quot;Unclear&quot; suggests there is at least one other natural way in which the expression could be interpreted.  Could you give an example of such an alternative?  Given that the left hand side explicitly announces that a probability is forthcoming, I'm having a hard time imagining how the values on the right hand side would be misunderstood." CreationDate="2011-04-11T16:04:54.277" UserId="919" />
  <row Id="16006" PostId="9266" Text="which is why I'm wondering about the 'min number of chains' formulation. that would naturally handle this case, by placing the elements in different chains. " CreationDate="2011-04-11T16:05:01.597" UserId="139" />
  <row Id="16007" PostId="9423" Text="@Hong Ooi  - The website's documentation talks about a process whereby older threads are occasionally bumped up at random to see whether anyone has anything new to add." CreationDate="2011-04-11T16:06:20.067" UserId="2669" />
  <row Id="16008" PostId="7024" Text="This is a resourceful approach and yet I wonder whether its benefits are outweighed by the problems it might pose for interpretability for some audiences." CreationDate="2011-04-11T16:07:54.090" UserId="2669" />
  <row Id="16009" PostId="9193" Text="I am currently reading The Elements of Graphing Data, http://books.google.com/books?id=KMsZAQAAIAAJ&amp;q. If you are familiar with it, do you know what the differences between these two books are (I can't seem to find a good description of either online)?" CreationDate="2011-04-11T16:27:04.600" UserId="1036" />
  <row Id="16010" PostId="9428" Text="Thanks for your help." CreationDate="2011-04-11T16:27:09.797" UserId="4108" />
  <row Id="16011" PostId="9435" Score="1" Text="You might want to also check out some of the other questions with the tag network, http://stats.stackexchange.com/questions/tagged/networks . They will have suggested graphics and tools to do those graphics as well." CreationDate="2011-04-11T16:28:57.377" UserId="1036" />
  <row Id="16012" PostId="9435" Text="So it was the case that it was a vocabulary issue. The &quot;edge-vertex graph visualization&quot; looks fantastic. Given that, what available libraries/utilities are there that I could use. Googling &quot;edge-vertex graph visualization&quot; doesn't yield many results..." CreationDate="2011-04-11T16:30:05.573" UserId="4112" />
  <row Id="16013" PostId="2719" Text="@Michael Here is another way to saying the above paragraph: Assuming we want to answer relevant questions—like evaluating a hypothesis by its posterior probability, which answers “given the observed data, what is the probability this hypothesis is true?” (this is what we want to do most of the time in Science) as opposed to the frequentist p-value, which answers the question “if this hypothesis is true (which it might not be), what is the probability of observing even more extreme data (which we didn't)?” (modified form of Dr. Wolpert's statement)" CreationDate="2011-04-11T16:32:33.410" UserId="1307" />
  <row Id="16015" PostId="9435" Text="@Jacques, unfortunately that's not standard terminology. I was just referring to some methods using that term in order to more easily distinguish them from some of the other approaches being discussed. The overlap in meanings of the term &quot;graph&quot;, especially in this visualization context is unfortunate and makes Google searches more challenging. " CreationDate="2011-04-11T17:20:12.363" UserId="2970" />
  <row Id="16016" PostId="9426" Text="If the goal is to represent a joint pmf of two discrete variables (which is how I interpreted &quot;probabilities of the outcomes&quot;), how is this presentation unclear? The brackets?" CreationDate="2011-04-11T17:21:17.800" UserId="26" />
  <row Id="16017" PostId="9439" Score="2" Text="Purely my opinion, but this might be better placed as a single line comment, especially in view of @whuber's remark. " CreationDate="2011-04-11T17:22:24.343" UserId="2970" />
  <row Id="16018" PostId="9315" Text="What do you mean by &quot;predicting the topic of a new document&quot;? Do you want to find which one single topic this document was generated from? Do you want to find a mixture of topics for the document? Do you want to label each word in the new document by the topic that word came from?" CreationDate="2011-04-11T18:17:11.573" UserId="3369" />
  <row Id="16019" PostId="9197" Text="+1 for caret.  If you want to use cross-validation to tune your nnet model, it is a very useful package.  If you want to just assess the performance of your nnet using cross validation, it it trivial to write your own loop." CreationDate="2011-04-11T19:13:43.577" UserId="2817" />
  <row Id="16020" PostId="8034" Text="How many products, I wonder, need to be involved before a simple correlation matrix is insufficient?" CreationDate="2011-04-11T19:42:37.223" UserId="2669" />
  <row Id="16021" PostId="9426" Text="Well, I said it was subjective and must bow to wiser heads. My concern stems from expecting there to be some matrix-operational reason to use matrix representation and not seeing one. But anyway, I'd best shut up before I make myself appear even stupider than I am." CreationDate="2011-04-11T20:12:22.630" UserId="174" />
  <row Id="16022" PostId="9392" Text="@GaBorgulya You are right. I need some kind of score that measures how good a combination of features is. Good/best is the most significant in the dataset. `{F1, F4, F9}` should be assigned a higher score than `{F5, F3}` based on how each of the features are scored in the dataset." CreationDate="2011-04-11T20:49:46.310" UserId="3111" />
  <row Id="16023" PostId="9392" Text="@chl Thanks. I have updated the question. I was/am unsure on how to frame the question. But I hope this updated version helps you guys help me better. Thanks" CreationDate="2011-04-11T20:51:25.590" UserId="3111" />
  <row Id="16024" PostId="9446" Score="2" Text="@user4119 I've tried to reword a little your post, and I corrected your $H_0$. It will be helpful if you can confirm that (a) parental response is the outcome, scored as a binary response (positive or negative &quot;case&quot;), (b) the scale you used is already a validated and unidimensional scale, that is working with a totalled score makes sense and reflect a single underlying construct (and all items were correctly reverse-scored, when needed). You said you cannot test for a gender effect because N's are even: it doesn't make sense at all." CreationDate="2011-04-11T21:08:47.490" UserId="930" />
  <row Id="16025" PostId="9449" Score="1" Text="http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval" CreationDate="2011-04-11T21:08:47.947" UserId="449" />
  <row Id="16026" PostId="8037" Text="Eh, you go to the gym to LOOSE weight ? :)" CreationDate="2011-04-11T21:09:08.783" UserId="704" />
  <row Id="16027" PostId="9447" Text="Welcome to the site! I tried to make title a bit more informative about the nature of the problem." CreationDate="2011-04-11T21:10:17.213" UserId="88" />
  <row Id="16028" PostId="9449" Text="Thanks for the link. My question is if I am getting $n$, the denominator, right for the different variations (and confirmation that the other instances are right)." CreationDate="2011-04-11T21:12:54.467" UserId="3186" />
  <row Id="16029" PostId="9448" Score="1" Text="Linear model? Every meteorologist will tell you that everything they do is just a small adjustment to &quot;tomorrow will be the same&quot; ;-)" CreationDate="2011-04-11T21:15:27.803" UserId="88" />
  <row Id="16030" PostId="9420" Text="@GaBorgulya Out of curiosity, what would be the rationale for displaying a table this way?" CreationDate="2011-04-11T21:29:08.197" UserId="930" />
  <row Id="16031" PostId="9392" Text="@yO2gO I'm still unsure how you mean “[best] combination of features … is the most significant in the dataset”. A way I can think about this is that you want to select a subset of features that contain most of the information contained in the dataset. In other words a good combination of features predicts all other features with good accuracy. You may also want to select the smallest number of such features. Is this what you want?" CreationDate="2011-04-11T21:34:57.050" UserId="3911" />
  <row Id="16032" PostId="9365" Text="@cardinal, nope, no particular interests -- the purpose was to branch out from the stuff I typically read, so I'm trying not to limit any answers. (This does maybe make the question a bit too broad, but I guess I'm looking for people's personal &quot;best of&quot; lists.)" CreationDate="2011-04-11T21:40:41.700" UserId="1106" />
  <row Id="16033" PostId="9392" Text="@GaBorgulya Yes, this is what I am looking for." CreationDate="2011-04-11T22:02:35.023" UserId="3111" />
  <row Id="16034" PostId="9420" Text="@chl The four frequencies (and probabilities) have a natural ordering, the 2 by 2 table. So the matrix notation is an appealing way to express the probability space. Also the determinant of the matrix is useful, see my comment below. So I speculated that although I had not met this notation, a similar notation might be in use. (I also like the concept of a [Karnaugh map](http://en.wikipedia.org/wiki/Karnaugh_map).)" CreationDate="2011-04-11T22:04:52.810" UserId="3911" />
  <row Id="16035" PostId="9306" Text="Why not compare mean frequencies during each period with a set of t.tests?" CreationDate="2011-04-11T22:17:46.833" UserId="776" />
  <row Id="16036" PostId="9443" Text="Thanks for the pointer, I'll check it out." CreationDate="2011-04-11T22:18:20.780" UserId="776" />
  <row Id="16037" PostId="9409" Text="I disagree that this question is ill formed. The 10 and 25 may be Poisson parameters, or observations, but even if they are only two observations they can be used as approximations of the parameters, so they can serve as basis of power calculation. (Alpha must be a mistake, but this does not make the question invalid.)" CreationDate="2011-04-11T23:54:41.083" UserId="3911" />
  <row Id="16038" PostId="1781" Text="You have already mentioned multiple imputation, the answer I'm thinking about. Why don't you use draws from truncated distributions?" CreationDate="2011-04-12T02:17:00.033" UserId="3911" />
  <row Id="16043" PostId="9447" Score="1" Text="I'm missing something. If we are predicting &quot;the average number of days in a year&quot; -- why would this necessarily be seasonal? Annual sunspot numbers are seasonal because that's an 11 year cycle, but it's not obvious to me that a seasonal component would be necessary here. " CreationDate="2011-04-12T04:51:12.940" UserId="3919" />
  <row Id="16044" PostId="9458" Text="@Jeromy: so, in a nutshell, you see nothing wrong with what I proposed (equal distance coding, t-test significance test), other than there are other options that may be more accurate?" CreationDate="2011-04-12T04:51:47.653" UserId="4123" />
  <row Id="16045" PostId="9458" Text="@Mac In my opinion, coming more from an applied perspective, it's a simple, easy to understand, easy to communicate, and generally reasonable approach. However, it's often worth considering measuring constructs of interest using scales rather than individual items." CreationDate="2011-04-12T05:20:08.110" UserId="183" />
  <row Id="16046" PostId="1781" Text="@Ga Thanks for the thought.  I'm concerned that the correlation of the truncated distributions could vary markedly from the actual correlation.  There's ample evidence of this in the environmetrics and chemometrics literature dating back to the mid-80s." CreationDate="2011-04-12T05:36:00.077" UserId="919" />
  <row Id="16047" PostId="9459" Text="@zbicyclist, since star usually means convolution in mathematical notation your decomposition model is far from classical :)" CreationDate="2011-04-12T05:43:59.430" UserId="2116" />
  <row Id="16049" PostId="9408" Text="Interestingly this question is the top hit when googling &quot;anova assumes infinite population&quot;." CreationDate="2011-04-12T05:59:26.047" UserId="4099" />
  <row Id="16050" PostId="9458" Text="@Jeromy: agreed. However, I believe for what I need this will do. Many thanks!" CreationDate="2011-04-12T06:07:09.933" UserId="4123" />
  <row Id="16051" PostId="9446" Text="Thank you Jeromy, the wording make sense now. My instructor placed emphasis on only having 4 males and suggested I do a pooled variance. That is one of my concerns. The likert items are also assigned as ordinal. I am really tring to understand all of us this." CreationDate="2011-04-12T06:15:18.983" UserId="4119" />
  <row Id="16052" PostId="9459" Text="I suspect * mean actually here + or . depending if the seasonality is additive or multiplicative" CreationDate="2011-04-12T06:19:21.687" UserId="1709" />
  <row Id="16053" PostId="9460" Text="I would also use seasonality for 1 year cycles (linked to the seasons) but wanted to check with you guys. Do you think that monthly patterns (for instance end of month effects) can be called seasonal patterns?" CreationDate="2011-04-12T06:22:55.193" UserId="1709" />
  <row Id="16054" PostId="9446" Text="With having a binary response (positive or negative) does this mean to do a descriptives and frequencies analysis for a total sum etc.? Will I need to do any other tests that correlations?" CreationDate="2011-04-12T06:25:34.403" UserId="4119" />
  <row Id="16055" PostId="9457" Text="I guess I win the award of the more useless question" CreationDate="2011-04-12T06:56:09.237" UserId="1709" />
  <row Id="16056" PostId="9464" Score="2" Text="@user4127 Here is a [related question](http://stats.stackexchange.com/questions/4756/confidence-interval-for-bernoulli-sampling) that might be helpful." CreationDate="2011-04-12T07:48:54.863" UserId="930" />
  <row Id="16057" PostId="9456" Text="Looks like a good candidate for a Community-Wiki question." CreationDate="2011-04-12T07:52:48.570" UserId="930" />
  <row Id="16059" PostId="9458" Text="just to note that the t test can be quite sensitive to differences in variance, so that would probably be something to check before you make your final decision" CreationDate="2011-04-12T08:21:24.277" UserId="656" />
  <row Id="16060" PostId="9344" Text="@Dmitrij Celov: Thanks a lot for the insights. It looks like ordinary ridge regression is the way to go for my application. I've posted a follow-on question about choosing a good value for $\delta$ (http://stats.stackexchange.com/questions/9467/regularized-fit-from-summarized-data-choosing-the-parameter)" CreationDate="2011-04-12T08:22:27.817" UserId="439" />
  <row Id="16061" PostId="9447" Text="Actually, you are right. Since I am not interested in predicting whether an actual day in future might fulfill the conditions I guess I can aggregate the data, however then I will only have 10 observations regarding the number of days in the past (i.e.: 10,14,12,...), which is quite a small sample. I will check if I can get more data, and maybe a simple a Gaussian distribution might enough." CreationDate="2011-04-12T08:27:26.093" UserId="4120" />
  <row Id="16062" PostId="9460" Text="The seasonal adjustment system [X-12-ARIMA](http://www.census.gov/srd/www/x12a/) deals with trading day and moving holiday effects, so you can reasonably call these seasonal effects so long as your reader is clear that is what you are doing.  " CreationDate="2011-04-12T08:34:49.330" UserId="2958" />
  <row Id="16063" PostId="9448" Text="Thanks for the hint, I've not much experience with multivariate Gaussian distributions, but I will definitely try that approach as well. Right know I am more interested in a quick &amp; dirty solution, which does not have to be very sophisticated and zbicyclist's comment reminded me that I can ignore the seasonality entirely." CreationDate="2011-04-12T08:38:11.093" UserId="4120" />
  <row Id="16064" PostId="9315" Text="Do you want to find which one single topic this document was generated from? Do you want to find a mixture of topics for the document?- I want to answer both of these questions actually...but my corpus is really big, so I cannot afford to re-train all of my model every time a new document is examined" CreationDate="2011-04-12T08:43:45.870" UserId="2986" />
  <row Id="16065" PostId="9327" Text="@chl. Thank you. I have rewritten the question and title; regarding your last sentence, I cannot say that it is usual, but I do not have a better idea." CreationDate="2011-04-12T08:46:12.980" UserId="2993" />
  <row Id="16066" PostId="9449" Score="1" Text="@Mitch This related question might be useful, [Statistical test for Positive and Negative Predictive Value](http://stats.stackexchange.com/questions/6309/statistical-test-for-positive-and-negative-predictive-value)." CreationDate="2011-04-12T10:15:15.913" UserId="930" />
  <row Id="16067" PostId="9458" Text="@Mac /cc @richiemorrisroe In my view, apart from the homoscedasticity assumption (which is somewhat circumvented with Welch's t test), the problem is mostly with asymmetric response distributions (ceiling or floor effect), which often arise when using Likert items." CreationDate="2011-04-12T10:47:43.797" UserId="930" />
  <row Id="16068" PostId="9468" Score="1" Text="@deltanovember, did you try standard volatility models? Since you take arithmetic means and do not estimate anything, the end result is again time series, so I do not see why standard models cannot be applied. Whether they will be appropriate is another matter, but that applies for any time series data." CreationDate="2011-04-12T10:49:43.757" UserId="2116" />
  <row Id="16069" PostId="9305" Text="This result cannot hold as stated. If it did, one could replace $Y$ by a smaller open subset $Z$ with $y_0$ on its boundary, apply the desired result to $Z$ and conclude that $c(n_0,y_0)=0$ for *every* $y_0$ in $Y$." CreationDate="2011-04-12T11:06:07.467" UserId="2592" />
  <row Id="16070" PostId="9032" Text="@mpiktas On the contrary, at the very next line after they define the set function, the authors mention that *it has to be extended to a probability measure* on the generated sigma-algebra." CreationDate="2011-04-12T11:16:53.343" UserId="2592" />
  <row Id="16071" PostId="9032" Text="@cardinal See comment above." CreationDate="2011-04-12T11:17:13.030" UserId="2592" />
  <row Id="16072" PostId="9327" Score="1" Text="@J. Winchester: It might be a crazy idea, but why not treat `abundace` as the dependent variable and add `species_identity` as a factor into the model? Then, when you have estimated the magnitude of `species_identity` you will be able to compare the abundance of both species in relative terms. This way you won't need the model comparison stage and things will be much more simple. I also don't see the purpose of `visit_day` as an estimable parameter; the species abundance will clearly depend on it as an exposure parameter, but then how one will interpret the `dryness * visit_day` interaction?" CreationDate="2011-04-12T11:34:46.440" UserId="3467" />
  <row Id="16073" PostId="9032" Text="@Didier Piau, @cardinal the book tells that the $Q$ is defined on $\cup \mathcal{F}_t$ which is algebra, and it has to be extended to $\mathcal{F}_{\infty}$, which presumably is $\sigma(\cup\mathcal{F}_t)$. I will fix the text." CreationDate="2011-04-12T11:45:03.600" UserId="2116" />
  <row Id="16074" PostId="9032" Text="@Didier Piau, @cardinal, I've found another typo, hope now everything is ok." CreationDate="2011-04-12T11:48:05.500" UserId="2116" />
  <row Id="16075" PostId="9032" Text="@Didier, thanks for joining the conversation. I didn't have a copy of the text handy, so could not check it myself." CreationDate="2011-04-12T11:55:34.183" UserId="2970" />
  <row Id="16076" PostId="9461" Text="As far as the 'causality' part goes, this sounds like testing for [mediation](http://en.wikipedia.org/wiki/Mediation_%28statistics%29), but as the lead of that Wikipedia article says, the possibility of / methods for testing for mediation using empirical data have been challenged. I haven't made a thorough study of this, but i'd number myself among the skeptics." CreationDate="2011-04-12T12:18:05.257" UserId="449" />
  <row Id="16077" PostId="9470" Text="@Lorenzo, please add link to the article" CreationDate="2011-04-12T12:24:17.343" UserId="2116" />
  <row Id="16078" PostId="9470" Score="2" Text="Google Scholar finds 229 papers authored in 2008 by a Blundell. Could you give us a bit more of a clue to which one you are referring, or maybe a URI? " CreationDate="2011-04-12T12:27:05.483" UserId="449" />
  <row Id="16079" PostId="7414" Text="It looks good. i would have loved it if it would accept non numericals. It seems one has to transform their data to a numerical list first. " CreationDate="2011-04-12T12:33:24.820" UserId="201" />
  <row Id="16080" PostId="6311" Text="@chl I had a look to this question again and I have seen that p-value doesn't refer neither PPV nor NPV, it refers to the entire row. I think the test they had should be associated to the whole contingency-table." CreationDate="2011-04-12T13:33:34.590" UserId="2719" />
  <row Id="16081" PostId="9456" Text="@chl, that makes sense. Go ahed if you have the privileges. " CreationDate="2011-04-12T14:01:02.640" UserId="4126" />
  <row Id="16082" PostId="9468" Score="1" Text="@deltanovember: Is your model actually a scaled Brownian motion with drift? If so, averaging the price process over $n$ observations to get a second series yields another Gaussian process and is determined by its mean and covariance, which you can readily calculate via the mean and covariance of the underlying price process. If your model is some (nonlinear) *function* of a Brownian motion, then things get more complicated. " CreationDate="2011-04-12T14:03:20.323" UserId="2970" />
  <row Id="16083" PostId="9364" Text="The conceptual issue I had is that if you are comparing enough models, you are effectively fitting on the validation data when you &quot;decide on a winner&quot; using the validation data.  Hence still may be a point in partitioning the validation data." CreationDate="2011-04-12T14:08:00.130" UserId="3567" />
  <row Id="16084" PostId="9473" Score="2" Text="In my experience *leave one out* cross validation almost always results in too little regularization. $k$-fold cross validation nearly always works better. " CreationDate="2011-04-12T14:08:37.307" UserId="2970" />
  <row Id="16085" PostId="9279" Text="I meant that you have scored subsets of the set of all elements. Some form of binary regression will assign scores to the individual elements. The form of regression depends on how you model the score of a set depending on its elements. You cannot build a unique hierarchy of sets, since you stated some overlapping sets are not in a hierarchical relationship. You could have an inclusion graph, but that's not a partition and has little to do with clustering." CreationDate="2011-04-12T14:29:11.680" UserId="2456" />
  <row Id="16086" PostId="9144" Text="The 1 is for prediction intervals. Leave it off for confidence intervals. Var($\hat{y}$) relates to confidence intervals." CreationDate="2011-04-12T14:44:03.293" UserId="159" />
  <row Id="16087" PostId="9384" Text="It probably comes from people using Ward simply because it is well known. I would say that Ward brings no gain compared to an average linkage in this settings. However, it is more computationally expensive (you need to compute the first two moments for each merge, or to precompute them). Thus, from a pragmatic standpoint, I would simply go for average linkage." CreationDate="2011-04-12T14:45:11.123" UserId="1265" />
  <row Id="16088" PostId="9475" Score="1" Text="In your application, what is a *trend*?" CreationDate="2011-04-12T14:47:45.573" UserId="2970" />
  <row Id="16090" PostId="9475" Score="2" Text="There are at least three other questions that cover this or very similar topics, http://stats.stackexchange.com/q/3331/1036 , http://stats.stackexchange.com/q/2777/1036 , http://stats.stackexchange.com/q/3238/1036 . " CreationDate="2011-04-12T15:23:58.980" UserId="1036" />
  <row Id="16091" PostId="9466" Text="Thanks for the explanation.  So then it seems that I have interpreted their results incorrectly as I thought they had run their experiment multiple times over the entire dataset.  I realize now that the confidence interval is based on a single experiment over the dataset.  " CreationDate="2011-04-12T15:28:33.410" UserId="4127" />
  <row Id="16092" PostId="9444" Score="1" Text="whubers comments to the original question already stated what it would look like if the variances were the same and the variables were uncorrelated. Perhaps giving an example of where this is not the case would be more enlightening." CreationDate="2011-04-12T15:51:25.737" UserId="1036" />
  <row Id="16093" PostId="9473" Text="(+1) @cardinal, nice addition, to be frank, I have little experience with cross validation methods. The usual simple things I used in practice are jack-knives (dropping up to $k$ subsequent observations) and out-of-samples for time series data. Though $k$-fold could be implemented for some time series models too, have to try it to build my own experience first. " CreationDate="2011-04-12T16:23:45.817" UserId="2645" />
  <row Id="16094" PostId="9451" Text="Argh...yes...I mixed up sensitivity and specificity." CreationDate="2011-04-12T16:31:55.157" UserId="3186" />
  <row Id="16095" PostId="9473" Text="There are some nice block bootstrap methods for stationary time series. Perhaps they could or have been modified for the purposes of selecting a regularization parameter. " CreationDate="2011-04-12T16:32:27.827" UserId="2970" />
  <row Id="16096" PostId="9478" Score="2" Text="This is essentially the same question as http://stats.stackexchange.com/questions/8285/normalizing-data-between-0-and-1 except your lower bound is -1, not 0.  So: take any of the answers there, multiply by 2, and subtract 1.  But first read the cautionary remarks!" CreationDate="2011-04-12T17:30:25.377" UserId="919" />
  <row Id="16097" PostId="9478" Text="This is essentially the same question as http://stats.stackexchange.com/questions/1112/how-to-represent-an-unbounded-variable-as-number-between-0-and-1/1113#1113 except your lower bound is -1, not 0. So Please have a look at my answer there. Note, I am referring to another question then @whuber" CreationDate="2011-04-12T17:33:03.423" UserId="442" />
  <row Id="16098" PostId="9478" Text="@Henrik Thanks for pointing that out.  Because these are substantially the same questions, I have merged them." CreationDate="2011-04-12T17:35:46.013" UserId="919" />
  <row Id="16099" PostId="9466" Text="Yes it is. I don't know if all datasets are the same cardinality ($N=50000$). By the way at least you can computer your own confidence interval for your data." CreationDate="2011-04-12T17:56:35.277" UserId="2719" />
  <row Id="16101" PostId="9477" Score="2" Text="@Adam If you get an answer here that is acceptable, I would suggest to remove your original post on SO." CreationDate="2011-04-12T18:25:30.297" UserId="930" />
  <row Id="16102" PostId="6311" Text="@Simone So, if I understand you correctly, you suggest that the authors provide PPV and NPV values but gave the p-value corresponding to a global association test of the 2x2 table? Is it related to this recent question, http://stats.stackexchange.com/questions/9464/can-someone-give-me-details-about-a-particular-confidence-interval-formula?" CreationDate="2011-04-12T18:37:37.720" UserId="930" />
  <row Id="16103" PostId="9032" Text="@cardinal Hi. I guessed so, no problem with that." CreationDate="2011-04-12T19:15:40.480" UserId="2592" />
  <row Id="16104" PostId="9470" Text="I would guess it was [Blundell, R., Pistaferri, L. and Preston, I. (2008). &quot;Consumption inequality and partial insurance&quot;, American Economic Review, 98(5), pp. 1887-1921.](http://gatton.uky.edu/Faculty/Ziliak/Blundell_Pistaferri_Preston_aer2008.pdf), though there are more possibilities [here](http://www.ucl.ac.uk/~uctp39a/pub.html)" CreationDate="2011-04-12T19:45:28.230" UserId="2958" />
  <row Id="16105" PostId="9469" Text="@opty Could you expand on how would bootstrap provide a more interesting approach for factor analysis?" CreationDate="2011-04-12T20:04:33.400" UserId="930" />
  <row Id="16106" PostId="9481" Text="@Gael Most of your responses are made CW (here, but also on metaoptimize.com/qa)... Any reason why?" CreationDate="2011-04-12T20:07:00.183" UserId="930" />
  <row Id="16107" PostId="6311" Text="@chl Yes, it would be related to that question if p-value were associated either to PPV or to NPV. And in that case you gave the solution. The test corresponds to the whole 2x2 table, I won't ever know what kind of test it is!" CreationDate="2011-04-12T20:11:34.430" UserId="2719" />
  <row Id="16108" PostId="9481" Score="1" Text="Because I think that it is good if people can fix typos, incorrect wordings... I like the idea of continuous improvements of answers. Is this against best practice?" CreationDate="2011-04-12T20:17:02.237" UserId="1265" />
  <row Id="16109" PostId="9481" Text="@Gael I fixed two or three typos in passing. Nothing to do with CW per se, unless you're thinking of the lower rep required to edit your post! Users can suggest any edits, though; but making your response CW won't allow you to earn rep from them. I guess you're not after rep, but as your responses were always well put, I was just wondering... Cheers." CreationDate="2011-04-12T20:48:53.253" UserId="930" />
  <row Id="16110" PostId="9481" Text="I wasn't aware of this. I am not terribly hunting rep, but it's always good to get some. On the other hand, I appreciate people fixing typos. I'll think it over :). Thanks for mentioning." CreationDate="2011-04-12T20:54:52.253" UserId="1265" />
  <row Id="16111" PostId="9459" Text="I believe that Trend is what's left over after removing seasonal and cyclical effects and smoothing out irregularity, so it doesn't quite fall into the frequency hierarchy of Cycles &lt; Seasons&lt; Irregularity, which makes it a bit tricky in my mind. I've also seen a combined TC (Trend-Cycle) factor used, where TC &lt; S &lt; I. (I.e. I is highest frequency, S is lower frequency, and TC is lowest frequency.)" CreationDate="2011-04-12T21:11:40.897" UserId="1764" />
  <row Id="16112" PostId="9364" Text="I think that the training-validation layer and the validation-testing layer serve different purposes in some sense, and that you do eventually have to compare models on a common validation set if you're going to declare a winner. So I'm not sure that additional layers help. (Though my knowledge isn't deep enough to really know.) The closest thing I can think of to your suggestion is how the Netflix competition was run. I believe they used partial test sets to keep teams from climbing the test set gradient, but I think it's different." CreationDate="2011-04-12T21:36:34.947" UserId="1764" />
  <row Id="16113" PostId="9482" Score="2" Text="Quick and dirty: `=MAX(($B6:$B18=1979)*IF(ISTEXT(E6:E18), 0, E6:E18))`" CreationDate="2011-04-12T22:10:05.277" UserId="919" />
  <row Id="16114" PostId="9306" Text="I think this question could be improved by providing more information. That will likely attract more interest as well. Here are some suggestions for additions: (a) consider providing a sample plot of the spectrogram, (b) explain what exactly you mean by &quot;put some numbers on it&quot;, (c) give details on the time windowing you are doing for the FFT analysis, (d) provide the number of subjects, sampling rate, and what the stimulus is. Is the overall power the quantity of interest, the power in some frequency band, or something else?" CreationDate="2011-04-12T23:21:35.947" UserId="2970" />
  <row Id="16115" PostId="9484" Text="Thank you for your answer. Unfortunately, I don't have the allele frequencies or counts, the authors did not show these data, they just put the SNP ID, OR and Confidence intervals (95%). I just extracted the SE value from each study, but I don't now to combine them (SE or CI)!!!! help!!" CreationDate="2011-04-12T23:52:32.287" UserId="4137" />
  <row Id="16116" PostId="9473" Text="You may find the following paper useful: Golub, G. H.; Heath, M. &amp; Wahba, G. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics, 1979, 21, 215-223. The criterion introduced by Golub et al. does not require any re-sampling." CreationDate="2011-04-13T00:09:48.357" UserId="530" />
  <row Id="16117" PostId="9484" Text="Oh, you're right - one more degree of freedom is needed here. Usually the authors give the allele freq. (sometimes buried in supp. info.). If not, you may find it from an external source like hapmap (assuming the GWAS was done on a similar population). &#xA;Another idea: the confidence interval itself can tell you the allele freq. All else being equal (sample size and OR), SNPs with low allele freq. have fewer carriers in both groups, hence a wider confidence interval. You can try different allele freqs., compute confidence interval for each one, and get the allele freq. matching what was reported" CreationDate="2011-04-13T00:21:17.793" UserId="3036" />
  <row Id="16118" PostId="9484" Text="I will try to do that, but in the meantime I'm curious about how PLINK can compute the pooled OR only  with this parameters:SNP idenitifier, OR   Odds ratio (or BETA, etc) and SE   Standard error of OR (or user-defined weight field). You can realize that PLINK did't ask for allele frequencies... so there is a way to perform this..." CreationDate="2011-04-13T00:33:52.827" UserId="4137" />
  <row Id="16119" PostId="9420" Text="@Ga I pointed out in a comment to your deleted post that a matrix notation for bivariate probabilities is (and long has been) in use.  Most statistical software will produce such tables, e.g.  It makes perfect sense and is quite clear: it's difficult to imagine someone misunderstanding what is meant." CreationDate="2011-04-13T01:32:37.623" UserId="919" />
  <row Id="16120" PostId="8703" Text="@Thomas Can you post the picture now?" CreationDate="2011-04-13T02:22:50.510" UserId="3911" />
  <row Id="16121" PostId="9469" Text="I would be interested in learning more about *why* the Mann-Whitney test would be favored over a t-test here." CreationDate="2011-04-13T02:35:13.717" UserId="919" />
  <row Id="16122" PostId="9487" Text="arghh - my screenshot blocks half of the actual table, but I hope the point gets across. Pivot tables can do quite a bit in terms of aggregation, so can definitely be worth the time investment if you find yourself working with Excel on a regular basis." CreationDate="2011-04-13T02:46:12.113" UserId="696" />
  <row Id="16123" PostId="9444" Text="Can you provide such an example?" CreationDate="2011-04-13T04:50:39.027" UserId="776" />
  <row Id="16124" PostId="9422" Score="1" Text="Please stop changing the title, you are obfuscating my purpose. Perhaps that suggests that this is a poorly defined problem. I'll update it again." CreationDate="2011-04-13T04:52:16.827" UserId="776" />
  <row Id="16125" PostId="9462" Text="Thanks Jeremy, I'll keep that in mind. Although, I don't think this is exactly what I'm looking for (the question about error was more of an aside to my primary question)" CreationDate="2011-04-13T05:09:43.987" UserId="776" />
  <row Id="16126" PostId="9490" Score="1" Text="@MYaseen208, please do not [cross-post](http://math.stackexchange.com/questions/32702/singular-value-decomposition-svd-of-a-three-dimensional-array)." CreationDate="2011-04-13T07:09:32.520" UserId="2116" />
  <row Id="16127" PostId="9448" Text="@mbq ;o) That is indeed the baseline meterologists need to beat!" CreationDate="2011-04-13T07:26:11.287" UserId="887" />
  <row Id="16128" PostId="9462" Text="@Brandon. Okay. It seems that your sample size for given questions is being reduced because of (a) random sampling within a random sample and (b) a certain question is only applicable to a subset of your sample. However, I don't see how this would change the process of deriving standard errors. You have a final sample size for that question, and that's that. Perhaps, I'm not understanding what you mean by a &quot;more robust sample of users&quot;." CreationDate="2011-04-13T07:39:11.717" UserId="183" />
  <row Id="16129" PostId="9462" Text="Essentially, I'd like a statistical method for sampling the population of users of Company X, to be able to say which other companies they are planning on doing more business with in the future. aka, Who's eating who's lunch." CreationDate="2011-04-13T08:33:34.560" UserId="776" />
  <row Id="16130" PostId="9475" Text="a trend in my application are  a group of  curves in a graph  of TS  with similar shape, i.e. a group of curve that show a seasonal trends, and another that show a long tern trend,......   " CreationDate="2011-04-13T09:42:28.273" UserId="4133" />
  <row Id="16131" PostId="9475" Text="yes this topic  [link]http://stats.stackexchange.com/questions/3331/is-it-possible-to-do-time-series-clustering-based-on-curve-shape_italic_ **bold** `code`   is very similar to what I want to do " CreationDate="2011-04-13T09:45:47.910" UserId="4133" />
  <row Id="16132" PostId="9462" Score="1" Text="@Brandon I still don't quite see how it is a special case. In survey research, we define a population, and attempt to sample from that population. In some cases the population is adults in a given country, in other cases it is people who use company X, in other cases it is people who use company Y, and in other cases it is people who use Company X, but are considering switching. In all these cases there are pragmatic challenges of actually getting a sample that is representative of the target population, and that is sufficiently large to draw relevant inferences." CreationDate="2011-04-13T10:20:22.027" UserId="183" />
  <row Id="16133" PostId="9421" Text="Downvoted.  How is the truncation issue relevant?" CreationDate="2011-04-13T10:30:54.367" UserId="3567" />
  <row Id="16134" PostId="9500" Text="PCA will give you different results depending on whether you use the covariance matrix or the correlation matrix." CreationDate="2011-04-13T10:59:04.160" UserId="3567" />
  <row Id="16135" PostId="9500" Text="I used the covariance matrix everywhere" CreationDate="2011-04-13T11:10:50.973" UserId="339" />
  <row Id="16136" PostId="9494" Text="Thanks for modifications Jeromy." CreationDate="2011-04-13T12:24:20.027" UserId="4145" />
  <row Id="16139" PostId="9501" Text="@Rosh, by $lognormal$ you mean probability density of log-normal distribution?" CreationDate="2011-04-13T12:45:12.607" UserId="2116" />
  <row Id="16140" PostId="9444" Text="@Brandon, all you need to do is generate the x and y values that are either correlated or have different variances. The different variances could be done right in the code as is. You could generate values from a specified covariance matrix using mvrnorm from the MASS package. Also I am not sure what the function &quot;dentist&quot; is in the above code, should it perhaps be &quot;density&quot;." CreationDate="2011-04-13T12:47:01.463" UserId="1036" />
  <row Id="16141" PostId="9501" Score="1" Text="This is expressible as a constant times a difference of two normal cdfs. Normal cdfs are efficiently computed using W. Cody's rational Chebyshev approximation. You shouldn't need and, almost undoubtedly shouldn't *prefer*, numerical-integration alternatives to this. If you need more details, I can post them. " CreationDate="2011-04-13T12:50:14.563" UserId="2970" />
  <row Id="16142" PostId="9500" Score="6" Text="The differences seem relatively small, though probably too big to be simple numerical issues. Could it be the difference between normalizing by $n$ or $n-1$, for example, when calculating an estimate of the covariance prior to computing the SVD or eigenvalues decomposition?" CreationDate="2011-04-13T12:58:34.340" UserId="2970" />
  <row Id="16143" PostId="9444" Text="That being said it is probably just as enlightening to work through the math to see why this is the case (and how manipulating the variance/covariances will change the distribution). It is not entirely clear for me why this is the case just by looking at the characteristic function mentioned by whuber. It looks like though a simple understanding of rules for adding, subtracting, and multiplying random variables will get you along way to understanding why that is." CreationDate="2011-04-13T13:03:57.360" UserId="1036" />
  <row Id="16144" PostId="9501" Text="@mpiktas, Yes, lognormal is the probability density function and lognormalCDF is the cumulative density function." CreationDate="2011-04-13T13:23:04.720" UserId="4146" />
  <row Id="16145" PostId="9501" Text="@cardinal, Do you mean the lognormal pdf can be expressed as a constant times the difference of two normal cdfs? I most certainly would be interested in any details you can provide. I need a solution to either integral of x*lognormal(x) or to the integral of lognormalCDF(x) (integration by parts rule). Can you explain how your suggestion would help solve this?" CreationDate="2011-04-13T13:25:28.743" UserId="4146" />
  <row Id="16146" PostId="9292" Text="Interesting, I'll certainly look into the list. Thanks!" CreationDate="2011-04-13T13:38:21.713" UserId="3280" />
  <row Id="16147" PostId="9512" Text="@Wazir, simply reestimating `arima_s` with updated `series` is not ok?" CreationDate="2011-04-13T14:03:31.213" UserId="2116" />
  <row Id="16148" PostId="9501" Score="3" Text="@Rosh $x$ has a lognormal distribution means that $\log(x)$ is normally distributed.  Thus, substitute $x = \exp(y)$ in your *original* integral.  The integrand is an exponential whose argument is a quadratic function of $y$.  Completing the square turns it into a multiple of a normal PDF, so your answer is written in terms of the normal CDF and the exponentials of the original endpoints.  There are many good approximations to the normal CDF (a multiple of the error function)." CreationDate="2011-04-13T14:12:48.707" UserId="919" />
  <row Id="16149" PostId="9500" Score="5" Text="@cardinal Nice guess!  Notice that the two different sequences of eigenvalues have *identical successive ratios.*  Thus, one set is a constant multiple of the other.  The multiple is 1.025 = 41/40 (*exactly*).  It's unclear to me where this comes from.  Maybe the dataset has 41 elements and the OP is revealing only the first 10?" CreationDate="2011-04-13T14:24:03.580" UserId="919" />
  <row Id="16150" PostId="6537" Text="Generally I would say a reason to not just used mixed models is that others are not familiar with the method and its outputs." CreationDate="2011-04-13T14:24:46.983" UserId="196" />
  <row Id="16153" PostId="9490" Score="1" Text="Could you explain what the analog of matrix multiplication might be for a &quot;three-dimensional array&quot; (presumably, a rank-three array requiring three independent indexes)?" CreationDate="2011-04-13T14:28:21.563" UserId="919" />
  <row Id="16154" PostId="6537" Text="P.S. @BR, the link you provide is talking about unbalanced mixed designs on the between subjects factor, I think that is a slightly different answer than the question being asked." CreationDate="2011-04-13T14:29:44.623" UserId="196" />
  <row Id="16155" PostId="9479" Text="thanks for you answer. I read some stuff about but still a bit confused about &quot;folding in&quot;. You are saying that I should keep the topic assignments of the old documents the same, this means that the word-topics assignments should be re-calculated? Is it possible for you to give me a more detailed steps of what should be done? or maybe referring me to a paper or a link that can actually help me clarify this &quot;folding in&quot; process. My first option is to do &quot;folding in&quot;. If unsuccessful I will go for the second method you proposed(not sure how well it works comparinfg to folding in).thanks." CreationDate="2011-04-13T14:34:14.683" UserId="2986" />
  <row Id="16156" PostId="9501" Text="@whuber I think I get what you are saying. @cardinal I guess you were saying the same thing. Problem is, its been ages since I did any kind of integration. Will work on this angle tomorrow and update here." CreationDate="2011-04-13T14:36:44.793" UserId="4146" />
  <row Id="16157" PostId="9512" Text="@mpiktas, It works, but change the parameter little bit.Actually i want to simulate with estimated parameter and initialize with this new value into the future" CreationDate="2011-04-13T14:44:59.650" UserId="3084" />
  <row Id="16158" PostId="9517" Text="@Jose: would it mean that if I write an answer to this question, will you then use it to sack employees who take longer than they &quot;should&quot;, according to my answer?" CreationDate="2011-04-13T13:53:38.370" />
  <row Id="16159" PostId="9517" Text="@Jose: Also, are all accounts the same? Or are some more difficult to &quot;resolve&quot; than others?" CreationDate="2011-04-13T13:55:30.273" />
  <row Id="16161" PostId="9500" Score="6" Text="@cardinal Indeed: Help page for `princomp`: &quot;Note that the default calculation uses divisor N for the covariance matrix.&quot; Help page for `prcomp`: &quot;Unlike princomp, variances are computed with the usual divisor N-1.&quot;" CreationDate="2011-04-13T15:04:27.163" UserId="1909" />
  <row Id="16162" PostId="9517" Score="1" Text="I think you should look at some histograms by employee to get a feel for the distribution. Since you report that the median is greater than the mean, it almost sounds like you might be *including* the &quot;unresolved&quot; cases for each employee in your estimate. There is also some (informative) censoring in your data." CreationDate="2011-04-13T15:06:59.833" UserId="2970" />
  <row Id="16163" PostId="9500" Text="@caracal, that appears to resolve the discrepancy. I am (and have been) commenting away from my computer, so couldn't check that. Thanks for doing the extra legwork to confirm our suspicions.  " CreationDate="2011-04-13T15:10:56.400" UserId="2970" />
  <row Id="16164" PostId="9500" Score="2" Text="@caracal, you should copy your comment into an answer (and maybe make it CW) so that it can be accepted and the question can be marked as resolved. " CreationDate="2011-04-13T15:14:42.230" UserId="2970" />
  <row Id="16165" PostId="9517" Text="The employee-specific data do not seem to address the question, due to passing unresolved accounts along at the end of a month: you have information about *how long each employee has an account* but this does not tell you directly *how long it takes to resolve each account.*  Therefore, *no* kind of average of the employee-specific data will be an appropriate solution." CreationDate="2011-04-13T15:16:32.957" UserId="919" />
  <row Id="16166" PostId="9518" Text="When the median exceeds the mean, the data are *negatively* skewed, not positively skewed as suggested in this reply.  This skewness almost surely is due to the one-month cutoff for unresolved accounts." CreationDate="2011-04-13T15:19:03.297" UserId="919" />
  <row Id="16167" PostId="9522" Score="1" Text="You might consider replacing &quot;guessed&quot; with &quot;already confirmed&quot; (see comment stream above.) You might also consider editing your answer to make it CW. Cheers. " CreationDate="2011-04-13T15:35:38.023" UserId="2970" />
  <row Id="16168" PostId="9516" Text="Looking at the chart, I think there may be some confusion between Fahrenheit and Celsius.  " CreationDate="2011-04-13T15:36:57.010" UserId="2958" />
  <row Id="16169" PostId="9522" Text="@cardinal I didn't see those comments.  I only saw those that had been up-voted.  Thanks.  Also, could you explain the rationale behind making the answer CW?  What are the rules / guidelines for that?" CreationDate="2011-04-13T15:38:23.857" UserId="1657" />
  <row Id="16170" PostId="9516" Text="Well spotted - I've been doing something similar for a few months and the data are in degree C. Was force of habit!" CreationDate="2011-04-13T15:40:25.197" UserId="1390" />
  <row Id="16172" PostId="9522" Text="Can anybody guess why the code is not simply `cv &lt;- cov.wt(z, method=&quot;ML&quot;)` making the 2 follwing lines unnecessary?" CreationDate="2011-04-13T15:42:51.997" UserId="1909" />
  <row Id="16173" PostId="9516" Text="@Henry - Thanks, now fixed" CreationDate="2011-04-13T15:43:35.117" UserId="1390" />
  <row Id="16174" PostId="9521" Text="+1 This seems appropriate *in conjunction with using the data correctly.*  Specifically, each *account* should be tracked, not accounts per employee (which is a biased approach).  The times to resolve accounts will still be censored: there will always be some open accounts (or accounts closed solely because they were deemed unresolvable).  Use censored methods to estimate the mean resolution time, multiply by the number of accounts to get total time required, and divide that by the number of employees to obtain mean &quot;length of time to resolve an account per employee.&quot;" CreationDate="2011-04-13T15:44:07.753" UserId="919" />
  <row Id="16175" PostId="9521" Text="Note that the K-M estimator is *not* appropriate for the account times per employee, because the most intractable accounts are shared among several employees.  This induces substantial *dependence* among the data, contrary to a fundamental assumption of independent data in the K-M approach." CreationDate="2011-04-13T15:46:06.440" UserId="919" />
  <row Id="16176" PostId="9487" Text="Thank you very much for the help. Saved lots of time" CreationDate="2011-04-13T15:50:48.060" UserId="4136" />
  <row Id="16177" PostId="9524" Text="+1 Cool!  (BTW, the links to the MaplePrimes site are returning a 404.)" CreationDate="2011-04-13T15:56:28.633" UserId="919" />
  <row Id="16178" PostId="9514" Text="OK, thanks it was a first try. I'm not familiar with ETS models. I thought the package might self-tune some of the parameters from the training data (thinking in NN models and such). I'll dig into it. Thanks again." CreationDate="2011-04-13T16:30:58.543" UserId="4134" />
  <row Id="16180" PostId="9522" Score="1" Text="@Joshua: My suggestion regarding making the answer CW was due to the fact that the answer appeared via a stream of comments and was generated by a &quot;community&quot; discussion. Since it was resolved in the comments, my thoughts are that it makes the most sense to reformulate it as an answer, marked as CW to indicate this collaboration, and this allows for the answer to be accepted and the question to be marked as resolved. (Otherwise, it will automatically get bumped back up by the software after a certain amount of time.)" CreationDate="2011-04-13T17:01:35.783" UserId="2970" />
  <row Id="16181" PostId="9513" Score="2" Text="Computer scientists call these &quot;Poisson trials&quot; to distinguish them from Bernoulli trials. In addition to Central Limit Theorem approximations, there are also good tail bounds available. [Here's](http://en.wikipedia.org/wiki/Chernoff_bound#Theorem_for_multiplicative_form_of_Chernoff_bound_.28relative_error.29) one. Google searches on &quot;Chernoff bounds for Poisson trials&quot; will turn up the results you might find in a typical CS treatment." CreationDate="2011-04-13T17:10:42.613" UserId="2970" />
  <row Id="16182" PostId="9521" Text="To add to @whuber's comments, in addition, the K-M estimator will (likely) be biased if used for account times per employee due to the informative nature of the censoring." CreationDate="2011-04-13T17:14:42.797" UserId="2970" />
  <row Id="16183" PostId="9501" Score="1" Text="Yes, @whuber and I were describing the same thing. You should get something like $e^{\mu + \frac{1}{2} \sigma^2} (\Phi(\beta) - \Phi(\alpha))$ where $\beta = (\log(b) - (\mu + \sigma^2))/\sigma$ and $\alpha = (\log(a) - (\mu+\sigma^2))/\sigma$ and $\Phi(\cdot)$ denotes the normal cdf. Note that, depending on the values of $a$, $b$, $\mu$ and $\sigma$, there are ways to rewrite this expression to be more numerically stable." CreationDate="2011-04-13T17:19:24.950" UserId="2970" />
  <row Id="16184" PostId="9513" Text="@Cardinal That nomenclature is interesting.  It would be valid for very small $p_i$, but otherwise seems misleading, because the distribution otherwise is not well approximated by Poisson distributions.  (There's another discussion on CV about this question, where &quot;16&quot; is replaced by 10,000 and we do examine tail probabilities, but I haven't been able to find it again.)" CreationDate="2011-04-13T17:23:58.290" UserId="919" />
  <row Id="16185" PostId="9513" Text="@whuber, yes, I agree on the name. I found it a bit odd when I first encountered it. I've given it here more as a useful term for searching. It appears computer scientists consider these probabilities often in dealing with certain algorithms. I'd be interested in reading that other question if you happen to find it. Is it [this one](http://stats.stackexchange.com/questions/5347/how-can-i-efficiently-model-the-sum-of-bernoulli-random-variables) perhaps?" CreationDate="2011-04-13T17:26:56.147" UserId="2970" />
  <row Id="16186" PostId="9522" Text="Thanks! Any idea how to do the same when using the PCA function?" CreationDate="2011-04-13T18:37:25.677" UserId="339" />
  <row Id="16187" PostId="9524" Text="@whuber: there was a temporary outage - I believe the links should work again." CreationDate="2011-04-13T18:41:48.747" UserId="2898" />
  <row Id="16188" PostId="9522" Text="@gd047: Again, I'm just looking at the source (you can do the same).  It doesn't look like `PCA` provides any option for that.  It uses the `svd.triplet` function, which doesn't contain an argument like `covmat`.  That said, you can scale the eigenvalues yourself." CreationDate="2011-04-13T19:15:57.663" UserId="1657" />
  <row Id="16189" PostId="934" Text="&quot;If statistics is all about maximizing likelihood, then machine learning is all about minimizing loss&quot; I disagree with your premise - strongly and in its entirety. Maybe it was true-ish of statistics in 1920, but it certainly is not today." CreationDate="2011-04-13T19:25:48.393" UserId="26" />
  <row Id="16190" PostId="9524" Text="Thanks.  The site's a little slow but it's working now.  I like the image examples." CreationDate="2011-04-13T19:49:50.447" UserId="919" />
  <row Id="16192" PostId="9492" Text="@Conjugate (+1) I must admit I prefer this approach :-)" CreationDate="2011-04-13T20:39:44.300" UserId="930" />
  <row Id="16193" PostId="9495" Text="Thank you very much for your response. The standard erros are based on the natural Log of the OR Ln(ORi). First I compute the weights for SE1(0.0937)= 10.67 and  for SE2(0.1029)= 9.71. So the computed SE under FEM is = 0.2215 . The pooled OR for these SNPs is = 0.7645, so the 95% confidence intervals are = (0.515-1.228). Am I correct??, if I do, I am worried because when I compare all results in a Forest plots, the combined intervals are too large compared with the originals in each study = study 1 = 95%CI(0.63-0.91) OR = 0.75, study 2 95%CI(0.69-1.04) OR = 0.85. It's everything ok?. Thanks" CreationDate="2011-04-13T21:59:48.260" UserId="4137" />
  <row Id="16194" PostId="9533" Text="Did you try to do PCA, project into a lower-dimensional space and estimate the covariance there?" CreationDate="2011-04-13T22:14:31.087" UserId="3369" />
  <row Id="16195" PostId="9526" Score="2" Text="Apart from the point that the standard deviation is the root mean square difference rather than the mean difference, it may depend on what you are using this for.  Another point to consider is in cases where one currency moves dramatically against the other so the scales are substantially different at different times; if so, it may be better to work with relative rather than absolute errors.  " CreationDate="2011-04-13T22:16:52.120" UserId="2958" />
  <row Id="16197" PostId="9533" Text="It is something that I have in mind, or regularized covariance estimation. I did play a bit with it, but it did not give me terribly good results, and made the results harder to interpret." CreationDate="2011-04-13T22:26:24.510" UserId="1265" />
  <row Id="16198" PostId="9495" Text="No, unfortunately not. Please note that my formula for $w$ was wrong, it is $1/(se^2)$ not $1/se$. As you can see, the 'pooled' 95% CI is [0.693; 0.9092]. I also wonder why your pooled OR is different (0.7645 vs 0.7938). Sorry, I have to go but I come back to it tomorrow..." CreationDate="2011-04-13T22:30:54.007" UserId="307" />
  <row Id="16199" PostId="9533" Text="By regularized covariance estimation, do you mean restricting the covariance to be diagonal (in effect, assuming feature independence)?" CreationDate="2011-04-13T22:46:55.413" UserId="3369" />
  <row Id="16200" PostId="9495" Text="Thank you very much!!!, that result is more consistent than mine. The pooled OR I give to you was in the PLINK output... now I'm very concerned about all my meta-analysis results... I'd better use R." CreationDate="2011-04-13T22:48:29.500" UserId="4137" />
  <row Id="16201" PostId="9513" Score="1" Text="@cardinal is right that we &quot;CS folks&quot; call them Poisson trials. in fact for this case, a standard Chernoff-Hoeffding bound will give you exactly the bound the OP is asking for. " CreationDate="2011-04-13T22:50:11.330" UserId="139" />
  <row Id="16202" PostId="9513" Text="@cardinal Yes, that's the one, thanks.  (I tagged it as `Binomial` to help with future searches.)  The replies include a reference to Le Cam's theorem, which is a good bound, as well as an efficient way to compute the exact distribution even when $n$ is quite large, plus various suggestions of other approximations.  @Suresh I don't see where the OP is asking for a bound: this question seems to ask for actual probabilities and the one Cardinal references asks for an &quot;exact probability.&quot;" CreationDate="2011-04-13T22:55:47.900" UserId="919" />
  <row Id="16203" PostId="9489" Text="Wow I can't believe it was that simple. Thanks... I checked this with other models that I knew the likelihood and it is correct. " CreationDate="2011-04-14T03:10:10.493" UserId="4027" />
  <row Id="16204" PostId="778" Text="@Dave - I disagree in part with your initial paragraph because the questions asks about the concept of Bayes theorem. The Frequentist-Bayesian debate *is relevant* to this part of the question. The Kolmogorov axioms do not give Bayes theorem the same conceptual importance as the &quot;probability as extended logic&quot; axioms do." CreationDate="2011-04-14T03:31:46.707" UserId="2392" />
  <row Id="16205" PostId="9533" Text="No, but shrinking it to diagonal (for instance using the Ledoit-Wolf estimator), which indeed boils down to the same ideas, but is a trade off." CreationDate="2011-04-14T04:44:44.857" UserId="1265" />
  <row Id="16206" PostId="9542" Score="4" Text="Your reasoning is good.  Actually, one would expect instability to occur when two or more eigenvalues are nearly coincident, for then although the eigenvalues are determined, the eigenvectors are not, and therefore neither are the loadings.  For numerical reasons, there is also instability in eigenvalues (and eigenvectors) that are very small in size compared to the maximum eigenvalue." CreationDate="2011-04-14T05:34:29.807" UserId="919" />
  <row Id="16207" PostId="9539" Score="2" Text="Why are you fitting a Poisson model using weighted least squares rather than maximum likelihood?" CreationDate="2011-04-14T06:30:38.027" UserId="449" />
  <row Id="16208" PostId="9516" Text="Thanks Gavin, a very nice and understandable answer. I'll try your suggestions. &#xA;Is it a good idea to plot the stl() trend component and make a linear regression?" CreationDate="2011-04-14T07:59:56.000" UserId="4147" />
  <row Id="16209" PostId="9516" Text="@pacomet - no, not really, unless you fit a model that accounts for autocorrelation in the residuals as I did above. You could use GLS for that (`gls()` in package nlme). But as the above shows for Cairo, and the STL suggests for your data, the trend isn't linear. As such, a linear trend would not be appropriate - as it doesn't describe the data properly. You need to try it on your data, but an AM like I show would degrade to a linear trend if that fitted the data best." CreationDate="2011-04-14T08:07:15.150" UserId="1390" />
  <row Id="16210" PostId="9522" Text="@Joshua Ulrich Well, I finally did it, making some modifications to the PCA and svd.triplet functions. Thank you." CreationDate="2011-04-14T08:25:24.340" UserId="339" />
  <row Id="16211" PostId="9549" Text="I usually use some sort of marking scheme, but I guess you *could* just sample them randomly from a normal distribution and see if anyone can tell the difference. It would certainly save time! (Perhaps this isn't what you meant. But I'm afraid I can't tell what you did mean.)" CreationDate="2011-04-14T10:19:54.320" UserId="449" />
  <row Id="16212" PostId="9549" Text="@user4167 I've tried to rephrase your question to convey what I think you might mean. Feel free to edit if I have misconstrued what you are asking." CreationDate="2011-04-14T10:44:22.627" UserId="183" />
  <row Id="16213" PostId="9555" Text="@mpiktas: Thanks for this. Just to clarify, I don't have $X$. I only have $X^TX$ -- see http://stats.stackexchange.com/questions/9341/regularized-fit-from-summarized-data" CreationDate="2011-04-14T10:59:15.293" UserId="439" />
  <row Id="16214" PostId="9553" Score="5" Text="@schenectady I'm not sure the OP is after PCR. PCA is also a good way to summarize multivariate datasets (not necessarily in order to perform data reduction for subsequent use in a modeling framework), that is approximate the VC matrix to a lower-order one while retaining most of the information. The question seems to be: Am I right when interpreting the first few eigenvalues and PCs (as linear combinations of the original variables) even if there were some collinearity effects? Your response doesn't seem to address directly the OP's question." CreationDate="2011-04-14T11:03:38.377" UserId="930" />
  <row Id="16215" PostId="9501" Text="@cardinal I arrived at the same with the help of this link on deriving the expected value of lognormal." CreationDate="2011-04-14T11:04:28.640" UserId="4146" />
  <row Id="16217" PostId="9555" Text="@aix, QR decomposition can still be applied, although Cholesky might be more appropriate." CreationDate="2011-04-14T11:07:12.273" UserId="2116" />
  <row Id="16218" PostId="9555" Score="1" Text="@mpiktas: To address your second point, I am not saying I am going to code up my own SVD/QR/Cholesky/whatever (unless I really have to, that is)." CreationDate="2011-04-14T11:08:09.040" UserId="439" />
  <row Id="16219" PostId="9542" Text="@probabilityislogic, @whuber comment answers your question,  but I would like to note that in case of 2 perfectly correlated variables, the PCA should not have any problems. The covariance matrix would be of rank 1, so there will be only 1 non-zero eigenvalue, hence only 1 PC.  The original variables will be the multiples of the this PC. The only issue may be the numerical stability." CreationDate="2011-04-14T11:13:36.803" UserId="2116" />
  <row Id="16220" PostId="9555" Text="@aix, I know that you are not saying, that is why I was asking :) " CreationDate="2011-04-14T11:16:46.393" UserId="2116" />
  <row Id="16221" PostId="9555" Text="@aix, I've updated the answer with the link to the book, which you might find interesting. Since your problem is pretty specific, I suspect that sooner or later you will be forced to read a book about numerical methods for least-squares problems, so why not start sooner :)" CreationDate="2011-04-14T11:21:26.793" UserId="2116" />
  <row Id="16222" PostId="9555" Text="@mpiktas: You wouldn't believe it, but I've actually ordered this very book this morning!" CreationDate="2011-04-14T11:22:20.443" UserId="439" />
  <row Id="16223" PostId="9539" Text="@mindless.panda, @onestop comment still applies, but maybe your question is how in general GLM models should be interpreted compared to OLS models?" CreationDate="2011-04-14T11:23:44.483" UserId="2116" />
  <row Id="16224" PostId="9555" Text="@mpiktas: By the way, thanks for the pointer." CreationDate="2011-04-14T11:24:34.293" UserId="439" />
  <row Id="16225" PostId="9501" Text="@cardinal Please ignore previous comment. I arrived at something similar with the help of this [link](http://math.kennesaw.edu/~agadidov/Spring06/Special%20topics/Lectures/Lecture%2012.doc) on deriving the expected value of lognormal. I don't understand how you got the values of the limits($\beta$ and $\alpha$) though? There is one conversion from lognormal to normal with mean $\mu$. Here the limits change to $\log(b)$. Then another conversion to normal with mean $\mu + \sigma^2$. Continued in next comment..." CreationDate="2011-04-14T11:24:59.770" UserId="4146" />
  <row Id="16226" PostId="9501" Text="At first, I thought the limits changed to $(\log(b) - (\mu + \sigma^2))/\sigma$ here, but apparently you have changed it to convert to the standard cumulative function $\phi()$, which is a third conversion. Can you explain this? In case it helps - $a, b, \mu and \sigma$ will always be positive." CreationDate="2011-04-14T11:25:34.107" UserId="4146" />
  <row Id="16227" PostId="9555" Text="@aix, &quot;great minds think alike&quot; :) Do not know where I heard or read this (context was humorous), but I think it applies here :D" CreationDate="2011-04-14T11:26:26.337" UserId="2116" />
  <row Id="16228" PostId="9552" Text="Thanks, I forgot about the repeated measures :)" CreationDate="2011-04-14T11:39:39.283" UserId="4168" />
  <row Id="16229" PostId="9553" Score="2" Text="@schenectady - good answer about PCA in general, but what about when PCA *is the final product*?  That is, the goal is to output a single PC.  @Chl is right on the money with his interpretation of the question" CreationDate="2011-04-14T12:01:10.597" UserId="2392" />
  <row Id="16230" PostId="9549" Text="Err, you can't sensibly 'scale to a Normal distribution'.  You can normalize to zero mean unit variance though.  Is that what is meant?" CreationDate="2011-04-14T12:33:05.890" UserId="1739" />
  <row Id="16231" PostId="9558" Text="Great answer, I had already downloaded it, but didn't get through the table of contents, and actually a chapter about hypothesis testing is there for me!" CreationDate="2011-04-14T12:36:48.447" UserId="4169" />
  <row Id="16232" PostId="9495" Text="I included a link to the Lipsey/Wilson book &quot;Practical meta-analysis&quot; (see References). I am a bit worried that PLINK's and my results differ. Do you know what meta-analysis methodology they use? You also should take into account that I have absolutely no idea about &quot;genome-wide association studies&quot;." CreationDate="2011-04-14T12:49:00.150" UserId="307" />
  <row Id="16233" PostId="9526" Text="Thanks Henry.  After doing a bit more research based on your comment I've decided to use the average absolute deviation." CreationDate="2011-04-14T12:57:22.167" UserId="4154" />
  <row Id="16234" PostId="9549" Text="I deeply dislike the idea that the grade of a specific student depends on the grade of any other student. Grades are for showing how well someone understood a topic, not for ordering people." CreationDate="2011-04-14T13:11:26.663" UserId="2860" />
  <row Id="16235" PostId="9549" Score="2" Text="Without a rubric, almost any grading scheme is acceptable, legally. Did you ever see the movie the &quot;Paper Chase&quot;. At the end, John Housemen didn't even read the finals; he assigned grades according to the student names! There is no reason for grades to have a normal distribution, regardless of the CLT. Have you found out how profs of previous year amalgamated grades?" CreationDate="2011-04-14T13:13:04.847" UserId="3805" />
  <row Id="16236" PostId="9549" Text="I assume the problem is: if a student got 5 correct answers on 10 questions, should he get the same grade INDEPENDENTLY of the average number of correct answers in the class? My answer would be yes. It should however depend on the difficulty of the test of course." CreationDate="2011-04-14T13:38:45.050" UserId="582" />
  <row Id="16237" PostId="9558" Text="However, the book is not covering the topic very widely." CreationDate="2011-04-14T13:39:19.243" UserId="4169" />
  <row Id="16238" PostId="9501" Text="@cardinal Your solution is correct. My mistake was thinking that a conversion in limits was required for the normal distribution with mean $\mu + \sigma^2$. This is not true as the variable u is not converted. If you post your comment as an answer, I can close this as resolved. Thanks!" CreationDate="2011-04-14T13:47:48.660" UserId="4146" />
  <row Id="16239" PostId="9549" Score="5" Text="I think that if any of those 800 students were to read this question, they might be offended. How well did they perform? How much learning was accomplished? That is what a grade should reflect, not some arbitrary statistical summary of their position in a group. IMHO this question should be recast in terms of teaching objectives, not statistical procedure, such as &quot;what is a good way to convert raw scores to grades in a way that respects student accomplishments and advances the learning objectives of this class?&quot; Statistics can help, but blind statistics--like standardization--will not." CreationDate="2011-04-14T14:46:24.183" UserId="919" />
  <row Id="16240" PostId="9566" Score="2" Text="Exploring the [change-point](http://stats.stackexchange.com/questions/tagged/change-point) tag might give you some ideas." CreationDate="2011-04-14T14:59:16.480" UserId="919" />
  <row Id="16241" PostId="9558" Text="@tunnuz - how do you mean not very widely? It seems to cover the basics, and if you grasp all of the material within, you should be well able to move on to more advanced topics." CreationDate="2011-04-14T15:26:26.303" UserId="656" />
  <row Id="16242" PostId="9569" Text="+1 Good reference, thanks.  The C-test is a more rigorous version of the one I sketched, so it's well worth considering.  The E-test relates a t-statistic to an appropriate distribution.  Calculating that distribution involves a double infinite sum that will take $O(n_1 n_2)$ calculations to converge: fairly easy to code, probably overkill for checking the newspaper!" CreationDate="2011-04-14T15:37:09.160" UserId="919" />
  <row Id="16243" PostId="9567" Text="Thanks (+1), that's just the right check for this kind of off-the-cuff thing.  It ended up being highly significant (p = 0.005) so the article is fine.  I hope you don't mind, though, that I accepted the other answer -- it's good to know the 'real' way to do it when it matters." CreationDate="2011-04-14T16:08:56.847" UserId="1378" />
  <row Id="16244" PostId="9561" Text="FWIW, article: http://health.usnews.com/health-news/family-health/brain-and-behavior/articles/2011/04/13/abusive-head-trauma-in-infants-doubled-during-recession-study" CreationDate="2011-04-14T16:09:34.570" UserId="1378" />
  <row Id="16245" PostId="8007" Text="That is just a guess. Certainly they are running around fetching parts and certainly they are sitting there assembling items and sometimes they are drinking coffee, but I don't know how all these times contribute to the overall time. I updated the question with some Excel charts of the actual data." CreationDate="2011-04-14T16:47:24.740" UserId="3615" />
  <row Id="16246" PostId="9561" Score="1" Text="Fine specimen of science journalism, there..." CreationDate="2011-04-14T18:01:16.853" UserId="71" />
  <row Id="16247" PostId="9561" Score="1" Text="Yeah... you can see why I wanted to check the statistics used." CreationDate="2011-04-14T18:04:58.590" UserId="1378" />
  <row Id="16248" PostId="8007" Text="Did you try simple regression, along the lines of this question: http://stats.stackexchange.com/questions/7531/multiple-regression-with-binary-predictors-component-value-analysis/7545#7545?" CreationDate="2011-04-14T18:52:28.680" UserId="3369" />
  <row Id="16249" PostId="7529" Score="2" Text="Instead of that (cumbersome) **sapply** call in your first code snippet, you can always just use **pmax**. " CreationDate="2011-04-14T19:00:19.520" UserId="2970" />
  <row Id="16250" PostId="8007" Score="1" Text="Reading the text I understand all the three graphs have &quot;time to complete an order&quot; as vertical axis, in other words $D_{order}$. However, the highest data point is at 40, 69 and 36, respectively. Why is this?" CreationDate="2011-04-14T19:23:33.163" UserId="3911" />
  <row Id="16251" PostId="9534" Text="@mpiktas, the updated answer looks more appropriate in a view of the point you raise eariler. thanks.." CreationDate="2011-04-14T19:36:35.373" UserId="3084" />
  <row Id="16252" PostId="9556" Text="(1) What do you mean by affecting a gene? Linkage? (2) ANOVA needs an approximately normally distributed dependent variable. How do you derive your dependent variable from the information on *tg*?" CreationDate="2011-04-14T19:41:41.527" UserId="3911" />
  <row Id="16253" PostId="9570" Text="+1 Nice reference." CreationDate="2011-04-14T20:55:04.737" UserId="919" />
  <row Id="16254" PostId="9562" Text="Thanks. I must admit that I didn't expect somebody to completely solve my problems. I am very happy that people come up with good suggestions. Energy based models are a good framework, but the question is which one? SVM can be seen as such a model (although with a very poorly usable energy), as well as multivariate Gaussians. Your suggestion to use an RBM is interesting: it is a probabilistic equivalent to SVMs. I did not know that RBMs could be used in outliers detection settings. Do you have a reference? Finally, we have had good experience with OC-SVMs for high-dimension outlier detection." CreationDate="2011-04-14T21:37:31.143" UserId="1265" />
  <row Id="16255" PostId="9549" Score="1" Text="The consensus here seems to be that norm-referenced assessments are uniformly bad and criterion-referenced ones, uniformly good.  I think a look at some basic examples of the vast assessment literature will show that each type has its place.  (But, yes, more often it is the latter method that will yield a more valid indicator of what a teacher is trying to accomplish.) " CreationDate="2011-04-14T21:42:48.897" UserId="2669" />
  <row Id="16256" PostId="4224" Text="`p &lt;- ggplot(data=x)` I get this error: `ggplot2 doesn't know how to deal with data of class phylo`. What am I missing?&#xA;" CreationDate="2011-04-14T23:42:11.797" UserId="3911" />
  <row Id="16257" PostId="8007" Text="Good point. That is because I just entered some scaling values that gave a nice picture. :/ The highest data point (in all three cases, of course) is actually 182 but that is just an outlier. So there are a few data points outside the visible range." CreationDate="2011-04-14T23:49:36.847" UserId="3615" />
  <row Id="16258" PostId="4111" Text="The National Park Service is no longer providing these estimates, as they are always controversial. They used to do this estimate by taking aerial photographs of a suitably chosen (non-random I guess) well-defined set of patches of real estate on the National Mall, counting the number of people in each square, and estimating total attendance from it. Clearly the time and order of pictures matters and contributes to the bias, especially with a moving crowd." CreationDate="2011-04-15T00:32:45.710" UserId="4062" />
  <row Id="16259" PostId="9524" Text="@Erik: Nice. Any idea about plotting the singular vectors. Thanks" CreationDate="2011-04-15T00:57:32.763" UserId="3903" />
  <row Id="16260" PostId="9575" Score="4" Text="Good explanation (+1). I would add, however, that the sample size needed for the distribution of means to approximate normality depnds on the degree of non-normalness of the population. For large samples there is no reason to prefer a t-test over a permutations test that makes no assumptions about the distributions." CreationDate="2011-04-15T02:01:58.697" UserId="1679" />
  <row Id="16262" PostId="9524" Text="@MYaseen208: I think you can do the same as with a regular SVD - create points $(d_1^\alpha u_{1i}, d_2^\alpha u_{2i})$, $(d_1^\alpha v_{1i}, d_2^\alpha v_{2i})$, $(d_1^\alpha w_{1i}, d_2^\alpha w_{2i}$ for $i = 1, \dotsc, n$, where $u_1, v_1, w_1$ are the 1st triple of &quot;singular&quot; vectors and $d_1$ the 1st singular value - similar for 2nd. (I had to look biplots up - hadn't seen them before.) By the way - I think you'd potentially get a more powerful (3D) graph if you include the third singular vector as a third coordinate..." CreationDate="2011-04-15T02:47:19.530" UserId="2898" />
  <row Id="16263" PostId="8271" Text="I think the confusion here is a matter of language. &quot;Time&quot; is your second IV. The DV does not have two levels. &quot;Time&quot; has two levels. Your DV, I am assuming, is metric." CreationDate="2011-04-15T03:48:25.513" UserId="183" />
  <row Id="16264" PostId="9424" Text="So averaging is the most recommended way? Can I conclude like that?" CreationDate="2011-04-15T04:03:24.207" UserId="4102" />
  <row Id="16265" PostId="9416" Text="n/w = Network. Sorry for using the abbreviation. I have transmitted 1MB of data to the peer system. like that I computed the bandwidth" CreationDate="2011-04-15T04:08:53.673" UserId="4102" />
  <row Id="16266" PostId="9576" Text="Great stuff. Thanks!" CreationDate="2011-04-15T06:13:12.723" UserId="4146" />
  <row Id="16267" PostId="9575" Score="1" Text="+1 although, as far as I know, t-test is fairly resistent to moderate deviations from normality. Also, an interesting related discussion: http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless " CreationDate="2011-04-15T07:02:48.503" UserId="582" />
  <row Id="16268" PostId="9424" Text="I am not sure what you want to do would be an accurate measure of the bandwidth. The bandwidth represents how much COULD be transmitted down the connection and not how much actually gets transmitted. If you want to get a measure of average tranfer speed then yes, averaging would be the way to go. Regardless, I would prefer to supply mean, median and a std deviation together, then you summarize your data better" CreationDate="2011-04-15T07:27:58.623" UserId="3014" />
  <row Id="16269" PostId="9562" Text="I always have a feeling that about any model that is optimized can be seen as an energy based model. After all, you can always formulate the loss as energy. The thing is, what do EBMs give you? And that is to create a model with no negative data, because you &quot;bootstrap&quot; by generating confabulations via MCMC. I have no idea if EBMs have been applied to outlier detection, but it sounds reasonable to me. To the SVM thing: high dimensional is fine, I am worried about the small dataset." CreationDate="2011-04-15T10:30:55.117" UserId="2860" />
  <row Id="16271" PostId="9556" Text="@GaBorgulya I updated my question with more explanation." CreationDate="2011-04-15T11:01:43.847" UserId="2885" />
  <row Id="16272" PostId="9556" Text="@probabilityislogic Hi, I think you are quite familiar with 3way-anova. Could you please have a look at my question?" CreationDate="2011-04-15T11:06:19.397" UserId="2885" />
  <row Id="16273" PostId="9581" Score="6" Text="For reference, this Q was asked on SO where I attempted an answer: http://stackoverflow.com/questions/5648383/clustering-in-r It would appear the OP has not attempted to improve the quality of their question in the meantime" CreationDate="2011-04-15T11:26:17.547" UserId="1390" />
  <row Id="16274" PostId="9581" Score="1" Text="What kind of clustering method would you like to use? " CreationDate="2011-04-15T11:27:53.057" UserId="582" />
  <row Id="16275" PostId="9581" Text="as u said 3 lines of code is enough for my dataset...could u please mention those 3 lines then directly i wil copy it and paste onto the console..." CreationDate="2011-04-15T11:30:13.500" UserId="4182" />
  <row Id="16276" PostId="9581" Text="@nico not a particular kind....in sas, knime and spss when i give the dataset that itself groups into clusters people who are having similar characterstics...in the same way i need those clusters here in r language also is it possible to get like same sas, knime and spss??" CreationDate="2011-04-15T11:32:07.753" UserId="4182" />
  <row Id="16277" PostId="9581" Text="@sridher: clustering data is not a trivial issue. You can cluser of data in several way and get several different results. It's up to you to understand which method to use and when. Unless you want us to do your work you should start by studying some books about clustering. Once you understand the concept then you can easily do it with the more than explicative code that was given you on StackOverflow." CreationDate="2011-04-15T11:33:11.323" UserId="582" />
  <row Id="16278" PostId="9581" Text="so is it must to do coding to get in r language to get clusters or are there any inbuilt methods to use directly for beginners to get the clusters for the dataset?" CreationDate="2011-04-15T11:36:27.690" UserId="4182" />
  <row Id="16280" PostId="9581" Text="@sridher I've tried to reword your question to reflect what I think you are asking. Feel free to edit it if I have misconstrued your question." CreationDate="2011-04-15T11:47:27.687" UserId="183" />
  <row Id="16281" PostId="9581" Text="@jeromy no it was correct only i wil edit if i feel so...:-)" CreationDate="2011-04-15T11:50:50.743" UserId="4182" />
  <row Id="16282" PostId="9581" Score="5" Text="@sridher Are you *incapable* of reading my answer? In one of the comments I told you exactly which three lines. But you *can't* just copy and paste it in, because I didn't have your data so used my own code/data and the objects don't have the same names as yours. All of this is included in my answer and the comments if you bother to look." CreationDate="2011-04-15T11:55:06.760" UserId="1390" />
  <row Id="16284" PostId="9581" Text="sorry to make u angry.....Yes its just my 2nd use for r language....i pasted those into console but giving errors....so wanted to see those exact 3 lines here without confusion...sorry really" CreationDate="2011-04-15T12:00:54.417" UserId="4182" />
  <row Id="16285" PostId="9581" Text="@Gavin but thanks alot for ur response...as u said i'm really incapable of getting u in this r clustering concept...." CreationDate="2011-04-15T12:13:29.713" UserId="4182" />
  <row Id="16288" PostId="9581" Text="I dont know what exactly you want to know. pleae explain with reproducible example, with small dataset.&#xA;Always a good way to know about the particulare methods in R, is to look the task views on CRAN. &quot;Cluster&quot; task view on CRAN. A quick overview of clustering in R can be found at&#xA;[R-Cluster Anlaysis][1] [1]: http://www.statmethods.net/advstats/cluster.html" CreationDate="2011-04-15T11:37:40.637" UserId="3084" />
  <row Id="16289" PostId="9584" Text="@gavin wow u r really amazing gavin....great to see good patience from u....this time i understood well....and when i pasted 3 lines into the console i got this error Error in colMeans(x, na.rm = TRUE) : 'x' must be numeric" CreationDate="2011-04-15T12:37:30.700" UserId="4184" />
  <row Id="16291" PostId="9581" Text="for example if we want to do target marketing we have to group the people who are having similar characterstics.....so i have the dataset which contains people age, sex, mon.sal like this more than 20 columns are there now i have to cluster for the target marketing...this is the task" CreationDate="2011-04-15T11:41:24.327" UserId="4182" />
  <row Id="16295" PostId="9584" Text="So your data contains non-numeric variables. Subset the data to only include the numeric ones." CreationDate="2011-04-15T12:50:40.783" UserId="1390" />
  <row Id="16297" PostId="9580" Text="+1 Good points.  The *power* of the t-test with skewed alternatives, though, can degrade severely (to the point where it is essentially zero even for huge effect sizes)." CreationDate="2011-04-15T14:20:08.723" UserId="919" />
  <row Id="16298" PostId="9578" Text="In the example you're discussing the power of the test, not its size.  The null, by the way, appears to be $p=0$, for which the distribution is degenerate (an atom at a single point): that's about as far from normality as one can be!" CreationDate="2011-04-15T14:22:42.123" UserId="919" />
  <row Id="16299" PostId="9586" Text="The question actually is more general than you suppose, because it really doesn't matter how the parameters were estimated.  The crux of the matter is that the parameters completely determine the distributions.  Thus, a set of simultaneous CIs on the parameters *is* a CI for the distributions." CreationDate="2011-04-15T15:02:06.817" UserId="919" />
  <row Id="16300" PostId="9586" Text="I understand that and perhaps it is that straightforward. Maybe what's nagging at me is that bootstrapping comes with it's own baggage and I was wondering if there was something about the procedure that introduces additional issues when used for this next step. On the other hand, it could just be what I had for breakfast that's gnawing.  Thanks for the quick comment. " CreationDate="2011-04-15T15:51:31.793" UserId="3591" />
  <row Id="16301" PostId="9575" Text="(+1 to the comments) Good points! My answer was very black-and-white because I don't really know the theory about the greyness in between. I added some simulations that may help address this though." CreationDate="2011-04-15T16:34:27.130" UserId="3874" />
  <row Id="16302" PostId="9586" Text="Gotta be the breakfast :-)" CreationDate="2011-04-15T17:23:29.330" UserId="919" />
  <row Id="16303" PostId="9588" Text="Some people will hang you out to dry for this. But in reality, you still need a metric that tells your story. " CreationDate="2011-04-15T17:43:33.370" UserId="776" />
  <row Id="16304" PostId="9584" Score="2" Text="@Gavin My very +1 for this instructive response (which nicely complements the other one you gave on SO, btw). For the OP @sridher who seems to wonder about understanding how Cluster Analysis works, there are a lot of clustering tutorials, including [this one](http://www.norusis.com/pdf/SPC_v13.pdf) or [this one](http://www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf) (both in PDF)." CreationDate="2011-04-15T18:06:33.117" UserId="930" />
  <row Id="16305" PostId="9586" Score="2" Text="There are at least two ways to bootstrap. The simplest way is to simply draw a random sample from the given observations and estimate the model b times (bootstrapping &quot;pairs&quot;). You can also bootstrap using the residuals from a model (&quot;residuals&quot; bootstrap). The first neglects the error structure in the data, which the second method implicitly assumes your model is correct. Efron &amp; Tibshirani (1993). &quot;An Introduction to the Bootstrap&quot; is the place to start." CreationDate="2011-04-15T19:58:36.427" UserId="3265" />
  <row Id="16306" PostId="8701" Text="@richie +1 OMG It is! I got it last week! I love it!" CreationDate="2011-04-15T20:18:27.733" UserId="3874" />
  <row Id="16307" PostId="9592" Text="I'm surprised that they give you different answers. It might help to clarify what you mean by &quot;answer&quot;." CreationDate="2011-04-15T20:27:55.350" UserId="3874" />
  <row Id="16308" PostId="9591" Score="1" Text="should this be $A = X + \epsilon Y$ not $Z + \epsilon Y$?" CreationDate="2011-04-15T20:28:41.980" UserId="795" />
  <row Id="16309" PostId="9591" Text="@shabby Yes, thank you.  (I had to change all the variable names in a draft to match the OP's names and messed this one up.)" CreationDate="2011-04-15T20:45:48.080" UserId="919" />
  <row Id="16310" PostId="9592" Score="2" Text="@Thomas The different answers are $p=ar+b$ and $p=(r-b')/a'$ from the two regressions.  The slopes *always* differ whenever the correlation is not 1 or -1: that's just another manifestation of &quot;regression to the mean.&quot;" CreationDate="2011-04-15T20:49:55.580" UserId="919" />
  <row Id="16311" PostId="9600" Text="That is basically the initial approach that I took, but I was informed that it wouldn't work in this case (still working on getting an explanation of why that's so).  The person who shot this approach down proposed the regression approach; I suggested the resampling approach as another possibility (perhaps less assumption-laden than the regression approach).  But thank you for your answer; it's reassuring to know this was somebody else's first thought.  Once I have some definitive words about why it wouldn't work, I'll update." CreationDate="2011-04-16T00:47:05.890" UserId="71" />
  <row Id="16312" PostId="9601" Text="Thanks for your answer - it's very helpful.  One point of clarification: the sampling took into account only x-ray status, because cultures take weeks to produce results.  Would I then build a model each for groups 1-6 and 7-9?" CreationDate="2011-04-16T00:53:59.490" UserId="71" />
  <row Id="16313" PostId="9601" Text="Yes, in that case the problem reduces to 2×3 groups of (`X-ray +, new +`, `X-ray +, new -`, `X-ray +, new nt`) and (`X-ray -, new +`, `X-ray -, new -`, `X-ray -, new nt`), everything else is analogous." CreationDate="2011-04-16T01:01:52.027" UserId="3911" />
  <row Id="16314" PostId="9542" Text="In fact, I think you'd be worse off if you had moderately correlated variables than when you've got really highly correlated variables. Numerical-wise too, if you're using an algorithm like NIPALS that removes PC's in order" CreationDate="2011-04-16T02:00:41.417" UserId="26" />
  <row Id="16315" PostId="9602" Text="Thanks for the input!  Unfortunately my data cannot fairly be approximated as normal.  I may have to post another question to discuss my shady binomial confidence intervals, but for now, I just want to combine oracle-given CI's." CreationDate="2011-04-16T05:42:10.033" UserId="4191" />
  <row Id="16316" PostId="9602" Text="@gaBorgulya - normal approx would be particularly bad for the 0-5% coin" CreationDate="2011-04-16T06:38:09.057" UserId="2392" />
  <row Id="16317" PostId="9607" Score="1" Text="If you match the mean and variance of each task with the normal distribution, then the expected value and standard deviation of the sum is unchanged.  You will loose all information about skewness though - and tails are likely to be thinner with normal" CreationDate="2011-04-16T11:00:25.753" UserId="2392" />
  <row Id="16318" PostId="9607" Text="mean sounds correct, but not sure with beta, i forgot to say how i calculate sd. I start form 90% confidence interval and mean. now i calculate the sd from a norm dist via z-score for 90%. here an error should already be introduced. but what can I say about the error of sd in comparision with beta-dist? " CreationDate="2011-04-16T11:25:41.560" UserId="778" />
  <row Id="16319" PostId="9607" Text="so you don't have a &quot;standard deviation&quot; specified, you have specified the quantiles - 5% and 95% I take it.  But in this case - if you use a beta distribution to fit the quantiles, and then use a normal to approximate that beta distribution, my comment above applies." CreationDate="2011-04-16T12:06:39.430" UserId="2392" />
  <row Id="16320" PostId="9607" Text="Sorry, can you clarify what you're trying to do? If you're summing beta-distributed random variables and want to know and calculate quantities of the distribution of the sum, there is no way those would be beta-distributed when you were done. So, in that context, it would seem that whether or not Google Spreadsheet had functions for the beta distribution would be irrelevant." CreationDate="2011-04-16T12:11:27.453" UserId="2970" />
  <row Id="16321" PostId="9607" Text="Also, it seems a little strange to use the beta distribution for a random variable with units of *days*. The beta distribution is concentrated on $[0,1]$, so are all tasks *guaranteed* to be completed in less than one ideal person day? Even if rescaled, it seems the upper bound is a strong assumption." CreationDate="2011-04-16T12:12:01.103" UserId="2970" />
  <row Id="16322" PostId="2700" Text="So is PCA equivalent to Total Least Squares if it optimizes orthogonal distances from points to the fit line?" CreationDate="2011-04-16T14:25:18.047" UserId="595" />
  <row Id="16323" PostId="9600" Text="Hmm... that is interesting. I'd love to see his or her explanation." CreationDate="2011-04-16T14:57:32.160" UserId="3874" />
  <row Id="16324" PostId="9607" Text="@probabilityislogic I get it, SD is SD - logic. but the skewness of the beta-dists will dissapear due the central limit theorem, right? With say 20 tasks normal is good approximated? and for the tails too?" CreationDate="2011-04-16T15:39:48.710" UserId="778" />
  <row Id="16325" PostId="9607" Text="@cardinal the PERT distribution is a version of beta and used to estimate durations of tasks http://www.vosesoftware.com/ModelRiskHelp/index.htm#Distributions/Continuous_distributions/PERT_distribution.htm" CreationDate="2011-04-16T15:46:52.980" UserId="778" />
  <row Id="16326" PostId="9607" Text="@cardinal Its not about normalized distribution. My concern is if I distort the information in the input variables for each task (5% Quantile-Average-95% Quantile) Project=Sum (Task). Project is clearly not beta-distributed due Central Limit Theorem. but what do I distord if Tasks are Norm not Beta? As a result I need P(Project&lt; x days) and the worst thing that can happen is that its way too high. So if tails estimated too small i have a problem" CreationDate="2011-04-16T15:56:44.380" UserId="778" />
  <row Id="16327" PostId="9598" Text="Here's a link to one of the references in the above top link http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.3325" CreationDate="2011-04-16T16:54:43.987" UserId="2775" />
  <row Id="16328" PostId="9604" Score="3" Text="Formally you can do, but from practical point of view are the discrete choice variables endogenously related in your model? It would be nice to hear a complete story, and in this ad hoc case it will be more or less clear if it is relevant. The estimation algorithm probably will be slow, since you have to deal with full (limited) information maximum likelihood method estimating the parameters." CreationDate="2011-04-16T17:01:39.403" UserId="2645" />
  <row Id="16329" PostId="9609" Text="@ probabilityislogic:  &quot;model misspecification is irrelevant when you are interpreting co-efficients in a model&quot;--how can it be irrelevant when the addition of new variables such as suppressors could change those coefficients substantially and meaningfully?" CreationDate="2011-04-16T17:01:58.940" UserId="2669" />
  <row Id="16330" PostId="9595" Text="@cchien, what in particular distributed lag scheme are you implementing (Almond or Koyck or some other)? I always thought that it is as it is, including lags means the loss of information, both by additional parameters to estimate and decrease in degrees of freedom. Well you may forecast backwards the values (by exponential smoothing or similar), but I personally would not be doing so." CreationDate="2011-04-16T17:10:31.617" UserId="2645" />
  <row Id="16331" PostId="9616" Text="+1, if series are stationary, then any lag of them is stationary also, so there is no depth here :)" CreationDate="2011-04-16T17:57:26.710" UserId="2116" />
  <row Id="16332" PostId="9625" Score="2" Text="Do you have a specific alternative hypothesis in mind?  Note, too, that comparing the mean to the median (in some way) often shows up as a test of *skewness*: that's a good way to search the literature.  Check out the recent paper in JSE that shows up as a top hit: [Investigating the Investigative Task: Testing for Skewness / An Investigation of Different Test Statistics and their Power to Detect Skewness](http://www.amstat.org/publications/jse/v18n2/tabor.pdf)  It reports some simulation results of several tests." CreationDate="2011-04-16T20:14:31.487" UserId="919" />
  <row Id="16333" PostId="9625" Text="@whuber: yes, I am thinking about this as a test of skewness. I recall seeing (sample mean - sample median) / sample standard deviation used some indicator of skewness, with the nice property that it is bound between -/+ 1. As far as the alternative hypothesis, I can go with a one-sided or two sided alternative." CreationDate="2011-04-16T20:25:24.260" UserId="795" />
  <row Id="16334" PostId="9627" Score="3" Text="What makes a correlation causal is a contentious issue in philosophy of science. The &quot;gold standard&quot; is to perform an experiment where the treatment variable is assigned at random, thus ensuring other potentially confounding covariates are not systematically related to the treatment. However, in many fields and for many questions, experiments are impossible. Some turn to so-called natural experiments for exogeneity. You may be interested in Paul Holland's classic article, &quot;Statistics and causal inference&quot;. Journal of the American Statistical Association, 81, 945-970.&#xA;" CreationDate="2011-04-16T21:02:10.530" UserId="3265" />
  <row Id="16335" PostId="9620" Text="@onestop, I suppose I might be interested in such derivations - I just don't quite get why we need to go to the characteristic function? Why is it easier than dealing directly with the pdf/cdf?" CreationDate="2011-04-16T21:10:16.063" UserId="1913" />
  <row Id="16336" PostId="9629" Score="1" Text="In the second case, are you assuming you know the total sample size? Or not?" CreationDate="2011-04-16T22:04:52.770" UserId="2970" />
  <row Id="16337" PostId="9624" Score="1" Text="Also, DK could have real meaning, ie those individuals are not health conscious and could be at-risk. " CreationDate="2011-04-16T22:19:23.440" UserId="776" />
  <row Id="16339" PostId="9609" Text="@rolando2 - new variables means a new model, so my comment still applies.  If you are looking to interpret the effect within in a class of models (such as additional variables), then you should take a weighted average of the co-efficient in each model, with the weights being equal to (or proportional to) the plausibility of each model" CreationDate="2011-04-16T22:52:02.150" UserId="2392" />
  <row Id="16340" PostId="9607" Score="1" Text="An exponential distribution may be a better idea for modeling instead of the beta distribution.  one because it is highly right skewed, two because it is always positive, three because it has &quot;fat tails&quot;, and four because the sum of exponentially distributed quantities has an Erlang or gamma or scaled chi-square distribution - which is no more difficult to handle than a normal, plus it is precise.  And exponential arises as a limiting form of a beta distribution.  Also exponential has maximum entropy (uncertainty) for a given mean - so you only need to ask for expected time to complete" CreationDate="2011-04-16T23:06:16.440" UserId="2392" />
  <row Id="16341" PostId="9607" Text="(cont'd) - expectations are much easier to get accurately compared to &quot;tail behaviour&quot; prior information.  One intuition is usually not too good with extreme parts of a distribution" CreationDate="2011-04-16T23:07:38.450" UserId="2392" />
  <row Id="16342" PostId="9620" Text="@Nick This has a bit of folklore element, like &quot;this is so elegant that this is THE representation of some distribution concept,...&quot;. Of course it helps with some maths, so it is not just a redundant toy, but for an everyday use it corresponds to a physicists forcing $\hbar$ to a classic problem just to use fine-structure constant." CreationDate="2011-04-16T23:41:26.540" UserId="88" />
  <row Id="16343" PostId="9627" Score="1" Text="You're asking very important questions, but it's doubtful anyone could give you a definitive series of steps to take or a nice, condensed recipe; mastering this issue is a long-term proposition.  Additional suggestions about terms and topics to study:  suppressor variables;  tolerance and variance inflation estimates; zero-order, partial, and semipartial (part) correlations; variable selection methods; crossvalidation." CreationDate="2011-04-17T00:28:15.560" UserId="2669" />
  <row Id="16344" PostId="9629" Text="@cardinal: yes I do know the total sample size." CreationDate="2011-04-17T00:30:16.227" UserId="2830" />
  <row Id="16345" PostId="9631" Text="@tuxette: Just to confirm, it will also work with expected proportions that are unequal to each other?" CreationDate="2011-04-17T00:31:49.050" UserId="2830" />
  <row Id="16346" PostId="9631" Score="3" Text="@tuxette, the test won't be meaningful unless you know the full sample size.  Proportions of 1.0 / 0.0 / 0.0 / 0.0 mean very different things if they are from a sample of size 1 as opposed a sample of size 100." CreationDate="2011-04-17T00:52:54.990" UserId="2975" />
  <row Id="16347" PostId="9629" Score="2" Text="@penyuan, then just multiply the proportions by the total sample size to transform into a table of counts, and apply the chi-sq. method corresponding to your first example." CreationDate="2011-04-17T00:53:52.573" UserId="2975" />
  <row Id="16348" PostId="9636" Score="1" Text="I'm pretty sure the expected *frequencies* can't be larger than 10. :)" CreationDate="2011-04-17T00:59:13.427" UserId="2970" />
  <row Id="16349" PostId="9636" Text="@cardinal - glad that this was your objection - for it means the rest of my answer must have been good :)." CreationDate="2011-04-17T01:04:44.310" UserId="2392" />
  <row Id="16350" PostId="9636" Text="Wow, I hope I'm not getting a reputation for being so picky/grumpy." CreationDate="2011-04-17T01:46:36.050" UserId="2970" />
  <row Id="16351" PostId="9636" Score="1" Text="I'm don't quite follow the &quot;exact&quot; terminology. Perhaps that is particular to Jaynes' work. Your $\psi$ is the log-likelihood-ratio test statistic though and so $2 \psi$ is asymptotically distributed as a $\chi^2$ distribution by Wilks' theorem. Also, $\chi^2 - 2 \psi \to 0$ in probability, which by Slutsky's theorem is enough to conclude that $\chi^2$ has the same distribution as $2\psi$. Finally, it turns out that $\chi^2$ is the scorte test statistic in this problem as well, which provides another connection between the two test statistics." CreationDate="2011-04-17T01:53:10.787" UserId="2970" />
  <row Id="16352" PostId="9636" Text="Also, Agresti (*Categorical Data Analysis*, 2nd ed., p. 80) claims that $\chi^2$ actually converges to a chi-squared distribution faster than $2 \psi$, which seems at odds with your recommendation. :)" CreationDate="2011-04-17T01:55:10.190" UserId="2970" />
  <row Id="16353" PostId="9635" Score="5" Text="Other items worth mentioned: **(a)** Recovery of moments via differentiation, **(b)** the fact that all distributions have characteristic functions (as compared to moment-generating functions), **(c)** The (essentially) one-to-one correspondence between distributions and their characteristic functions, and **(d)** the fact that many relatively common distributions have known characteristic functions but no known expression for the density (e.g., Levy stable distributions)." CreationDate="2011-04-17T02:02:53.893" UserId="2970" />
  <row Id="16354" PostId="9636" Text="@cardinal - you are focused on the distribution of the statistic.  What I am saying is that the likelihood ratio is already exact using $\psi$, you can compare hypothesis directly using it, rather than a p-value based on its distribution.  So if $\psi=ln(2)$ then this means that the &quot;perfect fit&quot; is twice as likely as the hypothesised fit - and I just realised my odds ratio is the &quot;wrong way around&quot;" CreationDate="2011-04-17T02:03:17.450" UserId="2392" />
  <row Id="16355" PostId="9636" Text="I didn't quite follow that last comment, but I think the odds ratio you use is correct." CreationDate="2011-04-17T02:07:40.847" UserId="2970" />
  <row Id="16356" PostId="9636" Text="@cardinal - they have the same asymptotic distribution, but different finite sample distributions, which is why I said use chi-square when sample size is large, and $\psi$ when sample size is small.  can show by easy example that $\psi$ is way better that $\chi^2$ in small expected count case." CreationDate="2011-04-17T02:08:08.647" UserId="2392" />
  <row Id="16357" PostId="9636" Text="That depends on how you use $\psi$. What Agresti is claiming is that if you use the reference chi-square distribution for inference (which nearly any practitioner will almost certainly do), then the Pearson $\chi^2$ will give you close to the nominal rate for smaller samples. Of course, how useful that is to you depends on the context; you might pay a price in lost power or in other areas." CreationDate="2011-04-17T02:11:18.767" UserId="2970" />
  <row Id="16359" PostId="9636" Text="consider the following example.  You have 2 people, both with different hypothesis, one says equal probs $H_{1}:p_1=p_2=p_3=\frac{1}{3}$, another says two dominant ones $H_{2}:p_1=p_2=0.499,p_3=0.002$.  Suppose you observed $x_1=x_2=14$ and $x_3=1$.  $\psi$ supports $H_{2}$ over $H_{1}$ by a large amount, but $\chi^2$ goes the other way, which is clearly wrong - the data support $H_{2}$ over $H_{1}$ just by looking at them, no test is required.  But $\chi^2$ fails in this case." CreationDate="2011-04-17T02:23:25.040" UserId="2392" />
  <row Id="16360" PostId="9609" Text="@ probabilityislogic  I see what you mean, thanks." CreationDate="2011-04-17T02:35:07.167" UserId="2669" />
  <row Id="16361" PostId="9636" Text="I don't follow. The OP is talking about goodness of fit testing and this is what this version of Pearson's $\chi^2$ statistic is geared towards. In the example you give, $\chi^2 \approx 11.65$. Using a chi-square with two degrees of freedom as the reference distribution gives a $p$-value of $p \approx 0.0031$, i.e., a highly significant value under the null assumption of uniformity." CreationDate="2011-04-17T02:37:02.120" UserId="2970" />
  <row Id="16362" PostId="9636" Text="Yes, but the chi squared under the alternative $H_{2}$ is $\chi^2\approx 15$, suggesting that a uniform is a better fit to the data than the one proposed.  this is where $\chi^2$ fails." CreationDate="2011-04-17T02:44:57.180" UserId="2392" />
  <row Id="16363" PostId="9570" Text="Question for all:  does the idea of doing a simple *T*-test, before vs. after, get compromised by the fact that the researcher had the benefit of seeing the entire series before choosing that Christmas split point?  Beyond that, are there methods simpler than GaBorgulya's that you would recommend?  And I'm not sure that fitting 2 ARIMA models would be much simpler." CreationDate="2011-04-17T02:46:38.420" UserId="2669" />
  <row Id="16364" PostId="9636" Text="Using $\psi$ you get $\psi_1\approx 35$ and $\psi_2\approx 8$" CreationDate="2011-04-17T02:46:42.543" UserId="2392" />
  <row Id="16365" PostId="9636" Text="The version of Pearson's chi-square that you are using is not for comparing two fixed distributions against one another. Maybe that is where you are confused. Seeing that single count for $x_3$ is heavy evidence in both cases of a lack of fit." CreationDate="2011-04-17T02:53:05.533" UserId="2970" />
  <row Id="16366" PostId="9636" Text="Yes, but if pearson's chi-squared is to be some &quot;universal&quot; GOF statistic (which many people claim), then it should be able to tell you if one hypothesis is a better fit to the data than another?  how else can you define &quot;goodness-of-fit&quot;?" CreationDate="2011-04-17T02:59:01.030" UserId="2392" />
  <row Id="16367" PostId="9636" Text="Easy. If you're considering two fixed distributions, as you are in your example, use the likelihood test and invoke Neyman-Pearson." CreationDate="2011-04-17T03:00:52.603" UserId="2970" />
  <row Id="16368" PostId="9616" Text="@Wesley Burr: I see what you are saying. But two time series are said to be co-integrated if there exists a linear combination of the two, so that the combination is stationary. The difference of the two identical time series are a all-zero time series. Is it stationary?" CreationDate="2011-04-17T03:07:44.440" UserId="4192" />
  <row Id="16369" PostId="9636" Text="@cardinal, and what if the two fixed distributions are my $H_1$ and my $H_2$?  Chi-squared is not good here as a GOF measure.  I would go further and say that you could not produce a single example where $\chi^2$ gives the right conclusion and $\psi$ gives the wrong one." CreationDate="2011-04-17T03:10:03.623" UserId="2392" />
  <row Id="16370" PostId="9636" Text="Then the log likelihood ratio statistic is most powerful and, as hinted at above, Pearson's $\chi^2$ is irrelevant. Which has been exactly my point. :)" CreationDate="2011-04-17T03:13:00.920" UserId="2970" />
  <row Id="16371" PostId="9636" Text="@cardinal - so you're saying that I should remove any reference to the $\chi^2$ test in my answer?  I think this is unwise because $\chi^2$ is closely connected to $\psi$, and almost every statistician knows what a $\chi^2$ test is.  They are both hypothesis tests against the &quot;sure thing&quot; hypothesis - but chi-square is not transient, whereas psi is transient.  So you don't need to invent another test when using $\psi$ either in specific hypothesis tests or as a GOF measure." CreationDate="2011-04-17T03:22:22.920" UserId="2392" />
  <row Id="16372" PostId="9636" Text="No, quite the contrary. I think your reference to, and discussion of, $\chi^2$ in the answer is good and entirely relevant. It addresses what the OP is asking about. However, the tangent you took when you introduced two separate fixed distributions to compare against is where the use of $\chi^2$ is not well-motivated. Indeed, I think that $\chi^2$ is essentially irrelevant in that context. That is *not* the problem that particular statistic is trying to address. I'm not sure what you mean by &quot;transient&quot; above, either." CreationDate="2011-04-17T03:30:56.997" UserId="2970" />
  <row Id="16373" PostId="9636" Text="As regards your final comment regarding $\psi$, it suffers from exactly the same &quot;defect&quot; as $\chi^2$ in the example in your comment: It is not the appropriate statistic to test two fixed distributions against one another. *However*, considering differences does work due to the fact that $\psi_1 - \psi_2$ is the log-likelihood statistic for $H_1$ vs. $H_2$ which, by Neyman-Pearson is optimal. (Maybe that's what you were getting at with the term &quot;transient&quot;??)" CreationDate="2011-04-17T03:34:14.373" UserId="2970" />
  <row Id="16374" PostId="9637" Score="3" Text="Can you provide more information on the problem you are trying to consider? As stated, it lacks enough detail to answer adequately." CreationDate="2011-04-17T03:36:04.603" UserId="2970" />
  <row Id="16375" PostId="9616" Score="1" Text="@David, yes, trivially so. Let $X_n = Y_n - Z_n$ where $Y_n$ and $Z_n$ are your &quot;two identical series&quot;. For any $k$ and any $n$, $\mathbb{P}( (X_1, X_2, \ldots, X_k) \in B ) = \mathbb{P}( (X_{1+n}, X_{2+n}, \ldots, X_{k+n}) \in B )$, i.e., $(X_t)$ is stationary." CreationDate="2011-04-17T03:40:20.543" UserId="2970" />
  <row Id="16376" PostId="9636" Text="@cardinal - both $\psi$ and $\chi^2$ are already hypothesis tests of the null against the perfect fit hypothesis, because $\psi_2=\chi^2_2=0$ in this case.  This is what a GOF statistic really is - and you can always show this (a GOF test can be shown to be a hypothesis test with an implicit alternative).  And by transient, I mean if A is a &quot;worse fit&quot; compared to C by an amount $X$ and B is a &quot;worse fit&quot; compared to C by $X-a$ for some $a&gt;0$ then A should also be a &quot;worse fit&quot; compared to B by $a$ (can think of this as &quot;consistent ranking&quot; in some sense)." CreationDate="2011-04-17T04:23:55.527" UserId="2392" />
  <row Id="16377" PostId="9636" Text="Maybe you meant *transitive*? Pearson $\chi^2$ can be motivated in a number of ways. One is as a Taylor series approximation to $\psi$, another is as a score test, which is essentially a way to get around having to do a Wald test based on a maximum likelihood estimator. In either case, you can think of Pearson $\chi^2$ as a local linearization of the desired statistic. Furthermore, an asymptotic distribution as opposed to the true distribution is used. So, putting those together, of course it won't be transitive in the sense you describe." CreationDate="2011-04-17T04:36:14.950" UserId="2970" />
  <row Id="16378" PostId="9636" Text="But, that is all still a bit beside the point for the example you give. The $\psi$ is special because $\exp(\psi_1 - \psi_2)$ for two *fixed* alternatives is the likelihood ratio, which you know is optimal by Neyman-Pearson. In that sense, it is &quot;special&quot;." CreationDate="2011-04-17T04:38:05.137" UserId="2970" />
  <row Id="16379" PostId="9636" Text="Also the &quot;perfect fit&quot; analogy is a bit contrived in this case as you're implicitly comparing a very specific parametric form to a nonparametric one." CreationDate="2011-04-17T04:41:39.173" UserId="2970" />
  <row Id="16380" PostId="9636" Text="@cardinal - yes I do mean transitive :) lol. my bad.  And my point is as before - that you focus on the distribution of the statistic - which puts the attention into data sets that might have been observed, but were not actually observed.  What should be the focus is how well the data is fit by the hypothesis - for the data set that we actually have - why should we care if an observed count for &quot;red&quot; might have been $12$, when we know that it is $15$?  This may be important for something like prediction, but not for testing how well a hypothesis fits the data." CreationDate="2011-04-17T04:50:20.457" UserId="2392" />
  <row Id="16381" PostId="9636" Text="@cardinal - the perfect fit is the perfect fit within the beroulli/binomial class.  If it was to all alternatives - then the way the observed data enters into the test would be different" CreationDate="2011-04-17T04:52:55.163" UserId="2392" />
  <row Id="16382" PostId="9627" Text="+1 for &quot;experiments are impossible&quot;" CreationDate="2011-04-17T04:58:39.473" UserId="776" />
  <row Id="16383" PostId="9625" Text="do you have this in mind with a specific model or set of models (some skewed some not skewed) or a non-parametric style test?  What will you do if the test comes back &quot;positive&quot; (mean not equal to median)?  What will you do if it comes back &quot;negative&quot; or &quot;inconclusive&quot;?" CreationDate="2011-04-17T05:01:30.127" UserId="2392" />
  <row Id="16384" PostId="9620" Text="We don't *need* to use them. I said only that they *can* be used. Sometimes they give a quicker derivation, sometimes they're no help at all. Whether a derivation is 'easier' depends on what you already know - if you don't already know about characteristic functions it won't be easier. In some cases moment generating functions provide an alternative, and have a more direct interpretation. " CreationDate="2011-04-17T05:02:59.230" UserId="449" />
  <row Id="16385" PostId="9636" Text="I'm not sure what that means, but I'm likely just tired. There are only three free parameters in the OP's model. You can't get a likelihood of one (i.e., &quot;perfect fit&quot;) for arbitrary data with a sample size larger than three. In fact, the best you can do is the MLE and that is precisely what both $\psi$ and $\chi^2$ are implicitly constructed to test against." CreationDate="2011-04-17T05:10:31.710" UserId="2970" />
  <row Id="16386" PostId="9636" Text="@cardinal - That's what I meant by &quot;perfect fit&quot; - where you set &quot;observed&quot;=&quot;expected&quot;.  but you are only testing GOF within a class of alternatives - not to everything.  So we are basically in agreement that $\psi$ and $\chi^2$ are already hypothesis tests for the hypothesis against the &quot;perfect fit&quot; - I think this is also called the &quot;saturated model&quot; in some contexts." CreationDate="2011-04-17T05:26:51.577" UserId="2392" />
  <row Id="16387" PostId="9636" Text="I think two different concepts are being conflated in your latest comment." CreationDate="2011-04-17T05:37:52.953" UserId="2970" />
  <row Id="16388" PostId="9607" Text="@probabilityislogic ad 4) I always suspected that a 'project' is also skewed, experience tells that its real effort exceed the expected in multiples but it's unlikelier to end much sooner. There is a steepness before the expected value and a long tail after. At least thats the intuition. I have to study your suggestion in depth. I see its hard to understand what distribution is right. I wonder if there is a common procedure I should follow..." CreationDate="2011-04-17T08:24:26.867" UserId="778" />
  <row Id="16389" PostId="9607" Text="@probabilityislogic maximum entropy,... really interesting stuff. " CreationDate="2011-04-17T08:30:45.537" UserId="778" />
  <row Id="16390" PostId="9636" Text="@cardinal, @probabilityislogic Comments are not a good place for discussions -- you can use chat instead." CreationDate="2011-04-17T09:37:43.163" UserId="88" />
  <row Id="16391" PostId="9616" Text="@Wesley Burr: I think you are slightly confusing when you say *If you have two identical time series, then the set of all linear combinations of those time series is equivalent to the set of all scalar multiples of one of them. In this case, if a time series has a scalar multiple of it that is stationary, then the original time series was itself stationary.*  Isn't the point here that even if the original times series is non-stationary, the difference between it and itself *is stationary* (indeed it is constant, since it is always zero)? So a time series is cointegrated with a copy of itself. " CreationDate="2011-04-17T11:38:08.587" UserId="2958" />
  <row Id="16392" PostId="9607" Text="In my opinion MaxEnt is the best general method for setting up models, and for setting up prior probabilities.  Its a very good &quot;ambiguity resolver&quot; - as is the theory of transformation groups." CreationDate="2011-04-17T11:45:53.007" UserId="2392" />
  <row Id="16393" PostId="9610" Text="For any normal (Gaussian) distribution, the skewness is $0$ since it is symmetric, and the excess kurtosis is also $0$ from the properties of a normal distribution.  For other distributions, the mean, variance, skewness and kurtosis are not enough to define the distribution, though examples can be usually be found." CreationDate="2011-04-17T11:46:00.217" UserId="2958" />
  <row Id="16394" PostId="9597" Text="Thank you bill_080 for the answer. I think you're saying that I should make sure the dependent variable (reflectivity) is in the right place: the $y$ axis. So you're suggesting that it is important to preserve the causal relationship? If I want to use regression (for some reason), then what? " CreationDate="2011-04-17T12:13:39.860" UserId="4187" />
  <row Id="16395" PostId="9593" Text="Thanks for that - you are spot on with your reading of the problem, and what you say about error makes sense. As bill_080 says in the next comment, I guess it comes down to whether or not preserving the natural causal relationship is important or not. In other words, what's more important: reducing the error, or representing the causality? " CreationDate="2011-04-17T12:17:50.637" UserId="4187" />
  <row Id="16396" PostId="9607" Text="@probabilityislogic reseached exponential distribution, problem: memorylessness . what finishes a task is work (effort, in my model), the more work the higher the probability of being finished. exponential distr would mean regardless of effort invested the probability of finishing is the same. Now thats fatalistic. Maybe its Weibull..." CreationDate="2011-04-17T12:28:38.013" UserId="778" />
  <row Id="16397" PostId="9643" Score="1" Text="Err, what happened to reading the response from the same Q you asked on SO? http://stackoverflow.com/q/5686940/429846 Use `prcomp()` instead! The help page `?princomp` even tells you this." CreationDate="2011-04-17T12:51:31.207" UserId="1390" />
  <row Id="16398" PostId="9631" Text="Yes, I DO know the total sample size." CreationDate="2011-04-17T14:08:58.690" UserId="2830" />
  <row Id="16399" PostId="9607" Score="1" Text="@roland - the &quot;memoryless&quot; property is a bit confusing because it is conditional, not absolute.  More work/effort would result in a higher rate parameter, and thus a smaller amount of time to finish a project.  the memorylessness would be assessed by looking at the distribution of the &quot;late tasks&quot;.  Now why would a task be &quot;late&quot;?  One way I can think of is that the amount of work required to finish it was under-specified - adding in these additional tasks - seems like we are back to the start - so memorylessness may be a good property." CreationDate="2011-04-17T14:29:38.793" UserId="2392" />
  <row Id="16400" PostId="9607" Text="(cont'd) the memorylessness can be seen as incorporating a degree of &quot;things not planned for&quot; in the tasks - where we expect the things not planned for to be of a similar size to the things which were planned for" CreationDate="2011-04-17T14:30:41.243" UserId="2392" />
  <row Id="16402" PostId="9646" Text="	&#xA;Hi Rolando, thanks for answer. Your guess is right. For each person I have single value covering the two months. The 44 people were the students that enrolled to this unit. It wasn't random sample, basically it was all the population that I had access. I'm planning to repete the experiment for future students... but right now I would like to grab some partial conclusion of the behavior observed... The last question I don't understand it... do you think that I should use other approach instead of decision trees?? thanks a lot for your help" CreationDate="2011-04-17T16:40:43.143" UserId="4203" />
  <row Id="16403" PostId="9635" Text="Good comments, @cardinal.  Please consider turning them into an actual reply." CreationDate="2011-04-17T16:55:44.017" UserId="919" />
  <row Id="16404" PostId="9597" Text="@Matt: I added my response as an edit above." CreationDate="2011-04-17T17:31:05.947" UserId="2775" />
  <row Id="16406" PostId="9616" Text="David (and Cardinal)'s points are well taken: the trivial case is obviously an extension. I was avoiding the case where the linear combination is to take the difference, as it didn't seem to be what the OP was asking. " CreationDate="2011-04-17T19:36:11.770" UserId="781" />
  <row Id="16407" PostId="9610" Text="@Henry Actually, in most $k$-parameter families of distributions with $k \le 4$, the first four moments--which can be recovered from the mean, variance, skewness, and kurtosis--are usually enough to identify the distribution." CreationDate="2011-04-17T20:05:17.223" UserId="919" />
  <row Id="16408" PostId="8391" Text="@JMS In some sense there are integrals hiding in many formulas.  For instance, the logarithm is usually defined as an integral.  Many other functions, like the exponential and the functions of mathematical physics, are (or can be) defined as solutions of differential equations--that is, generalizations of integrals." CreationDate="2011-04-17T20:07:58.000" UserId="919" />
  <row Id="16409" PostId="9646" Text="Sounds as if you could try to make the case that this year's students constitute a random sample of the population of all students who might enroll. That way you'd be justified in generalizing from sample to population...I was asking what was your basis for getting the tree to split into branches.  I'm familiar with programs that give you the option to form a split if certain criteria are met....How did you crossvalidate, though, if you only had 44 cases?" CreationDate="2011-04-17T20:51:53.737" UserId="2669" />
  <row Id="16410" PostId="9637" Text="[Geometric standard deviation](http://en.wikipedia.org/wiki/Geometric_standard_deviation)" CreationDate="2011-04-17T20:54:45.527" UserId="919" />
  <row Id="16411" PostId="8391" Text="@whuber Of course you're right; I was just poking a little fun. Although there is the somewhat larger point that we shouldn't be afraid of integrals we can't solve analytically if we can approximate them well as is the case for $\Phi$ above, or as you mention the menagerie of special functions arising from differential equations." CreationDate="2011-04-17T22:45:29.597" UserId="26" />
  <row Id="16412" PostId="26" Text="I agree with Joel. Standard deviation is an important concept in statistics. Would it not be absurd if you couldn't ask a question about it on a site about asking statistical questions." CreationDate="2011-04-17T23:00:09.423" UserId="3597" />
  <row Id="16413" PostId="9462" Text="@Jeromy, I can't go back to the population at this point (we have a lockout time and our population is well known to us) so I must make the best use of the sample that I have. In my previous statement, I made an error. I'm looking for a statistical method for sampling my sample of users of company X to be able to converge upon which companies they are planning on doing more business with in the future.  " CreationDate="2011-04-17T23:38:16.877" UserId="776" />
  <row Id="16415" PostId="9439" Score="1" Text="I strongly believe that my answer is one of the best solutions to the question asked. Would the one who downvoted it to negative explain why they think it's not useful?" CreationDate="2011-04-17T23:58:51.243" UserId="3911" />
  <row Id="16416" PostId="9630" Text="Would the one please who downvoted my answer to negative please explain? I believe that confidence intervals can be useful alternatives/supplements to significance tests. So although I know my reply did not give a full answer to the question I believe it has some value." CreationDate="2011-04-18T00:31:55.630" UserId="3911" />
  <row Id="16417" PostId="8371" Score="1" Text="A lot of non-experts (i.e. people who don't understand the statistical theory behind sampling) throw around the term &quot;quota sampling&quot; without really knowing what it means.  That may explain the claim that it's common practice in Britain and Southern Europe." CreationDate="2011-04-18T01:45:25.713" UserId="4110" />
  <row Id="16418" PostId="9610" Text="@whuber: That reads to me as being slightly circular: restricting distributions to a family where there are four or fewer parameters, knowing four statistics of the distribution often identifies the parameters. I agree. But one of my points was essentially that unrestricted there are different possibilities of distributions with substantially varying probability densities at particular points even with the same first four moments overall." CreationDate="2011-04-18T02:12:38.887" UserId="2958" />
  <row Id="16419" PostId="8371" Text="French opinion polling shortly before elections has more often than not been as accurate or more accurate than elsewhere in the world, though they do better when there are only two choices.  [Other polls conducted in March 2011](http://fr.wikipedia.org/wiki/%C3%89lection_pr%C3%A9sidentielle_fran%C3%A7aise_de_2012_%28sondages%29#Sondages_r.C3.A9cents) seem to be fairly consistent with the one which provoked this question." CreationDate="2011-04-18T02:28:03.500" UserId="2958" />
  <row Id="16420" PostId="9656" Score="1" Text="**First guess**: You're a C/C++/Java programmer used to zero-indexing. Matrices in $R$ aren't zero-indexed, but rather one-indexed. Also, you use the call `V(g)[i-1]` which the first time through the loop is `V(g)[-1]` and the second time is `V(g)[0]`, neither of which, I'm guessing, are what you actually intend to do." CreationDate="2011-04-18T02:30:06.687" UserId="2970" />
  <row Id="16421" PostId="9656" Text="Also, your call `graph.laplacian(g, normalized=TRUE)` returns the normalized graph Laplacian. In other words, if $D$ is a diagonal matrix of the degrees and $A$ is the adjacency matrix, then `graph.laplacian(g, normalized=TRUE)` returns $I - D^{-1/2} A D^{-1/2}$. You appear to be multiplying on the right and left by $D^{-1/2}$ again, which would yield $D^{-1} - D^{-1} A D^{-1}$. Is that *really* what you mean to calculate?" CreationDate="2011-04-18T02:33:55.073" UserId="2970" />
  <row Id="16425" PostId="9656" Text="@cardinal even if I try only the `graph.laplacian` I get almost the same results. Here the laplacian is  D-A, if it is normalized we will have 1 in the diagonal and `1/(sqrt(d[i])*sqrt(d[j])` for the other edges. Here I want to to do  D^1/2 * (D-A) * D^1/2" CreationDate="2011-04-18T03:35:21.720" UserId="2325" />
  <row Id="16426" PostId="9630" Text="(+1) Such a confidence interval would be useful and sensible. If a P value was wanted then, rather than using anything parametric, I would suggest a permutations test to give an exact value. (In fact I _did_ suggest it!)" CreationDate="2011-04-18T03:47:38.910" UserId="1679" />
  <row Id="16427" PostId="9661" Text="(meadian, not mode - cheers)" CreationDate="2011-04-18T04:13:00.410" UserId="253" />
  <row Id="16428" PostId="9656" Score="1" Text="Read the documentation carefully. There is a minus sign in front of `1/(sort(d[i])*sqrt(d[j]))`, from this it is clear that `graph.laplacian(g, normalized=TRUE)` already returns your desired quantity. You don't need to do any additional construction of `Dsqrt` or the `for`-loop, etc." CreationDate="2011-04-18T04:17:32.793" UserId="2970" />
  <row Id="16431" PostId="9656" Text="And, somehow, the first `sqrt` became `sort` above. Apologies." CreationDate="2011-04-18T04:42:41.997" UserId="2970" />
  <row Id="16432" PostId="9629" Text="I suspect you are asking about the &quot;goodness of fit&quot; test (using the chi square).  The use of which was explained bellow.  Cheers, Tal" CreationDate="2011-04-18T04:42:45.867" UserId="253" />
  <row Id="16433" PostId="9607" Text="@probabilityislogic I understand your point, now I will study exponential distribution and eventually build a new model based on it. thank you" CreationDate="2011-04-18T06:11:37.600" UserId="778" />
  <row Id="16436" PostId="9655" Text="(+1) great story about _aggressive cleanup_ and _type III error_, would be nice if this would be also relevant for economic time series. For deterministic models or models with low noise ratio small sample size IMHO won't be the greatest problem (compared to huge set of very noisy likely independent large sample data, even principal components are hard with these ones)." CreationDate="2011-04-18T06:57:31.967" UserId="2645" />
  <row Id="16437" PostId="9663" Text="You might want to look into this [book](http://books.google.co.il/books?id=dVP-RTea5wcC&amp;printsec=frontcover&amp;dq=%22A+First+Course+in+Order+Statistics%22&amp;source=bl&amp;ots=G4xgpvAAA4&amp;sig=OP2zEM--iAmYJoZPY4e8qF4T6o0&amp;hl=iw&amp;ei=_6o0TZvbHNOSjAeiiMG9Cg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=6&amp;ved=0CCwQ6AEwBQ#v=onepage&amp;q&amp;f=false)." CreationDate="2011-04-18T06:57:40.050" UserId="2116" />
  <row Id="16438" PostId="9663" Score="1" Text="@user4211, do you ask about distribution of minimum and maximum of any sample distribution, or only normal?" CreationDate="2011-04-18T06:58:32.233" UserId="2116" />
  <row Id="16439" PostId="9665" Text="+1, I've edited a small mistake in the second last formula." CreationDate="2011-04-18T06:59:46.567" UserId="2116" />
  <row Id="16440" PostId="9665" Text="@mpiktas: Thank you for the edit!" CreationDate="2011-04-18T07:01:09.110" UserId="3019" />
  <row Id="16441" PostId="9630" Text="@GaBorgulya, you've encountered the phenomenon of drive-by downvoting:) Do not worry, it happens. Note that sometimes people police their or other questions by downvoting the answers they feel are not apropriate.  In general your answer is useful, but in this particular case, I feel the OP wants the some kind of statistical test. The bootstrap might give you an idea, but in this particular case bootstrap is too crude as a solution." CreationDate="2011-04-18T07:11:16.367" UserId="2116" />
  <row Id="16442" PostId="9661" Text="+1 for the idea, but could you write down the actual test statistic?" CreationDate="2011-04-18T07:11:56.117" UserId="2116" />
  <row Id="16443" PostId="9664" Score="2" Text="+1, good question." CreationDate="2011-04-18T07:16:45.347" UserId="2116" />
  <row Id="16444" PostId="9655" Text="+1, the example is pure *uncontaminated* gold" CreationDate="2011-04-18T07:19:05.233" UserId="2116" />
  <row Id="16445" PostId="9653" Text="@even, I have an aversion to unnecessary mathematical notation, so I've edited the title, could you please check that I did not change the meaning by changing it?" CreationDate="2011-04-18T07:21:33.967" UserId="2116" />
  <row Id="16446" PostId="9666" Text="You can try running median for start." CreationDate="2011-04-18T07:36:51.120" UserId="88" />
  <row Id="16447" PostId="5430" Text="Package `seewave` has functions for finding peaks with a number of parameters to be tuned at your discretion." CreationDate="2011-04-18T08:17:39.793" UserId="144" />
  <row Id="16450" PostId="6994" Text="Why not gamma dispersion?" CreationDate="2011-04-18T09:14:23.873" UserId="1893" />
  <row Id="16453" PostId="9667" Score="1" Text="If you want the classical *theory*, it's hard to go wrong with Brockwell and Davis. " CreationDate="2011-04-18T09:26:26.253" UserId="2970" />
  <row Id="16455" PostId="9667" Score="1" Text="Possible duplicate of http://stats.stackexchange.com/questions/6275/good-introductions-to-time-series-with-r" CreationDate="2011-04-18T09:28:33.240" UserId="2116" />
  <row Id="16459" PostId="9667" Text="@mpiktas Let's keep it running here since the other question asked about applications with R." CreationDate="2011-04-18T09:39:51.383" UserId="930" />
  <row Id="16460" PostId="9667" Text="@chl, ok, this is one of those good duplicates. No worries, I just wanted an additional *moderated* opinion :)" CreationDate="2011-04-18T09:46:44.807" UserId="2116" />
  <row Id="16461" PostId="9674" Text="@user4212, what is your system locale? Sometimes you can have problems when working with one language but in different locale." CreationDate="2011-04-18T09:48:59.330" UserId="2116" />
  <row Id="16462" PostId="9674" Text="@user4212, the `txt_t` is undefined in your code. Can you include R output so we can see the error more clearly?" CreationDate="2011-04-18T09:51:49.043" UserId="2116" />
  <row Id="16463" PostId="9292" Text="Mistagged as mixed-models" CreationDate="2011-04-18T09:56:34.993" UserId="1893" />
  <row Id="16464" PostId="9674" Text="@user4212, I suspect that file  `russian.dat` is corrupted. Try asking package **tm** maintainers. " CreationDate="2011-04-18T09:57:17.337" UserId="2116" />
  <row Id="16465" PostId="9659" Score="2" Text="It sounds like the only reason you use beta is because its conjugate? uniform over the interval is fine if you have a reasonable amount of data ($\alpha=\beta=1$) - then you just have an incomplete beta function to evaluate, most software packages can do this for you.  Plus numerical integration is very quick because you have a short interval $[0.5,1]$ - uniform grid sampling would work well." CreationDate="2011-04-18T10:26:37.520" UserId="2392" />
  <row Id="16466" PostId="9658" Text="Sounds right to me - I usually think of compound poisson as a &quot;random sum&quot; - there are a &quot;random&quot; number of terms in the sum, and a &quot;random&quot; amount for each term within the sum.  Compound poisson comes from assuming the number of terms is poisson.  If you look up on insurance literature, you will probably find some good stuff." CreationDate="2011-04-18T10:32:10.523" UserId="2392" />
  <row Id="16468" PostId="3757" Text="you probably meant    print(NoCor &lt;- lmer(RT ~ Size + (0+Size|ID) +(1|Subject), data=rt)) as in this package example: A mixed-eﬀects model with independent random eﬀects&#xA;Linear mixed model fit by REML&#xA;Formula: Reaction ~ Days + (1 | Subject) + (0 + Days | Subject)&#xA;Data: sleepstudy; you should edit your post to fix your code." CreationDate="2011-04-18T10:48:27.507" UserId="1893" />
  <row Id="16469" PostId="9683" Text="Your idea is great and I will take it into consideration. But I tried also to make my question valuable for others by abstracting from my case and comparing normal with beta distribution. So I believe your answer is more a &quot;comment&quot; than a real answer. A moderator might explain if I am right or not. I just don't want to be selfish and adhere to the spirit of stackexchange..." CreationDate="2011-04-18T13:14:58.297" UserId="778" />
  <row Id="16470" PostId="9666" Text="@mbq Good point.  My first fear was that the median might be some outlier in between real modes, so I'll need some technique to detect that.  Also, given one mode, I wonder how best to identify the points that really are &quot;nearby&quot; that one, assuming a nice normal distribution.  Finally I could subtract that set out and iterate by going back to look for the median again." CreationDate="2011-04-18T13:47:34.573" UserId="2434" />
  <row Id="16471" PostId="9646" Text="Hi I used 10-fold stratified cross-validation. The software that I'm using have this option to train and test the model. Are you familiarized with this kind of cross-validation? " CreationDate="2011-04-18T14:54:59.857" UserId="4203" />
  <row Id="16473" PostId="8391" Text="@JMS Yep, that's exactly the point I was trying to provoke :-)  (I only wish I were as comfortable with, say, hypergeometric and theta functions as I am with trig and exponential functions...)" CreationDate="2011-04-18T15:24:14.163" UserId="919" />
  <row Id="16474" PostId="9674" Text="Wouldn't this question fare better on SO?" CreationDate="2011-04-18T15:29:33.380" UserId="919" />
  <row Id="16475" PostId="9681" Score="3" Text="Because this is a 1D, not multidimensional, integral, numerical methods (as suggested by @probabilityislogic) tend to be much faster and more accurate than Monte Carlo methods." CreationDate="2011-04-18T15:37:19.097" UserId="919" />
  <row Id="16476" PostId="9659" Score="1" Text="The indefinite integral can be computed exactly in terms of hypergeometric functions and gamma functions.  With integral values of $\alpha$, $\beta$, $n$, and $x$, this reduces to a polynomial in the limit of integration.  If you don't need to perform general analysis of the expressions, though, then numerical quadrature as suggested by @probability will be fast and effective.  It should be adaptive sampling, not uniform, though: the singularities at 1/2 and 1 for small values of $\alpha$ and $\beta$ might otherwise cause havoc." CreationDate="2011-04-18T15:43:31.197" UserId="919" />
  <row Id="16477" PostId="9688" Text="Graphical models may be similar to what you want: http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html" CreationDate="2011-04-18T15:45:02.103" UserId="3911" />
  <row Id="16478" PostId="9657" Text="Thanks for the detailed answer. I flagged the following points in it: (1) Quota sampling is used for Internet sampling in your example. Out of that context, there is no reason to use it over random sampling through RDD or equivalent techniques. (2) Even in the context you describe, it remains very controversial. So, to go back to my question, French pollsters justifying quota sampling for all their work are just being silly, or ignorant of sampling theory, or both." CreationDate="2011-04-18T15:51:45.430" UserId="3582" />
  <row Id="16479" PostId="9635" Text="For those of you who understand this topic, is it at all related to Characteristic Equations, as used with recurrence relations (i.e. in Knuth's Concrete Math)? My guess is that they're very different and only share the word &quot;characteristic&quot; by chance, but I thought I'd ask." CreationDate="2011-04-18T16:35:11.433" UserId="1764" />
  <row Id="16480" PostId="9689" Text="@bill_080: Thank you. This is more like a programming language, I would say. Of course you can *program* probability models with it, but it doesn't seem like an easy to use generator nor interpreter of probability formulas to me." CreationDate="2011-04-18T17:12:49.680" UserId="230" />
  <row Id="16481" PostId="9689" Text="@vonjd: R is a programming language, but it makes stats MUCH easier to understand (and ask questions). And, with the right book, learning R means you're learning statistics. Once you get rolling, you can also add Deducer http://www.deducer.org/pmwiki/pmwiki.php?n=Main.DeducerManual or Rcmdr http://en.wikipedia.org/wiki/R_Commander ....I honestly don't know of a better/faster way to learn statistics than with R." CreationDate="2011-04-18T17:40:25.160" UserId="2775" />
  <row Id="16482" PostId="9689" Text="@vonjd: I forgot to mention that R also has plenty of differential equation solvers.  I have used several packages and I can say that it is a lot easier to use R than typical C/Fortran/Basic versions of the same methods.   Here's soda http://svitsrv25.epfl.ch/R-doc/library/odesolve/html/lsoda.html   " CreationDate="2011-04-18T18:03:34.230" UserId="2775" />
  <row Id="16483" PostId="9677" Text="@Patrick, in your posted image, which is pred and new (ie true Y on the new data set)?" CreationDate="2011-04-18T18:42:39.393" UserId="2040" />
  <row Id="16485" PostId="9692" Text="@Brani You added the `spss` tag, but your syntax looks like you are looking for an R solution. Correct me if I am wrong..." CreationDate="2011-04-18T19:21:10.990" UserId="930" />
  <row Id="16486" PostId="9695" Text="Extreme is difficult to judge, every function approaches linear over some range; as we know from the Taylor Series decomposition. Why wouldn't Burnham and Anderson's information criterion approach to model selection  serve this problem well? " CreationDate="2011-04-18T19:22:27.657" UserId="1893" />
  <row Id="16487" PostId="9691" Score="1" Text="@penyuan You might find useful clues in this related question, [Multiple Chi-Squared Tests](http://stats.stackexchange.com/questions/1133/multiple-chi-squared-tests). Also, your question seems related to your previous one, [How to carry out multiple post-hoc chi-square tests on a 2 X 3 table?](http://stats.stackexchange.com/questions/9685/how-to-carry-out-multiple-post-hoc-chi-square-tests-on-a-2-x-3-table)." CreationDate="2011-04-18T19:23:29.317" UserId="930" />
  <row Id="16488" PostId="9646" Score="1" Text="I may well be missing something, but if you're splitting a sample of 44 cases into 10 subsamples, doesn't that leave only 4 or 5 per subsample?  If so, it's unfortunately a situation where the software makes something possible that doesn't have any practical worth.  Perhaps the procedure you're using is more like bootstrapping or jackknifing, where you select most of the 44 each time, sampling with replacement?" CreationDate="2011-04-18T19:31:39.463" UserId="2669" />
  <row Id="16489" PostId="9661" Text="@Michael - Can you elaborate? There are no labels; what would you permute?" CreationDate="2011-04-18T19:32:57.043" UserId="3432" />
  <row Id="16490" PostId="9697" Score="1" Text="@Patrick It would be interesting for the reader (and the OP) if you could provide details about the different R syntax (and the underlying tests) you gave, and especially how a Kruskal-Wallis test does compare to a log-linear model." CreationDate="2011-04-18T19:38:32.480" UserId="930" />
  <row Id="16491" PostId="9657" Text="&quot;French pollsters justifying quota sampling for all their work are just being silly, or ignorant of sampling theory, or both.&quot; Sounds about right to me." CreationDate="2011-04-18T19:40:34.750" UserId="4110" />
  <row Id="16492" PostId="9700" Text="+1: Wow, it worked right away! Thank you!" CreationDate="2011-04-18T19:42:36.050" UserId="230" />
  <row Id="16494" PostId="9700" Text="@vonjd I just checked: they are running R version 2.11.1. (Try `R.version()` at the command prompt.)" CreationDate="2011-04-18T19:45:36.763" UserId="930" />
  <row Id="16495" PostId="9699" Score="2" Text="A &quot;long installation procedure&quot;?  Only if you build from source.  There are binaries for most popular OS distributions and it takes &lt; 5 minutes to download and install them." CreationDate="2011-04-18T20:00:55.450" UserId="1657" />
  <row Id="16496" PostId="9699" Text="@Joshua: You are right - but &quot;long&quot; is relative, esp. when you compare it to just copy&amp;paste into a webinterface." CreationDate="2011-04-18T20:18:17.610" UserId="230" />
  <row Id="16497" PostId="9699" Score="5" Text="@vonjd: But you could have downloaded, installed, and run your script in less time than it took to write your question and wait for an answer. ;-)" CreationDate="2011-04-18T20:25:26.710" UserId="1657" />
  <row Id="16498" PostId="9646" Text="Mm is true... I'm only leaving 4 to 5 instances per subsample with this approach... it doesn't makes much sense. I'm going to check the bootstrapping and the jackkinifing to see what I can find. Thanks a lot for you answer! " CreationDate="2011-04-18T20:38:50.833" UserId="4203" />
  <row Id="16499" PostId="9699" Score="3" Text="@Joshua Good point.  But in some work environments it can take a lot of effort just to get the permission to install *any* executable on a machine.  And on Windows systems, most installations, even after a subsequent uninstall, litter the machine with detritus that accumulates over time and helps bog everything down, so if you aren't *sure* you're going to use some software there's incentive not to install it." CreationDate="2011-04-18T20:40:20.457" UserId="919" />
  <row Id="16500" PostId="9699" Text="@whuber: I was somewhat joking with my last comment.  I'm personally familiar with some of those constraints and I realize the benefits to a sandbox." CreationDate="2011-04-18T20:45:09.203" UserId="1657" />
  <row Id="16501" PostId="9711" Text="Thanks for looking, but these seem to be distribution-fitting applications rather than OLS regression." CreationDate="2011-04-18T21:08:36.733" UserId="919" />
  <row Id="16502" PostId="9659" Text="@probabilityislogic: if I can, I'd prefer to use something other than a uniform prior, since my prior is quite informative, and for speed, I'd like to observe as few actual binomial trials as possible. Beta seemed the natural alternative. But your answer induced me to consider a simple piece-wise uniform prior, which may have enough information for my purposes. " CreationDate="2011-04-18T22:04:34.313" UserId="4209" />
  <row Id="16504" PostId="9713" Text="That's great - thanks! How would I go about plotting the effects? I can see the summary tables from the aov() function but there doesn't seem to be a way to get plots out easily. I'm sure there is and I've missed it..!" CreationDate="2011-04-18T22:16:06.623" UserId="4204" />
  <row Id="16507" PostId="9619" Text="Thank You. It is a Survey Data and I am not sure if the it is MAR or MNAR. For Eg., there is a variable that says  1) &quot;whether a person has Diabetes or not?&quot; and another variable (2)Whether he is taking Insulin? I see that variable (2) has entries only when variable (1) is 'Yes' (i.e. a person is diabetic).Otherwise (2) is empty. Additionally (2) has 'yes', 'No', 'Don't know/Refused' as responses for the Diabetic case. So how do I treat the empty cells and 'Don't know/Refused' survey responses? &#xA;" CreationDate="2011-04-18T22:26:24.777" UserId="3897" />
  <row Id="16508" PostId="9619" Text="I would like to learn about multiple imputation and was looking for an online learning material. Could you suggest some learning material for MI?" CreationDate="2011-04-18T22:26:59.253" UserId="3897" />
  <row Id="16509" PostId="9641" Text="@Roman Luštrik : I tried your suggestion above, but got told that my chi square approximation might be incorrect, in fact the p-value is &quot;NA&quot;. What does that mean? I have appended the results to my original question. Thanks." CreationDate="2011-04-18T22:35:57.503" UserId="2830" />
  <row Id="16510" PostId="9661" Text="@lockedoff Whoops! I think that your implication is right: there is nothing to permute. I guess I didn't read the question well enough. The nearest thing to what I had in mind that would be possible is the bootstrap approach suggested by GaBogulya. Sorry." CreationDate="2011-04-18T22:42:07.527" UserId="1679" />
  <row Id="16511" PostId="9713" Text="No, from aov there is not an direct way to get plots out easy.  You need to get out your variables and plot them yourself.  I can see that you're trying ggplot2.  My answer was directed at what the error bar should be.  To get error bars on your plot you just need to add geom_errorbar() to your plot.  See the ggplot2 docs on that." CreationDate="2011-04-18T23:25:29.437" UserId="601" />
  <row Id="16512" PostId="9713" Text="An alternative is also to just use Mike Lawrence's ez package.  You should look at that." CreationDate="2011-04-18T23:26:03.603" UserId="601" />
  <row Id="16513" PostId="9715" Text="You can set the argument Hess=TRUE to get the Hessian back from multinom and then calculate the p-values manually. But I suggest you use the mlogit library (nnet can have convergence issues when the covariates are not scaled correctly). The vignettes for mlogit are pretty good, and should help you get your data set up correctly. The vignettes can be found at the usual place: http://cran.r-project.org/web/packages/mlogit/ " CreationDate="2011-04-18T23:26:50.677" UserId="3265" />
  <row Id="16514" PostId="9704" Text="I really appreciate you providing a code snippet, and I do have great respect for R, but I use it so infrequently that I forget the basics.  I eventually figured out that I need a library(strucchange) statement, but I'm still running into various odd errors with the format of clock.  Can you make this a complete example, even if with only a tiny amount of uninteresting data?" CreationDate="2011-04-18T23:27:32.403" UserId="2434" />
  <row Id="16515" PostId="9641" Score="1" Text="I suspect you're seeing this because of some low cell counts (some books I've read suggest a min. of 5 per cell). Maybe someone more knowledgeable on the subject can chip in?" CreationDate="2011-04-18T23:48:32.543" UserId="144" />
  <row Id="16516" PostId="9641" Score="1" Text="Also notice that you can get a p value if you make the last of your probability more than zero (but the warning still remains)." CreationDate="2011-04-18T23:50:59.890" UserId="144" />
  <row Id="16517" PostId="9641" Score="1" Text="Ott &amp; Longnecker (An introduction to statistical methods and data analysis, 5th edition) state, on page 504, that each cell should be at least five, to use the approximation comfortably." CreationDate="2011-04-18T23:55:54.907" UserId="144" />
  <row Id="16518" PostId="9641" Score="1" Text="@penyuan : You should've mentioned that you have quite some zero counts. Roman is right, using a Chi-square in this case just doesn't work for the reasons he mentioned." CreationDate="2011-04-19T00:40:09.283" UserId="1124" />
  <row Id="16519" PostId="9656" Text="Thank you, but I am still the same the problem, I think it is an approximation problem because all the negative values have at leat 9 zero after the period." CreationDate="2011-04-19T00:44:18.870" UserId="2325" />
  <row Id="16520" PostId="9656" Text="Are you now just doing `L &lt;- graph.laplacian(g, normalized=TRUE); eigen(L)$values` ? What is the size of $L$? Based on your description, it's quite possible it's some sort of numerical instability. Note that you're guaranteed to have at least one zero eigenvalue since $D^{1/2} \mathbf{1}$ is an eigenvector with corresponding eigenvalue of zero." CreationDate="2011-04-19T01:00:06.463" UserId="2970" />
  <row Id="16521" PostId="9699" Score="3" Text="You can put an entire Windows installation of R (and RStudio for ease of use) on a flash drive. Just copy the R folder. Then you can take it with you." CreationDate="2011-04-19T01:48:18.210" UserId="597" />
  <row Id="16522" PostId="9657" Text="Well, I am checking &quot;Answered&quot; on that one. Thanks for your time and detailed references." CreationDate="2011-04-19T02:51:23.113" UserId="3582" />
  <row Id="16523" PostId="9641" Text="@Joris Meys : is there an alternative to the chi square test in this case? Thanks." CreationDate="2011-04-19T04:20:38.790" UserId="2830" />
  <row Id="16524" PostId="9698" Text="Thank you so much. I really liked the use of expand.grid here. Is it ok to use this procedure in unbalanced datasets or when some cells are totally missing observations?" CreationDate="2011-04-19T06:11:50.863" UserId="632" />
  <row Id="16526" PostId="9682" Text="Thanks’ @aix. Source encoding `Koi8-u` displayed as `Iso8859-5`. I am changed `russian.dat` in `utf-8`. It's working." CreationDate="2011-04-19T06:17:03.253" UserId="4212" />
  <row Id="16527" PostId="9674" Text="Thanks all for help.&#xA;My system locale: `LC_COLLATE=Russian_Russia.1251  LC_CTYPE=Russian_Russia.1251` &#xA;&#xA;My output for control: `txt_cl[[1]] or txt_cl[[2]]`&#xA;&#xA;Yes. `txt_t` is undefined, correct - `c_txt`." CreationDate="2011-04-19T06:19:00.303" UserId="4212" />
  <row Id="16528" PostId="9693" Score="1" Text="I am pretty sure there has been question like this one already. However, you may have a look at papers to be published at AOS http://www.imstat.org/aos/ this should give you ideas of &quot;hot&quot; topics ? " CreationDate="2011-04-19T06:19:15.647" UserId="223" />
  <row Id="16529" PostId="9698" Text="Oh, I haven't used the effects package. Could you give me some tips on how it can be used here?" CreationDate="2011-04-19T06:21:05.213" UserId="632" />
  <row Id="16530" PostId="9722" Text="The behaviour of the order statistics bootstrap seems reasonable to me, given that the exponential distribution has a similar &quot;point mass&quot; at zero - The mode of an exponential distribution is 0, so it seems reasonable that the probability should be non-zero at the most likely value! The bootstrap would probably be something more like an geometric distribution which is a discrete analogue of the exponential. I wouldn't take this as a &quot;failure&quot; of the bootstrap here - for the estimated quantity of $\theta$ always lies in the appropriate interval $\theta\geq X_{(n)}$" CreationDate="2011-04-19T06:31:23.833" UserId="2392" />
  <row Id="16531" PostId="9725" Text="Because the question is about $p_2$ and $p_3$, not about trinomial probabilities *per se,* it really duplicates another question with several answers you will find useful.  If you have a question about the trinomial probabilities themselves, that could be worth starting a separate thread." CreationDate="2011-04-19T06:44:25.547" UserId="919" />
  <row Id="16532" PostId="9707" Text="@GaBorgulya: I tried to run an r script in a Sage webinterface but it didn't work - it said that there was an error creating a png (no X.11 available or sth.)" CreationDate="2011-04-19T07:46:00.617" UserId="230" />
  <row Id="16533" PostId="9564" Text="Well, I do not know how to fit the K-Means, which assumes the observations to be a d-dimensional real vector, to a co-occurance data. Also, since we're talking probabilities here, random initialization can be a bit of a pain, even tough I did end up using it. But doing some table testing with a dataset of just 5 observations did clear things up for me. Other than that, good general advices." CreationDate="2011-04-19T07:57:44.250" UserId="4141" />
  <row Id="16534" PostId="9678" Text="@probabilityislogic -1, I accidentally upvoted the answer earlier (blame Opera mini) so I had to edit it to be able to downvote, I am sorry for using such tactics. I did this only because I did not like the answer at first, but did not downvote because I wanted to prepare my arguments, which I will give in the following comment." CreationDate="2011-04-19T08:34:09.837" UserId="2116" />
  <row Id="16535" PostId="9678" Text="@probabilityislogic, for time-series processes the time plays important role, so distribution of vector $(X_t,X_{t+1})$ is different from $(X_{t+1},X_t)$. The resampling as done in naive bootstrap destroys this structure, so for example if you try to fit AR(1) model, after resampling you might get that you are trying to fit $Y_{10}$ as $\rho Y_{15}$, which is does not seem natural. If you google for &quot;bootstrapping time series&quot; [the second article](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.126.1402&amp;rep=rep1&amp;type=pdf) gives example of how estimate of variance of time series has..." CreationDate="2011-04-19T08:51:27.680" UserId="2116" />
  <row Id="16536" PostId="9710" Text="Sums of **equally** distributed bernoullis are binomial. They are slightly different in our case. We use $$p_i=p(1+i/N {\delta})$$ model now. But it should be some better model." CreationDate="2011-04-19T08:57:31.487" UserId="3670" />
  <row Id="16537" PostId="9678" Text="... wrong limit when naive bootstrap is applied (page 221). Furthermore ARIMA models do not have representation in trend and cyclical components as you have described. Usually ARIMA models are applied **after** trend and cyclical components are removed. In its standard definition ARIMA have constant mean, hence no trend. Also if the time series has a deterministic time trend, bootstrap it is more evident that naive bootstrap will fail. Pick $n$ points from a linear trend (i.e. line ) and shuffle them, the end result will definitely not look like the original line." CreationDate="2011-04-19T09:01:10.567" UserId="2116" />
  <row Id="16539" PostId="9728" Score="4" Text="@Zepti, please be more specific. Since you do not put any conditions on the model or what are you trying to predict. Take for example two independent random variables $X$ and $Y$, regression will not increase accuracy of prediction of $Y$ using $X$, since they are not related. But that is probably not the answer you are looking for." CreationDate="2011-04-19T09:34:19.513" UserId="2116" />
  <row Id="16540" PostId="9709" Score="2" Text="@nico And now there's [Rack](http://jeffreyhorner.tumblr.com/post/4723187316/introducing-rack)." CreationDate="2011-04-19T09:58:27.657" UserId="930" />
  <row Id="16541" PostId="9709" Text="@chl: cool, thanks for the link!" CreationDate="2011-04-19T10:03:28.163" UserId="582" />
  <row Id="16542" PostId="9678" Text="@mpiktas - thanks for the comments (at least you give reason for -1).  When you re-sample in a bootstrap, surely you would retain the time of the observation, as it contains important information.  In time series, the &quot;time&quot; is essentially a covariate.  So the situation you speak of with &quot;shuffling&quot; does not apply.  ARIMA models can be represented in the frequency domain, which is what the sine and cosine functions do, and they are of continuous time - so &quot;holes&quot; in the time series don't matter." CreationDate="2011-04-19T10:09:07.227" UserId="2392" />
  <row Id="16543" PostId="9678" Text="The reason why ARIMA models can be represented in frequency domain is that the ACF is a sufficient statistic for the model - because with MVN only the covariance matters - and the Fourier transform of the ACF is the equivalent &quot;frequency domain&quot; sufficient statistic.  Because Fourier transform is 1-to-1, the ARIMA model has an equivalent representation in terms of sine and cosine waves." CreationDate="2011-04-19T10:13:21.093" UserId="2392" />
  <row Id="16544" PostId="9678" Text="@probabilityislogic, I do not understand. You say that quote: &quot;when you re-sample in a bootstrap, you would retain the time of the observation&quot;, how can you do that? If we have the sample $X_1,...,X_n$, the naive bootstrapped sample is $X_1^*,...,X_n^*$, where each $X_i^*$ can be any of $X_1,...,X_n$, surely then the time of observation is lost? Furthermore I cannot agree that &quot;time&quot; is a covariate in time series. When we talk about covariates we speak about the iid sample of two variables, this is not the case for time series." CreationDate="2011-04-19T10:51:13.083" UserId="2116" />
  <row Id="16545" PostId="9678" Text="@mpiktas - time is a special covariate in a time series model - used in a particular way.  And in your data row, you would have $X$ in one column, and then the time $t$ in another column, plus other covariates in other columns.  How on earth could the time be lost???  You sample the whole row - just because you have relabelled $X^{*}_{1}$ as &quot;point one&quot; does not mean that it comes from time $t_{1}$.  It could well be that $X^{*}_{1}=\{X_{7},t_{7}\}$.  The notation $X_{i}$ is inappropriate in time series, because it obscures that time is important.  $\{x_{i},t_{i}\}$ is much better" CreationDate="2011-04-19T11:13:45.330" UserId="2392" />
  <row Id="16546" PostId="9678" Text="what you are describing in time series is analogous in OLS regression to bootstrapping $Y_{i}$ and $X_{i}$ independently." CreationDate="2011-04-19T11:14:53.160" UserId="2392" />
  <row Id="16548" PostId="9729" Text="@lee - the ME should be $1.645$ and not $0.82$" CreationDate="2011-04-19T11:45:06.737" UserId="2392" />
  <row Id="16552" PostId="9722" Text="@cardinal - the asymptotic distribution is not the appropriate benchmark - unless you have an infinite sample.  The bootstrap distribution should be compared to the finite sample distribution that it was designed to approximate.  What you want to show is that as number of bootstrap iterations goes to infinity, the bootstrap distribution converges to the *finite sampling distribution*.  letting $n\to\infty$ is an approximate solution not an exact one." CreationDate="2011-04-19T13:36:51.627" UserId="2392" />
  <row Id="16553" PostId="9678" Text="@probabilityislogic, ok so assume that we have a sample of vectors $(X_1,1),(X_2,2),(X_3,3)$, which comes from AR(1) model $Y_t=\rho Y_{t-1}+\varepsilon_t$. Suppose we estimate $\rho$ using linear regression without the intercept. For the given sample the estimate will be $(X_2X_1+X_3X_2)/(X_1^2+X_2^2)$. Suppose now we have a bootstrap sample $(X_1,1),(X_1,1),(X_3,3)$, how would one define the the estimate? Since time structure is kept, I should take it into account, but how? I cannot pair $(X_3,3)$ with $(X_1,1)$, since then I will estimate AR(2) model $Y_t=\alpha Y_{t-2}+u_t$. " CreationDate="2011-04-19T13:38:48.577" UserId="2116" />
  <row Id="16554" PostId="9678" Score="2" Text="@probabilityislogic, would it be possible for you to demonstrate your idea in your answer for naive bootstrap estimate of $\rho$ in AR(1) model $Y_t=\rho Y_{t-1}+u_t$? I do not think that it is possible, hence the basic reason for downvote. I would be glad to get proven wrong." CreationDate="2011-04-19T13:41:33.567" UserId="2116" />
  <row Id="16555" PostId="9722" Score="2" Text="@cardinal +1, I've upvoted the question earlier, but I just want to thank for a very good answer, examples and links to the articles." CreationDate="2011-04-19T13:45:08.990" UserId="2116" />
  <row Id="16556" PostId="9678" Text="No because the AR(1) model is recursive, so you have $Y_{t}=\rho(\rho Y_{t-2}+\epsilon_{t-1})+\epsilon_{t}$." CreationDate="2011-04-19T13:45:11.743" UserId="2392" />
  <row Id="16557" PostId="9722" Text="@probabilityislogic, of course in general application of asymptotic theory depends on the convergence rate, if it is slow, then it is not applicable. But you have then to demonstrate that the rate is slow, since I suspect that for example with uniform distribution taking sample size 100 you will encounter the problems @cardinal outlined." CreationDate="2011-04-19T13:48:33.870" UserId="2116" />
  <row Id="16558" PostId="9722" Text="@probabilityislogic, it's easy to see that your convergence statement is **never** true (modulo, perhaps, some trivial scenarios). This is because there are &quot;only&quot; $n^n$ possible bootstrap samples that could be drawn (and that's even considering the ordering; many statistics of interest are permutation invariant). So fixing $n$ and taking the number of bootstrap iterations $B\to\infty$ is not a useful sense of consistency. " CreationDate="2011-04-19T13:51:33.507" UserId="2970" />
  <row Id="16559" PostId="9678" Score="1" Text="@probabilityislogic, and? What will be the estimate of $rho$ in that case? I am sorry for pestering, but I genuinely do not see how can you show that naive bootstrap will not fail in this case." CreationDate="2011-04-19T14:00:27.317" UserId="2116" />
  <row Id="16560" PostId="9722" Text="@mpiktas, thanks for the kind remarks. " CreationDate="2011-04-19T14:04:34.363" UserId="2970" />
  <row Id="16561" PostId="9740" Text="+1 extrapolation and prediction intervals is an excellent point!  For regression of one scalar onto another, plotting the prediction intervals ought to be standard operating procedure." CreationDate="2011-04-19T14:17:20.057" UserId="887" />
  <row Id="16562" PostId="9661" Text="@Michael Makes sense ;)" CreationDate="2011-04-19T14:27:57.937" UserId="3432" />
  <row Id="16563" PostId="9739" Score="2" Text="You should check out http://cran.r-project.org/web/views/Spatial.html . There are a few packages that have cluster right in the name that would probably be of interest." CreationDate="2011-04-19T14:57:22.010" UserId="1036" />
  <row Id="16564" PostId="9678" Text="Seems like in the stationary case, the estimate is $0$. the likelihood is normal with mean $X_{3}-\rho^{2}X_{1}$ and variance $1+\rho^{2}$ - always has local mode at zero.  But the likelihood is explosive - diverges outside $\pm 1$.  Prob due to integration over $\epsilon_2$" CreationDate="2011-04-19T15:06:37.290" UserId="2392" />
  <row Id="16565" PostId="9308" Text="Any idea how to allow for &quot;finer&quot; tuning via two parameters? Where to put second parameter?" CreationDate="2011-04-19T15:19:15.423" UserId="333" />
  <row Id="16566" PostId="9736" Score="1" Text="Can you clarify what you mean by &quot;predetermined correlation&quot;? If you're thinking of the cross-correlation between two time series, it is a function of the lag; do you want the correlation between any two lagged inputs to be the same at 0.7? This doesn't seem reasonable." CreationDate="2011-04-19T15:39:19.533" UserId="781" />
  <row Id="16567" PostId="9738" Score="2" Text="Check out the comments and links available in this thread: http://r.789695.n4.nabble.com/allocMatrix-limits-td864864.html" CreationDate="2011-04-19T15:40:07.223" UserId="3265" />
  <row Id="16568" PostId="9746" Text="I thought that I have fair understanding of those concepts until now. Could you please give an example. &quot;How example, if you try to predict such and such using such and such, then such and such coefficients will be biased because of ...&quot; Thanks in advance" CreationDate="2011-04-19T15:56:41.757" UserId="333" />
  <row Id="16569" PostId="9672" Text="These are classical / good references for econometric time series, but I wouldn't recommend them for a general approach or a first learner. I'm especially not a fan of the Hamilton book: I purchased it because it seemed comprehensive, but it's not particularly well written. " CreationDate="2011-04-19T15:58:11.103" UserId="781" />
  <row Id="16570" PostId="9746" Text="also... when you say auto-correlations, do you mean auto-correlations of the variable that I want to predict or predictors which I use to do it? Many thanks" CreationDate="2011-04-19T15:58:33.253" UserId="333" />
  <row Id="16571" PostId="9713" Text="Great, thanks! I have spent the day getting to grips with ggplot2. It's not been all that easy - but I'm getting there :) Thanks! I'll take a look at the ez package as well." CreationDate="2011-04-19T15:59:41.897" UserId="4204" />
  <row Id="16572" PostId="9748" Text="Do you have two values for each week, (1) a continuous outcome measure and (2) a yes/no predictor?" CreationDate="2011-04-19T16:03:14.510" UserId="3911" />
  <row Id="16573" PostId="9308" Score="1" Text="@user333 yes. What about $f_{k,c}(x)=\frac{1}{1+exp(-k*x+c)}$ ? More generalized:  $f(x)=\frac{1}{1+exp(g(x))}$ where g(x) is any function you like. " CreationDate="2011-04-19T16:04:16.043" UserId="264" />
  <row Id="16574" PostId="9746" Text="The assumption in Stambaugh bis is that your predictor is an autoregressive process. The inputs into your regressive routine would then be lagged versions of this process. " CreationDate="2011-04-19T16:05:22.513" UserId="781" />
  <row Id="16575" PostId="9747" Text="+1. Brockwell &amp; Davis is a really strong text for general time series. " CreationDate="2011-04-19T16:14:36.920" UserId="1118" />
  <row Id="16576" PostId="9746" Text="OK. So if I want to use variable which is AR process (current value can be explained with linear combination of last couple of values)...why does that imply that inputs in my regression model are correlated? I have only one input in my regression model?" CreationDate="2011-04-19T16:18:42.527" UserId="333" />
  <row Id="16577" PostId="9746" Text="How having AR process as a predictor relates to the variable that I want to predict? That's the bit I'm missing." CreationDate="2011-04-19T16:20:02.807" UserId="333" />
  <row Id="16578" PostId="9731" Score="2" Text="If you don't find an answer in the existing threads on multiple testing for correlations, could you please explain how this question is different?  To find them, link on the &quot;multiple-comparisons&quot; tab and then add &quot;correlation&quot; to the search terms in the textbox at the upper right corner of the page.  You should get 8 results." CreationDate="2011-04-19T16:37:58.537" UserId="919" />
  <row Id="16580" PostId="9748" Score="1" Text="But aren't the values within each group autocorrelated and hence you have less information over the entire sample than you would if the data were independent?" CreationDate="2011-04-19T17:07:46.653" UserId="1390" />
  <row Id="16582" PostId="9748" Text="I'm not good enough to know the t-test answer to your question, but one thing to think about is: is the event &quot;random&quot; or scheduled? If it's scheduled, all kinds of interaction possibilities come to mind, including things that happen in the same week or things that happen the week before or after." CreationDate="2011-04-19T17:25:59.493" UserId="1764" />
  <row Id="16583" PostId="9749" Text="What were your &quot;unexpected results&quot; and did you do ${9 \choose 2} = 36$ pairwise comparisons? I would not be surprised by non-significant differences if so. You can model the effects of year in the binomial glm, e.g., if you wanted to just test for linear (or quadratic or whatever) components. It would be helpful if you indicated why you wanted multiple comparisons." CreationDate="2011-04-19T17:35:03.493" UserId="3432" />
  <row Id="16585" PostId="9749" Text="	&#xA;Thanks for your remarks @lockedoff. I would expect that yes, I did that many pairwise comparisons (I assume that's how the code works), however my data from the last 2 years graphically show such a dramatic decrease in clutch size, so I anticipated being able to see this statistically as well, but the 2006 data showed no difference when graphically, it should have the most! I wanted all the multiple comparisons to make sure I wasn't overlooking any other less-obvious differences in the clutch sizes. Here's a link to a jpg of the graph: [bit.ly/eJV2fK]. Hope this makes sense. Thanks!" CreationDate="2011-04-19T17:54:18.430" UserId="4238" />
  <row Id="16586" PostId="9443" Text="This package has methods built in for part of what I was looking for" CreationDate="2011-04-19T18:02:12.240" UserId="776" />
  <row Id="16587" PostId="9755" Text="For the first version of this project, I don't want to consider demographic or physical location.  What I want to look at is the transactional data for each customer.  For each customer, I have purchase history which includes the merchant where they made the purchase and amount.  I have grouped merchants into merchant types and I will be examining features that are counts of purchases at a merchant type over the life of the customer and dollar amount at a merchant type over the life of the customer." CreationDate="2011-04-19T18:14:37.330" UserId="4235" />
  <row Id="16588" PostId="9754" Text="So from multiple linear regression you end up with weight vector.  I am assuming that a larger weight means a stronger feature.  What I am hoping to find are some key drivers that help maximize the customer loyalty score.  " CreationDate="2011-04-19T18:21:05.123" UserId="4235" />
  <row Id="16589" PostId="9672" Text="@Wesley, note that I only claim that I found these books useful. Concerning Hamilton, it is a huge book, and probably some chapters are not good. On the other hand chapters on non-stationarity I found very good. In general I use this book as a reference, these type of books can be overwhelming at first, but in the end they are most useful, that is why I included it." CreationDate="2011-04-19T18:28:56.477" UserId="2116" />
  <row Id="16590" PostId="9749" Text="Can you paste your code for `model2`? I'm not familiar with your field, but I'm wondering why the binomial distribution is appropriate for clutch size (= number of eggs laid in one egg-laying period -- right?), instead of Poisson." CreationDate="2011-04-19T18:33:52.030" UserId="3432" />
  <row Id="16591" PostId="9754" Text="Re weights: if feature A and B are both predictive, but strongly correlated they may have relative low weights (weight shared between them). Re drivers: from observations you can only get associations, not cause-effect relationships; however your knowledge on the mechanisms may help you decide what are causes, effects or just correlates among the associations." CreationDate="2011-04-19T18:34:35.597" UserId="3911" />
  <row Id="16592" PostId="9759" Text="I can elaborate on the work I did in my MS thesis if needed, also." CreationDate="2011-04-19T18:46:53.780" UserId="4237" />
  <row Id="16593" PostId="9728" Score="2" Text="&quot;Not increase accuracy of prediction?&quot; Compared to what alternative?" CreationDate="2011-04-19T19:06:01.793" UserId="603" />
  <row Id="16594" PostId="9747" Text="+1, although I cannot understand disdain for econometrics. The OP mentions regression, for which I do not think spectral methods are useful. And I am curious about non-standard notation, could you elaborate more on that?" CreationDate="2011-04-19T20:25:45.407" UserId="2116" />
  <row Id="16595" PostId="9762" Score="4" Text="Normal distribution of the data *cannot possibly* be required for cluster analysis, because by definition a cluster would be a concentration of values that are distinctly *non* normal!  You might be thinking of some methods that expect the *variations in data relative to their cluster centers* to be *approximately* normal (after the fact)." CreationDate="2011-04-19T21:43:32.540" UserId="919" />
  <row Id="16596" PostId="9641" Score="1" Text="@penyuan : I added an answer giving you some options." CreationDate="2011-04-19T22:22:50.160" UserId="1124" />
  <row Id="16597" PostId="9747" Text="I'm coming at time series from an applied perspective, so I started as an engineer, then moved into statistics. All the (and by no means has my exploration been comprehensive) books and papers I've read that came at time series from an econometrics point of view have been 'different'. I'm not sure I can describe it any better, but there's always subtle differences in notation or approaches that don't jive with mainstream statistical time series analysis. It's not that one is better than the other, but simply that for best overall coverage, general approaches win." CreationDate="2011-04-19T22:26:03.423" UserId="781" />
  <row Id="16598" PostId="9731" Text="Thanks! I am new at this forum, and do as you suggest! Anyway, again...thanks for humble answer!" CreationDate="2011-04-19T22:30:44.327" UserId="4229" />
  <row Id="16599" PostId="9641" Text="@Joris Meys: Got it, thanks!" CreationDate="2011-04-19T23:11:37.473" UserId="2830" />
  <row Id="16600" PostId="8055" Text="Bump... I still cannot understand how to do this. Any takers on answering the original post?" CreationDate="2011-04-19T23:31:46.087" UserId="3645" />
  <row Id="16602" PostId="9722" Text="@cardinal - using your logic I could reject the use of any continuous distribution because we only have a finite number of observations.  Perhaps we would both be help by you producing a numerical example demonstrating that the bootstrap &quot;fails&quot; in the case of the maximum - I honestly don't think it will.  But happy to be proved wrong :)" CreationDate="2011-04-20T00:01:38.320" UserId="2392" />
  <row Id="16603" PostId="9678" Text="@mpiktas - the example you are describing should fail here with bootstrap because the sample size is small - just like my answer said.  If the time series was longer a more reasonable answer would be found." CreationDate="2011-04-20T00:06:13.713" UserId="2392" />
  <row Id="16604" PostId="9627" Text="If you were to tell us the purpose of this modeling, you might get even more helpful suggestions.  Multiple regression deals quite well with correlated independent variables, as long as they aren't too highly correlated, resulting in multicolinearity.  As others have said, assessing causality is difficult (but not impossible) outside of a randomized experiment.  See some of these links: http://www.delicious.com/MichaelBishop/causality for more on that topic." CreationDate="2011-04-20T00:10:14.400" UserId="3748" />
  <row Id="16605" PostId="9749" Text="Yes, I should have mentioned that part @lockedoff: this species only lays 1 or 2 eggs, so the poisson model wasn't working. I converted by 1-egg nests to 0s and the 2-egg nests to 1s. Here the code I used:&#xA;&#xA;`model2&lt;-glm(bin_clutchsize~bin_Year,binomial)`&#xA;&#xA;(Thanks @chi, for solving my account problem!)&#xA;" CreationDate="2011-04-20T00:22:56.083" UserId="4238" />
  <row Id="16606" PostId="9722" Text="I'll try to post an example in a day or so. But, I think you're likely misunderstanding. The logic is that the coverage implied by the bootstrap distribution is wrong, **even** asymptotically. That leaves little hope for it to work in finite samples. You might reread the second-to-last paragraph of that section, which should make it clear that the bootstrap can behave **arbitrarily badly** in terms of the approximation of the sampling distribution." CreationDate="2011-04-20T01:56:31.693" UserId="2970" />
  <row Id="16607" PostId="9722" Text="@probabilityislogic, One final note: Hopefully it is clear that this is not *my* logic, but rather that of the collective statistical community with 30+ years of studying the topic. Cheers." CreationDate="2011-04-20T01:57:13.127" UserId="2970" />
  <row Id="16608" PostId="9627" Text="The data itself is a series of network (graph) models of a particular neurological state. I'm measuring the &quot;clustering coefficient&quot; which describes the topology of each network as a whole (dependent variable here), and then seeing if the individual connectivities of four nodes in the larger 100+ network are driving the global clustering values (four independent variables). However, these nodes are part of a network, so sort of by definition it's possible they're correlated to some extent." CreationDate="2011-04-20T04:27:40.630" UserId="4139" />
  <row Id="16609" PostId="9632" Text="+1 for the classic examples and mentioning collinearity " CreationDate="2011-04-20T04:30:06.213" UserId="4139" />
  <row Id="16610" PostId="9720" Text="+1 for the links to the handbook, mentioning collinearity and IVF, and the specific solutions and even implementations in R. I am curious to hear your opinion about whether the data itself is not suited to regression analysis- I edited the question above to reflect that these are measurements of a network." CreationDate="2011-04-20T04:39:36.187" UserId="4139" />
  <row Id="16611" PostId="9739" Text="Thanks Andy, I'm gonna look and check it out" CreationDate="2011-04-20T06:57:38.543" UserId="4147" />
  <row Id="16614" PostId="9732" Text="Thanks a tonne, however the approach is only limited to 2 conditions? see the edited question above" CreationDate="2011-04-20T07:15:41.307" UserId="201" />
  <row Id="16615" PostId="6994" Score="1" Text="@Patrick you can definitely do that. But since I am taking log of the mean I prefer normal disp effect. A log normal distribution is another way to modeling distributions similar to gamma distribution. HTH." CreationDate="2011-04-20T07:54:54.767" UserId="1307" />
  <row Id="16616" PostId="9732" Text="I have edited my answer" CreationDate="2011-04-20T08:19:01.630" UserId="3019" />
  <row Id="16617" PostId="9750" Text="Thanks for your quick reply. The relation between the sequence of weeks and my event is a good point." CreationDate="2011-04-20T08:27:28.187" UserId="4233" />
  <row Id="16620" PostId="9771" Text="@mpiktas, I guess the author is interested in empirical tricks, though more context on the homework task would be great, if for instance I'm mistaking :) Anyway these are nice additions for the 3rd part from my list (+1)" CreationDate="2011-04-20T10:07:55.117" UserId="2645" />
  <row Id="16621" PostId="9766" Score="1" Text="@Beatrice, what does about zero mean? Generally when speaking of moments, the tail-behaviour is much more important than values around zero, so it is confusing to see both of the concepts used in the same sentence. Note the both the answers provided are about n-th moments, with no other conditions attached." CreationDate="2011-04-20T10:14:53.087" UserId="2116" />
  <row Id="16622" PostId="9774" Text="@Nil, I've changed the title so that statistical problem is immediately apparent. If you (or anyone else) think that the original meaning was lost, please change it back." CreationDate="2011-04-20T10:23:28.880" UserId="2116" />
  <row Id="16623" PostId="9763" Text="I am guessing that matrix $Q$ is stochastic, hence the maximal eigenvalue is 1. Since $Q$ elements are correlations, the second eigenvalue will be a maximum correlation in analogue to principal components, where the squared eigenvalue corresponds to the variance of principal component, which in turn is the linear combination of the columns of the matrix, or something to that effect." CreationDate="2011-04-20T10:29:14.927" UserId="2116" />
  <row Id="16624" PostId="9772" Score="3" Text="for recursive sequences function `filter` looks nicer than the loop" CreationDate="2011-04-20T10:31:07.233" UserId="2116" />
  <row Id="16625" PostId="9742" Text="@Andy Thank you for your formula based foundation of my gut feeling ;). Two questions: 1. What do you mean by &quot;(it is actually a good argument to prolong ..&quot; ? Do you mean this: By prolonging the test one gets more observations for a) special days and hence can identify corresponding effects or b) normal days and hence special days become less important ? 2. Where can I learn more about this science ? Is this the area called &quot;Design of Experiments&quot; ?" CreationDate="2011-04-20T11:46:02.803" UserId="264" />
  <row Id="16627" PostId="9774" Text="@Nil Do you mean profit or revenue ? If you mean profit there should be no additional costs. If you mean revenue, I see no problem on reporting back to a decision layer whether one group outperforms the other one significantly (regarding revenue). Calculating the costs (which includes you performing the analysis ;)) is not your job, isn't it ?" CreationDate="2011-04-20T12:05:56.063" UserId="264" />
  <row Id="16628" PostId="9775" Text="@Biorelated, if you have a function which checks for the pattern in the row, simple `apply` will work. Are you looking for something more complicated?" CreationDate="2011-04-20T12:13:08.323" UserId="2116" />
  <row Id="16629" PostId="9730" Score="1" Text="Look at the `cut` function" CreationDate="2011-04-20T12:22:20.450" UserId="46" />
  <row Id="16630" PostId="9743" Text="@Joris, You might reconsider your first statement, which I do not believe is correct. If $p_i = 0$ for some $i$ and the observed counts are zero (which they better be), then this just reduces to a submodel. The effect is that the number of degrees of freedom is reduced by one for each $i$ such that $p_i = 0$. For example, consider testing for uniformity of a six-sided die (that is $p_i = 1/6$ for $i \leq 6$). But, suppose we (strangely) decide to record the number of times that the numbers $1,\ldots,10$ show up. Then, the chi-square test is still valid; we just sum over the first six values." CreationDate="2011-04-20T12:24:22.493" UserId="2970" />
  <row Id="16631" PostId="9774" Text="Thanks for your response. I would traditionally do a test on either variance on reponse generated or revenue (which does not include additional costs) but have been asked to do tests on profit, which does include all the additional costs, as these costs are variable depending on what each group receives i find that the tests can already be biased before we start. Hope that makes sense?" CreationDate="2011-04-20T12:26:56.730" UserId="4248" />
  <row Id="16632" PostId="9775" Text="Hi mpiktas, i do not have such a function unfortunately.It would be much easier. My challenge is actually designing that function given that the pattern occurs across the data columns as opposed to within column." CreationDate="2011-04-20T12:30:31.487" UserId="201" />
  <row Id="16633" PostId="9742" Text="@steffen , for the first question it is not that special days become less important, it is that you have observations independent of those special day effects. Hence you observe the process in different settings so you know the observed treatment effect is not due to those special days. Similar to what svadali stated about generalizing the findings, if you only run the experiment on blue days you don't know if the results would be the same on normal days.  " CreationDate="2011-04-20T12:38:05.100" UserId="1036" />
  <row Id="16634" PostId="9722" Text="@cardinal - the reason I say the logic is a bit odd is that you first say $X_{(n)}\to\theta$ in probability, but then say that it is &quot;bad&quot; if we asymptotically have a point mass at $X_{(n)}$ in the bootstrap.  Doesn't that mean that we effectively have a point mass at the true value (or arbitrarily close to that value)?  How is that &quot;bad behavior&quot;?  Perhaps we are speaking of different things - you say its bad because the sampling distribution is &quot;wrong&quot; and I say &quot;who cares because we get the correct value of $\theta$ asymptotically&quot;." CreationDate="2011-04-20T12:43:35.157" UserId="2392" />
  <row Id="16635" PostId="9722" Text="@cardinal - and by the same argument, in order to make the interval arbitrarily small you need to make the sample arbitrarily large - meaning that the convergence in probability of $X_{(n)}$ will get arbitrarily better" CreationDate="2011-04-20T12:45:10.210" UserId="2392" />
  <row Id="16636" PostId="9742" Text="For the second question its difficult to recommend readings not knowing more about your work. My heart is close to this book, http://books.google.com/books?id=o7jaAAAAMAAJ&amp;q , but perhaps a book more focused on typical experimental designs is more appropriate (the above link is for a widely known book that mainly focuses on non-experimental research designs). The AB test as far as I can tell is no different than a factorial experiment with only one factor. " CreationDate="2011-04-20T12:45:30.147" UserId="1036" />
  <row Id="16637" PostId="9778" Score="2" Text="Perhaps moving window type clustering (for example SaTScan, http://www.satscan.org/ ) would meet your requirements. That specific program is for spatial/temporal data, so not really meant for higher dimensions, but maybe will give you some ideas or a place to start." CreationDate="2011-04-20T12:49:42.250" UserId="1036" />
  <row Id="16639" PostId="9722" Text="@probabilityislogic, No, the interval $[0,\varepsilon)$ stays fixed. The value of $\theta$ stays fixed, but, is large. The distribution of $n(\theta-X_{(n)})$ is nondegenerate. So, fix $\varepsilon &gt; 0$ and $0 &lt; \delta &lt; 1$. Then, there exists $\theta \equiv \theta(\varepsilon,\delta)$ such that $1-\exp(-\varepsilon/\theta) &lt; \delta$. Consider an i.i.d. sample from $\mathcal{U}[0,\theta]$. Then, the bootstrap distribution has mass $&gt; 1-e^{-1}$ in the interval $[0,\varepsilon)$, but the true limiting distribution has mass $&lt; \delta$." CreationDate="2011-04-20T12:56:06.333" UserId="2970" />
  <row Id="16640" PostId="9775" Score="1" Text="@Biorelated - perhaps a bit kludgy, but you could always collapse the columns together in a new column and then use `grepl` or the likes on the new column: `apply(data, 1, function(x) paste(x[1], x[2], x[3], x[4], sep = &quot;&quot;))`" CreationDate="2011-04-20T12:56:37.037" UserId="696" />
  <row Id="16641" PostId="9730" Text="@Biorelated Your last code is not working since you've messed up parentheses; it should be always `ifelse(cond,ifTrue,ifFalse)`, never `ifelse(cond) ifTrue, ifFalse`." CreationDate="2011-04-20T12:59:36.907" UserId="88" />
  <row Id="16642" PostId="9722" Text="@probabilityislogic, at first, I only saw the the latter of your two most recent comments. To address the former, you can see the first two sentences of the section above with heading &quot;What it means for the bootstrap to 'fail'&quot;, where this is addressed explicitly. The bootstrap is not about estimating the parameter. We assume we have a good way to estimate the desired parameter (in this case, $X_{(n)}$ works fine). The bootstrap is about knowing something about the *distribution* of the parameter so that we can do inference. Here, the bootstrap gets the distribution (**very!**) wrong." CreationDate="2011-04-20T13:06:32.943" UserId="2970" />
  <row Id="16643" PostId="9730" Text="Thanks! noted so.. :) " CreationDate="2011-04-20T13:08:48.803" UserId="201" />
  <row Id="16644" PostId="9743" Text="@cardinal : I just described the data, where the expected value is 0 but the observed doesn't have to be. It's what OP gave us (although on second thought it does indeed sound rather irrealistic). Hence adding a little bit to the p value to make it highly improbable instead of impossible will help, but even then the chi-square is in this case invalid due to the large amount of table cells with counts less than 5 (as demonstrated by the code). I added the consideration in my answer, thx for the pointer." CreationDate="2011-04-20T13:10:24.213" UserId="1124" />
  <row Id="16645" PostId="9722" Text="...and that should be &quot;... *distribution* of the parameter *estimate*...&quot; above. Sorry about dropping a word midstream." CreationDate="2011-04-20T13:36:42.617" UserId="2970" />
  <row Id="16646" PostId="9743" Text="@Joris, yes, I'd say if $p_i = 0$, but you observe a count for that cell, then you've got more serious problems on your hands, anyways. :)" CreationDate="2011-04-20T13:40:10.943" UserId="2970" />
  <row Id="16647" PostId="9786" Score="1" Text="@mala Are these two matched samples? What is the scale of measurement (e.g., ng/ml for Vitamin D level)?" CreationDate="2011-04-20T13:54:47.797" UserId="930" />
  <row Id="16649" PostId="9763" Text="@mpiktas Almost.  Actually, the rhs is in the form $P \cdot P^T$ where $P$ is stochastic.  This is needed to make $Q$ positive definite.  Now its maximum eigenvalue usually *exceeds* unity but its second one does not--and is guaranteed to lie between 0 and 1.  $Q$ really *is* a covariance matrix with a constant added to each term." CreationDate="2011-04-20T14:00:27.353" UserId="919" />
  <row Id="16650" PostId="9736" Score="1" Text="If you want to create a multivariate Gaussian with a covariance matrix $\Gamma$, find $A$ such that $A^T A=\Gamma$ using e.g. Cholesky decomposition.  Then if $Z$ is a vector of independent standard normals, $AZ$ has the desired covariance." CreationDate="2011-04-20T14:14:11.417" UserId="3567" />
  <row Id="16651" PostId="9731" Score="3" Text="@user4229 I'll second @whuber's remark and think you'll find useful clues about MT issues with correlation measures, if not an exact answer to most of your questions here. Now, if you're really interested in discussing FDR vs. single-step methods (such as the Bonferroni method), FP vs. FN issues in semi-exploratory studies, or multiple testing when dealing with correlated predictors, then I would suggest you rewrite your question in that particular direction, making this question a less obvious duplicate of some of the earlier threads." CreationDate="2011-04-20T14:20:26.557" UserId="930" />
  <row Id="16652" PostId="9780" Text="@mpiktas +1 and let me link to an older response of yours here, http://stats.stackexchange.com/questions/6005/an-r-function-for-performing-searches, because this particular question looks &quot;closely related&quot;." CreationDate="2011-04-20T14:35:07.200" UserId="930" />
  <row Id="16653" PostId="9774" Text="@Nil No, why should they. If the experiment is randomized (i.e. customers have selected randomly for mailing A or B and not dependent on costs), then the costs should have no effect." CreationDate="2011-04-20T14:35:27.980" UserId="264" />
  <row Id="16654" PostId="9780" Text="@chl, thanks. I've already forgot that one :)" CreationDate="2011-04-20T14:38:20.823" UserId="2116" />
  <row Id="16655" PostId="9776" Text="In order to use a Kalman filter you should have the model of your dynamical system (i.e. a system of differential equations) plus the measurements equations. Are they available? Are they linear?" CreationDate="2011-04-20T14:39:04.537" UserId="339" />
  <row Id="16656" PostId="9776" Text="No thats all the data I have" CreationDate="2011-04-20T14:47:34.250" UserId="4251" />
  <row Id="16657" PostId="9781" Score="2" Text="(+1) Welcome to the site, which I'm glad you've found. I look forward to your participation here. " CreationDate="2011-04-20T14:52:10.207" UserId="2970" />
  <row Id="16658" PostId="9781" Text="(+1) Good point about the Wilcoxon." CreationDate="2011-04-20T14:56:52.623" UserId="919" />
  <row Id="16659" PostId="9782" Text="This is exactly what I was looking for – I just didn't realize Somers generalized to ordinal Y. (I've used your `somers2()` function from the HMisc package, but it only accepts binary Y.) Thank you!" CreationDate="2011-04-20T15:33:53.847" UserId="1611" />
  <row Id="16660" PostId="9480" Text="@CHL Thank you so much for the detailed response. I wanted to ask, is there a way to make vertical confidence intervals at each level of the independent variable? Is there a way to remove the background and revert to 'old style' graph?" CreationDate="2011-04-20T16:08:19.990" UserId="1084" />
  <row Id="16661" PostId="9775" Text="Thanks but, The problem with that approach is that the columns are fixed. the search is to be defined by yet unknown pattern and the idea is to search for rows with such a pattern." CreationDate="2011-04-20T16:12:07.153" UserId="201" />
  <row Id="16662" PostId="9789" Text="The selection is completely randomised, but the costs associated to each group may be different. So even if there was not any difference in response or spend we would report a significant difference which is purely becasue one group had higher costs associated to it, which doesn't help us identify if there was any change in behaviour." CreationDate="2011-04-20T16:33:31.463" UserId="4248" />
  <row Id="16663" PostId="9789" Score="1" Text="The start of this reply looks good but I got lost in the second half, perhaps due to some lapses of grammar and the long sentence.  Could you clarify this part?" CreationDate="2011-04-20T16:39:50.533" UserId="919" />
  <row Id="16664" PostId="9789" Text="@Nil : exactly! Wouldn't you want to know that you can achieve the same result at less cost?" CreationDate="2011-04-20T16:50:48.893" UserId="279" />
  <row Id="16665" PostId="9793" Text="+1, just a note though that due to law of iterated logarithm the growth of unit root process is not exponential." CreationDate="2011-04-20T16:55:35.027" UserId="2116" />
  <row Id="16666" PostId="9736" Text="If what is desired is a contemporaneous correlation of 0.7, then the answer by charles.y.zhen is all you need, I think. (Normality of $Z$ is not required, so the answer is good in more general conditions than stated.) " CreationDate="2011-04-20T17:04:53.750" UserId="892" />
  <row Id="16667" PostId="9797" Score="1" Text="It appears there is a strong dependence in these data: namely, is it the case that if obs=1 for subject $i$ on day $t$ then necessarily obs=1 for subject $i$ on day $s$ whenever $s \ge t$?  If this is so, then you really have only four data values--one for each subject--and one of them is censored on the right." CreationDate="2011-04-20T18:13:24.680" UserId="919" />
  <row Id="16668" PostId="9797" Text="@whuber you are correct about the dependence (at least in the present within-year analysis); the data represent whether or not 'bud burst' has occurred prior to the observation date for each of four replicate trees. But I am not sure what you mean about on of the data values being censored on the right? " CreationDate="2011-04-20T18:19:09.817" UserId="1381" />
  <row Id="16669" PostId="9797" Score="1" Text="Here's a summary: subject 2 transitioned in the interval [1,8]; that is, 2--&gt;[1,8].  Also 3--&gt;[8,16], 4--&gt;[16,24], and 1--&gt;[24, infinity].  The latter means subject 1 was observed for 24 days without transitioning; it's the censored value.  You can frame this as a survival analysis problem and analyze it accordingly.  Incidentally, this dependence means the p-values in the logistic regression are misleadingly low." CreationDate="2011-04-20T18:24:44.670" UserId="919" />
  <row Id="16670" PostId="9797" Text="@whuber thank you for the insight, but does this mean that my approach if fundamentally flawed given that I am not interested in estimating p-values? Also, none of the data will be right-censored in a few weeks; I am just developing the analysis before the dataset is complete. I have altered the test data so that none of the subjects are right censored." CreationDate="2011-04-20T18:37:45.847" UserId="1381" />
  <row Id="16671" PostId="9797" Text="OK.  But you still have to cope with the interval-valued observations.  You can use [interval arithmetic](http://www.ramas.com/intstats.pdf) techniques to find a median (and compute a C.I. for it)." CreationDate="2011-04-20T18:43:06.883" UserId="919" />
  <row Id="16672" PostId="9795" Score="1" Text="Why do you perceive &quot;skewness&quot; as a problem that needs correction?" CreationDate="2011-04-20T18:50:56.097" UserId="919" />
  <row Id="16673" PostId="9797" Text="Your model is all wrong. You have repeated observation that are not being accounted for. As far as time goes you have only 5 sequential observations on only 4 subjects. You can say that the transition of 50% of the subject occurred on or before the 16th day. " CreationDate="2011-04-20T18:55:56.017" UserId="2129" />
  <row Id="16674" PostId="9797" Text="@DWin that is why I ask how I can account for repeated measures in the question. Are you suggesting that I can't make a stronger inference than '50% transitioned before day 16'?" CreationDate="2011-04-20T19:04:53.743" UserId="1381" />
  <row Id="16675" PostId="9802" Text="Awesome, I was looking for exactly that type of pdf. Thanks a ton!" CreationDate="2011-04-20T19:21:34.327" UserId="2091" />
  <row Id="16676" PostId="9803" Text="@GaBorgulya I think you have a typo; median(95%CI) = 16(5,28)" CreationDate="2011-04-20T19:45:16.860" UserId="1381" />
  <row Id="16677" PostId="9803" Text="@David Corrected, thanks." CreationDate="2011-04-20T19:58:36.387" UserId="3911" />
  <row Id="16678" PostId="9797" Text="You will probably find the [UseR! 2010 tutorial on Interval censored data analysis](http://user2010.org/tutorials/Fay.html) useful. The link to the slides is at the bottom of the page." CreationDate="2011-04-20T20:03:17.630" UserId="3911" />
  <row Id="16679" PostId="9480" Score="1" Text="@Adam I updated my response with the case of 2 categorical variables + a continuous response variable --  hope this is what you meant. I also added code to show how to customize `ggplot` theme. Generally, you can say `gp + theme_bw()` to just remove the grey background; here, I also removed the grid." CreationDate="2011-04-20T20:05:20.187" UserId="930" />
  <row Id="16680" PostId="9698" Text="@Brani (1) R remove statistical units with missing values before estimating the effects; compare the above results (especially the DF for the residuals) with `summary(lm1 &lt;- lm(y~A+B, data=df[-sample(1:nrow(df), 10),]))`. Of course, it is important to know whether data are missing at random or not, but see [Treatment of Missing Data](http://bit.ly/gISQy8) by Howell, for instance. (2) The `effects` package is useful to display predictions for complex models (e.g., incl. interaction). Here, you can try `plot(effect(term=&quot;A&quot;, mod=lm1))`, or browse the on-line, `help(effect)` for examples of use." CreationDate="2011-04-20T20:20:10.183" UserId="930" />
  <row Id="16681" PostId="9797" Score="3" Text="@DWin, @David This is *not* a repeated measurement situation.  The data format only makes it look like that.  The measurement for each subject consists of a single interval during which a transition was observed." CreationDate="2011-04-20T21:32:43.900" UserId="919" />
  <row Id="16682" PostId="9803" Text="You would do better with an ML fit of a reasonable distributional form to the interval data followed by an estimate of the median of the distribution." CreationDate="2011-04-20T21:33:11.543" UserId="919" />
  <row Id="16683" PostId="9802" Text="@Alexx If you like this answer, accept it." CreationDate="2011-04-20T21:39:49.863" UserId="88" />
  <row Id="16684" PostId="9803" Text="@whuber The &quot;reasonable distribution&quot; is the key question itself." CreationDate="2011-04-20T21:44:07.410" UserId="3911" />
  <row Id="16685" PostId="9803" Text="I agree.  It occurs to me that there must be nonparametric approaches, such as kernel smooths, that work with interval-valued data." CreationDate="2011-04-20T21:51:09.827" UserId="919" />
  <row Id="16686" PostId="9803" Score="3" Text="With four independent data points it is impossible to obtain a 95% CI for the median: no matter what the true median is, the four points will all be greater than it with probability $1/2^4$ = 6.25% and they will all be less than it with the same probability.  Therefore the range of the four observations *must* fail to cover the median 12.5% of the time.  This makes me suspicious of your approach." CreationDate="2011-04-20T21:54:59.073" UserId="919" />
  <row Id="16687" PostId="9806" Text="Nice idea! -- But could you perhaps explain how to obtain CIs for the median from a life table?" CreationDate="2011-04-20T21:56:49.230" UserId="919" />
  <row Id="16688" PostId="9798" Text="This is an excellent response; thank you very much for the elaboration.  I just found another reference for that book today, and I will certainly track it down now that I know I'll need NLME." CreationDate="2011-04-20T22:22:29.613" UserId="4237" />
  <row Id="16689" PostId="9760" Text="Thanks as well.  Charlie's comment was useful.  Cheers." CreationDate="2011-04-20T22:24:23.557" UserId="4237" />
  <row Id="16692" PostId="9807" Score="1" Text="You should check out the resources on the SPSS tag wiki, http://stats.stackexchange.com/tags/spss/info . The article, Programming and Data Management for IBM Statistics would be a good start to answer your questions." CreationDate="2011-04-20T23:06:53.470" UserId="1036" />
  <row Id="16693" PostId="9327" Text="@ils Thank you, I have tried that and updated the question. I wonder what is the difference between your suggestion, and using `lmer(cbind(sp_a, sp_b)~elevation*dryness ... )` which is a format used at the bottom of the lmer help page." CreationDate="2011-04-20T23:38:24.383" UserId="2993" />
  <row Id="16694" PostId="9803" Text="@whuber Thanks for pointing this out. Either more data or more knowledge/assumptions are needed to have a meaningful result. (I put a remark into my answer.)" CreationDate="2011-04-20T23:41:19.247" UserId="3911" />
  <row Id="16695" PostId="9807" Score="1" Text="Usually independent questions are asked separately, so you have a better chance to get answers of you ask them one by one. You will find that for some there have been  very similar questions already answered." CreationDate="2011-04-20T23:58:22.907" UserId="3911" />
  <row Id="16696" PostId="5396" Text="I assume you have found the answers yourself in the meantime?" CreationDate="2011-04-20T23:58:24.590" UserId="3760" />
  <row Id="16697" PostId="9810" Score="2" Text="Do you want a global test for the three groups with the null hypothesis that the three medians are equal? Or do you want pairwise comparisons? All the tree pairs (control-prior, control-after, prior-after)? Will you draw the same conclusion if one or an other pairwise comparison shows a difference?" CreationDate="2011-04-21T00:06:49.153" UserId="3911" />
  <row Id="16698" PostId="9797" Text="@whuber: you are saying that no measurement is made until a transition occurs. I disagree. Just because the state is _measured_ as zero does not mean that no measurement was made." CreationDate="2011-04-21T00:20:20.853" UserId="2129" />
  <row Id="16699" PostId="9797" Text="@DWin Think about it: you watch the tree.  You can record its state every week, every day, every nanosecond if you like.  That frequency is irrelevant.  What matters is the time at which a single, irreversible event (the bud burst) occurs.  My second comment (above) shows a straightforward way to record the full dataset, demonstrating there is one interval-valued observation per experimental unit (the trees)." CreationDate="2011-04-21T00:34:41.047" UserId="919" />
  <row Id="16700" PostId="9797" Text="If your point is that in some experimental or observation there can be only one transition per subject then I agree. This might be one of htose, although the OP provided rather skimpy background. If the transition can go both ways then I would not accept your formulation." CreationDate="2011-04-21T01:47:51.980" UserId="2129" />
  <row Id="16701" PostId="9807" Score="1" Text="In SPSS you can go to Help...Tutorial or Help...Index (and once you've chosen a topic, click Show Me to get the tutorial for that topic).  A couple of hours spent on this should answer most of your questions in a visual and/or verbal way." CreationDate="2011-04-21T01:52:55.847" UserId="2669" />
  <row Id="16702" PostId="9795" Text="This question sounds rather similar (http://stats.stackexchange.com/questions/9398/supervised-learning-with-rare-events-when-rarity-is-due-to-the-large-number-of) and Dikran gave a good answer to it." CreationDate="2011-04-21T05:57:58.413" UserId="264" />
  <row Id="16703" PostId="9809" Score="1" Text="@brianjd, calculating expectations almost always involves integration. Unfortunately there are no universal simple rules how to calculate integrals." CreationDate="2011-04-21T06:54:53.390" UserId="2116" />
  <row Id="16704" PostId="9802" Text="Oh, I thought I could do it myself now, but I can't. Can you tell me if my steps are right or if I should take &quot;another way&quot; to solve this?" CreationDate="2011-04-21T08:12:37.693" UserId="2091" />
  <row Id="16705" PostId="9802" Text="@Alexx Hardt: My first equation in the edit is the same as your very last equation in the particular case where p = 2. So, you can mimic my calculations for components 3, 4, ..., p. " CreationDate="2011-04-21T08:20:38.627" UserId="3019" />
  <row Id="16706" PostId="9816" Text="Ace, thanks a lot." CreationDate="2011-04-21T10:01:17.387" UserId="2425" />
  <row Id="16707" PostId="9796" Text="Thanks a lot, that's useful." CreationDate="2011-04-21T10:01:45.357" UserId="2425" />
  <row Id="16708" PostId="9802" Text="Thanks again :) I think I'll actually use all three suggestions. I'm building a .pdf which explains and sums up basic stats matrix algebra, because I somehow never wanted to learn it when I learned it in my classes. To solve it with three different ways will help me understand it better, I hope." CreationDate="2011-04-21T10:38:30.990" UserId="2091" />
  <row Id="16710" PostId="9327" Text="@J. Winchester: What is `in_out` in the new data frame? I feel that the variables are somewhat blindly &quot;thrown&quot; into the model, so you need to clarify the question and add the terms into the model according to the aim of the analysis. I am not sure about the differences in using `lmer` as noted in the help file and as suggested, though." CreationDate="2011-04-21T11:07:21.507" UserId="3467" />
  <row Id="16711" PostId="9802" Text="Oh, but this is for p=2 and n=2, right? I'll write it down with n=3 I think." CreationDate="2011-04-21T11:12:08.320" UserId="2091" />
  <row Id="16712" PostId="9793" Text="@mpiktas, indeed you are right. In my opinion, from the context of question follows the assumption for random walk _with drift_. Moreover to have exponential _explosion_ we also need the original time series process to be logarithmically transformed (or by some power Box-Cox type of transformation)." CreationDate="2011-04-21T11:16:29.047" UserId="2645" />
  <row Id="16713" PostId="9327" Text="@J. Winchester: I also don't see the `dryness` variable included in the model." CreationDate="2011-04-21T11:32:34.333" UserId="3467" />
  <row Id="16714" PostId="9802" Text="@Alexx Hardt: Yes, n=2 as well in my answer. You can also have a look at http://www.amazon.com/Matrix-Algebra-Useful-Statistics-Probability/dp/0471866814. This book is a standard in statistics I think." CreationDate="2011-04-21T11:36:48.823" UserId="3019" />
  <row Id="16715" PostId="9813" Score="2" Text="@Durrett, According to your examples you are averaging $Z_{I_{k-1}+1},\dots,Z_{I_k}$. May be you could add some story behind the formal setup? And what do you want to do with it ? Possible tasks would be to disaggregate $Z$ from $X$ or at least to estimate the distribution it comes from, but since $X$s distributions are the convolutions of $Z$'s distribution, it won't be easy. " CreationDate="2011-04-21T11:36:53.560" UserId="2645" />
  <row Id="16716" PostId="9802" Text="(+1) Well and clearly done." CreationDate="2011-04-21T11:43:11.010" UserId="919" />
  <row Id="16717" PostId="9815" Text="+1 mpiktas: Your solution is more ingenious than mine and I think it should be used in more complex practical situations." CreationDate="2011-04-21T11:44:04.687" UserId="3019" />
  <row Id="16718" PostId="9815" Text="@ocram, thanks. I would not call it ingenious, it is a standard application of existing formulas. You just need to know the formulas :)" CreationDate="2011-04-21T11:52:42.747" UserId="2116" />
  <row Id="16719" PostId="9801" Text="@GaBorgulya, thanks for the edit, did not know about `smallmatrix`, so did not try to edit, since usual solution of breaking the formula in several lines would not have worked here." CreationDate="2011-04-21T11:57:48.277" UserId="2116" />
  <row Id="16720" PostId="9817" Text="What do you mean by &quot;number of free parameters of G&quot;? Do you mean the number of parameters in the sampling distribution of the data for a particular G?" CreationDate="2011-04-21T12:54:56.237" UserId="495" />
  <row Id="16721" PostId="9817" Text="I feel obligated to tell you that learning the optimal structure of a Bayesian Network (which is a DAG) is in NP (http://en.wikipedia.org/wiki/NP_%28complexity%29) and hence not feasible. Does this question has any statistical background / aim or it is &quot;just&quot; a exercise in Combinatorics ?" CreationDate="2011-04-21T13:10:52.993" UserId="264" />
  <row Id="16722" PostId="9817" Text="@steffen: It is certainly feasible if the number of vertices p is relatively small (p&lt;6 can be done by simple enumeration, and I think some of the more recent optimisation algorithms work for p&lt;50). With a few restrictions on the type of graphs (say, limiting the number of parents of any vertex), you can easily step it up to even larger graphs." CreationDate="2011-04-21T13:21:33.750" UserId="495" />
  <row Id="16723" PostId="9817" Text="@Simon With the introduction of restrictions, how can you be sure that the OPTIMAL structure will be found ? Nevertheless, I am interested in such optimisation algorithms ;) ... can you provide a link ?" CreationDate="2011-04-21T13:27:49.440" UserId="264" />
  <row Id="16724" PostId="9817" Text="@steffen: Well, it would be would be optimal from the class you consider. How do you know that any DAG is an appropriate model? (all models being wrong etc.) As far as algorithms go, the PC algorithm and its variants seems to be what all the cool kids are using these days." CreationDate="2011-04-21T13:41:45.760" UserId="495" />
  <row Id="16725" PostId="9821" Text="_hidden_ variables usually are not observed, so, I think, the author do not have $Z$ values." CreationDate="2011-04-21T13:42:12.377" UserId="2645" />
  <row Id="16726" PostId="9813" Text="@Dmitrij, You're right - I fixed the typo. I also added a story and background." CreationDate="2011-04-21T13:51:56.610" UserId="3891" />
  <row Id="16727" PostId="9821" Text="@user4272 - Indeed, as Dmitrij said, I do not know the $Z$. I actually do know the $I$'s for each observation. I clarified this in the question." CreationDate="2011-04-21T13:55:52.770" UserId="3891" />
  <row Id="16728" PostId="9824" Text="Fortunately the forum rules allow adding +1 to every answer. Thanks for the education, guys!" CreationDate="2011-04-21T14:17:56.447" UserId="2129" />
  <row Id="16729" PostId="9824" Text="@DWin, did you mean to post this under the comments to the question?" CreationDate="2011-04-21T14:20:41.007" UserId="2970" />
  <row Id="16730" PostId="9822" Text="I've included some R code by way of explanation. So you're saying I'm not applying the wiki formula correctly? Presumably by assigning the probabilities, no?" CreationDate="2011-04-21T14:20:43.013" UserId="3306" />
  <row Id="16731" PostId="9823" Text="So my error is not re-upping during losing years and taking profits on winning years? " CreationDate="2011-04-21T14:23:24.507" UserId="3306" />
  <row Id="16732" PostId="9822" Text="Now I see: you did not divide your initial amount into three parts (that is what the Wikipedia formula would be for), the three returns are just the returns each year. In your case, the formula is simply   ((100(1+0.25))(1-0.4))(1+0.25). There is no probability involved in this way." CreationDate="2011-04-21T14:29:35.250" UserId="4257" />
  <row Id="16733" PostId="9817" Text="@steffen: The question does have a statistical background and I am using a greedy optimization algorithm that starts with an empty graph and then adds/reverses/deletes edges according to the BIC.  I am also limiting the number of parents of vertices, as Simon mentioned.  The number of vertices I'm currently working with is relatively small (&lt;20), but in the future I will extend it to &gt; 200." CreationDate="2011-04-21T14:31:44.127" UserId="3595" />
  <row Id="16734" PostId="9827" Text="@GaBorgulya Thank you!  I just couldn't figure out how to change the lm command." CreationDate="2011-04-21T14:35:32.137" UserId="4274" />
  <row Id="16735" PostId="9817" Text="@Simon - Sorry for the lack of clarity.  I need the number of free parameters of G for inclusion in the BIC.  The number of nodes is fixed, but the number of edges can vary arbitrarily (with optional limits on the number of parents).  I could think of a few different reasonable ways to consider the number of free parameters (free parameters = # edges seems to be a weak criteria, free parameters = 1 parameter (for the number of edges) + # possible locations for these edges tends to work better), but do not know the mathematically correct model." CreationDate="2011-04-21T14:37:15.260" UserId="3595" />
  <row Id="16736" PostId="9742" Text="@Andy Thanks again !" CreationDate="2011-04-21T14:37:29.317" UserId="264" />
  <row Id="16737" PostId="9817" Text="@hammer:I understand that, but you said you were fitting by maximum likelihood: what is your likelihood?" CreationDate="2011-04-21T14:41:22.153" UserId="495" />
  <row Id="16738" PostId="9771" Text="Thanks, this was exactly what I was looking for." CreationDate="2011-04-21T14:56:51.503" UserId="3062" />
  <row Id="16739" PostId="9829" Score="1" Text="(1) I have a funny feeling that this is homework. (2) I see no question to the community. but wait, ahh, now I see. You want us to do your homework. I dont think so..." CreationDate="2011-04-21T15:16:07.847" UserId="442" />
  <row Id="16740" PostId="9829" Text="No Henrik- wanting some guidance- a different thing." CreationDate="2011-04-21T15:19:02.150" UserId="4276" />
  <row Id="16741" PostId="9829" Score="1" Text="Then, what did you try to solve this problem? At which point do you need some guidance? Do you have an idea, where to start?" CreationDate="2011-04-21T15:21:05.393" UserId="442" />
  <row Id="16742" PostId="9824" Text="I suppose I could have. I had sequentially gone through the question and then all the answers (after the processing of the MathML stopped jerking around) and found each of the answers informative. I just dropped my comment on yours because it was where I stopped reading." CreationDate="2011-04-21T15:23:02.850" UserId="2129" />
  <row Id="16743" PostId="9829" Text="OK- should I approach this from using a hypergeometric formula?" CreationDate="2011-04-21T15:25:08.417" UserId="4276" />
  <row Id="16744" PostId="9829" Text="Which formal tests for statistical significance do you know? Which of them do you think could in principal answer this question?" CreationDate="2011-04-21T15:27:23.237" UserId="442" />
  <row Id="16745" PostId="9824" Score="1" Text="@DWin, yes, the rendering is a bit funky. I thought you might have intended the comment for another post since this one has no votes (up or down) and so the comment seemed to be out of place. Cheers. " CreationDate="2011-04-21T15:30:22.233" UserId="2970" />
  <row Id="16746" PostId="9829" Text="Thats's just it.  We have covered so many formulas that they are beginning to all run together.  I apologize, that while I have enjoyed learnng this subject matter, it is incredibly difficult." CreationDate="2011-04-21T15:30:28.740" UserId="4276" />
  <row Id="16747" PostId="9829" Score="1" Text="@Victor : go back to your course, and check which tests allow you to compare continuous,paired values with 3 explanatory categorical variables. Think interactions, random effects,... And please, don't just copy-paste the assignment as a question. " CreationDate="2011-04-21T15:36:28.347" UserId="1124" />
  <row Id="16748" PostId="9829" Text="Joris- thank you.  This is actually very helpful.  It rings a bell and I have some direction. Thank you." CreationDate="2011-04-21T15:42:19.837" UserId="4276" />
  <row Id="16749" PostId="8784" Score="2" Text="Starting a bounty - seems like an important question with three very different answers, none of which has *a single vote*.  We can do better.  Andy W's [paper link](http://dx.doi.org/10.1093/esr/jcp006) on [this related question](http://stats.stackexchange.com/questions/8718/comparing-logistic-regression-coefficients-across-models/) seems pertinent." CreationDate="2011-04-21T16:09:51.997" UserId="71" />
  <row Id="16750" PostId="9824" Text="Just more interface funkiness I guess. I did click it and it did turn color but it didn't &quot;take&quot;. Fixed (I hope)." CreationDate="2011-04-21T16:21:23.813" UserId="2129" />
  <row Id="16751" PostId="9824" Text="@DWin: I see. Yes, I believe the interface may limit the number of votes you can submit within a certain time span. So, if you went through and voted on each in rapid succession, it may have simply blocked later ones. That's probably to keep people from writing programs to automate voting. " CreationDate="2011-04-21T16:36:27.603" UserId="2970" />
  <row Id="16752" PostId="9813" Text="$\forall k&gt;1$ :), for the first subset it was ok. I liked the story, because I may think of many similar cases for micro level data in (socio)economics (firms, households, individuals) that may be given as in your scheme, though I would go for equal subsets (and have a feeling that further analysis will be much easier in this case)." CreationDate="2011-04-21T17:12:03.907" UserId="2645" />
  <row Id="16753" PostId="9735" Text="In the frequentist viewpoint, Original either beats Combination, or it doesn't.  There is no &quot;chance&quot; or probability involved." CreationDate="2011-04-21T17:45:15.253" UserId="3567" />
  <row Id="16754" PostId="9837" Text="@ocram- Thanks for the clarification. It seems the method requires some intensive computations, maximizing the profile likelihood. Just wondering why not simply resort to the bootstrap method if the estimator is not normally distributed. " CreationDate="2011-04-21T17:46:04.037" UserId="2264" />
  <row Id="16755" PostId="9735" Text="@charles.y.zheng hm ... you can calculate the power of a test i.e. the probability that the Null-Hypothesis is rejected assuming the true parameters. How would you call that ? " CreationDate="2011-04-21T18:34:37.600" UserId="264" />
  <row Id="16756" PostId="9823" Text="@Milk It's not that simple.  Suppose you invest 100 dollars. After year 1, you take 25 out, leaving 100.  After year 2, you have lost 40, so you put the $25 back in *plus* another 15.  After year 3, you have 125.  *In toto* you were forced to have 115 in cash (**not** 100), on which you made only 10 after 3 years.  All this assumes you cannot earn interest on the unused cash sitting around and you cannot borrow money.  This only begins to hint at the mistakes that will be made when using oversimplified formulas for expectation.  Reading a good text on financial calculus will serve you well." CreationDate="2011-04-21T18:48:30.383" UserId="919" />
  <row Id="16757" PostId="9823" Text="@Milk An excellent starting point is *Stochastic Calculus for Finance; Volume I: The Binomial Asset Pricing Models* by Steven E. Shreve, Springer-Verlag, New York, 2005." CreationDate="2011-04-21T18:50:51.520" UserId="919" />
  <row Id="16758" PostId="9836" Score="2" Text="this question was asked on [SO](http://stackoverflow.com/questions/5681630/market-mix-modeling-with-r). Coincidentally, the book referenced [there](http://www.amazon.com/gp/search/ref=sr_nr_n_0?rh=n%3A283155%2Ck%3Amarket+mix+modeling%2Cn%3A!1000%2Cn%3A3&amp;bbn=1000&amp;keywords=market+mix+modeling&amp;ie=UTF8&amp;qid=1302897164&amp;rnid=1000) is still applicable here. " CreationDate="2011-04-21T18:51:03.873" UserId="696" />
  <row Id="16759" PostId="9841" Text="It's not possible to apply logistic regression correctly to these data because the dependent variable is not binary (and, when it is forced to look binary, it induces strong dependencies among the records.)  See the comments at http://stats.stackexchange.com/questions/9797/how-can-i-use-logistic-regression-to-estimate-the-time-at-which-50-of-a-binomial ." CreationDate="2011-04-21T18:53:01.347" UserId="919" />
  <row Id="16760" PostId="9810" Text="The reference group are not &quot;paired&quot; to the first two groups. That is the problem.  In that case I guess Friedman would have been a good test." CreationDate="2011-04-21T18:54:40.293" UserId="4229" />
  <row Id="16761" PostId="9841" Text="Do you mean that essentially you have two data points: the day of pupation in 2005 and the day of pupation in 2006? Not much statistics can be done on two values. Or is there something more hidden in the ...?" CreationDate="2011-04-21T19:42:45.043" UserId="279" />
  <row Id="16762" PostId="9841" Text="@whuber thank you for your suggestion.  In my attempt to simplify the biology of the question I think I misrepresented the system.  I have edited the question to reflect the change.  In reality we are looking at whether a pupa that forms on a given day is or is not in diapause.  This seems to resolve the non-binary problem you mention and since all of the butterflies have pupated by the end of the experiment then there would not be the right censored problem.  thanks again" CreationDate="2011-04-21T19:49:59.553" UserId="4048" />
  <row Id="16763" PostId="9841" Text="@Aniko, I have edited the question to hopefully make the system clearer.  In each year we have observations for ~50 individuals: the day of pupation and whether the pupa is in diapause.  Thanks." CreationDate="2011-04-21T19:53:11.017" UserId="4048" />
  <row Id="16764" PostId="9841" Text="I don't see any individuals in your data. Does one animal keeps moving from being in diapause to not and back? " CreationDate="2011-04-21T19:57:15.023" UserId="279" />
  <row Id="16765" PostId="9841" Text="@Aniko, each row is a different individual.  I have edited the question make that clearer " CreationDate="2011-04-21T19:59:49.573" UserId="4048" />
  <row Id="16766" PostId="9841" Text="@Kenny Thank you for the clarification!  Could you also indicate how these individuals were selected?  Of possible importance are whether they are identified prospectively (and then followed through observation of pupation) and whether they can be viewed as an independent random sample of a population.  In other words, information about your experimental design would be welcome and useful." CreationDate="2011-04-21T20:01:06.763" UserId="919" />
  <row Id="16767" PostId="9842" Text="Please note that NNs are rather... obsolete. " CreationDate="2011-04-21T20:09:55.017" UserId="88" />
  <row Id="16768" PostId="9810" Text="The pairing structure is clear. But please tell more details - otherwise the answers have to consider many ways that may or may not be appropriate in your setting." CreationDate="2011-04-21T20:12:50.963" UserId="3911" />
  <row Id="16769" PostId="9841" Text="@whuber yes, the individuals can be viewed as a random sample of a wild population.  I have edited the question to reflect this.  Thanks" CreationDate="2011-04-21T20:13:13.627" UserId="4048" />
  <row Id="16770" PostId="9844" Text="@mpiktas Sorry, simultaneous edit :-|" CreationDate="2011-04-21T20:14:30.413" UserId="88" />
  <row Id="16771" PostId="9844" Text="@bit-question, I've latexified your question, but it is clear that definition for $K$ is wrong. What is $(x,y)$?" CreationDate="2011-04-21T20:15:27.313" UserId="2116" />
  <row Id="16772" PostId="9844" Text="@mbq, $(x|y)$ is physics notation for scalar product?" CreationDate="2011-04-21T20:16:22.617" UserId="2116" />
  <row Id="16773" PostId="9844" Text="@mpiktas Rather $\langle x\mid y\rangle$... I thought this is the math one (-;" CreationDate="2011-04-21T20:18:09.350" UserId="88" />
  <row Id="16774" PostId="9844" Text="@mbq, I'll revert to my version and change definition of $K$, since you left some of the gramatical errors :p :)" CreationDate="2011-04-21T20:18:12.707" UserId="2116" />
  <row Id="16775" PostId="9843" Text="@IrishStat, thank you for the answer. I don't want to post the data yet since I am doing this as a class assignment. Give me pointers to articles/papers that I can read and implement for now. I will be posting the data when I am done with the assignment. I am open to other methods other than holt-winters too, infact like you said holt-winters may not be the best way for this problem." CreationDate="2011-04-21T20:19:08.387" UserId="4280" />
  <row Id="16776" PostId="9844" Text="@mpiktas O`right, but I have corrected eigenfunctions." CreationDate="2011-04-21T20:21:43.327" UserId="88" />
  <row Id="16777" PostId="9844" Text="@mbq, I've checked the notation of scalar product in wikipedia" CreationDate="2011-04-21T20:22:48.680" UserId="2116" />
  <row Id="16778" PostId="9844" Text="@GaBorgulya, sorry I've rejected your edit, your edit came in the middle of edit-war between me and mbq :) Your effort is very welcome and appreciated." CreationDate="2011-04-21T20:24:04.903" UserId="2116" />
  <row Id="16779" PostId="9823" Text="@whuber, BOT, thanks" CreationDate="2011-04-21T20:28:58.693" UserId="3306" />
  <row Id="16780" PostId="9844" Score="1" Text="@bit-question, is this exact citation? Could you please give a reference? " CreationDate="2011-04-21T20:36:53.270" UserId="2116" />
  <row Id="16781" PostId="9843" Text="@jason,You might consider constructing an equation which incorporates dummies to handle the &quot;seasonal effect&quot; and to create a set of tentative model residuals.These &quot;residuals&quot; may then be used to determine how many trends there are in the data or how many level shifts i.e.intercept changes. This might requires either significant analysis on your part or the programming the algorithms of R.Tsay (Journal of Forecasting Vol pp 1-20,1988 ) or investing in commercially available software. I have been involved in developing these fairly proprietary procedures for AUTOBOX. " CreationDate="2011-04-21T20:49:56.643" UserId="3382" />
  <row Id="16782" PostId="2929" Text="@Ample It sounds like the [Ripley K-function](http://www.stat.iastate.edu/preprint/articles/2001-18.pdf) (or its relative, the L-function) would provide the information you want." CreationDate="2011-04-21T20:56:07.073" UserId="919" />
  <row Id="16783" PostId="9836" Text="Thanks Chase! I post it the same ques here expecting somebody gives me some more resources. I got one (Bayesian) &amp; that book is good. " CreationDate="2011-04-21T20:57:51.857" UserId="4278" />
  <row Id="16784" PostId="9842" Text="@jason, NN have been characterized as &quot;regression without ethics&quot; as they not only over-fit but make the mistake of &quot;believing the data&quot; rather than &quot;challenging the data for consistency of signal&quot;" CreationDate="2011-04-21T21:06:07.457" UserId="3382" />
  <row Id="16785" PostId="9844" Text="You must mean either &quot;of total degree *at most* d&quot; or else you need to use $p+1$ in place of $p$ and mean &quot;*homogeneous* of degree d.&quot;" CreationDate="2011-04-21T21:07:54.210" UserId="919" />
  <row Id="16786" PostId="9842" Score="1" Text="NNs are totally not obsolete. They hold the best scores on several important benchmarks currently tackled by the ML community. Also, they are the best multi purpose differentiable function approximator around. Check out the work of Bengio's, Hinton's and lecun's groups of the last 5 years." CreationDate="2011-04-21T21:12:39.510" UserId="2860" />
  <row Id="16787" PostId="9814" Text="Could you describeyour data and/or the model you want to fit in a little more detail? Your reference seems to be behind a paywall. From your quote it seems you're after mixed effect models; if so then there are a multitude of R packages which do so as there are many models which fall under this umbrella." CreationDate="2011-04-21T23:34:48.717" UserId="26" />
  <row Id="16788" PostId="9823" Text="I'm accepting this but need to say the wiki equation is awkward at best. It's application to trading is limited. There are other expected return formulae that are better suited to examine trade system characteristics." CreationDate="2011-04-22T00:18:29.407" UserId="3306" />
  <row Id="16789" PostId="9845" Text="If the volume remains constant, then for an [ideal gas](http://en.wikipedia.org/wiki/Ideal_gas_law), the product of the temperature (measured from absolute zero, e.g. in Kelvin) and the pressure should be constant.  " CreationDate="2011-04-22T00:19:24.943" UserId="2958" />
  <row Id="16791" PostId="5220" Text="I am also trying to do fixed-effect and random-effect modeling ( panel data) using SPSS. Can anybody tell me how to do it in SPSS. Thank you for you help." CreationDate="2011-04-22T01:22:57.970" UserId="4285" />
  <row Id="16792" PostId="5220" Text="Welcome to this site, @G!  Please take a moment to read over the FAQ and acquaint yourself with the conventions here.  In particular, because replies are treated so much differently than comments, we are careful to distinguish them.  Your &quot;I have this question too&quot; is a useful comment but is not a helpful reply: that's why I have turned it into a comment." CreationDate="2011-04-22T01:51:28.880" UserId="919" />
  <row Id="16793" PostId="9852" Score="2" Text="Are the models nested (i.e. are the models the same except for the one variable in the four variable model?)" CreationDate="2011-04-22T02:49:59.250" UserId="1036" />
  <row Id="16794" PostId="9852" Text="Good Q.. No they aren't, but close.  One model uses four variables, W X Y and Z.  The other model has three variables, W X and (Y+Z)/2.  Although Y and Z may or may not be weighted equally in the second model. " CreationDate="2011-04-22T03:15:03.820" UserId="1977" />
  <row Id="16795" PostId="9852" Text="@Jeff , you should update your question with this information, attempt to write out the models you are fitting mathematically, and be as explicit as possible about the transformation to &quot;Y and Z&quot; and what you are trying to accomplish with that transformation. " CreationDate="2011-04-22T03:26:04.107" UserId="1036" />
  <row Id="16796" PostId="9852" Text="Well let's just stick with a simple average for now...  Q has been updated, thanks!" CreationDate="2011-04-22T03:40:52.167" UserId="1977" />
  <row Id="16797" PostId="9859" Score="1" Text="@ocram, just a note, the estimators from the likelihood can also be biased, the classical example is likelihood variance estimate for normal sample." CreationDate="2011-04-22T07:19:46.720" UserId="2116" />
  <row Id="16798" PostId="9859" Text="@mpiktas: Thanks for your comment. Indeed, the classical mle can also be biased. I will edit the question to make things clearer." CreationDate="2011-04-22T07:23:55.320" UserId="3019" />
  <row Id="16800" PostId="9814" Text="@JMS: Thanks for your comment. In multienvironment trials, we have a matrix genotype-by-environment interaction effects and implement **singular value decomposition (SVD)**, equivalent to **PCA**, on this matrix assuming both genotypes and environments fixed. Most often one or both of genotypes and environments are random results in the use of **Factor Analytic (FA) structure**. Please have a look on the eqs (12) and (13) on page 453 in the given reference. I'd highly appreciate if you help me on this. Thanks" CreationDate="2011-04-22T08:03:09.007" UserId="3903" />
  <row Id="16801" PostId="9810" Text="I need a global test for the three groups with the null hypothesis the medians are equal. Kruskal-Wallis may be an option, but this test is not less powerful in comparing the paired &quot;before&quot;-&quot;after&quot;. Subsequently, I need post-hoc tests comparing &quot;before&quot;-&quot;after&quot;, &quot;control&quot;-&quot;prior&quot; and &quot;control-after&quot;. Which post-hoc test I am not sure. I have been thinking of doing  wilcoxon test with adjusted p-values..." CreationDate="2011-04-22T08:41:39.567" UserId="4229" />
  <row Id="16802" PostId="9735" Text="@steffen:  that is called the significance level, or $\alpha$.  The power of a test is how often it rejects the null hypothesis when the alternative is true." CreationDate="2011-04-22T09:24:20.903" UserId="3567" />
  <row Id="16803" PostId="9735" Text="@charles.y.zheng I knew that ;). If you think that such a probability cannot be calculated by frequentists, why not submitting it as answer. If the community agrees, I am happy to accept it :)." CreationDate="2011-04-22T09:26:39.730" UserId="264" />
  <row Id="16804" PostId="9735" Text="@steffen: The significance level of a test is easy to obtain by calculation or simulation.  The power level of a test is only defined with respect to a specific alternative.  That is why it is not possible to calculate a general &quot;power&quot; of a test; such a notion cannot be defined." CreationDate="2011-04-22T09:34:09.690" UserId="3567" />
  <row Id="16805" PostId="9735" Text="@charles.y.zheng I suggest to submit it as answer then." CreationDate="2011-04-22T09:47:53.087" UserId="264" />
  <row Id="16806" PostId="9859" Text="@ocram, what is the assymptotic bias? Are you talking about non-consistent estimators?" CreationDate="2011-04-22T10:02:40.417" UserId="2116" />
  <row Id="16807" PostId="9850" Score="1" Text="That really depends on you've done the clustering :) If you show a little example of the data you have i'm sure you'll get an answer" CreationDate="2011-04-22T10:40:45.773" UserId="3732" />
  <row Id="16808" PostId="9826" Text="@lodonnell What about an example ? I think the OP might appreciate that (given he/she seems to have only basic knowledge of statistics). An example metric, some artificial data and an exemplary execution of a test ... this would make the answer really complete." CreationDate="2011-04-22T12:05:20.500" UserId="264" />
  <row Id="16809" PostId="9859" Text="@mpiktas: Yes, this is what I should have said..." CreationDate="2011-04-22T12:40:20.530" UserId="3019" />
  <row Id="16810" PostId="9850" Score="1" Text="Using different colors and markers tends to be the simplest/easiest to read.  If all you have is 2 clusters, then you can just print 0/1 or O/X for the different values." CreationDate="2011-04-22T13:15:36.470" UserId="595" />
  <row Id="16811" PostId="9842" Text="I read the Bengio &amp; Lecun paper and it does change my mind a bit, but what they are doing is quite different from historical neural nets. The OP wants to work with time series, where there are several methods that can work with univariate time series and actually give you useful information about the time series (DLMs come to mind). If you have data beyond the time series itself, you can use a variety of other methods (LMs, etc) that are straightforward and also illuminating. Why use a black box with unlabeled dials when you can do something understandable?" CreationDate="2011-04-22T13:27:13.467" UserId="1764" />
  <row Id="16812" PostId="9851" Text="How is wunderground not &quot;legit&quot;? It apparently has nothing to do with amateur observations, which you want, but what is it?" CreationDate="2011-04-22T14:02:40.300" UserId="1764" />
  <row Id="16813" PostId="9874" Score="1" Text="I think N26 is suggesting a thought experiment. I.e., if when designing a study you have a choice between two scales, should you prefer, prima facie, the one with the greater variance. Also, having two predictors that represent the same construct, but are measured differently does not violate the assumption of independence of observations." CreationDate="2011-04-22T14:32:56.543" UserId="183" />
  <row Id="16814" PostId="9852" Text="@Jeff Yes, the models are nested.  To see this, you can rewrite model 1 in terms of W, X, (Y+Z)/2, and (say) (Y-Z)/2, showing that model 2 just eliminates one variable." CreationDate="2011-04-22T14:35:41.143" UserId="919" />
  <row Id="16815" PostId="9814" Text="Like I said, your reference is behind a paywall. If you reproduce the equations here maybe we can help; again, packages exist for doing factor analysis and PCA (eg, factanal)." CreationDate="2011-04-22T14:46:37.407" UserId="26" />
  <row Id="16817" PostId="9851" Text="I'm pretty sure that wunderground recruits people to report directly to them, so I don't see how you'd do that. And I'm not sure what the difference would be. Are you looking for more details than you can get through them?" CreationDate="2011-04-22T15:25:06.623" UserId="1764" />
  <row Id="16818" PostId="9842" Text="@Wayne: Well said !" CreationDate="2011-04-22T16:13:16.037" UserId="3382" />
  <row Id="16819" PostId="9879" Text="The two multivariate Gaussian distributions seem to have different means but the same covariance matrix (or is that a typo?).  Are they independent of each other?" CreationDate="2011-04-22T18:38:05.577" UserId="2958" />
  <row Id="16820" PostId="9879" Text="This question is ill-posed at the moment. The information provided is not enough to establish a (unique) distribution for the sum. " CreationDate="2011-04-22T18:43:58.847" UserId="2970" />
  <row Id="16821" PostId="9880" Score="3" Text="Focusing this question on a specific application will help generate helpful answers.  In its current state it's almost too vague for a response and is likely to be closed." CreationDate="2011-04-22T19:27:02.873" UserId="919" />
  <row Id="16822" PostId="9877" Text="(+1) A couple brief comments: Though I too would be more inclined to try regression splines first over binning, where possible, I don't see immediately how regression splines help avoid introducing &quot;arbitrariness&quot; into the model any more than binning. One is left to choose both the number and placement of the knots, neither of which is necessarily obvious or trivial, even for a specific application. Also, perhaps there is a more directed link you could provide, as the one currently given seems a bit broad and perhaps difficult to determine what you are pointing the OP to. Cheers. " CreationDate="2011-04-22T19:48:44.383" UserId="2970" />
  <row Id="16823" PostId="9879" Text="@cardinal What information are you missing?" CreationDate="2011-04-22T19:51:57.737" UserId="919" />
  <row Id="16825" PostId="9877" Score="1" Text="+1 for the idea of splines.  One has a lot of flexibility in their use: in my experience the model tends to be less sensitive to the positions of the knots and numbers of splines than it would be to the choices of bins.  It allows for more continuous transitions (by construction).  As @cardinal rightly points out, though, the fundamental question is still out there: what guidance is there concerning the numbers of splines and locations of the knots?  To what extent might selecting them *post hoc* affect p-values?" CreationDate="2011-04-22T19:57:11.007" UserId="919" />
  <row Id="16827" PostId="9879" Score="1" Text="@whuber: The joint distribution of two separate multivariate Gaussian vectors is not necessarily distributed as a multivariate Gaussian." CreationDate="2011-04-22T20:07:32.877" UserId="2970" />
  <row Id="16828" PostId="9882" Text="This assumes independence; it is not clear from the question whether this is the case." CreationDate="2011-04-22T20:18:07.857" UserId="892" />
  <row Id="16829" PostId="9879" Score="2" Text="@cardinal Good point.  We have to assume either the joint distribution is Gaussian *or else* that the two variables are independent.  One might, with some generosity and suitable caution, make that latter assumption for the OP." CreationDate="2011-04-22T20:19:08.087" UserId="919" />
  <row Id="16830" PostId="9882" Text="@F That is true (and you are in good company with @cardinal): I explicitly make this assumption and even italicize it for emphasis.  Which approach is constructive: to withhold or deprecate replies because the OP has not explicitly named all necessary assumptions; or to point out the natural, conventional assumptions needed for a useful answer and to provide that answer?" CreationDate="2011-04-22T20:24:12.950" UserId="919" />
  <row Id="16831" PostId="9879" Score="1" Text="@whuber: Yes, I probably tend to be (overly) pedantic about such matters. On the other hand, I think that it can be instructive to draw attention to such matters since it can help the OP understand the boundaries of their own question. Thanks for your patience with me in this regard. Also, your second statement (independence) is just a subset of the first (multivariate Gaussianity), of course. " CreationDate="2011-04-22T20:31:29.290" UserId="2970" />
  <row Id="16832" PostId="9879" Score="1" Text="@Cardinal You are a mathematician, I see :-).  I completely agree that it's worthwhile to draw attention to implicit assumptions and boundaries, because that heads off potential misuse and abuse.  It's hard to know where to draw the line, too: when too many assumptions are left unsaid, some prodding is required to make sure we're answering the right question.  On the other hand, we're not the math.se site (where rigor is the watchword and heaven help the poor person with a slightly flawed question) and, as practitioners, we tend to want to help questioners.  So: perhaps &quot;friendly pedantry&quot;?" CreationDate="2011-04-22T20:39:58.973" UserId="919" />
  <row Id="16833" PostId="9879" Text="@whuber: While you will find me listed by the math genealogy project, I doubt that any serious mathematician would consider me worthy of the title." CreationDate="2011-04-22T21:58:00.413" UserId="2970" />
  <row Id="16834" PostId="9882" Text="(+1) Might be worth clarifying the meaning of $\Sigma_1 \oplus \Sigma_2$." CreationDate="2011-04-22T21:58:24.797" UserId="2970" />
  <row Id="16835" PostId="9882" Text="@Cardinal Thanks.  I didn't feel like TeXing it out... :-)" CreationDate="2011-04-22T21:59:57.567" UserId="919" />
  <row Id="16836" PostId="9842" Text="Of course the neural nets used back in the 80s/90s are different from what you use today and they are still a very active area of research. Furthermore, you *never* use neural networks when you care about interpretability. You use them when you care about prediction error. Neural networks are fast and they solve problems other methods fail at. They are nice because they are simple from a conceptual perspective due to the lack of any assumptions about the data you are modeling (except Gaussian noise when used with the squared error). They have their own merits and flaws." CreationDate="2011-04-22T22:03:18.183" UserId="2860" />
  <row Id="16837" PostId="9880" Text="Ah, fair enough. I did wonder if it was too broad, but wasn't sure! So let's start at what the end of the process will be. Say that I have a set of data produced by a computational model, and I want to compare the output to actual data collected from real people. What's the best way to do that, in terms of statistical methods and techniques?" CreationDate="2011-04-22T22:52:29.160" UserId="4204" />
  <row Id="16838" PostId="9886" Score="2" Text="You probably know much more about the mechanisms generating the “number of hits” than written here – that mechanism may be a key point in building the right statistical model. Times series often show autocorrelation or periodicities, modelling should include these if they are present. “Significant decrease” may then be defined as deviation from what was predicted, but it's important to have a good prediction model. " CreationDate="2011-04-22T23:24:55.760" UserId="3911" />
  <row Id="16839" PostId="9880" Text="@vize You're getting there.  Why don't you edit the question to reflect this refinement and see what responses you get?" CreationDate="2011-04-22T23:51:27.157" UserId="919" />
  <row Id="16840" PostId="9848" Text="Thank you so much " CreationDate="2011-04-23T02:50:02.963" UserId="4282" />
  <row Id="16841" PostId="9848" Text="Let me put it this way, I am working on safety Systems. I wanna calculate each term in Bayes'rule: P(Temp|Pressure)= P(Pressure|Temp)* P(Temp) / P(Pressure). I wanna use MATLAB. This Bayesian Network is assumed to be used to predict accidents in as Nuclear, Natural gas, etc. How to calculate the prior, likelihood and evidence" CreationDate="2011-04-23T03:05:06.373" UserId="4282" />
  <row Id="16842" PostId="9846" Text="Thank you so much. I wanna calculate each term in Bayes'rule: P(Temp|Pressure)= P(Pressure|Temp)* P(Temp) / P(Pressure). I wanna use MATLAB. This Bayesian Network is assumed to be used to predict accidents in as Nuclear system, Natural gas, etc. How to calculate the prior, likelihood and evidence. I wanna know what is the probability that Temp will change if pressure changes" CreationDate="2011-04-23T03:12:12.157" UserId="4282" />
  <row Id="16843" PostId="9895" Text="The notation is strange: the only independent variable is $N_o$ and you are trying to model $N_e$ in terms of itself!  What really are you trying to do here?" CreationDate="2011-04-23T04:05:06.190" UserId="919" />
  <row Id="16844" PostId="9895" Text="@Whuber: I'd appreciate if you have a look on this link http://www.esapubs.org/Archive/ecol/E088/187/appendix-B.htm. (B.1). This notation also confused me first." CreationDate="2011-04-23T04:16:04.667" UserId="3903" />
  <row Id="16845" PostId="9704" Text="Thanks for the sample clock data and other updates.  But I get an error on function &quot;med.filter&quot; not being found.  Is that in a different library?" CreationDate="2011-04-23T05:29:50.813" UserId="2434" />
  <row Id="16846" PostId="9884" Score="2" Text="Kevin, it's better to split questioins into 'answerable' bites and the programming questions are probably better asked at stackoverflow or similar (though established stats.se members might disagree). But I found [this tutorial](http://www.stat.auckland.ac.nz/~paul/RGraphics/rgraphics.html) helpful for understanding base R graphics" CreationDate="2011-04-23T05:29:57.643" UserId="3732" />
  <row Id="16847" PostId="9895" Text="I don't know if this is the case, but everytime I've used `nls` and I've got those error it was due to problems in the start parameters... I'll leave a proper answer to the real statisticians ;)" CreationDate="2011-04-23T06:36:13.090" UserId="582" />
  <row Id="16848" PostId="9884" Score="1" Text="@Kevin I've removed the general question about R programming; I can't see that it has any relationship with your question about pairs(). I encourage you to ask your R programming question separately either on this site or, as David W says, on StackOverflow with the R tag." CreationDate="2011-04-23T06:42:42.887" UserId="183" />
  <row Id="16850" PostId="9887" Text="@GaBorgulya From what I seem to remember, for each user you have his/her vote with date + question ID." CreationDate="2011-04-23T06:57:45.983" UserId="930" />
  <row Id="16851" PostId="9881" Text="@Chris I don't think this can explain the warning or error messages reported by the OP." CreationDate="2011-04-23T06:59:02.473" UserId="930" />
  <row Id="16852" PostId="9887" Text="(+1) for vote prediction. Great idea!" CreationDate="2011-04-23T07:19:58.443" UserId="264" />
  <row Id="16853" PostId="9895" Text="This is a numerics problem -- in both cases you have no true convergence. Can you post the data you're fitting to, and/or at least SAS output? And if this $N_e$ is the same on the both sides, you should solve the eq for it first." CreationDate="2011-04-23T08:04:40.063" UserId="88" />
  <row Id="16854" PostId="9882" Text="@whuber: Sorry I did not notice the italized assumption in your reply (in fact, I focused on Method 2). I did not mean to deprecate any answer. Whether the assumption is natural or conventional, I would not venture to say: I know nothing about the underlying problem the OP has in mind." CreationDate="2011-04-23T08:22:41.313" UserId="892" />
  <row Id="16855" PostId="9895" Text="@mbq: I've posted the data. Given model is very commonly used in ecology. I also tried to solve the equation for $N_e$ but it did not turn out nicely. I'd appreciate if you and/or anyone help me to figure out this problem. Thanks" CreationDate="2011-04-23T08:29:32.737" UserId="3903" />
  <row Id="16856" PostId="9899" Text="please feel free to correct me. I don't think the eigen vecs are perpendicular (they must be in theory; may be I am plotting something wrong?)." CreationDate="2011-04-23T09:19:58.623" UserId="1307" />
  <row Id="16857" PostId="9899" Text="@suncoolsu: Thanks for your reply. This ellipse looks nice but something is missing there. I tried your given code with this matrix &#xA;$\begin{equation*}&#xA;\mathbf{A} = \left(&#xA;\begin{array} {cc}&#xA;1 &amp; -5\\&#xA;-5 &amp; 1&#xA;\end{array} \right)&#xA;\end{equation*}$ and it gave correct ellipse. I'm wondering why it is not providing the correct ellipse for different variances in the matrix. Any comment!" CreationDate="2011-04-23T10:01:36.027" UserId="3903" />
  <row Id="16858" PostId="9900" Text="@caracal: Would you mind to plot the eigenvalues and eigenvectors on this ellipse? Thanks" CreationDate="2011-04-23T10:05:22.123" UserId="3903" />
  <row Id="16859" PostId="9899" Text="@suncoolsu Just set `asp=1` to have an aspect ratio of 1 and perpendicular arrows. Changing your code to `evs &lt;- sqrt(eigens$values)` gives the same ellipse as my answer." CreationDate="2011-04-23T10:47:00.900" UserId="1909" />
  <row Id="16860" PostId="9899" Score="1" Text="@MYaseen208 Your new matrix is not positive definite: it has negative eigenvalues and isn't a possible covariance matrix. I don't know what ellipse to draw in that case." CreationDate="2011-04-23T10:49:17.260" UserId="1909" />
  <row Id="16861" PostId="9880" Text="@whuber I think @Jeromy Anglim has done a better job than I ever could! Thanks to you both for the help in getting my question across properly :)" CreationDate="2011-04-23T11:12:05.370" UserId="4204" />
  <row Id="16862" PostId="9256" Text="One more question about this technique. Can be same method applied to find $C$ and $\nu$ for $\nu$-SV regression?" CreationDate="2011-04-23T12:56:25.877" UserId="4051" />
  <row Id="16863" PostId="9256" Text="It should be O.K., I haven't used $\nu$-SVR, but I regularly use these sorts of methods for optimising the hyper-parameters of a wide range of kernel models and Gaussian processes (both regression and classification).  As long as you apply a suitable transformation to get an unconstrained optimisation problem, it seems pretty reliable." CreationDate="2011-04-23T13:00:06.520" UserId="887" />
  <row Id="16864" PostId="9902" Text="@Conjugate Prior,  Thank you for your sugestions.  I am not sure exactly what you mean by &quot;ideal inferential endpoint&quot; so I an not sure how to clarify further. The 'pairs of years' is basically because there is a large gap in samples between the 2 sets of years, we opted to consider each pair of years independently as opposed to having Year as a factor with all the data combined.  " CreationDate="2011-04-23T13:02:22.917" UserId="4048" />
  <row Id="16865" PostId="9900" Text="@MYaseen208 I edited my answer to show the eigenvectors as the ellipse's axes. Half the axes' length equals the square-root of the corresponding eigenvectors." CreationDate="2011-04-23T15:05:54.377" UserId="1909" />
  <row Id="16866" PostId="9865" Text="Thanks for the answer." CreationDate="2011-04-23T15:42:05.630" UserId="3891" />
  <row Id="16867" PostId="9850" Text="Please tell what you mean by &quot;a set of marks.&quot;  How many variables do you have with which to characterize the clusters?  Also, are you confident that 2 is the best number of clusters to use?  Many times one has to use cluster analysis programs iteratively; at the outset one might get just 2, but with some adjustments one might get a more interesting and informative higher number." CreationDate="2011-04-23T15:53:40.100" UserId="2669" />
  <row Id="16868" PostId="9882" Score="1" Text="@F Your reminder that we don't know the underlying problem is well taken.  By &quot;natural&quot; I meant that typically when people refer to two distributions as separate entities, as in the statement of this question, they take them to be independent.  It is wise to be mindful that this is usually an important assumption: that's how I read the comments you and @cardinal have kindly provided." CreationDate="2011-04-23T16:08:21.047" UserId="919" />
  <row Id="16869" PostId="9905" Text="Wow, great! This is the kind of thing I was looking for. It's time to start reading and learning then - thanks! :)" CreationDate="2011-04-23T16:49:12.267" UserId="4204" />
  <row Id="16870" PostId="9902" Text="@KennyPeanuts By 'ideal inferential endpoint' I meant what you'd really like to be able to say or show or do after you've completed the analysis. Basically I was struggling to parse your third paragraph and looking for some context. Thinking in your initial model formulation, the first sentence suggests you would like to report a shift in the intercept of a logistic regression model (which moves the diapause prob. up or down by year). But the second suggests you would like to show a shift in the slope (which makes the effect on diapause of being a day later more pronounced by year). Or both..." CreationDate="2011-04-23T18:00:34.150" UserId="1739" />
  <row Id="16871" PostId="9899" Text="@caracal thanks! ... yeah -- I missed the sqrt part!" CreationDate="2011-04-23T18:18:48.237" UserId="1307" />
  <row Id="16872" PostId="9902" Text="@Conjugate Prior, Oh, I see what your queston is now.  Basically we would like to know if the prob of going into diapause on a given day changes between years.  It seems that this could be due to either a change in intercept or slope.  As I understand your initial comments on an interaction term, a sig year effect without sig interaction would mean a shift in intercept with essentially no change in slope, while a sig year effect and interaction would indicate a shift in intercept and slope.  Is that the correct interpretation? " CreationDate="2011-04-23T18:25:36.247" UserId="4048" />
  <row Id="16875" PostId="9907" Score="1" Text="You can't, at least not with the information you've given. Even if $F$ and $H$ are (marginally) independent, they may not be conditionally independent given $G$." CreationDate="2011-04-23T20:44:18.317" UserId="495" />
  <row Id="16876" PostId="9889" Text="@user4294 Your two recent posts let me wonder what you really expect from the SE community. They were removed as this is not a very constructive way of envisionning SE Q&amp;A sites in general, nor a very elegant way to address stats.SE users. You asked your question less than 24h ago, so it would be fair to wait a little bit. I hope you will get some prompt responses for the same question you posted 3 hours after on [stackexchange](http://stackoverflow.com/questions/5762128/r-cointegration-multivariate-co-ja-johansen). And I close this one right now." CreationDate="2011-04-23T20:49:19.057" UserId="930" />
  <row Id="16877" PostId="9892" Score="2" Text="I don't think you can: GLMSELECT only does linear models. If you are willing to use R or Matlab, there is the glmnet package." CreationDate="2011-04-23T20:56:05.847" UserId="495" />
  <row Id="16878" PostId="9913" Text="Migrating to Stats.stackexchange where this is more suitable." CreationDate="2011-04-23T21:31:36.400" />
  <row Id="16879" PostId="9889" Text="@user4294, why would you think that voltage variables would be cointegrated in the first place? I don't see how that makes any sense." CreationDate="2011-04-23T21:51:43.103" UserId="2970" />
  <row Id="16880" PostId="9913" Score="2" Text="The result follows directly from the formula for elasticity in the [Wikipedia article](http://en.wikipedia.org/wiki/Elasticity_%28economics%29) because the form of the rhs shows $B_1$ is the derivative of $log(Y)$ with respect to $log(X)$." CreationDate="2011-04-23T21:54:04.837" UserId="919" />
  <row Id="16881" PostId="9905" Score="1" Text="Just to play devil's advocate here -- ABMs have a tendency to suffer from a garbage-in-garbage-out problem.  ABMs are based on assumptions about how agents make decisions. Through simulation, the modeler can then explore system-level emergent properties of interacting agents.  The problem is that the key assumptions about decision-making are rarely tested.  And if those assumptions are wrong, then the model conclusions will likely be wrong as well.  Bottom line: ABMs let you explore the implications of a given model of decision-making, but they aren't as good at testing the model itself." CreationDate="2011-04-23T21:57:21.833" UserId="4110" />
  <row Id="16882" PostId="9913" Score="1" Text="BTW, I believe you need to fix the formula for $E_{y,x}$: it should be a quotient, not a product." CreationDate="2011-04-23T22:04:50.677" UserId="919" />
  <row Id="16883" PostId="9890" Text="+1 Nice reference." CreationDate="2011-04-23T22:06:57.507" UserId="919" />
  <row Id="16884" PostId="9809" Text="@mpiktas I believe anyone who has taught a freshman calculus course might consider disagreeing a little :-)." CreationDate="2011-04-23T22:07:45.743" UserId="919" />
  <row Id="16885" PostId="9911" Text="Clarifying question: if rows in your matrix are &quot;events&quot; and columns are &quot;trials,&quot; then what is an entry in the matrix?  A more typical design would have rows as &quot;individuals&quot; (say, people in a repeated experiment), and columns as &quot;treatments&quot; (a series of experimentally manipulated treatments), with entries in the matrix denoting &quot;responses&quot; (how each individual reacted to each treatment).  Is that right, or are you doing something special?" CreationDate="2011-04-23T22:16:16.990" UserId="4110" />
  <row Id="16886" PostId="9905" Score="1" Text="I agree with the first half of that but I would point out for the second half you can run a simulation as many times as you like, changing assumptions until you find something that reasonably approximates reality. How difficult that is can depend on the complexity of what you're trying to model." CreationDate="2011-04-24T00:04:02.020" UserId="3597" />
  <row Id="16888" PostId="9895" Text="The model may have difficulties when $N_e = 0$, since the expression in square brackets is presumably positive, making the right hand side positive even when the left hand side is zero.  What are you trying to minimize? Perhaps the sum of squares of the differences between the left and right hand sides?" CreationDate="2011-04-24T00:43:51.080" UserId="2958" />
  <row Id="16889" PostId="9575" Text="@thomas - good answer, although there is one small detail that you missed: the distribution of the data must have finite variance.  T-test is hopeless for comparing difference in location of two Cauchy distributions (or student with 2 degrees of freedom), not because it is &quot;non-robust&quot;, but because for these distributions there is additional relevant information in the sample beyond the means and standard deviations which the t-test throws away." CreationDate="2011-04-24T00:55:46.113" UserId="2392" />
  <row Id="16890" PostId="9603" Text="While we may not know if the sample mean and variance are independent, we always know that they are *uncorrelated*.  This is because the sample mean is a function of bivariate sums $x_{i}+x_{j}$, and sample variance is a function of bivariate differences $x_{i}-x_{j}$ (these are called &quot;U statistics&quot;) and we have $cov(x_{i}+x_{j},x_{i}-x_{j})=var(x_{i})-var(x_{j})+cov(x_{i},x_{j})-cov(x_{j},x_{i})=0$ as long as the distribution is &quot;homogenous&quot; $var(x_{i})=var(x_{j})$, which is part of the problem statement." CreationDate="2011-04-24T01:08:35.387" UserId="2392" />
  <row Id="16891" PostId="9911" Text="Well, it is a standard setup. Rows are entry from repeated  experiments (K). Each experiment measures M events, so each column in the matrix is a one event through repeated trials. For example, matrix 2x5 would mean 2 repeated experiments (trials) with 5 events. Now, I would like to pick N events (N&lt;5) among these 5 events, so that I can represent dataset. Answer should be e.g. 1st, 2nd, and 4th element are good enough (with some error) as representatives." CreationDate="2011-04-24T01:13:46.460" UserId="1313" />
  <row Id="16892" PostId="8403" Text="@rolando2 the research is investigating the effect of brain surgery on different psychophysiological measures, including heart rate. The groups are defined according to areas of the brain that have been lesioned. For most DVs the smallest group is n=4 and I am using nonparametric tests, but for HR I wanted to covary the effect of baseline HR out so I used ANCOVA. However due to medications some patients had to be excluded, resulting in smaller ns, so I used planned comparisons with SPSS syntax to compare three theoretically related smaller groups together with the two larger control groups." CreationDate="2011-04-24T03:58:09.913" UserId="2025" />
  <row Id="16893" PostId="9918" Score="1" Text="If you are familiar with R, there is a package `vegan` with functions `anosim` or preferably, `adonis` (permutational MANOVA)." CreationDate="2011-04-24T05:15:18.643" UserId="144" />
  <row Id="16895" PostId="9901" Score="1" Text="@mbq: The LHS must be $$\log\left(\frac{N_0-N_e}{N_0}\right)$$" CreationDate="2011-04-24T06:29:03.193" UserId="3903" />
  <row Id="16896" PostId="9920" Score="1" Text="@Lisanne (+1) This is a well formulated question. However, it now looks partly duplicated from your [preceding question](http://stats.stackexchange.com/questions/4009/inter-rater-reliability-using-intra-class-correlation-with-ratings-for-multiple-o) (that you've updated recently), though it's another question. Would it be possible to balance the content of your two questions, with one target question in each? If you always update your question (instead of asking new ones), it becomes difficult to provide acceptable response." CreationDate="2011-04-24T07:30:16.383" UserId="930" />
  <row Id="16897" PostId="9906" Text="@Henry Thanks! It seems that this method is not quite common or I'm just reading the wrong papers/books. Is it really not common? Is there a more standard way to solve the problem? Can you suggest keywords/references where I can learn more about this?" CreationDate="2011-04-24T07:48:24.683" UserId="3074" />
  <row Id="16898" PostId="9908" Score="1" Text="@Jason, do us a favour and link to your crosspost on SO. :)" CreationDate="2011-04-24T08:16:22.960" UserId="776" />
  <row Id="16899" PostId="9902" Score="1" Text="@KennyPeanuts Yes, that's what I was suggesting.  &#xA;&#xA;You'd want to compare the *marginal effects* of day between years rather than the model coefficients directly to really make the inference.  A good paper discussion and figures is here: http://homepages.nyu.edu/~mrg217/interaction.html  &#xA;&#xA;The other issue is that you don't say what the distribution of diapause is over days of a year, so I assumed as a default that it is rising or falling.  If in fact it peaks at some time of year then you may want days^2 in the model as well (or some other polynomial or smooth term).  " CreationDate="2011-04-24T08:54:08.397" UserId="1739" />
  <row Id="16900" PostId="9881" Text="@chi you could be right, its not easy to say without seeing the original data. My experience with pvclust is that it tends to throw error messages that are not very easy to interpret." CreationDate="2011-04-24T13:11:25.870" UserId="1475" />
  <row Id="16901" PostId="9908" Text="@Brandon: http://stackoverflow.com/questions/5765772/how-to-print-a-single-column-in-this-r-datatype" CreationDate="2011-04-24T13:47:25.437" UserId="2958" />
  <row Id="16902" PostId="9906" Text="@ianalis: The median of sample medians works well, and is still close to the &quot;true&quot; median of 0; it is not the same as the overall median but there is no reason to expect that it would be.  The mean of sample means does give the overall mean (if the sample sizes are equal).  Combining measures of dispersion from different samples is more complicated, especially with median or quantile based measures: with variances the answer is to use the denominator $n-1$ to get unbiased estimates, as shown here.     " CreationDate="2011-04-24T14:24:18.027" UserId="2958" />
  <row Id="16903" PostId="9902" Text="@Conjugate Prior,  thank you for all your input and for the ref.  I appreciate it." CreationDate="2011-04-24T14:53:34.913" UserId="4048" />
  <row Id="16904" PostId="9919" Text="@Lisanne:  could you show a screen shot of what the output looks like before and after this step?" CreationDate="2011-04-24T15:38:17.443" UserId="2669" />
  <row Id="16905" PostId="9915" Text="@ cardinal. Thats a wonderful answer, and given the availability of all this data, it would make a fascinating project." CreationDate="2011-04-24T15:40:37.480" UserId="656" />
  <row Id="16906" PostId="9926" Score="3" Text="**[Renyi entropy](http://www.google.com/search?q=Renyi+entropy)**" CreationDate="2011-04-24T15:46:43.790" UserId="2970" />
  <row Id="16907" PostId="5245" Text="+1 Nice, comprehensive answer.  I'm going to nitpick on 2 points.  &quot;If your residuals are correlated with your independent variables, then your model is heteroskedastic&quot;--I would say that if the *variance* of your residuals depends on the level of an independent variable, then you have heteroscedasticity.  Also, I have heard the Kolmogorov-Smirnov/Lilliefors tests described as &quot;notoriously unreliable,&quot; and in practive I have certainly found this to be true.  Better to make a subjective determination based on a Q-Q plot or a simple histogram." CreationDate="2011-04-24T15:57:03.650" UserId="2669" />
  <row Id="16908" PostId="8898" Text="I don't know anything about ROC, but what little I've read (https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf) suggests that getting a significance measure is messy." CreationDate="2011-04-24T17:13:04.417" UserId="3874" />
  <row Id="16909" PostId="9901" Text="Thanks all, good point with $N_0$. My answer still doesn't solve the problem, but I hope it will be more useful now." CreationDate="2011-04-24T18:06:13.040" UserId="88" />
  <row Id="16910" PostId="9928" Text="It is better to use $\LaTeX$ instead of Mathematica syntax for maths, the first is interpreted here." CreationDate="2011-04-24T18:16:29.400" UserId="88" />
  <row Id="16911" PostId="9928" Text="Setting $p(x) = 0$ for $x &gt; M$ is definitely *not* fat-tailed by any standard definition." CreationDate="2011-04-24T18:52:29.330" UserId="2970" />
  <row Id="16912" PostId="9927" Text="@rolando2 Thanks for your response.  I've managed to calculate Cronbach's alpha, but how does one calculate Pearson correlation coefficients in this case?  I could calculate them in a pairwise fashion for each individual characteristic, but I'd like to know how to calculate correlations between groups of characteristics.  Characteristics within a group will have similar scores for each observation.  I'm going to edit my question to make this a little clearer." CreationDate="2011-04-24T18:55:01.153" UserId="4301" />
  <row Id="16913" PostId="4367" Text="@barrycarter, The mean of a Cauchy is *not* zero. In fact, it doesn't exist. Thus, neither does any of its central moments." CreationDate="2011-04-24T18:55:54.227" UserId="2970" />
  <row Id="16914" PostId="4369" Text="There is a difference between the mean and variance not existing and them being infinite." CreationDate="2011-04-24T18:56:16.627" UserId="2970" />
  <row Id="16915" PostId="9901" Text="@mbq: The following SAS code was used by the author for this dataset:   PROC NLIN DATA=NOTONEC2;&#xA;PARMS BHAT= 0.001 0.01 0.1 CHAT= 0.001 0.01 0.1 DHAT= 0 THHAT=3.0; &#xA;BOUNDS BHAT&gt;0,CHAT&gt;=0, THHAT&gt;0;&#xA;T=72;&#xA;X=NE;&#xA;A=(DHAT+BHAT*N0)/(1+CHAT*N0);&#xA;C1=EXP(-A*T);&#xA;C2=A*THHAT;&#xA;H=N0*C1*EXP(C2*X)+X-N0;&#xA;ITER=0;&#xA;DO WHILE(ABS(H)&gt;0.0001 AND ITER&lt;50);&#xA;	X=X-H/(N0*C1*C2*EXP(C2*X)+1);&#xA;     H=N0*C1*EXP(C2*X)+X-N0;&#xA;     ITER=ITER+1;&#xA;END;&#xA;MODEL NE=X;&#xA;Run;" CreationDate="2011-04-24T19:44:22.273" UserId="3903" />
  <row Id="16916" PostId="9901" Text="On the revised graph, these are not straight lines, but curves of the form $y=\log(1-\frac{x}{k})$.  They may look fairly straight, but some are obviously concave." CreationDate="2011-04-24T20:12:54.640" UserId="2958" />
  <row Id="16917" PostId="9901" Text="@MYaseen208 I have no access to SAS so I can't check, however I'm pretty sure that this model does not fit this data and thus the SAS solution is either unstable or degenerated (as your R one)." CreationDate="2011-04-24T20:18:54.380" UserId="88" />
  <row Id="16918" PostId="9931" Text="I say exponentially because I want to control the weight of old samples just like using weighted moving average. But I want the old samples to be less weighted, exponentially less, than newer samples" CreationDate="2011-04-24T18:57:52.350" UserId="4306" />
  <row Id="16920" PostId="9901" Text="@Henry Why do you think so? Obviously they would be straight only for perfect model realization, but this is *ad absurdum*..." CreationDate="2011-04-24T20:26:15.623" UserId="88" />
  <row Id="16921" PostId="9901" Text="@mbq: Thanks for all your help. This example has been taken from Juliano, S.A. (2001) Non-linear curve-fitting: predation and functional response curves. Design and Analysis of Ecological Experiments http://books.google.com/books?id=ZVX7Un6GGysC&amp;printsec=frontcover&amp;dq=Design+and+analysis+of+ecological+experiments&amp;source=bl&amp;ots=OsimzJB8vb&amp;sig=EoogNJutzKz6dVCtxWsJVmpjP_4&amp;hl=en&amp;ei=W4i0TejbAZGDtgfpvoTqDg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=7&amp;ved=0CFAQ6AEwBg#v=onepage&amp;q=nonlinear%20curve%20fitting&amp;f=false. And number of articles has been published using this model." CreationDate="2011-04-24T20:35:22.370" UserId="3903" />
  <row Id="16922" PostId="9901" Text="Model and the fitting procedure seems reasonable, but this example can be just unfortunate. " CreationDate="2011-04-24T20:47:47.033" UserId="88" />
  <row Id="16923" PostId="9901" Text="With the given SAS program I can reproduce the results given in book. But I'd like to do this in R." CreationDate="2011-04-24T20:54:17.893" UserId="3903" />
  <row Id="16924" PostId="9901" Text="That's obvious, there is no stochasticity here, so same code+data gives same results -- however I think the SAS results for *this* data is worthless (while you do the optimization with constrains there, some optimum is almost guaranteed). BTW post the SAS fit here if you can." CreationDate="2011-04-24T21:06:43.123" UserId="88" />
  <row Id="16925" PostId="9933" Text="Never heard of R. Seems to be what I want, but I don't understand all of the syntax. Can you explain how that would work with this linear regression formula? (NΣXY - (ΣX)(ΣY)) / (NΣX^2 - (ΣX)^2). The formula works perfectly for my problem, but I'm writing a research paper and I need a more acceptable exponential moving form of this to work as samples come in." CreationDate="2011-04-24T21:47:31.450" UserId="4306" />
  <row Id="16926" PostId="9933" Text="It seems like the seq command in that syntax means it keeps track of the last 10 samples. I need something like exponentially weighted moving average that decreases all past weights approaching but never reaching 0" CreationDate="2011-04-24T21:49:15.833" UserId="4306" />
  <row Id="16929" PostId="9926" Text="$f$ is a pdf, right?" CreationDate="2011-04-24T22:19:08.083" UserId="919" />
  <row Id="16930" PostId="9934" Text="Thanks, I'll look into this, and see how it goes." CreationDate="2011-04-24T23:09:49.787" UserId="4301" />
  <row Id="16931" PostId="9928" Text="@barrycarter, I have tried to fully rewrite and reformat your question without altering the content. To do so, I've introduced the variable $Y$. But, that more clearly brings out a bit of ambiguity in the original question regarding whether you are interested in the stated property for $X$ itself or for $Y$. I actually believe it is for $Y$, but I've strived hard to make the question a bit more reader-friendly without altering the content. You're welcome to roll back the revision. In any event, I **strongly** recommend you *edit* it to clarify whether the main object of interest is $X$ or $Y$." CreationDate="2011-04-24T23:36:39.270" UserId="2970" />
  <row Id="16932" PostId="9928" Text="@cardinal - setting $p(x)=0$ for $x&gt;M$ is not unrealistic for large enough $M$.  I would say this is the only truly realistic way to model anything, because infinity doesn't exist in the real world.  Its just hard to say exactly what that $M$ is, and by taking the limit as $M\to\infty$ is a useful approximation which opens up analytic methods.  But one cannot claim for it to be &quot;exact&quot; - even if no mathematical approximations have been made." CreationDate="2011-04-25T00:30:48.127" UserId="2392" />
  <row Id="16933" PostId="9928" Score="1" Text="@probabilityislogic, your point is well taken. However, my intent was to address terminology in order to (hopefully) reduce confusion and (possible) misinterpretation. A density function supported on a compact set is pretty much the antithesis of &quot;fat-tailed&quot;." CreationDate="2011-04-25T00:34:19.617" UserId="2970" />
  <row Id="16934" PostId="9928" Text="@cardinal - I would disagree here.  For if I put a mass $p_M$ at $x=M$ and then distribute the rest according to a pdf defined on $0&lt;x&lt;M$ I have arbitrarily large probability mass in the tail of the distribution, and hence &quot;fat tails&quot;.  A less pathological example is a Cauchy distribution, restricted to $|x|&lt;M$.  This density still has &quot;fat tails&quot; relative to densities defined on $|x|&lt;M$ - just that there is some densities with fatter tails defined on a larger support" CreationDate="2011-04-25T00:40:56.807" UserId="2392" />
  <row Id="16935" PostId="9928" Text="@probabilityislogic: [Heavy tailed distributions](http://en.wikipedia.org/wiki/Heavy-tailed_distribution). Precision is used in mathematical terminology so that everyone can agree on what's being discussed. Many times the terms used are meant to be evocative of some intuition, but also end up being a little at odds with colloquial meanings." CreationDate="2011-04-25T00:49:24.793" UserId="2970" />
  <row Id="16936" PostId="9936" Score="1" Text="This is a good start to an answer to the OP's question, I think. The notation should be clarified somewhat, particularly the meaning of $\Gamma[t;\,1,v]$, and the OP's specific queries regarding symmetry (for what values of the parameters?) and the finiteness of $\mathbb{E} e^Y$ should be addressed. There are several freely available references on using these models to price options, so providing those in the answer as well would go along way to wrapping up all loose ends." CreationDate="2011-04-25T00:54:45.397" UserId="2970" />
  <row Id="16937" PostId="9933" Text="I used 10 for the example; replace that with N where you determine N as the length of your dataset.  Also, this is simply  an application of **weighted least squares** which any decent regression textbook will cover in more detail. " CreationDate="2011-04-25T01:19:25.267" UserId="334" />
  <row Id="16938" PostId="9937" Score="2" Text="`match(&quot;bar&quot;,names(x))` also works, though `match` is much more useful when the first argument is also a vector." CreationDate="2011-04-25T01:35:06.657" UserId="2970" />
  <row Id="16939" PostId="9926" Text="Yes, $f$ is a density." CreationDate="2011-04-25T02:34:33.807" UserId="3567" />
  <row Id="16940" PostId="9862" Text="tnx for the help." CreationDate="2011-04-25T05:25:20.050" UserId="4286" />
  <row Id="16941" PostId="9918" Text="I've updated your question to try to use standard terminology (i.e., variables not characteristics; groups of variables rather than &quot;groups&quot;)" CreationDate="2011-04-25T05:28:10.360" UserId="183" />
  <row Id="16942" PostId="9932" Score="2" Text="Could you elaborate on how the distinction between mediators and moderators is relevant to whether a partial correlation can be greater than a zero-order correlation?" CreationDate="2011-04-25T05:49:30.027" UserId="183" />
  <row Id="16943" PostId="9882" Text="@whuber Thanks for ur explanation." CreationDate="2011-04-25T06:10:22.240" UserId="4290" />
  <row Id="16944" PostId="9915" Text="@cardinal: Most of your suggestions seems practical and ML-related to me. Anyway, some of them should deal with erroneous and incomplete data. Sadly I don't have deep knowledge of data mining and cleansing nor enough time to learn it. I hope other members of stats do some work about these ideas and make a contribution to SE community and impress them :) " CreationDate="2011-04-25T06:27:53.060" UserId="2148" />
  <row Id="16945" PostId="9887" Text="This project seems great, especially if we predict the vpvote count very soon for the user. A further work can be to tell the user what is holding his/her question back and what improvement can make his question popular. Anyway, as always the choice of feature is really an important and challenging task and the performance of such predictions highly depends on this selection. **TL;DR I like your idea**" CreationDate="2011-04-25T06:32:51.057" UserId="2148" />
  <row Id="16946" PostId="9747" Text="Thanks, Wish me happy journey through time series." CreationDate="2011-04-25T07:04:39.897" UserId="861" />
  <row Id="16947" PostId="9940" Score="1" Text="That initial probability $n\Phi(c)$ only applies if the collection of $X_{i}$ are independent of $Y$ and $Z$ - so if that applies, then you are correct.  But you mentioned correlations, so it is likely your initial formula is not applicable.  If you have conditional independence given $Y$ and $Z$, then you would need to modify the constant $c$ in $\Phi(c)$ so that it is a function of $Y$ and $Z$." CreationDate="2011-04-25T07:30:11.500" UserId="2392" />
  <row Id="16948" PostId="9943" Text="@probabilityislogic: Thanks for pointing out the variance part.Its mistake while it typed it. Many thanks for ur answer. Got to know the importance of CF." CreationDate="2011-04-25T07:59:16.933" UserId="4290" />
  <row Id="16949" PostId="9940" Text="Say I keep the constant $c$ as it is. Then on average not more than a constant (any constant) amount of the $X_i$ should be under the threshold still, right?" CreationDate="2011-04-25T08:17:47.237" UserId="4312" />
  <row Id="16950" PostId="9926" Text="@cardinal Answer!" CreationDate="2011-04-25T08:24:37.310" UserId="88" />
  <row Id="16951" PostId="9911" Text="Sampling is more about balancing the impact of reducing sample size, so IMO it has nothing to do with your problem." CreationDate="2011-04-25T09:00:37.543" UserId="88" />
  <row Id="16952" PostId="9947" Text="@Arun If I understand you correctly, you want to match each (estimated) $k_i$ with only one (predefined) $c_j$?" CreationDate="2011-04-25T09:11:59.953" UserId="930" />
  <row Id="16953" PostId="9947" Text="@chi yes i wanted to have one-one mapping between estimated $k_i$ and predefined $c_j$" CreationDate="2011-04-25T09:17:13.853" UserId="4290" />
  <row Id="16954" PostId="9948" Score="2" Text="@Chethan As cross-posting is not encouraged, it would be better to remove your question on SO if you feel you'll be more likely to get a response there. As your question was asked few hours ago, it is fair to wait a little bit before cross-posting, though. Anyway, let's wait and see. If you get an acceptable response here, don't forget to remove the SO duplicate." CreationDate="2011-04-25T09:23:41.563" UserId="930" />
  <row Id="16955" PostId="9948" Score="2" Text="@chl definitely I will do that. I will remove one post once I get a favorable response in either of the sites." CreationDate="2011-04-25T09:31:21.053" UserId="4313" />
  <row Id="16956" PostId="9940" Text="If anything is unclear please let me know, and I will clarify as much as I can." CreationDate="2011-04-25T10:03:26.547" UserId="4312" />
  <row Id="16957" PostId="9943" Score="1" Text="@Arun, it should be noted that this result holds much more generally, in particular, when $\alpha$ is a **matrix** (of appropriate dimensions). Only the expression for the covariance chances slightly to accommodate this. Note that @probabilityislogic is (rightly) treating $\beta$ as an arbitrary **vector** of the correct dimension here. " CreationDate="2011-04-25T11:43:56.943" UserId="2970" />
  <row Id="16958" PostId="9935" Text="@Henry: The given model might be wrong but numerous articles in prestigious journals has been published based on this model. And this is one of the commonly used models in ecology." CreationDate="2011-04-25T11:58:50.723" UserId="3903" />
  <row Id="16959" PostId="9926" Text="@mbq: Ok, I'll try to type something up a bit later that is worthy of an answer. :)" CreationDate="2011-04-25T12:05:58.067" UserId="2970" />
  <row Id="16960" PostId="9935" Text="I am not suggesting that the model is wrong as such, but that it is unhelpful in that it is difficult to get accurate estimates of the parameters: different parameters would produce similar results, and you cannot easily get estimates of $N_e$ from $N_o$ since $N_e$ is an input to the calculation of itself." CreationDate="2011-04-25T12:20:44.230" UserId="2958" />
  <row Id="16961" PostId="9943" Score="1" Text="(+1) A couple minor quibbles: (a) Generalize to the matrix case mentioned above, (b) The *distribution* and not the pdf is characterized by the ch.f., so the last sentence should be changed and (c) Using `\exp` will get $\exp$ to render properly. " CreationDate="2011-04-25T13:22:02.197" UserId="2970" />
  <row Id="16962" PostId="9895" Text="Rookie question here: Is the N_e on the LHS interpreted more as an N_t and the N_e on the RHS more as an N_(t-1)?" CreationDate="2011-04-25T13:43:42.113" UserId="1764" />
  <row Id="16963" PostId="9937" Text="Do you actually need the column's index? You can also use x[,&quot;bar&quot;] to get the entire column, similar to the usual x\$bar. And in both cases, subscript the rows: x[2:3, &quot;bar&quot;] or x\$bar[2:3]. If you need to feed the index to a routine that requires an index, that's not helpful, of course." CreationDate="2011-04-25T14:57:10.190" UserId="1764" />
  <row Id="16964" PostId="2419" Text="@mbq, @Joris Meys, @Andre Holzner or @Gael, I am running into the same problem. Could any of you point me to those R boosted decision trees packages? (is it just rpart or are there others?) Thanks!" CreationDate="2011-04-25T15:35:09.563" UserId="2798" />
  <row Id="16965" PostId="9955" Text="that got me where I needed thanks!" CreationDate="2011-04-25T15:50:08.827" UserId="4306" />
  <row Id="16966" PostId="9955" Text="Apparently this site won't let me hit the up arrow cause I'm too new, but definitely what I needed" CreationDate="2011-04-25T15:52:43.567" UserId="4306" />
  <row Id="16967" PostId="9955" Text="@brandon Now, you can (15 rep needed)." CreationDate="2011-04-25T15:57:02.003" UserId="930" />
  <row Id="16968" PostId="9951" Score="1" Text="Well I do not understand the question, do you want to simplify the informal description of stationarity concept you quoted here? If you ask is this explanation correct, yes it is. Though non-stationarity has much more forms, any time-dependence in up-to second order moments will be the source for such in weak case (structural changes could be added, for example). Are you looking for some other quotes? " CreationDate="2011-04-25T16:37:32.280" UserId="2645" />
  <row Id="16969" PostId="2419" Text="@AmV Please, post another question. " CreationDate="2011-04-25T16:48:01.450" UserId="88" />
  <row Id="16970" PostId="9948" Score="2" Text="As promised I have deleted this question from SO. Therefore deleted the first part in the question now - &quot;I have also posted this question in StackOverflow....&quot;" CreationDate="2011-04-25T17:12:56.067" UserId="4313" />
  <row Id="16971" PostId="9948" Text="@Chethan Thanks." CreationDate="2011-04-25T17:23:35.717" UserId="930" />
  <row Id="16972" PostId="9955" Text="FYI, The Crane and Crotty reference (http://www.jstor.org/pss/2627674) is a valid URL, but when trying to get the PDF I get a message that &quot;the file is damaged and could not be repaired&quot;." CreationDate="2011-04-25T17:34:24.423" UserId="3919" />
  <row Id="16973" PostId="9963" Text="If you look closely at the figure (about one-third from the left) you can see where the minimum-distance heuristic is clearly violated.  The trick is to keep the *largest* distances pretty small.  Note, too, that this formulation is insensitive to the dimension of the original K-means problem: all that matters are the mutual distances between the two sets of coefficients." CreationDate="2011-04-25T17:58:50.380" UserId="919" />
  <row Id="16974" PostId="9952" Text="Thanks for your contribution.  It leaves some questions hanging, though: (a) In what sense do you see this as a multi-objective optimization problem?  (b) What do you specifically propose as the &quot;energy&quot; to minimize?  (c) Because there are $n!$ possible pairings, an exhaustive search will be prohibitively long once $n$ is much larger than $13$ or so (at which point there are over 6 billion pairings to examine...)." CreationDate="2011-04-25T18:15:16.060" UserId="919" />
  <row Id="16976" PostId="9962" Text="@Rana This might help: http://www.ats.ucla.edu/stat/r/dae/mlogit.htm." CreationDate="2011-04-25T19:05:03.513" UserId="930" />
  <row Id="16977" PostId="9951" Text="Thanks. I was hoping to get feedback pointing to important aspects of the stationarity (as a concept) which aren't there..." CreationDate="2011-04-25T20:04:25.033" UserId="333" />
  <row Id="16978" PostId="9704" Text="Oops, forgot the library (robfilter). I believe it calculates medians for runs of at least 50 samples." CreationDate="2011-04-25T20:14:39.717" UserId="1764" />
  <row Id="16979" PostId="9966" Text="There is a package in R called lars that handles common procedures related to least angle regression and lasso.  http://cran.r-project.org/web/packages/lars/index.html" CreationDate="2011-04-25T20:19:52.860" UserId="1118" />
  <row Id="16980" PostId="9958" Text="This is great, thanks Mike! I'll have a read and work out what I can from this stuff. I've heard of diffusion models before in various guises, but not really thought about trying to implement one. Do you know what is the standard? " CreationDate="2011-04-25T20:21:52.250" UserId="4204" />
  <row Id="16981" PostId="9905" Text="Thanks to you both - clearly I'm going to have to be very careful about how I approach the modelling process. As I understand, it's best to make as few assumptions as possible to avoid getting swamped in too many possibilities." CreationDate="2011-04-25T20:22:02.313" UserId="4204" />
  <row Id="16982" PostId="9966" Score="1" Text="What have you tried and why hasn't it satisfied you?" CreationDate="2011-04-25T20:46:18.300" UserId="1265" />
  <row Id="16983" PostId="9965" Text="Do you mean events have $N$ outcomes or $N$ *possible* outcomes?  What do you mean by &quot;testing&quot; an event?  (&quot;Tests&quot; seem to have &quot;values&quot; that might or might not be &quot;correct.&quot;  Does this mean a &quot;test&quot; is a measurement of something?)  Who &quot;chooses&quot; a test result for an incorrect test?  How is the correctness of a test determined?  Is it part of the data or just something hypothetical or theoretical?  What is your &quot;hypothesis&quot;?  It seems to refer to a probability, but it does not appear to make any assertions about the probability.  Do you mean you want to *estimate* $p$ and check for model fit?" CreationDate="2011-04-25T21:24:25.927" UserId="919" />
  <row Id="16984" PostId="9958" Text="I'm not sure what you mean when you ask about a standard, but here is a link to an R implementation of the diffusion model (written by Andrew Heathcote of Newcastle University): https://gist.github.com/cf6a07d42f70f9c4a6b7" CreationDate="2011-04-25T21:27:59.903" UserId="364" />
  <row Id="16985" PostId="9958" Text="And here's code for LBA: https://gist.github.com/941314" CreationDate="2011-04-25T21:49:11.150" UserId="364" />
  <row Id="16986" PostId="9958" Text="Neat, I just came across this project implementing a hierarchical Bayesian approach to fitting both diffusion and LBA: http://code.google.com/p/hddm/" CreationDate="2011-04-25T21:53:50.893" UserId="364" />
  <row Id="16987" PostId="9943" Text="Cardinal - thanks for the comments - just one minor counter quibble, I thought CFs were just a fourier transform of the PDF - hence you can get the PDF from the CF by inverting the Fourier transform.  How then is the PDF not uniquely defined by the CF?" CreationDate="2011-04-25T22:53:27.220" UserId="2392" />
  <row Id="16988" PostId="9915" Text="@Isaac, the list I've provided was not intended to overwhelm. It was intended simply to potentially help with brainstorming. Depending on the nature of the project, I would think that handling 1-2 of them would be the most that could be expected. Cheers." CreationDate="2011-04-25T22:56:56.530" UserId="2970" />
  <row Id="16989" PostId="9965" Text="@whuber: Each test is a measurement (with N possible outcomes, yes) which may be correct or incorrect.  I can only discover the correctness or incorrectness through the measurements, but it is true that exactly one outcome is correct, never 0 or 2." CreationDate="2011-04-25T23:13:25.183" UserId="1378" />
  <row Id="16990" PostId="9965" Text="&quot;Who 'chooses' a test result&quot; is a part of what I'm trying to test -- the guess is that all are equally likely if there is a measurement failure.  The hypothesis is that the model itself is a reasonable fit for the data, for any p (which I would then want to estimate); if for k = (1, 2, 3) I had (100, 100, 300) then the model would be a very bad match, regardless of what p value I chose and the number of discarded measurements." CreationDate="2011-04-25T23:21:18.153" UserId="1378" />
  <row Id="16991" PostId="9943" Score="1" Text="@probabilityislogic: Certainly the characteristic function is uniquely defined once the density is specified. The reverse is not (quite) true. First there is the issue of the existence of the pdf in the first place. Second, we can change the pdf on any set of measure zero and obtain the same characteristic function. Note that this includes dense(!) subsets of $\mathbb{R}$, for example, the rationals. The *distribution function* **is** uniquely determined by the ch.f., though." CreationDate="2011-04-26T00:15:56.650" UserId="2970" />
  <row Id="16992" PostId="6647" Text="This looks like a homework problem..." CreationDate="2011-04-26T00:22:41.647" UserId="4110" />
  <row Id="16993" PostId="9966" Score="1" Text="Have you looked at the [\[lasso\]](http://stats.stackexchange.com/questions/tagged/lasso) tag? There are several relevant questions there." CreationDate="2011-04-26T00:54:56.973" UserId="2970" />
  <row Id="16994" PostId="9972" Score="1" Text="It sounds as if you already have decided the 10 or so items that will feed into (comprise) your scale score. (You use the term &quot;dependent variable&quot; as if this scale score is going to be eventually explained or predicted by some other variables, but as I read you you have not indicated what they are--only the components of the dependent variable itself. Is that right?)  Further, are you looking for help designing the survey; computing your Engagement scale; or explaining/predicting it?" CreationDate="2011-04-26T01:43:50.287" UserId="2669" />
  <row Id="16995" PostId="9943" Text="@cardinal - I see your point, but my statement basically implies that a PDF exists (unless there is a way to derive a CF from a non-existent PDF - the vice versa).  I wouldn't mind seeing a specific example with numbers of the &quot;changing the PDF on a set of measure zero&quot; - this sounds odd to me, like one of the set theory annoyances (sounds related to the axiom of choice and some of its consequences)." CreationDate="2011-04-26T02:21:15.107" UserId="2392" />
  <row Id="16996" PostId="9943" Score="1" Text="@probabilityislogic, the pdf need not exist for the ch.f. to exist. Indeed, a ch.f. exists for *all* distribution functions $F$ (not just those with densities). This is a simple consequence of monotonicity of expectation, i.e., $|\mathbb{E} e^{i X}| \leq \mathbb{E}|e^{iX}| \leq 1$." CreationDate="2011-04-26T02:38:34.163" UserId="2970" />
  <row Id="16997" PostId="9943" Score="1" Text="@probabilityislogic, For an example of changing a pdf on a dense subset of the real line, let $\varphi(x)$ be the density of a standard normal and define $\bar{\varphi}(x) = \varphi(x)$ for all $x \not\in \mathbb{Q}$ and $\bar{\varphi}(x) = \varphi(x) + 1/q(x)$ for every $x \in \mathbb{Q}$ where $q(x)$ is the denominator of $x$ expressed in lowest terms. Then $\bar{\varphi}(x)$ is continuous at every irrational number $x$ and discontinuous at every rational number, yet the characteristic functions of $\varphi$ and $\bar{\varphi}$ coincide. " CreationDate="2011-04-26T02:39:45.703" UserId="2970" />
  <row Id="16998" PostId="9926" Text="@mbq: I've submitted an answer, though, unfortunately it's suboptimal. :)" CreationDate="2011-04-26T02:42:24.483" UserId="2970" />
  <row Id="16999" PostId="2927" Text="Testing a set of variables is not a form of regularization (shrinkage).  And testing gives one the temptation to remove variables, which has nothing to do with reducing overfitting." CreationDate="2011-04-26T02:47:21.043" UserId="4253" />
  <row Id="17000" PostId="9943" Text="@cardinal - I understand what you are saying here, but your proof depends on being able to distinguish a rational number from one that is irrational - and this can't be done without the axiom of choice (I think).  This is is what I meant by a specific numerical example.  For any irrational number can be represented as a limit of a rational one, just by adding more decimal places to the fraction.  Your proof relies on the comparison of two infinite quantities - this usually cannot be done without specifying how the &quot;infinities&quot; are to be generated." CreationDate="2011-04-26T03:43:38.310" UserId="2392" />
  <row Id="17001" PostId="9943" Text="@cardinal [this article](http://arxiv.org/ftp/math/papers/0305/0305326.pdf) provides a proof of what I am talking about - basically proves that if axiom of choice is accepted, then cardinality of rationals equals cardinality of irrationals or is bigger, hence the set $\mathbb{Q}$ does not have a measure zero, and your example doesn't apply" CreationDate="2011-04-26T06:07:44.660" UserId="2392" />
  <row Id="17002" PostId="9824" Score="1" Text="@cardinal +1, useful trick. This question turned out to be a pretty good reference." CreationDate="2011-04-26T06:30:46.773" UserId="2116" />
  <row Id="17003" PostId="9978" Text="+1 Thanks for the reference." CreationDate="2011-04-26T06:35:24.887" UserId="919" />
  <row Id="17004" PostId="9879" Text="@cardinal, I think Paul Erdos gave a good definition of mathematician, one, who has published an article in mathematical journal. It is not a perfect definition, but I think it is a very good one. " CreationDate="2011-04-26T07:11:50.887" UserId="2116" />
  <row Id="17005" PostId="9915" Text="all ideas are good or great, but I like &quot;Biclustering of users and topics&quot; best ... a recommender system for potentially interesting questions would be awesome." CreationDate="2011-04-26T07:12:20.080" UserId="264" />
  <row Id="17006" PostId="9943" Text="@probabilityislogic, the importance of axiom of choice is well described in wikipedia page, if you want to disregard it you have to have serious reasons. I've read the article in your link, and I suspect that it is false. The assumptions for the proof for example are a tad suspicious for me. Furthermore axioms are generally assumed not proven and given this result's enormous implication it is strange that half of the proof involves historical citations." CreationDate="2011-04-26T07:34:39.713" UserId="2116" />
  <row Id="17007" PostId="9915" Text="@cardinal +1, great list." CreationDate="2011-04-26T07:58:03.493" UserId="2116" />
  <row Id="17008" PostId="9943" Score="1" Text="@probabilityislogic, it's good to be able to recognize a crank when you see one. The countability of $\mathbb{Q}$ and uncountability of $\mathbb{R}$ are among the first things one typically learns in any introductory real analysis course. The proofs are elementary and the arguments beautiful." CreationDate="2011-04-26T08:15:02.753" UserId="2970" />
  <row Id="17009" PostId="9943" Text="@probabilityislogic, the proof that $\mu(\mathbb{Q}) = 0$ is (almost) equally elementary and beautiful. ($\mu$ here being Lebesgue measure or any measure absolutely continuous with respect to Lebesgue measure.)" CreationDate="2011-04-26T08:22:58.627" UserId="2970" />
  <row Id="17010" PostId="9943" Text="@cardinal - but they are both dense sets, so given any two real numbers, we can find a rational one in between them - countability is irrelevant.  But you can never actually &quot;write down&quot; an irrational number (should be called the fake numbers I think :) ), except as a limit of a sequence of rational numbers (such as square roots, $\pi$, euler's number)." CreationDate="2011-04-26T09:41:01.433" UserId="2392" />
  <row Id="17011" PostId="9967" Text="I don't know, I posted it on theoretical CS site, I got voted down. And they refer to this site" CreationDate="2011-04-26T10:07:09.417" UserId="4319" />
  <row Id="17012" PostId="9954" Text="@whuber, can you please elaborate on how \sum_{i}\alpha[i+h]\beta[i] is obtained. I am not able to understand how to get it! Also it would greatly help me if you can point me to a resource which explains the above concepts in a simple manner." CreationDate="2011-04-26T10:07:48.200" UserId="4313" />
  <row Id="17013" PostId="9975" Text="I was indeed using the Renyi entropy as a substitute for the Shannon entropy; it is nice to see confirmation of my intuition.  Thank you for the enlightening response." CreationDate="2011-04-26T10:11:09.660" UserId="3567" />
  <row Id="17014" PostId="9943" Text="@probabilityislogic, I fail to see your point or the relevance to the question, your answer, or my comments. Can you clarify? Have you had any exposure to (introductory) real analysis? If you're really disputing the countability of $\mathbb{Q}$ or its implication in establishing the Lebesgue measure of $\mathbb{Q}$, then I'm afraid you'll find you're in very sparse and (mathematically) naive company. That discussion may be better suited to a different medium than the comments to this answer." CreationDate="2011-04-26T10:12:04.590" UserId="2970" />
  <row Id="17015" PostId="9975" Score="1" Text="Many (but not all!) of the properties and usefulness of Shannon entropy arise from its convexity. If you look at the build up of the basics results in information theory, they more or less hinge on Jensen's inequality. So, in a certain (vague) sense, there is not too much that is (terribly) special about $-\log x$ as the particular nonlinearity that leads to a notion of &quot;information&quot;." CreationDate="2011-04-26T10:17:20.200" UserId="2970" />
  <row Id="17016" PostId="9984" Text="(+1) It might be worth noting *why* this is the case, as it may not be clear to all readers. For example, it follows from Markov's inequality." CreationDate="2011-04-26T10:28:34.740" UserId="2970" />
  <row Id="17017" PostId="9975" Text="I see.  Specifically, I need the property that the maximum entropy joint distribution which satisfies given marginals is the product of the marginals (what you would get from independence.)" CreationDate="2011-04-26T10:33:30.070" UserId="3567" />
  <row Id="17018" PostId="9984" Text="@cardinal, I think more pedestrian reason that the integrand function must at least converge to zero at infinity if we integrate over all line is sufficient. That is why I did not add any notes." CreationDate="2011-04-26T10:48:35.853" UserId="2116" />
  <row Id="17019" PostId="9939" Text="The code does not work for me. Line 5 looks strange and Emacs tells me that some parentheses are missing." CreationDate="2011-04-26T10:50:05.040" UserId="307" />
  <row Id="17020" PostId="9943" Text="@cardinal, agreed (discussion finished).  Basically a tangent gone too far from a proposed counter example." CreationDate="2011-04-26T10:50:47.643" UserId="2392" />
  <row Id="17021" PostId="9981" Score="2" Text="Can you provide the citation or link to the text you quote?" CreationDate="2011-04-26T10:51:01.467" UserId="1390" />
  <row Id="17022" PostId="9893" Text="I can't really answer your question, but this might be a place to start: http://en.wikipedia.org/wiki/Statistical_power" CreationDate="2011-04-23T03:17:32.700" UserId="4265" />
  <row Id="17023" PostId="9939" Text="There's a missing &quot;return&quot; - must have accidentally deleted it.  should be good now." CreationDate="2011-04-26T11:00:29.413" UserId="2392" />
  <row Id="17024" PostId="9973" Text="Fair enough, we could debate about whether or not the algorithm is statistics-related for a long time. The reason I mentioned that is that I know some researchers that work on this algorithm, and they never worry about its statistical properties (consistency, recovery, ...). The community revolving around it seems to be more of an optimization community than a statistical inference one." CreationDate="2011-04-26T11:08:48.363" UserId="1265" />
  <row Id="17025" PostId="9939" Text="Thanks, works now! However, I am having some difficulties to understand the plot: is it really $r_{xz}$ on both axes?" CreationDate="2011-04-26T11:31:09.653" UserId="307" />
  <row Id="17026" PostId="9984" Text="Sorry, I skimmed over your answer a bit too fast. I tend to think of heavy-tailed distributions in terms of upper-tail probabilities, which is where my remark originated from. For me, thinking about upper tails avoids having to make distinctions regarding whether a density exists or not." CreationDate="2011-04-26T11:47:26.670" UserId="2970" />
  <row Id="17028" PostId="9943" Text="@probabilityislogic, actually my counterexample is a bona fide one and I can provide even &quot;stranger&quot; ones if you'd like. For example, I can give you a distribution that has no atoms and yet has derivative of zero almost everywhere, so it has no density either. Yet, its characteristic function exists." CreationDate="2011-04-26T11:51:43.967" UserId="2970" />
  <row Id="17030" PostId="9958" Text="Whoa, that's awesome - thanks for the code and links! I think it's time to have some fun getting on with modelling my datasets!" CreationDate="2011-04-26T11:56:32.483" UserId="4204" />
  <row Id="17031" PostId="9824" Text="@mpiktas, yes, I agree that this question has turned out better than I would have initially guessed." CreationDate="2011-04-26T12:14:56.453" UserId="2970" />
  <row Id="17032" PostId="9939" Text="dog gone it! need to fix my code... again - plot() is wrong.  Ah the glorious tedium of computer code" CreationDate="2011-04-26T12:19:08.207" UserId="2392" />
  <row Id="17033" PostId="9084" Text="Would a radial kernel also work on this kind of data?" CreationDate="2011-04-26T12:54:34.107" UserId="2817" />
  <row Id="17034" PostId="9084" Text="@Zach I think so; also polynomial one should be OK" CreationDate="2011-04-26T13:04:03.917" UserId="88" />
  <row Id="17036" PostId="9989" Text="@GaBorgulya Thanks a lot for your quick response, but still it doesn't answer the basic question. How to define those hard-limits??&#xA;" CreationDate="2011-04-26T13:21:13.817" UserId="4331" />
  <row Id="17037" PostId="9987" Text="@Jeromy Anglim: Thanks for the editing and making me correct." CreationDate="2011-04-26T13:21:48.607" UserId="4331" />
  <row Id="17039" PostId="9993" Text="If that's your desire then you need to find out what the limits are for your data.  There are no general rules." CreationDate="2011-04-26T13:28:29.760" UserId="601" />
  <row Id="17040" PostId="9989" Text="There are no &quot;hard-limits&quot;. If you have no more information than a sample of 7 data points you probably don't know enough about the problem to achieve meaningful results." CreationDate="2011-04-26T13:32:23.263" UserId="3911" />
  <row Id="17041" PostId="9936" Text="@probabilityislogic, sorry to pester. That's not my intent. The argument to the ch.f. is treated as a real number and so, lacking more info, your update seems a bit heuristic. Indeed, in the complex plane, the ch.f. Is defined for every point on the imaginary axis only. The mgf, when it exists, is defined on an interval of the *real axis* around the origin. So, for an argument like yours to work, you'd need to know something more than just the ch.f. Do you mind providing the reference you've used for the development so far?" CreationDate="2011-04-26T13:34:48.943" UserId="2970" />
  <row Id="17045" PostId="9809" Text="@whuber, the key word here is universal. There are simple rules which can help a lot for a large class of integrals. I just wanted to make a point, that there are no similar rules like rules for differentiation. Rereading the question, I see that I missed the point :)" CreationDate="2011-04-26T14:17:55.970" UserId="2116" />
  <row Id="17046" PostId="9979" Text="Thanks for the answer Jeromy, I think you sum up the issue well.  If I were to reorganize the data with biomass type as an independent variable would that inflate my degrees of freedom? I have 70 experimental patches, can each plant group be though of as an independent group within those patches (n = 280)? I imagine I would need to nest plant group within patch? " CreationDate="2011-04-26T14:25:19.363" UserId="4326" />
  <row Id="17048" PostId="9954" Text="@Chethan The last example shows explicitly how this sum is computed.  I learned about the FFT method from the geostatistics literature long ago, but it's a stretch to characterize this as &quot;simple.&quot;  You can find an explicit description by Denis Marcotte in a [1996 paper](http://w3eos.whoi.edu/12.747/problem_sets/PS05/marcotte_96.pdf) in Computers &amp; Geosciences.  A [2010 PowerPoint presentation by Jim Jennings](http://reproducibility.org/wikilocal/docs/data_flows.ppt) summarizes the approach (starting p. 14)." CreationDate="2011-04-26T14:33:45.487" UserId="919" />
  <row Id="17051" PostId="9988" Text="&quot;Cluster Analysis&quot; means?" CreationDate="2011-04-26T14:39:51.487" UserId="919" />
  <row Id="17052" PostId="9936" Text="@cardinal - you are right in my argument being a bit heuristic.  My references are the article linked in, and my course notes from university - textbook for the course was *Options, Futures and Other Derivatives (6th Edition)* by John C Hull.  I can make it more precise by noting that the expected value of $X^{R}$ is equal to the $R$th derivative of the CF evaluated at $0$ divided by $i^{R}$.  Can show that for VG process, the CF is infinitely differentiable at zero - hence all moments exist, hence MGF exist, hence my quick way should give same answer." CreationDate="2011-04-26T14:42:45.260" UserId="2392" />
  <row Id="17053" PostId="9936" Text="(cont'd) you can also show more directly by first conditioning on the gamma process $E[\exp(X)|\Gamma]$ which is just a log-normal expectation.  Then you can integrate this conditional expectation w.r.t a gamma distribution, and all is good" CreationDate="2011-04-26T14:44:33.200" UserId="2392" />
  <row Id="17054" PostId="9988" Text="The answer to the first question is &quot;certainly.&quot;  Consider the geometric interpretation of PCA and apply that to (say) a pair of strongly negatively correlated variables.  The principal component must have a positive loading from one and a negative loading from the other." CreationDate="2011-04-26T14:44:39.627" UserId="919" />
  <row Id="17055" PostId="9991" Text="I think this response needs some qualification. So far the OP has not stated the target of the estimation--the estimand--so we cannot possibly know whether a statistic is biased or not.  For instance, the median is an *unbiased* estimator of the geometric mean of a (usually highly skewed) lognormal distribution." CreationDate="2011-04-26T14:46:57.503" UserId="919" />
  <row Id="17056" PostId="9994" Text="+1 For emphasizing the need to frame a question." CreationDate="2011-04-26T14:47:57.800" UserId="919" />
  <row Id="17057" PostId="9987" Score="1" Text="Asking for a &quot;hard-limit&quot; is misguided when you haven't specified the purpose of the statistics: the replies are going to push back on this.  Even when the research objective is placed in evidence, there will always be mitigating factors modifying any rule of thumb.  Using statistical procedures effectively and correctly always requires *judgment.*  It's not blind application of algorithms." CreationDate="2011-04-26T14:52:05.440" UserId="919" />
  <row Id="17058" PostId="9993" Text="@user4331 It sounds to me like you really should not add an auto-switch to your routine. It's an admirable goal, but as has been pointed out, it's a tricky question that depends on a knowledge of the data, your goals, and what is &quot;normal/usual&quot; for the data, and an auto-switch can't possibly know those things. So your auto-switch routine could be terribly misleading to its users, who would assume that the routine could somehow know better than they when to switch." CreationDate="2011-04-26T14:53:25.737" UserId="1764" />
  <row Id="17059" PostId="9991" Text="That's right for a large enough N (depends on degree of skew what large enough is).  But if the N is very small (like the examples posted) then medians are pulled by the tail more than means.  Bias in the mean is constant with N but varies with N for median.  That's what the poster is talking about." CreationDate="2011-04-26T14:54:34.987" UserId="601" />
  <row Id="17060" PostId="9993" Text="I might also add that if you want help for a specific kind of data for which you have some estimate of skewness expected and help for a specific plan for handling that data then you should post that question.  That might get you the answers you want." CreationDate="2011-04-26T14:57:16.870" UserId="601" />
  <row Id="17061" PostId="9936" Text="@probabilityislogic, that's starting to sound closer along with the verification of some necessary continuity assumptions. I still think you may only get an mgf for certain values of the parameters, though. Hull is a pretty good reference for intuition. His math tends to be very hand-wavy, though, if I recall. (I don't know the book well. It must be in its 20th edition or so, by now.)" CreationDate="2011-04-26T15:02:42.733" UserId="2970" />
  <row Id="17062" PostId="9084" Text="@mbq Thank you!" CreationDate="2011-04-26T15:50:19.067" UserId="2817" />
  <row Id="17063" PostId="9981" Text="@Gavin Simpson: Sure. I have updated my question with the link to the paper that contains this text. " CreationDate="2011-04-26T16:23:55.283" UserId="2164" />
  <row Id="17064" PostId="9986" Text="@Gavin Simpson: I was curious if the technique is a standard or they did something more. This clears it up. Thank you for your time and explanation. Really appreciate it. " CreationDate="2011-04-26T16:25:32.567" UserId="2164" />
  <row Id="17065" PostId="9937" Text="Bah, it keeps messing up my dollar signs. The bottom line is that there are two ways to use a column's name the dollar sign method and including it as a subscript." CreationDate="2011-04-26T16:53:15.203" UserId="1764" />
  <row Id="17066" PostId="9991" Text="@John Nothing of what you say comports with my understanding of these estimators, but perhaps that's because nobody has yet stipulated *what* is being estimated, nor have they clearly stated any distributional assumptions except that the distribution is &quot;highly skewed.&quot;" CreationDate="2011-04-26T18:25:50.740" UserId="919" />
  <row Id="17067" PostId="9994" Text="+1 For trimmed mean. It works really well in practice." CreationDate="2011-04-26T18:28:30.567" UserId="1265" />
  <row Id="17069" PostId="9999" Text="On the other hand, the additional information that is being picked up could be just a fluke of the data, and your accuracy estimator will be overly optimistic because of overfitting." CreationDate="2011-04-26T19:26:46.417" UserId="279" />
  <row Id="17070" PostId="7674" Text="@Henry Thanks for these kind words." CreationDate="2011-04-26T19:52:02.737" UserId="930" />
  <row Id="17071" PostId="8675" Text="That seems a bit extreme.  Type I and Type II is defined for both approaches.  The problem, it would seem to me, is that to say much about the comparison in error terms they'd have to know what sort of non-Normal they're up against (and if they could do that well enough, they might as well build a model and dump the bootstrapping *and* ranks :-)" CreationDate="2011-04-26T20:14:27.390" UserId="1739" />
  <row Id="17072" PostId="10003" Text="If you exclude the editorial board, the citation you provide could be used for description of stats.SE. " CreationDate="2011-04-26T20:24:23.587" UserId="2116" />
  <row Id="17073" PostId="10005" Score="6" Text="Your question is pretty vague and as it stands is more of a request for an introduction to plotting with R and less of a specific problem. If you provide sample data either by making it up or by using `dput()` to paste the contents of your actual data into the question - people can provide specific tips. Otherwise, [this tutorial](http://www.harding.edu/fmccown/r/) addresses lines and bar charts well." CreationDate="2011-04-26T20:24:27.690" UserId="696" />
  <row Id="17074" PostId="9990" Score="1" Text="Terms random effects, fixed effects, latent growth might mean different things in different contexts. Concerning the second one Andrew Gelman had a blog post with examples of several definitions. So it would be great if you provided the links to the definitions of these models. In general I think you are right in your assumption. Time trends are usually treated separately, since the usual assumption that variance of the regressors is bounded does not hold, so you have to show that for time trend this does not actually change anything in terms of model estimation and interpretation." CreationDate="2011-04-26T20:35:45.167" UserId="2116" />
  <row Id="17075" PostId="10003" Score="1" Text="@mpiktas I definitely appreciate the value of this site, but the editorial board has the potential to give the articles a more widely accepted currency of validity (e.g. C.V. value, pardon the pun) than the posts here. Furthermore, the format is fundamentally different, and the quality of answers on this site vary widely." CreationDate="2011-04-26T20:38:27.490" UserId="2750" />
  <row Id="17076" PostId="10003" Text="Statistical societies endorsed it; tenure committees, however... It seems to me that they only partly solved the problem of getting academics to contribute to sites like this (that is, editorial control and conspicuous attribution). " CreationDate="2011-04-26T20:41:15.087" UserId="26" />
  <row Id="17077" PostId="10003" Text="Also, the board listed on the site doesn't seem to be accurate; in the blog post you cite Christian Robert indicates that he, Nando de Freitas, and George Casella all resigned -- last year -- but are still on the StatProb list! " CreationDate="2011-04-26T20:43:14.693" UserId="26" />
  <row Id="17078" PostId="10008" Text="Exactly what do you plan to do with this correlation once you compute it?" CreationDate="2011-04-26T21:14:10.830" UserId="919" />
  <row Id="17079" PostId="10008" Text="We would deduce whether the two users according to their sports interest are related or not. Something like If I like soccer and football, and some one likes just football, still we both are related by some percentage. So i want to calculate that percentage of similarity" CreationDate="2011-04-26T21:24:49.500" UserId="4341" />
  <row Id="17080" PostId="9986" Text="@Gavin Simpson: I hope you won't mind if I ask a related question. Why is ranking important? Is there some effect is it addressing here?" CreationDate="2011-04-26T21:58:24.583" UserId="2164" />
  <row Id="17081" PostId="9986" Text="@Legend in the paper you link to, the authors state &quot;Since the test is based on ranks, it is robust to outliers and requires no special distributional assumptions.&quot; So there are two reasons. I'm not really that familiar with the CUSUM literature though; my experience is with these tests for structural changes in regression models, so the CUSUM is on residuals and there the actual residuals are used, not ranks, but in other respects the CUSUMs are calculated in the manner described.&#xA;" CreationDate="2011-04-26T22:10:28.580" UserId="1390" />
  <row Id="17082" PostId="10008" Text="Do you have a large matrix where rows are users and sports are columns and values are measures how much someone likes a sport?" CreationDate="2011-04-26T22:21:45.157" UserId="3911" />
  <row Id="17083" PostId="10008" Text="yes, i have a matrix for individual users where columns are different sports and the one they like have a value one." CreationDate="2011-04-26T23:12:31.037" UserId="4341" />
  <row Id="17084" PostId="10011" Score="3" Text="This is a [mixture modeling](http://en.wikipedia.org/wiki/Mixture_model) problem.  Naming it is half the battle - I'll let somebody else take care of the other half." CreationDate="2011-04-26T23:37:32.127" UserId="71" />
  <row Id="17085" PostId="10011" Text="If you're comfortable with R, check out the [CRAN Task View on clustering](http://cran.r-project.org/web/views/Cluster.html) - just searching the page for &quot;mixture&quot; will help you find related packages." CreationDate="2011-04-26T23:40:19.563" UserId="71" />
  <row Id="17086" PostId="10011" Text="I don't want to use unsupervised clustering here. What about using Likelihood-ratio test? Is it a possible solution? if so, is there anyway to do that in R?" CreationDate="2011-04-27T00:00:39.527" UserId="2725" />
  <row Id="17087" PostId="10002" Text="I still don't fully understand lme4 syntax, so I'm curious to see the answer. But I have a hunch that it's related the following difference: P is the amount of time a student spends doing homework, R is a treatment at the class level and F is the student. (We'd should also have a random effect for the class itself.) If all students are subject to all treatments R at different times, the levels of F are comparable across classes. If we measure a whole school all at once, we have different students in each class, so the levels of F in different classes don't have anything to do with each other." CreationDate="2011-04-27T00:23:47.937" UserId="3874" />
  <row Id="17088" PostId="9988" Text="Cluster analysis doesn't generate factors.  It finds groups of cases that have multiple characteristics in common.  The words &quot;Cluster Analysis can only cluster into the same factor&quot; don't make sense to me." CreationDate="2011-04-27T01:41:13.197" UserId="2669" />
  <row Id="17089" PostId="9969" Text="I have to reread your answer again to make sure I really understand. But in the first part you seem to construct a counterexample; yet in your last paragraph you say &quot;if the expected number of X_i below the threshold c, conditional on Y≤c, is much larger than pn, then the probability that Y≤c would have to be greater than Φ(c), implying Y does not have a standard Normal distribution.&quot; So you are suggesting that one can not have many more X_i under the treshold than pn, and thus if p=3/n also no more than a constant amount of X_i will be under it?" CreationDate="2011-04-27T02:10:06.667" UserId="4312" />
  <row Id="17090" PostId="9988" Text="Based on your (@whuber and @rolando2's) comments it sounds like the person who sent me an email may be speaking about statistics in a psychological, possibly SPSS-centric, way. @whuber &quot;Cluster analysis&quot;: http://en.wikipedia.org/wiki/Cluster_analysis; @rolando2 It is possible to cluster variables e.g., with SPSS hiearchical cluster analysis by variables." CreationDate="2011-04-27T04:47:34.083" UserId="183" />
  <row Id="17091" PostId="9987" Text="@whuber Thank-you whuber for sharing the valuable thoughts. I totally agree with you that there is no rule of thumb, to decide whether to use mean/median for average. But it makes me to think when I used this tool: http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MeanTest.htm&#xA;&#xA;Here, he is coming up with conclusions how far mean holds good with a given distribution. May be he is using Standard-Error, but what are these limits which changes the &quot;conclusion&quot; in his tool." CreationDate="2011-04-27T06:46:37.163" UserId="4331" />
  <row Id="17092" PostId="9994" Text="@Jeromy Anglim Thank you Jeromy for writing and understanding me more precise what I'm asking about. But as I said Jeromy, I want those hard-limits, and I came across with those hard-limits at:&#xA; For mean: http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MeanTest.htm&#xA; For median:http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/MediansTest.htm&#xA;&#xA;He has designed something around what I'm talking about. But I really couldn't find any pointers which states how to achieve this kind of solution, what he did in his tools." CreationDate="2011-04-27T06:55:43.753" UserId="4331" />
  <row Id="17093" PostId="10017" Score="3" Text="@ricardh, curse upon all economists for not checking whether the term they want to use is already used in statistics. Cluster in this context means group and is totally unrelated to cluster analysis, this is why rseek  gave you unrelated results. Hence I removed the clustering tag. For panel data analysis check out [package plm](http://cran.r-project.org/web/packages/plm/index.html). It has a nice vignette, so you may find what you want. As for your question it is not clear what you want. Within group standard errors? " CreationDate="2011-04-27T07:03:02.493" UserId="2116" />
  <row Id="17094" PostId="9989" Text="@GaBorgulya: I agree with you, that these 7 data-points are almost meaning-less if there is no context around them. But GaBorgulya, I'm not very concerned about the information around these sample data-points, but what I'm concerned about is- how these data-points should be treated in-general. And as I quoted in few of my comments, I found these hard-limits at:  &#xA;http://bit.ly/ibQhWo, &#xA;http://bit.ly/fWhRTD&#xA;&#xA;I'm not saying what he is doing is very optimal or good solution, but what I'm saying is he came up with interesting conclusions, which are really making some sense for me." CreationDate="2011-04-27T07:03:36.197" UserId="4331" />
  <row Id="17095" PostId="10017" Text="@ricardh, it would help a lot if you could link to some manual of Stata where this `cluster` option is explained. I am sure it would be possible to replicate in R. " CreationDate="2011-04-27T07:04:33.257" UserId="2116" />
  <row Id="17096" PostId="9999" Text="@aniko - while true cross validation doesn't fix this problem - it relies on the data being a &quot;realistic&quot; version of the data one is likely to predict from.  No method which uses only the data can account for this (because by definition, it is based only on the data)." CreationDate="2011-04-27T07:09:20.963" UserId="2392" />
  <row Id="17097" PostId="10013" Text="@user4338, as I've said in the answer to one of the questions you link, there is nothing fancy in using exogenous variables in VECM. If they really are exogenous with asymptotically bounded variance (no time trends) then they can be treated as dummy variables.  " CreationDate="2011-04-27T07:11:04.997" UserId="2116" />
  <row Id="17098" PostId="9993" Text="@John I'm talking about context-less data. I'm talking about any data in-general." CreationDate="2011-04-27T07:12:43.100" UserId="4331" />
  <row Id="17099" PostId="9994" Text="@user4331 as far as I can see these links go to applets for doing significance testing related means and medians, and do not tell you when you should use one over the other." CreationDate="2011-04-27T08:15:45.900" UserId="183" />
  <row Id="17100" PostId="10017" Score="1" Text="+1 for that comment. economists colonize terminology like crazy. Though sometimes it's hard to pick the villain. Ii took a while e.g. until I realized `factor` had nothing to do with `factanal` but refers to categorized variables. However cluster in R refers to cluster analysis, k-means is just THE partitioning method: http://www.statmethods.net/advstats/cluster.html . I don't get your question, but I also guess cluster has nothing to do with it. " CreationDate="2011-04-27T09:17:06.240" UserId="704" />
  <row Id="17101" PostId="10012" Text="@Thomas The OP said the number of liked objects is not the same for the two raters. How would you apply Cohen's $\kappa$ in this case?" CreationDate="2011-04-27T09:33:22.430" UserId="930" />
  <row Id="17102" PostId="10017" Text="@mpiktas, @ran2 -- Thanks! I hope I clarified the question. In short, why does &quot;standard error clustering&quot; exist if it is just fixed effects, which already existed?" CreationDate="2011-04-27T09:33:51.543" UserId="1445" />
  <row Id="17103" PostId="10017" Text="@ricardh, the link you gave mentions `cluster` explicitly in robust covariance matrix estimates context. Your question it seems is not about covariance matrices. Your first code is least squares dummy variables regression and the second fixed effects model. They give the same coefficient estimates, but since you use `lm` for estimation of second model, it calculates degrees of freedom as if you estimate simple linear model, which is not the case. You say that you came across some research, can you link to it?" CreationDate="2011-04-27T10:13:41.023" UserId="2116" />
  <row Id="17104" PostId="10021" Text="@Arun, I am not sure that I understand even the univariate case. By $\mu_2=\mu_1\pm 2\sigma_1$ you mean that both answers are ok?" CreationDate="2011-04-27T10:24:00.717" UserId="2116" />
  <row Id="17105" PostId="10021" Score="1" Text="@Arun, what is the context of this problem? As it stands now, it is impossible to answer, since in multivariate case the means are vectors and the distance between them is a number, but the covariance matrix is not." CreationDate="2011-04-27T10:26:01.213" UserId="2116" />
  <row Id="17107" PostId="9994" Text="@Jeromy Anglim Yes, they are doing significance testing. But do you noticed that if you club these techniques and apply them in the illustrated fashion, than they can give you some admirable outputs:-&#xA;for-ex: Start with a sample-distribution, find whether it passes the mean-test or not(using those applets); If 'yes' than go with mean; If 'no' than go for median test;Now find whether it passes the median test or not; If 'yes' than go with median; Else go with IQR. &#xA;&#xA;And I believe this approach can work out to find a better average, for in-general sample-distributions." CreationDate="2011-04-27T10:46:24.837" UserId="4331" />
  <row Id="17108" PostId="10017" Text="@mpitkas -- It's not explicitly about vcov matrices, but in economics the focus is on the estimate and the standard error. Since it seems to me that &quot;clustering&quot; standard errors gives the same results (in terms of estimates and standard errors) as fixed effects, why does the practice of clustering exist? There must be some difference that I'm missing?" CreationDate="2011-04-27T10:59:08.240" UserId="1445" />
  <row Id="17109" PostId="10011" Text="If this is your only variable, you're going to do some sort of unsupervised learning, and since you're trying to divide it into two groups, you're going to do some sort of clustering. The relevant techniques that aren't called &quot;unsupervised clustering&quot; were probably just invented by statisticians instead of machine-learning people." CreationDate="2011-04-27T10:59:18.533" UserId="3874" />
  <row Id="17110" PostId="9999" Text="The number of times you repeat the whole cross-validation process is also important.  My machine learning colleagues feel that 10-fold cross-validation should be repeated 100 times to achieve good precision.  A single 10-fold cross-validation yields an accuracy measure that has a high variance.  The choice of accuracy measures is also important.  You are using an improper scoring rule that is also discontinuous, so expect more problems." CreationDate="2011-04-27T11:37:16.883" UserId="4253" />
  <row Id="17111" PostId="10011" Score="1" Text="@Jana you tagged your question &quot;statistical-significance&quot;. Does your question have such an aspect as well?" CreationDate="2011-04-27T11:44:03.400" UserId="3911" />
  <row Id="17112" PostId="10010" Text="Do your spectra have their peaks at the same locations, but varying powers? Or do the peaks shift? Are the peaks independent or interrelated? Such a priory information may help choosing a much powerful model than feeding your raw data into a generic data mining algorithm." CreationDate="2011-04-27T12:03:25.020" UserId="3911" />
  <row Id="17114" PostId="9986" Text="@Legend, @Gavin, the basic idea behind the CUSUM tests is that under null hypothesis  you sum up zero mean independent identically distributed random variables. This sum converges to random walk. If there is a change point, then the convergence does not happen. So ranking is important here, since under null hypothesis ranks are independent. Note that the actual convergence should be proved, since for funky samples you can get that there is no convergence even in case of no change point. Time trends come into mind." CreationDate="2011-04-27T12:25:31.157" UserId="2116" />
  <row Id="17115" PostId="9988" Text="@Jeromy - that's a useful reminder to make, but still, clustering variables wouldn't result in a &quot;factor&quot; in the usual sense, i.e., there'd be no analog to factor loadings for the different variables (although there would be factor scores).  Is that right?" CreationDate="2011-04-27T12:33:47.287" UserId="2669" />
  <row Id="17116" PostId="9986" Text="@mpiktas but residuals are assumed independent random variables so ranks are not a requirement - the `strucchange` package doesn't use ranked residuals IIRC." CreationDate="2011-04-27T12:45:04.017" UserId="1390" />
  <row Id="17117" PostId="9988" Text="@Jeromy The point, amply illustrated by your reference to Wikipedia, is that &quot;cluster analysis&quot; means a huge number of different things.  Without specifying which kind of cluster analysis your student has in mind, the question is vague." CreationDate="2011-04-27T12:53:07.597" UserId="919" />
  <row Id="17118" PostId="9896" Text="Great answer and brought up issues I hadn't thought about.  Perhaps I'll use your equation and comments to stress how important it is that the sample size grows to reflect increase in factors." CreationDate="2011-04-27T12:54:21.433" UserId="4346" />
  <row Id="17119" PostId="9969" Text="@Josiane That's correct.  In fact, the expectation of the number of $X_i$ less than $c$ is at most $pn + O(\sqrt{n})$, so if $p$ changes with $n$ like $O(1/n)$, the number of such $X_i$ is a constant plus $O(n^{-1/2})$." CreationDate="2011-04-27T12:55:36.597" UserId="919" />
  <row Id="17120" PostId="10026" Text="@Frank. To confirm, your approach is to compute L1 as the reduction in deviance (-2*) resulting from the inclusion of the four social variables, adjusted by the df of these four variables? And likewise in turn for L2 and L3?" CreationDate="2011-04-27T12:57:11.893" UserId="2040" />
  <row Id="17121" PostId="9986" Text="@Gavin, strictly speaking residuals are not independent, but that does not matter. Note that in the article original data is ranked not residuals. I think this may be a crucial assumption. The approach described would not work on residuals." CreationDate="2011-04-27T13:19:47.080" UserId="2116" />
  <row Id="17122" PostId="10030" Score="1" Text="@Massimo, please provide link, internet is a large place." CreationDate="2011-04-27T13:22:55.580" UserId="2116" />
  <row Id="17123" PostId="9986" Text="@mpitkas I think we are talking at crossed purposes. I thought you are commenting on the comments here? CUSUM can be applied to model residuals - see the `strucchange` package - and I only mentioned that this was my experience with CUSUM tests. Note I said _assumed_  - from a linear regression say we might CUSUM on the assumed IID ~N(0, $\sigma$) residuals." CreationDate="2011-04-27T13:32:35.847" UserId="1390" />
  <row Id="17124" PostId="9988" Text="@rolando2 and @whuber good points." CreationDate="2011-04-27T13:41:56.573" UserId="183" />
  <row Id="17125" PostId="10031" Score="2" Text="Your title and text seem to contradict to each other: are there no events in the outcome variable, or is there an input (predictor) variable for which one potential value has not been observed? Please clarify." CreationDate="2011-04-27T14:01:29.147" UserId="279" />
  <row Id="17126" PostId="10032" Text="Wow, good sleuthing - I would have never thought of that. Based on the short description of the data, it seems unlikely that this was the intention (especially for `b`), but clearly that's what the online calculator is doing." CreationDate="2011-04-27T14:12:20.140" UserId="279" />
  <row Id="17127" PostId="10032" Text="@Henry +1 for replicating the value of KS test. I would like to add that I suspect that R does not think the way you suggested. `c(0,1,1,9,4)` is simply the vector with values 0,1,1,9,4. I do not find any suggestion in ks.test manual page to suggest otherwise." CreationDate="2011-04-27T14:18:09.490" UserId="2116" />
  <row Id="17128" PostId="9985" Text="@Roah: Right! This is kind of the idea. Maybe should put up a wiki toward which everyone can contribute. (My next project once I'm done with my dissertation :&gt;) Was never aware of Crofton's method btw. Pretty cool!" CreationDate="2011-04-27T14:21:07.380" UserId="3577" />
  <row Id="17129" PostId="9890" Text="@JS01: Definitely a nice reference. Didn't find what I was looking for...but I'll hold onto this. I'm sure it'll come in handy at some point :)" CreationDate="2011-04-27T14:22:07.113" UserId="3577" />
  <row Id="17130" PostId="9847" Text="@JMS: Thanks for the info. I've been avoiding buying Mathematica for some time. But after the advice here I started checking some videos and looking at their site in more detail. I may have to buy it. Some great features. I'll just add that [link](http://reference.wolfram.com/mathematica/ref/Expectation.html) is a nice list of Mathematica's capabilities in regards to solving expectations" CreationDate="2011-04-27T15:16:54.377" UserId="3577" />
  <row Id="17131" PostId="9847" Text="@brianjd Nice link, thanks; when I've needed expectations I've just been typing out the integral but I think I'd be better off using the probability distribution facilities (via ProbabilityDistribution[]) which takes a pdf + support and gives you not just expectations but also moments, chf, etc. Needless to say I haven't properly learned the software yet :)" CreationDate="2011-04-27T15:46:39.600" UserId="26" />
  <row Id="17132" PostId="10040" Text="I have been trying to connect R to the database and have been failing mightily. Any suggestions? " CreationDate="2011-04-27T15:56:00.880" UserId="1514" />
  <row Id="17133" PostId="10043" Text="can that do ANOVA?" CreationDate="2011-04-27T16:04:56.423" UserId="1514" />
  <row Id="17134" PostId="10040" Score="1" Text="http://stackoverflow.com/questions/1408897/connecting-gnu-r-to-postgresql" CreationDate="2011-04-27T16:13:50.763" UserId="3911" />
  <row Id="17135" PostId="10043" Text="As suggested by GaBorgulya, you are better off using a stat package. Unless PostGre is able to do multiple regression (which looks like it does not), you are limited to testing the difference between 2 groups since your two groups could be dummy coded into 1 variable.    http://psych.unl.edu/psycrs/statpage/coding_eg.pdf" CreationDate="2011-04-27T16:43:41.690" UserId="3489" />
  <row Id="17136" PostId="9928" Text="@cardinal Thanks for editing! I cleaned it up further and removed my erroneous comment re P(x)=0 for x&gt;M being fat-tailed." CreationDate="2011-04-27T16:45:33.797" UserId="1566" />
  <row Id="17137" PostId="10041" Text="@Simon: Exactly what I was looking for!  " CreationDate="2011-04-27T16:51:40.110" UserId="1643" />
  <row Id="17138" PostId="9936" Text="@probabilityislogic Thanks! The variance gamma distributions might do exactly what I want. I saw the Levy distribution on http://en.wikipedia.org/wiki/Heavy-tailed_distribution but ignored it, since it was one-tailed. I'm guessing it can be adjusted to be symmetric around 0?" CreationDate="2011-04-27T16:53:36.580" UserId="1566" />
  <row Id="17139" PostId="10046" Text="@Suresh: Thanks, it's intuitive that objective function decreases, but I wasn't sure that it decreases strictly. I wanted to make sure that there is no patological case just like in linear programming" CreationDate="2011-04-27T16:55:33.457" UserId="1643" />
  <row Id="17140" PostId="10047" Text="I knew about Google Fusion Tables from Google Maps' API, but didn't realize there was a free standalone version. Thanks!" CreationDate="2011-04-27T17:29:12.667" UserId="1566" />
  <row Id="17141" PostId="9851" Text="It turns out you're right. wunderground confirmed they are primary. Maybe there are no other sources?!" CreationDate="2011-04-27T17:51:24.760" UserId="1566" />
  <row Id="17142" PostId="9974" Text="Thanks a lot, I figured it out now." CreationDate="2011-04-27T18:24:10.130" UserId="4320" />
  <row Id="17143" PostId="9847" Text="@JMS: One follow-up question for you: Does what I'm doing above (added more to the question in marked edits) look like something that might be able to be done in Mathematica? I'd like to avoid a dead end if possible. But it does look promising with the existence of the `erf` function. " CreationDate="2011-04-27T18:46:37.780" UserId="3577" />
  <row Id="17144" PostId="10049" Score="4" Text="I can obtain a misclassification rate of just 10% with these data, sight unseen, by always predicting 0!  Surely *any* method you try will improve on this, right?  Perhaps some more information about the quality of your results would help us understand the situation better." CreationDate="2011-04-27T18:57:54.193" UserId="919" />
  <row Id="17145" PostId="10051" Text="More information on the nature of the repetitions would be helpful. Perhaps clustered survival data methods including frailty models apply (and perhaps not)." CreationDate="2011-04-27T19:16:12.553" UserId="279" />
  <row Id="17146" PostId="10046" Text="Well yes and no. While it converges, it can take exponential time, much as how simplex does. Moreover, for both problems, you can show that &quot;smoothed&quot; variants converge in polynomial time" CreationDate="2011-04-27T19:26:32.740" UserId="139" />
  <row Id="17147" PostId="10005" Text="@Owusu This a duplicate of your [preceding question](http://stats.stackexchange.com/questions/10005/how-to-convert-a-table-into-a-graph-in-r). Please consider updating the former one instead (and delete this one afterwards). And don't forget to register your account, as indicated on the [FAQ](http://stats.stackexchange.com/faq). Finally, I removed the `graph-theory` tag in your preceding post because it really has nothing to do with [graph theory](http://en.wikipedia.org/wiki/Graph_theory)." CreationDate="2011-04-27T19:58:29.507" UserId="930" />
  <row Id="17148" PostId="10027" Text="@GaBorgulya +1 for adding context and the interesting points you raised. Is this a validated scale? If so, I assume a cutoff has already been established because it should not be sample-dependent, for interpretation purpose. Good point about the PPV and prevalence." CreationDate="2011-04-27T20:05:08.360" UserId="930" />
  <row Id="17149" PostId="10028" Text="@mbq +1 and thanks for the links." CreationDate="2011-04-27T20:17:07.787" UserId="930" />
  <row Id="17150" PostId="9988" Text="@rolando2 @whuber As @Jeromy said, we can apply (hierarchical) cluster analysis (CA) onto the variables, as we would do for highlighting homogeneous groups of individuals, based on a certain metric. Another way of constructing scales is to apply *item clustering*, as proposed by William Revelle (Multivariate Behavioral Research 1979 14:57-74), based on [Cronbach's alpha](http://bit.ly/mKsFdi). Like for multidimensional scaling, the object of interest (variables or individuals) is choosen by the researcher. Now, I must admit that the question looks vague because nothing is said about CA..." CreationDate="2011-04-27T20:32:49.687" UserId="930" />
  <row Id="17151" PostId="10036" Text="I don't know the formula you are trying to implement, but `dnorm` cannot possibly be right: it is the normal density, which will be very small at most points. " CreationDate="2011-04-27T20:33:01.263" UserId="279" />
  <row Id="17153" PostId="10051" Text="There are n subjects, each one has k tasks timed.  I wish to compare the timing of the tasks between each other (are there timing different or not).  But they are dependent through the subject.  (Thanks)" CreationDate="2011-04-27T20:52:13.440" UserId="253" />
  <row Id="17154" PostId="10058" Score="2" Text="The plot suggests that the effect of wet/dry is different for the two species. How about a `species:type2` interaction term?" CreationDate="2011-04-27T20:53:13.783" UserId="279" />
  <row Id="17155" PostId="10060" Text="Yes, but all of your explanatory power goes out the window in this setting.  Besides, if it is this obvious why do so many academic papers ignore it?" CreationDate="2011-04-27T20:53:48.960" UserId="4352" />
  <row Id="17156" PostId="10035" Text="Thank you I will read a bit about this" CreationDate="2011-04-27T20:59:23.290" UserId="4342" />
  <row Id="17157" PostId="10010" Text="@GaBorgulya I added some more information to the question" CreationDate="2011-04-27T20:59:34.973" UserId="4342" />
  <row Id="17158" PostId="10049" Text="@whuber: Yes, we can always predict 0 and get misclassification rate(MIR) of &lt; 10%. But, the models that I am using make prediction by reducing error, they do not look at the proportion of the classes (i.e.) 90:10. The dataset I am working on is BRFSS 2009 dataset which is available here in [csv format](http://www.hofroe.net/stat579/brfss%2009/brfss-2009-clean.csv). The target variable is DIABETE2.&#xA;http://stats.stackexchange.com/posts/10049/edit. I am not sure about what you mean on information about the quality of my results." CreationDate="2011-04-27T21:08:06.853" UserId="3897" />
  <row Id="17159" PostId="10032" Text="@mpiktas: that is what I meant by &quot;one value of 0, two values of 1, one value of 9, and one value of 4&quot;." CreationDate="2011-04-27T21:12:00.433" UserId="2958" />
  <row Id="17160" PostId="10058" Text="@Aniko, you are very observant. The interaction term is indeed significant. The two species seem to be mutually exclusive." CreationDate="2011-04-27T21:15:04.253" UserId="144" />
  <row Id="17161" PostId="9847" Text="Perhaps; $p(X,Y)$ is nice enough. Of course the problem as ever is that there may not be a simpler expression than that integral :) Why is a numeric solution no good?" CreationDate="2011-04-27T21:37:27.000" UserId="26" />
  <row Id="17162" PostId="10060" Text="@SpeedBoots Maybe the academic papers you are reading are focused on explaining the past rather than predicting the future.  Furthermore, it could be that there are certain variables that predict a stock's price going up.  The January effect would be an example of one such model." CreationDate="2011-04-27T21:46:35.540" UserId="2817" />
  <row Id="17163" PostId="10005" Text="Some data seems to be in the later almost duplicate question at http://stats.stackexchange.com/questions/10055/how-to-convert-a-table-into-a-graph-in-r " CreationDate="2011-04-27T21:48:03.993" UserId="2958" />
  <row Id="17164" PostId="10053" Text="@Gavin: should it be closed or could it be moved?" CreationDate="2011-04-27T18:44:20.237" UserId="1657" />
  <row Id="17165" PostId="10053" Text="@Joshua If it can be moved by a mod, moved would be best as @DrewConway has done a reasonable job with the Q and reproducibility. How do we achieve a migrate? Flag the post?" CreationDate="2011-04-27T18:48:11.420" UserId="1390" />
  <row Id="17166" PostId="10053" Text="Damn, I was writing an answer and had to move for 10 mins from machine. And now I came back and saw your comments. =)" CreationDate="2011-04-27T18:56:40.273" UserId="1356" />
  <row Id="17167" PostId="10053" Text="Thanks for the help" CreationDate="2011-04-27T19:02:52.090" UserId="302" />
  <row Id="17168" PostId="10049" Score="1" Text="By &quot;quality&quot; I meant more details about the misclassification rate.  For instance, there are two kinds of misclassification: predicting the 0's as 1's and predicting the 1's as 0's.  Also, how are you doing the classifying?  With such an imbalance of 0's and 1's it's unlikely that setting the classification threshold to a logit of 0 is best: you will need a different threshold.  " CreationDate="2011-04-27T22:06:45.897" UserId="919" />
  <row Id="17169" PostId="10047" Text="Turns out Fusion Tables are limited to 100M/table and 250M/user, and my data's a lot bigger than that. Checking out Google Public Data now." CreationDate="2011-04-27T22:08:50.333" UserId="1566" />
  <row Id="17170" PostId="10005" Score="1" Text="This question has been merged with its duplicate.  The new text is now marked by the &quot;Edit&quot;.&#xA;" CreationDate="2011-04-27T22:11:09.200" UserId="919" />
  <row Id="17171" PostId="9847" Text="@JMS: Oh, just because a closed form solution would be nice and convincing. I do have something of a numeric solution. I can find $\alpha$ for particular values of the parameters. I suppose the next step is to do this for many values of the parameters and do some sort of polynomial fitting to get the relationship between $\alpha$ and the other parameters. This is at the periphery of my knowledge though. " CreationDate="2011-04-27T22:12:56.410" UserId="3577" />
  <row Id="17172" PostId="9840" Text="@GaBorguyla, thanks for the pointer. My roommate is actually one of William Stein's students...so I should probably have a chat with him :)" CreationDate="2011-04-27T22:22:53.140" UserId="3577" />
  <row Id="17173" PostId="10065" Text="What do you mean by *inverted list*?" CreationDate="2011-04-27T22:55:07.920" UserId="2970" />
  <row Id="17174" PostId="10065" Text="I meant inverted index, that is for each for w I have a list of documents in which a particular word w occurs.&#xA;inverted_index[w] = {doc1, doc2, doc6}. In a similar fashion as index of terms at the back of the book." CreationDate="2011-04-27T23:23:34.100" UserId="1371" />
  <row Id="17175" PostId="10059" Text="You mean like clustering on a sample to quickly get good starting centroids for the 'full' pass?  Or more like two different methods altogether?" CreationDate="2011-04-28T00:06:17.823" UserId="595" />
  <row Id="17176" PostId="9840" Text="@GaBorguyla: I did talk to my roommate. He said that Sage isn't great for this yet. " CreationDate="2011-04-28T00:59:57.280" UserId="3577" />
  <row Id="17177" PostId="9920" Text="@chl yes, it is similar to the last question, I was told that you are more likely to get a response when you ask a new questions, hence the new question (as noone answered the updated question and I don't have enough points for a bounty yet!), I will try to adjust the old question so that they differ more, and only ask new questions in future." CreationDate="2011-04-28T01:47:06.340" UserId="2025" />
  <row Id="17178" PostId="10026" Text="I didn't use the best notation.  I mean the likelihood ratio $\chi^2$ statistic, which is the change in -2 log likelihood upon removing the set of variables being tested." CreationDate="2011-04-28T01:54:09.477" UserId="4253" />
  <row Id="17179" PostId="9850" Text="I used kmeans where I have to give the number of clusters explicitely" CreationDate="2011-04-28T02:06:06.223" UserId="2721" />
  <row Id="17180" PostId="10053" Text="This question has been merged with its near-duplicate." CreationDate="2011-04-28T02:10:21.553" UserId="919" />
  <row Id="17181" PostId="10059" Score="1" Text="Are you referring to the clustering algorithm which SPSS calls two-step? http://www.spss.ch/upload/1122644952_The%20SPSS%20TwoStep%20Cluster%20Component.pdf" CreationDate="2011-04-28T02:13:22.650" UserId="183" />
  <row Id="17182" PostId="10054" Score="1" Text="@User3897:  Whatever you do, definitely save some of your half million cases as a holdover sample for crossvalidation...I'm perplexed by your inability to find a better-predicting decision tree.  Have you fully explored SAS's options, e.g., on your inputted rules for node size requirements for when a parent node can be split, or when a child can be formed?  For the statistical significance of such splits?  For the program's ability to group the values of continuous predictors into effective subsets? " CreationDate="2011-04-28T02:49:20.897" UserId="2669" />
  <row Id="17183" PostId="9969" Text="But if that's plus $\sqrt n$ then it grows without bounds? I.e., the number of $X_i$ is not bounded by a constant?" CreationDate="2011-04-28T03:28:13.797" UserId="4312" />
  <row Id="17184" PostId="9969" Text="Say if I had to prove it formally that there is no more than a constant amount, could I formulate X_i as the weighted sum of a RV dependent on Y (and Z) and another one which is independent of it? Maybe that could help." CreationDate="2011-04-28T03:34:49.533" UserId="4312" />
  <row Id="17185" PostId="10059" Text="Yeah! I'm taking about spss kind of 2-step clustering.But I want to do it using R." CreationDate="2011-04-28T04:08:42.043" UserId="4278" />
  <row Id="17186" PostId="10078" Text="Why do you have C(2) and not just C?" CreationDate="2011-04-28T06:34:01.357" UserId="2958" />
  <row Id="17187" PostId="10078" Score="2" Text="@user4341 The SD of y is zero." CreationDate="2011-04-28T06:40:11.323" UserId="930" />
  <row Id="17188" PostId="10030" Text="www.sixsigmain.it/ebook/Capu7-15.html " CreationDate="2011-04-28T06:40:40.293" UserId="4345" />
  <row Id="17189" PostId="10030" Text="sorry, in italian language" CreationDate="2011-04-28T06:41:09.700" UserId="4345" />
  <row Id="17190" PostId="10078" Text="In math, cov(x,y) means the covariance of the two variables x and y. In MATLAB, cov(x,y) calculates the covariance matrix of x and y. Here cov is a function and x and y are the inputs.cov returns a square matrix and we need the off diagonals." CreationDate="2011-04-28T06:43:21.910" UserId="4341" />
  <row Id="17191" PostId="10078" Text="Yes, SD for y is zero and I want to know if we have such a scenario how to calculate Pearson coefficient." CreationDate="2011-04-28T06:45:31.910" UserId="4341" />
  <row Id="17192" PostId="10032" Text="@Henry, I've misread it, sorry :)" CreationDate="2011-04-28T06:46:55.107" UserId="2116" />
  <row Id="17193" PostId="10030" Text="@Massimo, @Henry answer is the correct one. The data in your link comes in tabular format, how many times certain values are observed, `ks.test` expects simple sample, not the table, hence the difference." CreationDate="2011-04-28T06:49:53.197" UserId="2116" />
  <row Id="17194" PostId="10080" Text="Your answer looks great. Can I use it efficiently for my data. Couldnt get time to try it. " CreationDate="2011-04-28T07:15:21.497" UserId="2721" />
  <row Id="17195" PostId="10030" Text="@mpiktas, and then ... how write with R for compare a with b?" CreationDate="2011-04-28T07:47:00.070" UserId="4345" />
  <row Id="17196" PostId="10069" Text="+1, could also look at `last_plot() + facet_grid(. ~ variable)` to isolate each scenario [depending on exactly what the OP is trying to visualize]&#xA;" CreationDate="2011-04-28T07:51:27.750" UserId="3732" />
  <row Id="17197" PostId="10030" Text="@Massimo, is @Henry answer not clear? " CreationDate="2011-04-28T08:24:20.193" UserId="2116" />
  <row Id="17198" PostId="10078" Score="2" Text="@user4341 [Pearson correlation of data sets with possibly zero standard deviation?](http://stats.stackexchange.com/questions/9068/pearson-correlation-of-data-sets-with-possibly-zero-standard-deviation)" CreationDate="2011-04-28T08:39:20.567" UserId="930" />
  <row Id="17199" PostId="10032" Text="@Henry: only now I understand. While I wrote the number of bird, R wanted the distribution of the hours." CreationDate="2011-04-28T08:48:48.947" UserId="4345" />
  <row Id="17200" PostId="10073" Text="@Jeromy (+1) There seems to be some additional information on this thread: http://forums.spss.co.in/forum_posts.asp?TID=193 (can't check myself)." CreationDate="2011-04-28T08:57:56.830" UserId="930" />
  <row Id="17201" PostId="10032" Text="@Henry, @mpiktas, @Aniko, ...... thank of all" CreationDate="2011-04-28T09:07:23.900" UserId="4345" />
  <row Id="17202" PostId="10072" Text="@ukw This site supports Markdown syntax for editing, and also LATEX for mathematical expressions. Further information can be found here: http://stats.stackexchange.com/editing-help. " CreationDate="2011-04-28T09:47:01.050" UserId="930" />
  <row Id="17203" PostId="10031" Text="Thanks Aniko, I've modified the question." CreationDate="2011-04-28T10:07:33.903" UserId="3280" />
  <row Id="17204" PostId="10086" Text="The widely used random number generators today are of high quality that will not allow to distinguish between them using small samples." CreationDate="2011-04-28T10:55:01.430" UserId="3911" />
  <row Id="17205" PostId="10084" Text="In R `df` is used for *the density function of the F distribution*. Assigning new values to built in names is possible, but it may lead to unpredictable results." CreationDate="2011-04-28T10:58:46.340" UserId="3911" />
  <row Id="17206" PostId="10089" Score="2" Text="Are subjects randomised into the three arms? Is it a valid assumption that here is no difference between the groups at baseline? Is it a valid assumption that the during-treatment value is between the pre-treatment and post-treatment values (a monotonic relationship)? Are the pre/during/post values available for each subject? What is your null hypothesis? What kind(s) of difference(s) are you interested in, what should the test be sensitive for? " CreationDate="2011-04-28T11:35:35.433" UserId="3911" />
  <row Id="17207" PostId="10084" Text="@TMOD, I do not understand what such adjusted data will be used for, it will not provide the solution that maximizes tonnes in every wagon? Residuals, as GaBorgulya suggested, shows just the deviations from the average. Well it is not what you are actually looking for, isn;t it? I think, you have to play with some mean value. For example $\max(tonnes)$ will be just intercept term of your regression `113.92`, but it is hardly reachable. You may try to fill the gap between `mean(df$tonnes)` and this value by some technically reasonable proportion. " CreationDate="2011-04-28T11:43:57.027" UserId="2645" />
  <row Id="17208" PostId="10089" Text="Does the survey lead to one or to multiple outcome variables? If there are multiple do they have to be combined or evaluated independently?" CreationDate="2011-04-28T11:46:49.777" UserId="3911" />
  <row Id="17209" PostId="10092" Score="4" Text="Could you please give a www link how 20/ln(2) is applied? Thanks!" CreationDate="2011-04-28T12:04:49.030" UserId="3911" />
  <row Id="17211" PostId="10090" Text="My random number generators generate real numbers in the range from -180 to 180. I understand that equal means do not mean that distributions are the same. But if the means are different, then the distributions are also different. So, it is why I want to check the means. My problem is that the difference between means is smaller than the difference caused by chance. I cannot understand how it is possible." CreationDate="2011-04-28T12:23:33.333" UserId="2407" />
  <row Id="17212" PostId="10092" Text="Looks like a simple transformation (on a linear scale) http://bit.ly/klhGCr " CreationDate="2011-04-28T12:49:51.953" UserId="930" />
  <row Id="17213" PostId="10064" Text="Thanks for the re-post from SO, but how do you recommend I deal with the zeroes given that I still want to estimate a GOF?" CreationDate="2011-04-28T13:04:06.553" UserId="302" />
  <row Id="17214" PostId="10089" Text="@Kate I've updated your title with what I seem to understand from your rather vague question (see the points raised by @GaBorgulya). Feel free to update if I missed something." CreationDate="2011-04-28T13:04:44.437" UserId="930" />
  <row Id="17215" PostId="10090" Text="Like I said: I agree that comparing two distributions should certainly not give less different means than identical ones (at least not consistently so). Post some code and/or example data, and maybe we can help pick up what went wrong? Note that when the means are the same, the distribution could still be very different (think of a bimodal distribution centered around the mean of a standard normal one, or a more skewed distribution)." CreationDate="2011-04-28T13:46:04.037" UserId="4257" />
  <row Id="17217" PostId="9969" Text="@Josiane You have to work in the other direction: you cannot have the $X_i$ depend on $Y$ or $Z$, because that would violate their assumed independence: $Y$ and $Z$ have to depend on the $X_i$.  Construct $Y$ and $Z$ as mixtures of disjoint truncated Normal distributions (so that they each become univariate Normal).  Arrange it so that when $n$ of the $X_i$ are less than $c$, then so are $Y$ and $Z$ (if possible).  Then repeat for $n-1$, $n-2$, ..., etc., until you have completely accounted for $\Pr(Y\le c, Z\le c)$.  This maximizes the conditional expectation of the number of $X_i \le c$." CreationDate="2011-04-28T14:11:11.400" UserId="919" />
  <row Id="17218" PostId="10096" Text="@lindelof, did you look at the [changepoint](http://stats.stackexchange.com/questions/tagged/change-point) tag?" CreationDate="2011-04-28T14:13:20.337" UserId="2116" />
  <row Id="17219" PostId="10095" Score="2" Text="Something is seriously wrong when you have to use Box-Cox transformations this strong.  Moreover, there's not much practical difference between -2.8 and -2.3: you could safely use -2.5 in both situations.  I suspect you may have one or more outliers to deal with.  Box-Cox transformations really shouldn't be used in such an automated way: they are more suited for exploration.  Draw a ladder of probability plots of transformed residuals for each model, varying $\lambda$ from $-1$ to $2$ in units of $1/2$, to see what might be happening." CreationDate="2011-04-28T14:26:41.193" UserId="919" />
  <row Id="17220" PostId="9092" Text="@prob I question why it makes sense *as a generic procedure* to average over all correlations.  In effect this procedure substitutes the definite and possibly quite wrong answer &quot;zero!&quot; for the correct answer &quot;the data don't tell us.&quot;  That difference can be important for decision making." CreationDate="2011-04-28T14:32:12.017" UserId="919" />
  <row Id="17221" PostId="10075" Text="Take a look at an [ANOVA table](http://www.itl.nist.gov/div898/handbook/prc/section4/prc433.htm)." CreationDate="2011-04-28T14:36:10.533" UserId="919" />
  <row Id="17222" PostId="10096" Text="@mp This isn't a changepoint problem because the times of change are exactly known.  It's really a question about regression with strongly correlated multiple measurements." CreationDate="2011-04-28T14:38:35.210" UserId="919" />
  <row Id="17223" PostId="10049" Text="@whuber I have provided the confusion matrix [here](https://docs.google.com/document/d/1ud3bif91BIE2YLEs1jtl_UhUbjIjjWIkoxwyJCMSkK4/edit?hl=en&amp;authkey=CPCGnKIC#) Could you suggest a way of improving the accuracy based on this?" CreationDate="2011-04-28T14:49:36.187" UserId="3897" />
  <row Id="17224" PostId="10049" Text="Have you considered using a negative threshold?" CreationDate="2011-04-28T14:55:27.193" UserId="919" />
  <row Id="17225" PostId="10091" Text="@GaBorgulya Could you please elaborate on your answer. I couldnt get your point of **use various clustering algorithms and their corresponding methods to find out the number of clusters**. If I choose to run k-means on the new dataset, i have to provide **k** as an input. My Q is what value of **k** of best for the original dataset. " CreationDate="2011-04-28T15:01:54.933" UserId="4290" />
  <row Id="17226" PostId="10088" Score="2" Text="see this related question on stopping criteria for cluster analysis: http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice" CreationDate="2011-04-28T15:08:53.733" UserId="183" />
  <row Id="17227" PostId="10054" Text="@rolando2 I have provided my SAS Enterprise Miner output [here](https://docs.google.com/viewer?a=v&amp;pid=explorer&amp;chrome=true&amp;srcid=0B_a7XsRmdVaENWM2NWQyMzUtZGI1Ni00NDBjLWI3ZDctMTUwZWQ2MWViNDZh&amp;hl=en&amp;authkey=CO3ij40B). This contains the tree settings that I used. Could you suggest what I need to improve upon." CreationDate="2011-04-28T15:10:33.290" UserId="3897" />
  <row Id="17228" PostId="9533" Text="@Gael, could you post your data someplace ? or a generator with a couple of parameters would be even better, for various people playing with high-dim ML." CreationDate="2011-04-28T15:17:49.593" UserId="557" />
  <row Id="17229" PostId="10052" Text="I am sorry for the misnomer of &quot;independent X's&quot;. What I meant was &quot;independent Y's conditional on X's&quot;. Namely, the residuals must be independent. This assumption is violated in longitudinal observations. Reference: Modeling longitudinal data, II: standard regression models and extensions. Methods Mol Biol. 2009;473:61-94." CreationDate="2011-04-28T15:39:34.620" UserId="4349" />
  <row Id="17230" PostId="10100" Score="1" Text="Your p-value is not 0.9615: it is 0.07698.  This is because your z-statistic is -1.7685 and a standard normal variate has a 0.07698 chance of exceeding 1.7685 in absolute value.  Note that this p-value is high enough that in many applications it would not be considered significant evidence of a difference anyway.  The additional uncertainty in your situation should *raise,* not lower, this p-value." CreationDate="2011-04-28T16:06:58.393" UserId="919" />
  <row Id="17231" PostId="10086" Text="Do you really mean &quot;random number generators&quot;, i.e. numerical algorithms that aim to give observations with a known distribution (usually uniform(0,1)), or just &quot;random variables&quot;, i.e. sources of data with unknown distributions." CreationDate="2011-04-28T16:11:25.540" UserId="279" />
  <row Id="17232" PostId="10104" Score="2" Text="In addition to the lognormal family there are many other common distributions with positive essential support: gamma, Weibull, F, half-normal, etc.  Moreover, *no* distribution is truly &quot;continuous&quot; if you're going to be this strict about definitions: your measurements must be real numbers with finite representations and as such are necessarily *discrete*!" CreationDate="2011-04-28T16:17:46.997" UserId="919" />
  <row Id="17233" PostId="10096" Text="@lindelof I changed the title to try to make it clearer that this is NOT a changepoint problem. Feel free to change it to something better." CreationDate="2011-04-28T16:24:22.997" UserId="183" />
  <row Id="17234" PostId="10115" Score="1" Text="@Nyota A bit of context would certainly help... Otherwise, I'm afraid we'll have to close it as not a real question." CreationDate="2011-04-28T18:28:13.597" UserId="930" />
  <row Id="17235" PostId="10112" Text="Thanks! Could you expand on the 'dynamic linear model'? In your own words to someone who has no idea about dlm. I did google it and found a bunch of PDFs but I would happy to read an explanation in plain English." CreationDate="2011-04-28T18:29:32.053" UserId="4240" />
  <row Id="17236" PostId="10115" Score="1" Text="@Nyota, there are companies which specializes to answer these type of questions. They are quite expensive. On the other hand this question can be rephrased into a very good one: how to explain usefulness of forecasting models. " CreationDate="2011-04-28T18:34:21.480" UserId="2116" />
  <row Id="17237" PostId="10111" Text="@chl thanks for edits. Quite useful." CreationDate="2011-04-28T18:43:23.533" UserId="2885" />
  <row Id="17238" PostId="10111" Text="@Pegah It looks like you're confused about F-test for testing two variances (in the NRC++ 3rd ed., I read `prob = 2.0*beta.betai(0.5*df2,0.5*df1,df2/(df2+df1*f))`, where `f=var1/var2`, in the section entitled &quot;14.2.2 F-Test for Significantly Different Variances&quot;) vs. one- or two-way ANOVA. What you described suggests you're interested in a one-way model (you have one response variable and one explanatory variable, or factor) and want to conduct a statistical test about the effect of the factor with an F-test while providing a measure of effect-size with the eta-squared. Could you confirm?" CreationDate="2011-04-28T19:02:33.150" UserId="930" />
  <row Id="17239" PostId="10105" Text="(+1) nice suggestions, and thanks for Kelley proper links ^_^" CreationDate="2011-04-28T19:04:11.713" UserId="2645" />
  <row Id="17240" PostId="9991" Text="Doing a quick simulation of looking at medians across N with skewed distributions will show it." CreationDate="2011-04-28T19:08:33.637" UserId="601" />
  <row Id="17241" PostId="9993" Text="Then your question has been answered.  There are no rules and having such a switch is very unlikely to be a good idea." CreationDate="2011-04-28T19:09:33.243" UserId="601" />
  <row Id="17242" PostId="10091" Text="@Learner I've edited my post to reply to your comment." CreationDate="2011-04-28T19:13:43.973" UserId="3911" />
  <row Id="17243" PostId="10117" Text="You seem to want to do forecasting using time-series or longitudinal data; presumably modelling, not data mining.&#xA;You have data for each student at multiple time points. Are these data collected at regular time intervals? What are your predictors like (the &quot;inputs&quot;)? Are the outcome variables aggressions,  self-injurious behaviours counts? Do they account for a given time period? Is drop binary (yes/no)?" CreationDate="2011-04-28T20:49:20.523" UserId="3911" />
  <row Id="17244" PostId="10117" Text="Well, I believe that doing forecasting is probably the more advanced of the two things I'd like to do. I'd be happy to start with just some basica anomaly detection outside of just doing standard deviation. To answer your questions, the data is collected at various time intervals although it is usually aggregated into uniform intervals (e.g. hourly, daily, etc.) Right now the predictors are the behaviors being tracked (aggression, self-injurious behaviors, etc.), although we will be adding more in the future along the lines of sleep, meals eaten, special incidents, medical information..." CreationDate="2011-04-28T21:00:56.680" UserId="4379" />
  <row Id="17245" PostId="10117" Text="I guess in theory I was wondering if the inputs and the outputs can be similar (e.g. can more aggressions predict more self injurious behaviors)? Drop is actually duration, but could be simplified down to frequency as well for these purposes. Again, I'm not at this point as concerned with our specific situation, I Just wanted to give a basic example to make sure I'm on the right track. I'd rather learn how to do this in general with some basic examples then hopefully become more proficient and apply it to our exact circumstances." CreationDate="2011-04-28T21:02:48.443" UserId="4379" />
  <row Id="17246" PostId="9936" Text="@barry - it isn't one tailed - conditional on the gamma process, it is just plain old ordinary brownian motion (with drift and volatility).  Its just that instead of using calendar time, it uses a stochastic process to measure time - usually called &quot;market time&quot;.  You can also think of a brownian motion with &quot;jumps&quot;" CreationDate="2011-04-28T21:43:51.730" UserId="2392" />
  <row Id="17247" PostId="10087" Text="Hi GaBorgulya, it seems that the way you've written those, df$locationAdjustedTonnes returns the difference between actual values and the model. Was that intentional? If so, no worries, I've changed it to `df$locationAdjustedTonnes = df$tonnes - resid(lm(df$tonnes ~ df$dist_fr_middle))`.&#xA;" CreationDate="2011-04-28T22:09:16.743" UserId="827" />
  <row Id="17248" PostId="10087" Text="Apologies for not knowing what I'm doing, but I was hoping that a model wouldn't fit all values of df$tonnes exactly onto the regression line. I thought it would instead adjust all values of df$tonnes by adding the mean residual for a particular dist_fr_middle to each value at that dist_from_middle " CreationDate="2011-04-28T22:16:13.353" UserId="827" />
  <row Id="17249" PostId="10087" Text="You wanted &quot;distance-from-middle-adjusted tonnes&quot; and (I believe) that means the observed tonnes minus the expected tonnes considering distance-from-middle." CreationDate="2011-04-28T22:22:35.247" UserId="3911" />
  <row Id="17250" PostId="10086" Text="Also see http://stats.stackexchange.com/questions/30/testing-random-variate-generation-algorithms, http://stats.stackexchange.com/questions/40/pseudo-random-number-generation-algorithms" CreationDate="2011-04-28T22:32:35.050" UserId="3911" />
  <row Id="17251" PostId="10101" Text="Cheater! /10char :)" CreationDate="2011-04-28T22:40:24.343" UserId="776" />
  <row Id="17252" PostId="10117" Text="You may want to take a look at a somewhat similar question: http://stats.stackexchange.com/questions/242/using-time-series-analysis-to-analyze-predict-violent-behavior" CreationDate="2011-04-28T22:43:01.407" UserId="3911" />
  <row Id="17253" PostId="10121" Score="1" Text="@Richard, thanks for your question. Unfortunately, the method you describe above will *not* produce entries that are normally distributed. The diagonals are 1 with probability one and the off-diagonals are bounded between $-1$ and $+1$. Now, the *rescaled* entries will converge asymptotically to a normal distribution centered around zero. Can you give us more information about the problem you're actually trying to solve? And, why do you want &quot;normally distributed&quot; off diagonals?" CreationDate="2011-04-28T22:47:59.763" UserId="2970" />
  <row Id="17254" PostId="10121" Text="@cardinal, thank you for your comment.  The purpose is to simulate the correlations of assets, which often have normal look distribution.  The previous described matrix  look quite close to normal when looking at qqplot.  But you are rWould you explain the meaning of &quot;rescaled&quot; entries?" CreationDate="2011-04-28T23:01:36.123" UserId="4383" />
  <row Id="17255" PostId="10121" Text="@Richard In your followup to @cardinal, please explain how you plan to relate *correlation* to *standard deviation*: the two are different and almost independent." CreationDate="2011-04-28T23:07:11.843" UserId="919" />
  <row Id="17256" PostId="10121" Text="@Richard, what I mean was, suppose $X = (X_1,X_2,\ldots,X_n)$ and $Y = (Y_1,Y_2,\ldots,Y_n)$ are two independent vectors such that the entries of each are i.i.d. standard normal. Compute $\hat{\rho}_n = s_{xy} / (s_x s_y)$; that is, the sample correlation between $X$ and $Y$. Then $n^{1/2} \hat{\rho}_n$ converges in distribution to a standard normal random variable. By &quot;rescaled&quot;, I meant the multiplication by $n^{1/2}$ which is what is required to obtain a non degenerate limiting distribution." CreationDate="2011-04-28T23:11:45.040" UserId="2970" />
  <row Id="17257" PostId="10121" Text="@whuber sure.  Say we have 50 assets or random variables.  Then the correlation matrix is 50 by 50 matrix.  There are 50x49 off diagonal entries.  Those entries look normally distributed." CreationDate="2011-04-28T23:12:07.367" UserId="4383" />
  <row Id="17258" PostId="10121" Text="@Richard, the essence of the &quot;problem&quot; is that by making two restrictions (a) that the norms of each row are 1 and (b) that the entries are generated from a random sample, you necessarily are forcing the correlations to be quite small (on the order of $n^{-1/2}$. The reason is that you can't have arbitrarily large correlations between rows and still get the norms of each row to be 1 in the presence of so much independence." CreationDate="2011-04-28T23:18:02.103" UserId="2970" />
  <row Id="17259" PostId="10121" Text="@cardinal, this is an interesting part.  I thought the  entries in the correlation matrix defined earlier are dependent.  So looking at the distribution of one entry cannot represent the distribution of all entries.  although each entry has the ditribution as you describe." CreationDate="2011-04-28T23:18:36.040" UserId="4383" />
  <row Id="17260" PostId="10121" Text="@Richard, yes, the off-diagonals are necessarily dependent. In fact, by renormalizing each row of the generating matrix, you *start out* with dependent random variables! But, the dependence is necessarily weak and *must* grow weaker as the dimension (number of stocks) increases." CreationDate="2011-04-28T23:20:16.287" UserId="2970" />
  <row Id="17261" PostId="10121" Text="...now, you can get larger correlations in magnitude by *first* correlating the rows among themselves before renormalizing. But, you essentially only have one parameter to play with, so both the asymptotic mean and variance will be tied to that parameter. So, that probably won't give you the flexibility you seem to want, either." CreationDate="2011-04-28T23:22:13.347" UserId="2970" />
  <row Id="17262" PostId="10121" Text="@cardinal, that's a good point.  Another source of dependency comes from matrix multiplication." CreationDate="2011-04-28T23:29:08.530" UserId="4383" />
  <row Id="17263" PostId="10121" Text="@cardinal, when you say first correlating the rows then renormalize, could you explain that?" CreationDate="2011-04-28T23:31:17.523" UserId="4383" />
  <row Id="17264" PostId="10121" Text="@cardinal, I guess you mean calculate the covariance matrix then correlation matrix " CreationDate="2011-04-28T23:34:21.600" UserId="4383" />
  <row Id="17265" PostId="10121" Text="Sure, let's take a simple case. Call the generating matrix $X$, which we'll assume to be $m \times n$ without loss of generality. Now, generate the *columns* of $X$ as i.i.d. *vectors* such that the elements of each vector are standard normal random variables that are equicorrelated with correlation $\rho$. Now, use the procedure you have been. Let $\hat{\rho}_{ij}$ denote the sample correlation between the $i$th and $j$th *row* of $X$. Then for fixed $m$, letting $n \to \infty$, $n^{1/2} (\hat{\rho}_{ij} - \rho)$ converges in distribution to a $\mathcal{N}(0,(1-\rho^2)^2)$ random variable." CreationDate="2011-04-28T23:38:49.423" UserId="2970" />
  <row Id="17266" PostId="10121" Text="@cardinal, I have a question here, also related to the 3rd comments you made.  my procedure basically calculate the correlation entries by taking dot product of any two rows.  Is it the same as the sample correlation you described here?" CreationDate="2011-04-28T23:57:11.700" UserId="4383" />
  <row Id="17267" PostId="10121" Text="@cardinal, yes, essentially. The sample correlation would normally remove the sample mean from each vector before normalizing and taking the dot product, but that is a minor detail of little consequence here." CreationDate="2011-04-29T00:00:04.000" UserId="2970" />
  <row Id="17268" PostId="10121" Text="@cardinal, thank you for your help.  I need to be off line now.  I will post further questions later. We will continue our discussion.  Thanks!" CreationDate="2011-04-29T00:00:58.110" UserId="4383" />
  <row Id="17269" PostId="10113" Text="I am not a statistics expert. Could you please explain more about the plotting idea?" CreationDate="2011-04-29T02:16:58.390" UserId="2721" />
  <row Id="17270" PostId="10121" Score="1" Text="Why the down vote on this question?" CreationDate="2011-04-29T02:44:14.793" UserId="2970" />
  <row Id="17271" PostId="1965" Text="@shabbychef - given that the reference provided speaks of *non-asymtotic* minimax rates... then no asymptotics should be done.  But you essentially have the answer anyway, just $L=n^{-n\beta}exp(n^{2})$.  So (+1) from me" CreationDate="2011-04-29T03:00:52.207" UserId="2392" />
  <row Id="17272" PostId="9838" Text="Thanks dmk8, very useful. Some follow-up points/questions: is this what is often meant when referring to varying the variable of interest (the economy from bad to good for example) while holding all control variables at their means? What do you mean by deterministically? How do I determine the confidence intervals around the probabilities?" CreationDate="2011-04-29T04:06:47.387" UserId="3883" />
  <row Id="17273" PostId="10097" Text="Thanks, this is exactly the kind of information I was looking for." CreationDate="2011-04-29T05:25:52.143" UserId="4359" />
  <row Id="17274" PostId="10099" Text="Thanks, this was also very helpful." CreationDate="2011-04-29T05:26:36.837" UserId="4359" />
  <row Id="17275" PostId="10086" Text="@Aniko, you are right, it is better to say that I have a random variable. I just wanted to abstract from the details. In my case the real numbers are created by different person (one &quot;random number generator&quot; in my case is a human)." CreationDate="2011-04-29T07:59:40.620" UserId="2407" />
  <row Id="17277" PostId="10064" Text="Hi Drew, sorry for my late answer. Well, in case of zero frequencies, it is advised (at least I was taught so) to either merge categories, since Chi-square allows some 0 frequencies (less than 20% frequencies can be 0 when the factor has more than 2 levels), if that's reasonable, or to drop Chi-square and choose an alternative technique. As I usually drop the thing, so you may want to ask another question regarding appropriate goodness-of-fit test." CreationDate="2011-04-29T08:18:50.697" UserId="1356" />
  <row Id="17278" PostId="10111" Text="@chl thanks for the answer. I am using 2way-anova, and I have 2 factors ( I did not get what you mean by response variable). I am interested in the variance of one of the factors in comparison to the total variance (both factors) which forms the eta squared." CreationDate="2011-04-29T08:19:34.300" UserId="2885" />
  <row Id="17285" PostId="10096" Text="@Jeromy I'm not sure I understand the title to my question any longer :-)" CreationDate="2011-04-29T08:59:02.273" UserId="4370" />
  <row Id="17286" PostId="5421" Text="cuz you need to start somewhere :) " CreationDate="2011-04-29T09:22:18.197" UserId="704" />
  <row Id="17287" PostId="10141" Text="the binary case might be a sufficient first start..." CreationDate="2011-04-29T09:48:11.643" UserId="704" />
  <row Id="17289" PostId="10145" Score="2" Text="A few questions: (1) &quot;level of disclosure&quot; and &quot;index&quot; imply a variable with more than two levels; have you considered using an ordinal scale as the dependent variable? (2) Does disclosure vary over time or is it operationalised as constant over time?" CreationDate="2011-04-29T12:43:51.720" UserId="183" />
  <row Id="17290" PostId="10124" Text="Thanks for the great answer. Definitely gives me some specific terms to start Googling to hopefully start making heads and tails of all of this." CreationDate="2011-04-29T13:25:28.117" UserId="4379" />
  <row Id="17291" PostId="10086" Text="When you say &quot;reference differences&quot; are smaller than the &quot;test differences&quot;, do you mean that as in -10 is smaller than a bunch of numbers in the -5 to 5 range, or do you mean that in absolute value? The latter would imply that half are below and half are above, because the &quot;test differences&quot; should be symmetrically distributed around 0 (without an absolute value)." CreationDate="2011-04-29T13:30:24.513" UserId="279" />
  <row Id="17292" PostId="10143" Score="1" Text="+1 R is a good suggestion.  Be very careful about using software not specifically created for mathematical or statistical calculation!  In particular, do not use Excel--it makes mistakes--and do not rely on Web apps." CreationDate="2011-04-29T13:49:25.983" UserId="919" />
  <row Id="17293" PostId="10143" Text="@whuber Thank you. Point taken." CreationDate="2011-04-29T14:14:51.877" UserId="4331" />
  <row Id="17294" PostId="10143" Text="@Nick Sabbe: I'm trying to look into it, and will let you know. Thank you Nick." CreationDate="2011-04-29T14:16:18.863" UserId="4331" />
  <row Id="17295" PostId="10113" Text="The basic idea is that many variables are correlated with each other and everything can be reduced to only two variables which are uncorrelated with each other and explain &quot;most&quot; of the variation in the data. You need to read up on principal components analysis and apply a package which allows you to implement it.                        http://en.wikipedia.org/wiki/Principal_component_analysis" CreationDate="2011-04-29T14:32:56.883" UserId="3489" />
  <row Id="17296" PostId="10132" Text="@GaBorgulya - Thanks! That's really helpful. I did not realize the interpretation could be taken down to the random student level, as my unit of analysis is the school." CreationDate="2011-04-29T14:48:10.163" UserId="3309" />
  <row Id="17297" PostId="10135" Text="Thanks for the tip!" CreationDate="2011-04-29T15:18:35.037" UserId="4240" />
  <row Id="17300" PostId="10133" Text="Interesting question, but probably better suited for [stats.stackexchange.com](http://stats.stackexchange.com)" CreationDate="2011-04-29T15:49:59.750" UserId="71" />
  <row Id="17301" PostId="10052" Text="You can find out more about the 'generalizations of least squares' mentioned above from [Lewis and Linzer 2005](http://pan.oxfordjournals.org/content/13/4/345.abstract)" CreationDate="2011-04-29T15:59:28.370" UserId="1739" />
  <row Id="17302" PostId="10147" Score="3" Text="This sounds like homework" CreationDate="2011-04-29T16:19:18.143" UserId="2817" />
  <row Id="17304" PostId="10044" Text="If X's (independent variables) are measured variables, then it is a &quot;random variable&quot; instead of a &quot;fixed value&quot;. According to Wikipedia (http://en.wikipedia.org/wiki/Random_variable):&#xA;&quot;In probability and statistics, a random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process.&quot; Am I correct?" CreationDate="2011-04-29T04:59:18.027" UserId="4389" />
  <row Id="17305" PostId="10044" Text="Please don't use replies to ask questions.  You could continue this conversation in comments.  Another option is to re-post this as a question and link back here." CreationDate="2011-04-29T16:32:05.063" UserId="919" />
  <row Id="17306" PostId="10143" Score="1" Text="I have some that I did in Excel so my students could use them.  I also have just a simple spreadsheet that you put the alpha/confidence value in, add degrees of freedom, and get the required value. Email me at zbicyclist at yahoo dot com   Note whuber's comment -- Excel will work for standard values likely to be see in an introductory course, but will flop if you go beyond them -- e.g. z score of 8. " CreationDate="2011-04-29T16:33:35.520" UserId="3919" />
  <row Id="17307" PostId="10069" Text="Hello David, many thanks for your help. I tried it out and it worked. however if I remove the factor(ie if I make it x=V1) it does not work!" CreationDate="2011-04-29T16:40:46.070" UserId="4340" />
  <row Id="17308" PostId="10133" Text="I originally submitted this to stats.stackexchange.com and am viewing it from there...is this showing up elsewhere?" CreationDate="2011-04-29T16:45:58.050" UserId="4386" />
  <row Id="17309" PostId="10146" Score="1" Text="A first check would be to calculate the rank of your design matrix. If it's less than the number of columns, you probably need to combine and/or recode appropriately." CreationDate="2011-04-29T16:47:04.667" UserId="2970" />
  <row Id="17310" PostId="10150" Text="This is a very general question that no one can answer without more information. It really depends on the nature of your data. Do you have a reason to believe that past behavior can predict future behavior? Do you have other variables you can use to help make predictions (for example, if you are trying to predict future temperature, knowing the pressure could help)?" CreationDate="2011-04-29T17:57:52.387" UserId="2965" />
  <row Id="17311" PostId="10151" Text="Thanks, this worked.  I'm a new user and not allowed yet to mark it as the correct answer." CreationDate="2011-04-29T18:12:08.187" UserId="4400" />
  <row Id="17312" PostId="10150" Text="Hm. The nature of the data tends to be mostly oscillating, but not entirely and not always. Sometimes it tends to be more linear. No, I don't really have any other variables that could be applied. Predicting this could just be too difficult..." CreationDate="2011-04-29T18:15:58.713" UserId="4399" />
  <row Id="17313" PostId="10151" Text="@jd01 Glad it helped. You should be able to accept it by clicking on the check mark if you like; I don't think it takes any rep to accept an answer to your own question." CreationDate="2011-04-29T18:18:36.057" UserId="26" />
  <row Id="17314" PostId="10133" Text="Oh, man, it has been a long week. I habitually open a bunch of StackOverflow and CrossValidated questions in adjacent tabs, and this was at the end of a string of StackOverflow questions. Then, when I flagged for attention, it gave me the option to suggest that it move to stats.stackexchange - except that was actually META.stats.stackexchange. I should probably go to bed now." CreationDate="2011-04-29T18:20:09.243" UserId="71" />
  <row Id="17315" PostId="10147" Text="@Nyota As I said in my previous comment (cf. the question @Jeromy linked to), it would help if you could clarify what you mean by a *forecasting model* (forecasting what? which info is available? etc.) or at the very least what would qualify as a &quot;good&quot; response." CreationDate="2011-04-29T18:44:56.463" UserId="930" />
  <row Id="17316" PostId="10147" Text="well you could look at other examples of similar domestic chains that have expanded into different countries, maybe even China. Look at KFC for example, I think they were the first to enter china in the early 90's. without knowing what forecasting methods you're using, im not sure how to complete the answer though. and what are you forecasting? sales?" CreationDate="2011-04-29T16:45:08.360" UserId="2644" />
  <row Id="17317" PostId="10147" Text="@Nyota I'm inclined to close this question quickly unless it is amplified and clarified: it reads too much like a homework question that has been copied and pasted." CreationDate="2011-04-29T18:47:18.313" UserId="919" />
  <row Id="17318" PostId="10151" Text="@jd01 I gave you a +1 on your question so that you could even upvote the answer you ticked as the one which solved your problem." CreationDate="2011-04-29T19:01:28.873" UserId="930" />
  <row Id="17319" PostId="10151" Text="note: importing a couple of useful notes from stackoverflow: in my example I'm using the complete population's SD (2), not the sample SD (2.23); Ah, well then you should say so in your Q, because the scale() answers will only work if you do: scale(x, center=4, scale=2)" CreationDate="2011-04-29T19:03:01.057" UserId="4400" />
  <row Id="17320" PostId="10151" Score="1" Text="@chl: thanks - upvoted jms' answer as soon as i got enough reputation." CreationDate="2011-04-29T19:06:51.270" UserId="4400" />
  <row Id="17321" PostId="10151" Score="1" Text="In re: population vs sample SD, the only real benefit of scale(x) where the input x is a vector and not a matrix is that it calculates mean/sd for you. If you calculate the sd separately or have it in hand, (x-mean(x))/2 will do the same." CreationDate="2011-04-29T19:18:41.627" UserId="26" />
  <row Id="17322" PostId="10147" Text="@ whuber: thanks. I think you can close it now. Yes this was a homewor, I was trying to find something to compare with what I did and make sure I am on right truck. That's how the question was asked. I guess like Zach said and that's what exactely I tried forcasting. Thank yo all." CreationDate="2011-04-29T19:50:51.320" UserId="4395" />
  <row Id="17323" PostId="10152" Score="2" Text="Before you start modeling a special term for `1`, make sure that there is a logical reason for the `1`s to be over-represented. When you figure out why are there too many `1`s, you might find the most appropriate model." CreationDate="2011-04-29T19:55:33.110" UserId="279" />
  <row Id="17324" PostId="10012" Text="@chi I was thinking of the users each rating each of a few dozen sports as one of two levels (like or not). Does that answer your question? If not, I should probably review how Cohen's $\kappa$ works." CreationDate="2011-04-29T20:21:55.000" UserId="3874" />
  <row Id="17325" PostId="10154" Text="@mbq: many thx for editing my question :)" CreationDate="2011-04-29T20:33:31.127" UserId="3856" />
  <row Id="17326" PostId="10155" Text="Reading this may help: http://stats.stackexchange.com/questions/9330/" CreationDate="2011-04-29T21:17:42.667" UserId="3911" />
  <row Id="17327" PostId="9710" Text="Can you paste the first few rows of your data set?" CreationDate="2011-04-29T21:29:12.817" UserId="1893" />
  <row Id="17328" PostId="10155" Text="I am not sure I understand how are you using a binomial family with a response variable that is a percentage. Do you have the numerator and denominator counts?" CreationDate="2011-04-29T21:57:28.450" UserId="279" />
  <row Id="17329" PostId="10145" Text="1) The measurements of voluntary disclosure is indicated by a ratio from 0-1. 1 representing complete disclosure, 0 non. &#xA;&#xA;2) Level of disclosure vary over time  " CreationDate="2011-04-29T23:13:07.593" UserId="4394" />
  <row Id="17330" PostId="9697" Text="You can see this by copying and pasting the code into the R console." CreationDate="2011-04-29T23:15:33.937" UserId="1893" />
  <row Id="17331" PostId="9677" Text="It doesn't matter." CreationDate="2011-04-29T23:17:18.163" UserId="1893" />
  <row Id="17332" PostId="10140" Score="1" Text="@mark999's sounds like a good plan.  I'd add that the power problem would stem not just from having continuous predictors being multiplied, but from having &quot;social science&quot; (i.e., error-prone) variables.  Limited reliability gets compounded when multiplying two such variables." CreationDate="2011-04-30T00:27:45.133" UserId="2669" />
  <row Id="17333" PostId="10132" Text="I'm skeptical about this interpretation and also about the conclusion that logistic regression must be used when modelling percentages. Seems to amount to saying &quot;we can't model at the student level, only at the school level.&quot; Which, by the way, seems to argue against @GaBorgulya's interpretation, neat as it is. I'd like to hear from other people more experienced than myself." CreationDate="2011-04-30T00:44:09.130" UserId="2669" />
  <row Id="17334" PostId="10136" Text="@Jeromy - I enjoyed reading this clear, helpful response and am going to share it with a colleague.  " CreationDate="2011-04-30T00:54:20.807" UserId="2669" />
  <row Id="17335" PostId="10026" Text="@Frank Harrell - would you also grant that there's a risk, in devising a purely statistical solution, of missing a possible overarching problem whereby all 3 groups of predictors could be measuring characteristics/behaviours occurring at the same time.  Without an earlier-causes-later sort of basis for a causal chain, might it be impossible to definitively disentangle causal relationships in this situation--whatever our calculations might be?  (I'm trying to think the way James Davis does in The Logic of Causal Order.) " CreationDate="2011-04-30T01:05:52.933" UserId="2669" />
  <row Id="17336" PostId="10160" Score="1" Text="There's something wrong here: the normal curve overlaid on the bar chart does not agree with the bars." CreationDate="2011-04-30T01:23:31.717" UserId="919" />
  <row Id="17337" PostId="10145" Text="Thus, your dependent variable is not &quot;binary&quot; and not &quot;dummy&quot;, it is measured on a scale from zero to one with many values in between? " CreationDate="2011-04-30T03:10:02.020" UserId="183" />
  <row Id="17338" PostId="10145" Text="@BEF I have updated the question and question title to try to reflect the points you have made in the comments; feel free to edit your question, if I have misconstrued what you are saying." CreationDate="2011-04-30T03:27:14.657" UserId="183" />
  <row Id="17339" PostId="10158" Score="2" Text="Building on this, even a graph like what they use for high/low daily stock prices could work well." CreationDate="2011-04-30T05:22:45.417" UserId="776" />
  <row Id="17341" PostId="10161" Score="3" Text="@henry - $F(1)=1-1^{-\alpha}=1-1=0$, so support can't be below 1 (as CDF is a non-decreasing function)" CreationDate="2011-04-30T06:59:39.203" UserId="2392" />
  <row Id="17342" PostId="10167" Text="@Flavio I've tried to add a little more context to your question. Feel free to modify, if I have misconstrued what you are asking." CreationDate="2011-04-30T07:14:05.087" UserId="183" />
  <row Id="17343" PostId="10162" Score="1" Text="See http://stats.stackexchange.com/questions/8000/proper-way-of-using-recurrent-neural-network-for-time-series-analysis and http://stats.stackexchange.com/questions/9842/getting-started-with-neural-networks-for-forecasting" CreationDate="2011-04-30T08:01:48.600" UserId="2860" />
  <row Id="17344" PostId="9082" Text="@MYaseen208 Sure, look here: http://stats.stackexchange.com/questions/6890/plotting-a-heatmap-given-a-dendrogram-and-a-distance-matrix-in-r/6893#6893. You'll need to get the raw result from `biclust` and then plug it into a lattice heatmap with added dendrograms." CreationDate="2011-04-30T08:25:59.147" UserId="930" />
  <row Id="17345" PostId="9814" Text="@JMS: I've put the equations. Thanks" CreationDate="2011-04-30T08:45:17.847" UserId="3903" />
  <row Id="17346" PostId="10161" Text="@probabilityislogic: You may be correct in terms of the book.  I will change my response." CreationDate="2011-04-30T09:49:36.357" UserId="2958" />
  <row Id="17347" PostId="10146" Text="Given that all variables are categorical, one alternative option is to use contingency table methods.  I.e. you have a five way contingency table.  This can be done using a poisson glm (log-linear model), which may be more stable (may not be though).  Could also be a &quot;separation problem&quot; - your response can be perfectly predicted from the covariates - makes computers freak out when this happens because variance is zero." CreationDate="2011-04-30T11:01:31.493" UserId="2392" />
  <row Id="17348" PostId="10171" Score="3" Text="Do you mean something like, e.g., where the entries of a single column of the design matrix are permuted, holding the response and other covariates fixed? If you have a particular reference you are using, it might be helpful to list it." CreationDate="2011-04-30T13:36:37.273" UserId="2970" />
  <row Id="17349" PostId="10175" Text="In general, I would be cautious about using &quot;general&quot; special functions like $_1F_1$ for these types of numerics. In fact, it's quite hard to find a really good implementation of $_1F_1$, for example. The more specialized functions do just fine and have received much more attention from the numerical-analysis and algorithms community." CreationDate="2011-04-30T13:44:32.193" UserId="2970" />
  <row Id="17350" PostId="10142" Text="Just to reemphasize what others have already alluded to, it is generally *much* preferred to rely on standard packages and software for computing such things. A lot of the algorithms and code have decades worth of thought and maintenance associated with them. It's certainly worthwhile to understand how they arise mathematically and even some of the details behind the numerical implementation. But, similar to numerical linear algebra, I'd highly discourage anyone from attempting to roll their own implementation." CreationDate="2011-04-30T13:49:25.403" UserId="2970" />
  <row Id="17351" PostId="10168" Text="(+1) Particularly for the sharp-eyed recognition that the given support was incorrect." CreationDate="2011-04-30T14:00:14.373" UserId="2970" />
  <row Id="17352" PostId="10162" Text="Thanks I'll read thought those." CreationDate="2011-04-30T14:50:46.763" UserId="4403" />
  <row Id="17353" PostId="9814" Text="Hmm, it looks to me like you ought to be able to fit this with one of the mixed effects packages in R (for example) but I'm afraid I haven't done anything similar. Are there constraints on the $\Lambda$'s? (you mentioned PCA &amp; the SVD so I'm wondering if they're supposed to be orthogonal)" CreationDate="2011-04-30T15:47:30.317" UserId="26" />
  <row Id="17354" PostId="10152" Text="@Aniko makes a good point, you ought to be sure that there really are excess ones beyond what you can chalk up to sampling variability or you risk overfitting. And also that there are *enough* excess ones that you need the more flexible model." CreationDate="2011-04-30T15:51:57.973" UserId="26" />
  <row Id="17355" PostId="10177" Score="1" Text="+1 This is a real pain." CreationDate="2011-04-30T15:53:00.460" UserId="88" />
  <row Id="17356" PostId="10167" Text="Thanks, it is exactly what I want to know" CreationDate="2011-04-30T16:05:37.930" UserId="4406" />
  <row Id="17357" PostId="10176" Text="@chi That book looks fantastic! Although I do agree with the reviewer that there are some conspicuous absences (simulated annealing &amp; the various stochastic EM flavors). Kind of odd since it's in their statistics series, but c'est la vie" CreationDate="2011-04-30T16:16:37.530" UserId="26" />
  <row Id="17358" PostId="10176" Text="Also, are you familiar with Harville's matrix algebra book? I'd be curious to know how it compares with Gentle's. I find Harville a nice reference, but very dense. Just from the TOC of Gentle's book I like the whole part 2 being devoted to &quot;selected applications&quot;" CreationDate="2011-04-30T16:19:48.260" UserId="26" />
  <row Id="17359" PostId="2300" Text="I agree, a very nice book. But I would love to see another refresh (last ed was in 1999). Most recent treatments don't stack up IMO." CreationDate="2011-04-30T16:23:39.833" UserId="26" />
  <row Id="17360" PostId="10176" Text="@JMS Nope. I only have Gentle's textbook. (Because I only make a moderate usage of mathematical textbooks in general, except this one which I found pretty handy for multivariate data analysis.) Part 2 is about application (section 9) and Part 3 about software issues. The homepage is http://mason.gmu.edu/~jgentle/books/matbk" CreationDate="2011-04-30T16:25:01.200" UserId="930" />
  <row Id="17361" PostId="10173" Text="@probabilityislogic: I've posted on data example. Would you like to answer specifically to this problem. Thanks" CreationDate="2011-04-30T16:36:02.927" UserId="3903" />
  <row Id="17362" PostId="10133" Text="@don:  you don't sound like a novice data miner to me." CreationDate="2011-04-30T16:39:01.880" UserId="2669" />
  <row Id="17364" PostId="9814" Text="@JMS: If you like I can send the said article for your consideration. Thanks" CreationDate="2011-04-30T16:40:27.307" UserId="3903" />
  <row Id="17365" PostId="9090" Text="@probabilityislogic:  If you could humor a nonmathematical statistician/researcher...would it be possible to summarize or translate your answer to a group of dentists or high school principals or introductory statistics students?" CreationDate="2011-04-30T16:53:00.370" UserId="2669" />
  <row Id="17366" PostId="10133" Text="@rolando2: I suppose it's all relative, heh. I still feel like I've only scraped the surface of the subject..." CreationDate="2011-04-30T16:55:40.130" UserId="4386" />
  <row Id="17367" PostId="10180" Score="1" Text="@brett: See if you can get a little more precise: &quot;when enough data would be collected&quot; for what? For x% confidence, perhaps? You're asking about the link between sample size and precision of estimates, which is a matter of statistical power. Confidence intervals (CI) for the number of sales (or for the % of clicks that results in sales) will, as you indicate, depend in large part on sample size. The basic formula for a CI for a proportion is given at http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval" CreationDate="2011-04-30T17:02:00.970" UserId="2669" />
  <row Id="17369" PostId="10176" Text="Yeah, looking at it more it seems to have more from the applied side. Harville's book is very theorem-proof, but focused on results that are important in statistics; I think they probably complement each other pretty well despite the overlapping material." CreationDate="2011-04-30T17:27:19.980" UserId="26" />
  <row Id="17370" PostId="9814" Text="Honestly it would take more time than I have in order to really get my head around it. Good luck though!" CreationDate="2011-04-30T17:28:57.860" UserId="26" />
  <row Id="17371" PostId="10169" Text="@AndyW No, only the math-mode subset is supported (The engine is called MathJax, see [its page](http://www.mathjax.org/) for detailed docs). Yet I think a proper way to post tables is plain text in code block, so it could be copyable. " CreationDate="2011-04-30T18:22:56.897" UserId="88" />
  <row Id="17372" PostId="9944" Text="@chl Thank you!  Especially for the references." CreationDate="2011-04-30T18:28:16.863" UserId="4301" />
  <row Id="17373" PostId="10162" Text="[Prediction using neural networks](http://www.obitko.com/tutorials/neural-network-prediction/introduction.html) and bayer's previou in the thread [Proper way of using recurrent neural network for time series analysis](http://stats.stackexchange.com/questions/8000/proper-way-of-using-recurrent-neural-network-for-time-series-analysis) helped to improve my understanding. " CreationDate="2011-04-30T17:22:25.227" UserId="4403" />
  <row Id="17374" PostId="10182" Text="@dd3 Please, take the time to look at the [reliability](http://stats.stackexchange.com/questions/tagged/reliability) or [inter-rater](http://stats.stackexchange.com/questions/tagged/inter-rater) tags. The ICC is based on an ANOVA table, but the idea is merely to analysis components of variance rather than produce a single test statistic with know distributional properties, like the F-test. What application do you have in mind?" CreationDate="2011-04-30T19:07:34.570" UserId="930" />
  <row Id="17375" PostId="10182" Text="@chl I'm looking to analyze individual scores for grouped data.  I've seen a few papers examining differences between parents and children's scores, using the ICC to tell if there was a significant difference in the responses of parents vs. children.  I think the ICC is what I want here, but as I mentioned, I don't really understand the difference between the two.  I hesitate to ask more, but do you know of any good (basic) references?  My statistics background stopped at linear regression, and I feel like I'm asking questions that aren't well-formulated. Thank you." CreationDate="2011-04-30T19:33:43.803" UserId="4301" />
  <row Id="17376" PostId="10182" Text="@dd3 It looks like you have paired data. When considering separate groups (parents vs. their children) and using an ICC to report scores reliability, you'll throw away part of the information, i.e. whether ratings from parents and their relative act in a consistent way. Your two ICCs will only tell you whether both series of scores, assumed to be independent, are &quot;reliable&quot;, in the sense that a significant part of the variance might be accounted for by the rater effect. (...)" CreationDate="2011-04-30T19:40:20.317" UserId="930" />
  <row Id="17377" PostId="10182" Text="@dd3 (...) In sum, if you want to show that parental ratings are more reliable than children ones, then using ICC is fine; if, on the other hand, you want to study how parental ratings relate to children ratings, then you might resort to other kind of analysis (precisely, analysis of dyadic data)." CreationDate="2011-04-30T19:41:18.660" UserId="930" />
  <row Id="17378" PostId="10161" Text="Thanks for the response. What does f(x) represent? The probability density function? Is the derivative of the cdf always f(x)?" CreationDate="2011-04-30T20:23:26.533" UserId="4401" />
  <row Id="17379" PostId="10161" Score="1" Text="$f(x)$ is indeed supposed to be the probability density function.  If the cdf has a derivative then it is the density, though there are distributions (for example discrete) where the cdf does not have a derivative everywhere " CreationDate="2011-04-30T20:24:25.180" UserId="2958" />
  <row Id="17380" PostId="10168" Text="Thanks for the response. I fixed the question. I meant to put x&gt;=1. How did you know to first differentiate the cdf to get the density function?" CreationDate="2011-04-30T20:37:51.850" UserId="4401" />
  <row Id="17381" PostId="10161" Text="@Henry On a related question, $Var(X) = E(X^2) - E(X)^2$. How can I find $E(X^2)$? Thanks." CreationDate="2011-04-30T21:02:51.420" UserId="4401" />
  <row Id="17382" PostId="10178" Text="The ymd, dmy, etc. functions in this package look close, but don't quite solve the problem.  Better than the base POSIX commands, though..." CreationDate="2011-04-30T21:05:16.287" UserId="4110" />
  <row Id="17383" PostId="10179" Text="Yeah, it seems like a system like this would either be really brittle or have lots of problems with error trapping.  Maybe this is a pipe dream and I should just get comfortable the process." CreationDate="2011-04-30T21:07:09.763" UserId="4110" />
  <row Id="17384" PostId="10161" Score="1" Text="@styfle: If it exists then $E[X^2] = \int_{1}^{\infty}  x^2 f(x)\,dx$, and similarly for the expectations of other functions of $x$. " CreationDate="2011-04-30T22:08:27.160" UserId="2958" />
  <row Id="17385" PostId="10184" Text="(+1) A nice feature of the logistic versus probit is that it is relatively insensitive to a small number of responses that get incorrectly coded. This is, of course, related to the fact that it has fatter tails." CreationDate="2011-04-30T22:49:45.240" UserId="2970" />
  <row Id="17386" PostId="10112" Text="@Wayne: +1 I appreciate your efforts to answer my question. DLM does not seem to be the method I am looking for but it may be useful in one of my other projects. Thanks anyhow!" CreationDate="2011-04-30T23:01:23.697" UserId="4240" />
  <row Id="17387" PostId="10169" Text="When you say *homogeneity*, are you trying to test whether the treatments have the same response proportions of $R/(R+NR)$ with gender as a confounding variable, or that the treatments have the same proportions as each other and for both genders?   " CreationDate="2011-05-01T00:34:35.957" UserId="2958" />
  <row Id="17388" PostId="10168" Text="@styfle - because that's what a PDF is, whenever the CDF is continuous and differentiable.  You can see this by looking at how you have defined your CDF.  Differentiating an integral just gives you the integrand when the upper limit is the subject of the differentiation." CreationDate="2011-05-01T01:00:51.240" UserId="2392" />
  <row Id="17389" PostId="10168" Score="1" Text="@styfle - the PDF can also be seen as the probability that a RV lies in an infinitesimal interval.  $Pr(x&lt;X&lt;x+dx)=F(x+dx)-F(x)\to \frac{dF(x)}{dx}dx=f(x)dx$ as $dx\to 0$.  This way holds more generally, even for discrete RV and RV without a density (the limit is just something other than a derivative)" CreationDate="2011-05-01T01:04:12.157" UserId="2392" />
  <row Id="17390" PostId="10169" Text="Just my curiosity - any particular reason that the M-F split is so unbalanced?  This seems odd to have a *fixed total* of 274 for F and only 46 for M.  Or are these based on &quot;population&quot; proportions - there is a 274:46 split in the population of F to M?" CreationDate="2011-05-01T01:47:59.187" UserId="2392" />
  <row Id="17391" PostId="9090" Text="@rolando2 - the first thing I wouldn't do is show them the above formulas - enough to scare most people away, or at least make their eyes glaze over and the &quot;smile and nod&quot; would happen.  This would be best done using a graph of the integrated likelihood for various prior ranges $U_{\mu},L_{\mu},U_{\sigma},L_{\sigma}$ - one because this is what matters and two because graphs are easy to understand and to explain.  This will answer exactly *how much* the data and prior information say about the correlation.  my point was that its not nothing." CreationDate="2011-05-01T02:06:18.883" UserId="2392" />
  <row Id="17392" PostId="10185" Text="No idea if this is what is causing your problem or just a copy/paste error, but the syntax you are using in the assignment of `jags.inits` is wrong.  It should be `list(foo = bar)`, not `list(foo &lt;- bar)`." CreationDate="2011-05-01T02:58:29.423" UserId="2975" />
  <row Id="17393" PostId="10192" Score="3" Text="Start by thinking about centers of mass." CreationDate="2011-05-01T04:38:01.340" UserId="2970" />
  <row Id="17394" PostId="10170" Text="Thanks a lot Chl! It seems to be a very rare package as I didn't come across it after doing huge amount of searches." CreationDate="2011-05-01T06:11:56.903" UserId="4278" />
  <row Id="17395" PostId="10170" Text="@user4278 I also found some hits with Weka (e.g., [here](http://weka.sourceforge.net/doc/weka/datagenerators/BIRCHCluster.html)) but I didn't look further." CreationDate="2011-05-01T07:17:44.057" UserId="930" />
  <row Id="17396" PostId="10194" Text="Hmm.. I get the attributes-consideration-for-finding-results-based-on-clusters part..but The data Ive given isnt a dataSET per se...Each row is a session..And each column represents the number of times that particular entry has occurred in that session. So is this data useful enough to form clusters? " CreationDate="2011-05-01T08:02:02.860" UserId="4402" />
  <row Id="17397" PostId="10145" Text="@Jeromy Anglim - Yes, its true. " CreationDate="2011-05-01T09:17:31.363" UserId="4394" />
  <row Id="17398" PostId="9697" Text="@Patrick Sure. Responses come from themselves by running the code, of course." CreationDate="2011-05-01T09:41:46.293" UserId="930" />
  <row Id="17399" PostId="10204" Text="It seems like some clustering, but can you elaborate a bit more? What do you mean by &quot;levels of that behavior&quot;?" CreationDate="2011-05-01T11:35:59.933" UserId="88" />
  <row Id="17400" PostId="10192" Text="This is probably about the concept of detail scale separation -- if you have a cloud of particles, you first locate it (~mean), than describe the size (~variance), elongation (~skewness) and so on, going into even more subtle details." CreationDate="2011-05-01T11:45:26.967" UserId="88" />
  <row Id="17401" PostId="10121" Text="@cardinal,  From your comment, it seems we can get sd of normal close to 1. " CreationDate="2011-05-01T11:53:55.973" UserId="4383" />
  <row Id="17402" PostId="10184" Text="Logistic is basically indistinguishable from a t-distribution with about 7-8 degrees of freedom.  If you wanted &quot;super-robustness&quot; you should use the arctan link function - so you have a cauchy distribution" CreationDate="2011-05-01T12:10:55.580" UserId="2392" />
  <row Id="17403" PostId="10204" Text="The levels of the behavior are the frequencies (occurrence of behavior/minute) that were exhibited by the individuals. " CreationDate="2011-05-01T13:46:28.970" UserId="4420" />
  <row Id="17404" PostId="10206" Text="One thing that is not explicitly stated is that $V$ needs to be finite, so you would have to impose some sort of conditions on the covariance structure.  This is usually what those &quot;technical conditions&quot; are doing, making sure that the variance doesn't get &quot;too big&quot;" CreationDate="2011-05-01T14:14:24.977" UserId="2392" />
  <row Id="17405" PostId="10206" Text="@user4422, hand-waving is good, but I did not see any proof. All the technical details usually come up, after you start to write rigorous proof. As the text stands now it is just an idea or strategy of proof, not a proof. The 2 things which stand out is the first claim that omiting some elements changes nothing, another thing is the claim that the remaining sequence is iid, it is not and assymptotic closeness to iid does not immediately allows application of CLT for iid variables." CreationDate="2011-05-01T14:38:44.823" UserId="2116" />
  <row Id="17406" PostId="10207" Score="2" Text="added homework tag, due to original wording of question (no context, poor wording, etc)" CreationDate="2011-05-01T14:42:39.263" UserId="2116" />
  <row Id="17407" PostId="10204" Score="1" Text="@Orp Is the behaviour measured on a single variable or multiple variables?" CreationDate="2011-05-01T14:47:01.037" UserId="183" />
  <row Id="17408" PostId="10204" Text="@Orp there seem to be a lot of different ways to approach this... some more information is needed to best develop an answer.  How are the data distributed? Do you have an a priori reason to think there are 2 groups and only 2 groups?  etc..." CreationDate="2011-05-01T15:20:23.233" UserId="4048" />
  <row Id="17409" PostId="10028" Text="@mbq And you can add this one: [rt-rank](https://sites.google.com/site/rtranking/home) (just found it, not tested)." CreationDate="2011-05-01T15:30:56.513" UserId="930" />
  <row Id="17410" PostId="10206" Text="OK. Right, we must ensure that V stays finite. Is there any counter-example where V goes to infinity for a stationary ergodic process? " CreationDate="2011-05-01T15:50:31.190" UserId="4422" />
  <row Id="17411" PostId="10210" Score="2" Text="Please note that there is an important difference between methods [clustering](http://en.wikipedia.org/wiki/Cluster_analysis) and [classification](http://en.wikipedia.org/wiki/Classification_%28machine_learning%29): for the first you don't know what your clusters are, for the second they are defined and you want to predict membership. What you mean by ““some sense”-ish point of view” may be the difference between the two." CreationDate="2011-05-01T16:15:43.430" UserId="3911" />
  <row Id="17412" PostId="10210" Text="I imply the clustering task, that is the classes/groups/clusters/topics are unknown. The intent of adding the &quot;classification&quot; tag was to maximise visibility of the question ;)" CreationDate="2011-05-01T16:31:34.000" UserId="4425" />
  <row Id="17413" PostId="10200" Text="The first argument describes a property of any location family of distributions whose pdfs have monotonically decreasing right tails.  The second argument is not correct unless you add assumptions (such as the two distributions have the same shape and location)." CreationDate="2011-05-01T17:14:29.337" UserId="919" />
  <row Id="17415" PostId="10210" Text="@dorserg I have retagged accordingly :-) It looks like you want to put a priori constraints on your cluster solution, right?" CreationDate="2011-05-01T17:21:23.160" UserId="930" />
  <row Id="17417" PostId="10184" Text="@probabilityislogic: How sure are you about your first statement?" CreationDate="2011-05-01T19:34:11.310" UserId="2970" />
  <row Id="17418" PostId="10213" Score="1" Text="See this question: http://stats.stackexchange.com/questions/8485/a-good-gibbs-sampling-tutorials-and-references" CreationDate="2011-05-01T20:05:45.453" UserId="2970" />
  <row Id="17420" PostId="10204" Text="Thanks Jeromy, I am not sure I understand your question. The behavior scores range from 0 occurrences per minute to 8 per minute on a continuous scale." CreationDate="2011-05-01T20:08:21.877" UserId="4420" />
  <row Id="17421" PostId="10204" Text="KennyPeanuts - the data come from a sample of 15 individuals, so I do not assume normality. My aim is to compare the scores of subjects that exhibit high levels of the observed behavior with subjects that show low levels of the behavior on  a different score. Therefore I would like to cluster all of them into two distinct groups based on their level of expression (high vs. low). Thank you. " CreationDate="2011-05-01T20:33:11.610" UserId="4420" />
  <row Id="17423" PostId="10216" Text="Wow. Upvoted for the originality of the examples. It brought back many memories!" CreationDate="2011-05-01T21:29:55.263" UserId="2044" />
  <row Id="17424" PostId="10204" Text="I'm not sure that I understand the variables in your study. It seems that you have one continuous independent variable and at least one continuous response variable. You say that you would like to cluster the independent variable. Why not just use it as a continuous variable without clustering? I think this is like what KennyPeanuts asked." CreationDate="2011-05-01T22:13:25.013" UserId="3874" />
  <row Id="17425" PostId="10212" Text="First of all thanks for your long and careful answer, I really appreciate that. As for the example, it is very interesting, but I am struggling to understand why the process is ergodic: the notion of ergodicity I have in mind, coupled with stationarity, implies strong mixing, i.e. asymptotic independence of the terms of the sequence, which in this example seem perfectly dependent." CreationDate="2011-05-01T22:56:00.370" UserId="4422" />
  <row Id="17426" PostId="10211" Text="@Behacad - well-explained problem.  You are estimating a heck of a lot of parameters using very sparse data.  So inference is going to be awfully shaky.  But I'd like to step back and ask - are you using these 42 to infer relationships among a larger population?  If so, is the 42 a random sample, or a least a demonstrably representative one?" CreationDate="2011-05-01T23:02:33.483" UserId="2669" />
  <row Id="17427" PostId="10212" Text="Sorry, I was wrong again. Mixing is stronger. Then it might be ergodic even if it's not mixing." CreationDate="2011-05-01T23:05:29.520" UserId="4422" />
  <row Id="17428" PostId="10218" Text="There is also an [accompanying paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.65.2585&amp;rep=rep1&amp;type=pdf) for the particle filter talk" CreationDate="2011-05-01T23:23:14.397" UserId="4360" />
  <row Id="17429" PostId="10212" Text="OK. Now I understood why it is ergodic. Trivially all shift invariant subsets have either probability zero or one." CreationDate="2011-05-01T23:33:23.007" UserId="4422" />
  <row Id="17430" PostId="10160" Text="Correct, the curve is built using only the mean and std dev of the observed data and is meant to show what the data would look like if it were normal.  The frequency bars are the actual observations. Given the statistics shown to the left of the chart indicating positive skewness and fat tails the point is made fairly well.  There is more green than red and there are frequency bins above the normal curve tails." CreationDate="2011-05-02T00:11:35.580" UserId="933" />
  <row Id="17432" PostId="10218" Text="@Thies: +1 and thanks for your answer. I know the first video by heart :) As for your idea in the last paragraph, they perform essentially the same approach in the paper &quot;Force platforms as ergometers&quot; I refer to just below the graph." CreationDate="2011-05-02T00:19:55.633" UserId="4240" />
  <row Id="17433" PostId="10163" Text="Great answer.  Thanks whuber.&#xA;Found a good page that quickly explains how to read the chart.&#xA;http://www.skymark.com/resources/tools/normal_test_plot.asp" CreationDate="2011-05-02T00:44:07.613" UserId="933" />
  <row Id="17434" PostId="10216" Text="Ditto, +1 for getting D&amp;D into a stats thread." CreationDate="2011-05-02T01:45:32.953" UserId="1569" />
  <row Id="17435" PostId="10211" Text="Thank you for your comment rolando2! The sample includes 42 university students, and I am looking at the relationship between a number of factors and anxiety. The relationships I am looking to infer would be among the general population. My inferences are limited because participants are all relatively young students, but I am not looking for any specific population (e.g., individuals suffering from an anxiety disorder). I am interested in broadly stating for instance that X is indirectly associated with Y in a non-clinicam sample. Does that answer your questions?" CreationDate="2011-05-02T02:41:14.497" UserId="3262" />
  <row Id="17436" PostId="10217" Text="I agree with this answer : ) " CreationDate="2011-05-02T02:52:18.087" UserId="3262" />
  <row Id="17437" PostId="10216" Text="Err what is a buff?" CreationDate="2011-05-02T03:08:04.753" UserId="2965" />
  <row Id="17438" PostId="10219" Text="I respectfully disagree regarding your second recommendation. Though certainly an interesting book, it is almost surely *not* the place to start for a non-statistician. Especially someone concerned about clear delivery." CreationDate="2011-05-02T03:43:08.913" UserId="2970" />
  <row Id="17439" PostId="10219" Text="You're right, it's certainly not a good first read on the topic and not written in a concise way. Still, i thought for a scientist interested in how a consistent way of doing statistical analysis might be built, it makes an interesting read. But definitely not an introduction. I'll update my answer to reflect that." CreationDate="2011-05-02T04:17:28.497" UserId="4360" />
  <row Id="17440" PostId="10220" Text="Your question is pretty vague; some context about what sort of process you're trying to simulate would be very helpful." CreationDate="2011-05-02T04:27:20.150" UserId="26" />
  <row Id="17441" PostId="10220" Text="@JMS: Well I'm just looking for the various distributions so I can study their relevance. Basically when confronted with a problem I wish to choose an appropriate distribution. Specifics: I'm looking to perform risk analysis in a software project using monte-carlo distributions. The type of variable I may need to model is very context dependent e.g.: Risk of the project being called off midway through development - how best to capture this as a distribution to model it? There could be others, so I'm just looking for a layman's descriptions of different type of distributions..." CreationDate="2011-05-02T04:39:20.463" UserId="4426" />
  <row Id="17442" PostId="10220" Text="Your question remains too vague IMO. Any good introductory probability text (Sheldon Ross's or Jim Pitman's, for example) will give not only mathematical definitions of probability distributions but also their properties and physical processes which give rise to them. You will almost certainly learn more distribution theory there than here." CreationDate="2011-05-02T04:56:54.290" UserId="26" />
  <row Id="17443" PostId="10220" Text="@JMS: Agreed. I was more keen on knowing what other applicable distributions have people tried and if I could gleam anything from their experience/suggestions. I'll look up the books you suggested for sure. However, if the moderators deem this question to be too vague and would like to close it, I'd accept the decision!" CreationDate="2011-05-02T05:04:12.977" UserId="4426" />
  <row Id="17444" PostId="10222" Text="@NRH: Great! I looked through the contents and a sample chapter and it seems under the application section for each distribution it lists where/how/why was this distribution chosen and to model what phenomena. That's pretty much what I was looking for. I've already gone through the wikipedia links :) I guess I should be able to take it from here - else the forum is always there to help with the clarifications :D" CreationDate="2011-05-02T06:30:02.637" UserId="4426" />
  <row Id="17445" PostId="10217" Text="That was very informative, and makes sense. Thanks so much!" CreationDate="2011-05-02T06:35:41.460" UserId="4420" />
  <row Id="17446" PostId="10223" Score="2" Text="@lmsasu Compared to the message posted on [weka ml](http://comments.gmane.org/gmane.comp.ai.weka/25217) (notwithstanding the cross-post on [MetaOptimize](http://metaoptimize.com/qa/questions/5656/model-selection-in-weka-through-cross-validation-for-regression-problems)), do you mean you want to compare different regression models, or just to optimize a given model with e.g., cross-validation?" CreationDate="2011-05-02T06:40:45.307" UserId="930" />
  <row Id="17447" PostId="10221" Text="Jeromy, thanks for taking the time. It is very helpful." CreationDate="2011-05-02T06:48:27.590" UserId="4420" />
  <row Id="17448" PostId="10184" Text="@cardinal - if you're referring to logistic and t-dist.  [here is a link](http://www.google.com.au/url?sa=t&amp;source=web&amp;cd=8&amp;ved=0CEAQFjAH&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.78.7165%26rep%3Drep1%26type%3Dpdf&amp;rct=j&amp;q=logistic%20equals%20student%20t&amp;ei=q1m-TaiCAYHIrQfll_j5BQ&amp;usg=AFQjCNG5FtPgA4RxzHHw1aWYyNT_j8yONg&amp;sig2=yiWmzDA0qWJXXxZikMAc9A&amp;cad=rja)" CreationDate="2011-05-02T07:17:29.597" UserId="2392" />
  <row Id="17449" PostId="10204" Text="@Orp I've altered this question in line with the comments with the aim of making it clearer for others in the future. Feel free to modify if I have misconstrued what you are asking." CreationDate="2011-05-02T07:55:21.700" UserId="183" />
  <row Id="17450" PostId="10228" Score="2" Text="This sort of question is probably more suited for [Stack overflow](http://stackoverflow.com), as there is no statistics involved. I answered two of them because the answers were short, but please send more of these type of questions over there. Better yet: read the introduction to R, which holds the answer to at least your previous question..." CreationDate="2011-05-02T07:58:31.760" UserId="4257" />
  <row Id="17452" PostId="10231" Score="1" Text="What do you define as the best result? How do you suggest this should be automated as per user2721's question?" CreationDate="2011-05-02T08:13:05.013" UserId="183" />
  <row Id="17453" PostId="10230" Text="You may find this related question on number of clusters useful: http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice" CreationDate="2011-05-02T08:14:02.423" UserId="183" />
  <row Id="17454" PostId="10224" Text="Wouldn't the skewness be a more appropriate measure here?  I say this because this seems to be what you are interested in.  So rather than take the average time $\overline{t}$, take $\frac{1}{n}\sum_{i}(t_{i}-\overline{t})^{3}$, or why not take both mean and skewness?  Any particular reason you can't take two numbers?" CreationDate="2011-05-02T08:18:34.967" UserId="2392" />
  <row Id="17455" PostId="10221" Text="@Orp You can now upvote @Jeromy's response." CreationDate="2011-05-02T08:19:03.030" UserId="930" />
  <row Id="17456" PostId="10230" Score="1" Text="Just to clarify, you are looking for a feature that will automate the determination of number of clusters? or are you simply looking to batch process the running of a set of cluster analyses where number of clusters are known? What is your current manual approach to deciding whether two or three clusters is appropriate? do you wish to continue to use this rule or are you interested in other procedures for determining number of clusters?" CreationDate="2011-05-02T08:19:34.857" UserId="183" />
  <row Id="17457" PostId="10226" Text="@JTT The OP explicitly said he doesn't want to compute the distance matrix and the dendrogram again, so I'd suggest making this point clearer in your response." CreationDate="2011-05-02T08:20:53.263" UserId="930" />
  <row Id="17458" PostId="10231" Score="1" Text="He indicates that sometimes he 'needs' 2 and sometimes 3. So apparently, he has some criterion, which he doesn't mention. It surely is not uncommon to run clustering over a set of possible cluster numbers, and then evaluate." CreationDate="2011-05-02T08:30:40.253" UserId="4257" />
  <row Id="17459" PostId="10229" Text="A bit shorter version: `M[sample.int(nrow(M)),]`" CreationDate="2011-05-02T08:34:40.230" UserId="88" />
  <row Id="17460" PostId="10231" Score="1" Text="@Jeromy [Silhouette measure](http://en.wikipedia.org/wiki/Silhouette_(clustering)) is quite usable, at least for start; it often offers clear minimum, so it is good for automatic optimization." CreationDate="2011-05-02T08:41:48.900" UserId="88" />
  <row Id="17461" PostId="10223" Score="1" Text="@chl In fact he should do both to get reliable results =)" CreationDate="2011-05-02T08:45:20.427" UserId="88" />
  <row Id="17462" PostId="10223" Text="@mbq is right. However, the first step is to optimize some specific regression models. @chl: unfortunately, the weka ml community didn't come with a useful response. " CreationDate="2011-05-02T09:33:09.643" UserId="976" />
  <row Id="17463" PostId="10171" Text="I think this link http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.9299&amp;rep=rep1&amp;type=pdf refers to the right technique. I am currently trying to get back in touch with the lecturer who told me about this method ..." CreationDate="2011-05-02T09:51:45.690" UserId="3896" />
  <row Id="17464" PostId="10216" Text="+1 (should be +10) - Best explanation I've ever heard :]" CreationDate="2011-05-02T10:12:59.020" UserId="22" />
  <row Id="17465" PostId="10224" Text="I understand you have n problems grouped by difficulties. But what do you mean when you say “each benchmark”? By “interval averages” do you mean averages within a difficulty-group?" CreationDate="2011-05-02T10:16:27.267" UserId="3911" />
  <row Id="17466" PostId="10236" Score="1" Text="40000/12e5=3.3%, this does not look a very low rate to me." CreationDate="2011-05-02T11:33:37.640" UserId="3911" />
  <row Id="17467" PostId="10236" Score="1" Text="Thanks ga..in case people need more context for deciding low and high event rate, this data is of  insurance sector." CreationDate="2011-05-02T11:41:45.820" UserId="1763" />
  <row Id="17468" PostId="10196" Score="1" Text="+1 to this answer, but also adding that even if the most recent 3 months are the only ones that correlate to the answer, there could also be factors like the same month of the year in the last 5 previous years that are useful inputs (e.g. if Decembers tend to look like previous Decembers, typically)." CreationDate="2011-05-02T12:59:06.533" UserId="2917" />
  <row Id="17469" PostId="10236" Score="2" Text="You might be interested in [Logistic Regression in Rare Events Data](http://gking.harvard.edu/category/research-interests/methods/rare-events)." CreationDate="2011-05-02T13:37:43.363" UserId="307" />
  <row Id="17470" PostId="10160" Text="@Golly the problem goes beyond that: the curve is uniformly too high.  You need to rescale it so its total area equals the total area of the bars." CreationDate="2011-05-02T14:14:43.003" UserId="919" />
  <row Id="17471" PostId="10032" Text="To get to the data format used here... rep(1:5, c(0,1,1,9,4))." CreationDate="2011-05-02T14:30:26.920" UserId="601" />
  <row Id="17472" PostId="10232" Score="1" Text="The problem isn't picking a model beforehand, it's not doing the model checking afterwards! If you just pick a model to fit your data, without using any substantive knowledge of the problem, you're all but guaranteed to overfit it. Or you might miss important features that aren't present in your dataset (heavy tails, for example) A better &quot;workflow&quot; is 1) Pick a model that may reasonably represent your problem 2) Fit the model to the data 3) Check the model 4) Rinse and repeat." CreationDate="2011-05-02T14:40:50.237" UserId="26" />
  <row Id="17473" PostId="10220" Text="Oh I wouldn't vote to close; I just thought with some extra information we might be able to help more. Hopefully you've got a good jumping off point now :)" CreationDate="2011-05-02T14:43:00.000" UserId="26" />
  <row Id="17474" PostId="10238" Text="I added the `multinomial` tag because it will provide useful related links at the right of this page." CreationDate="2011-05-02T16:18:44.503" UserId="919" />
  <row Id="17475" PostId="10245" Text="Yes, I will add more variables to my model. However, I was planning on doing it one at a time, ie considering each time crimes VS income, crime VS age, crime VS ethnicity, etc. Doing everything at the same time may be faster, but will be harder to visualize (plus, I don't need relations between the other variables, just with the crimes)." CreationDate="2011-05-02T16:35:53.193" UserId="3699" />
  <row Id="17476" PostId="10244" Text="Well I have many questions that I would like to answer, for instance what is the distribution of crimes Y given X (where X is the average income for instance). Doing a regression will give me something like `Y = mX + b`, but I'd be more interested in getting `P(Y|X) = N(X, 1)` for example." CreationDate="2011-05-02T16:39:42.990" UserId="3699" />
  <row Id="17478" PostId="10152" Text="@Aniko The `1` value is significantly over-represented because it is the minimum value representing the simplest case for the data I am measuring. I guess I could transform the data (x' = x-1) and then use the zero-inflated model, but I wasn't sure what effect that would have and if the interpretation of the results would be the same." CreationDate="2011-05-02T16:51:37.410" UserId="1973" />
  <row Id="17479" PostId="10232" Text="@Ian: That's a great point. My only concern is that sometimes we really *don't* have any input data and have to model it with certain assumptions. Like the example I gave in a comment above - the risk of a project being called off midway. Past history may be only 2-3 data points but to 'estimate' risk in this case would require a 'model fit' as in @JMS step 2. I agree steps 3 and 4 are equally important!" CreationDate="2011-05-02T17:00:10.553" UserId="4426" />
  <row Id="17480" PostId="10232" Text="@Ian: You mention that *...there are better ways of looking at the problem...* Could you please throw more light on that?" CreationDate="2011-05-02T17:00:54.330" UserId="4426" />
  <row Id="17482" PostId="10241" Score="3" Text="I would use a scatterplot, it's hard to visualize what is going on with the lines everywhere. " CreationDate="2011-05-02T17:36:38.290" UserId="2965" />
  <row Id="17483" PostId="10160" Text="Gotcha.  Yet another reason this isn't an ideal solution.  " CreationDate="2011-05-02T17:45:53.243" UserId="933" />
  <row Id="17484" PostId="10220" Text="@JMS: Absolutely! Good some good points/directions...pretty much what I was looking for! Thanks." CreationDate="2011-05-02T18:15:21.767" UserId="4426" />
  <row Id="17485" PostId="10152" Text="Then it means that you have no 0's? Subtracting 1 from the data sounds like a good idea, but you could also consider the _positive negative binomial distribution_. It all depends on the actual meaning of the data points." CreationDate="2011-05-02T18:20:51.293" UserId="279" />
  <row Id="17486" PostId="10249" Score="2" Text="According to the legend for that graphic, &quot;simple coincidence&quot; is synonymous with &quot;expected from pure chance.&quot;  The latter might mean that if we assume the answers to all three questions are independent and if a random man's answers are matched to a random woman's answers, then they should match 32%/3.7 of the time." CreationDate="2011-05-02T18:25:04.897" UserId="919" />
  <row Id="17487" PostId="10232" Text="@Nupul Do you really only have 2 or the data points? What about thinking in terms of Pr(project *not* called off)? Also, Ian is talking about fitting a model too but his answer (the way I read it) would have you get some data, pick a model based on that data, and then fit that model to said data. This is overfit city; you can always find a model to fit your data arbitrarily well, but it won't perform out of sample." CreationDate="2011-05-02T18:29:22.930" UserId="26" />
  <row Id="17488" PostId="10249" Score="3" Text="@whuber: What would you bet that figure hasn't been corrected for the (rather large) number of multiple comparisons present?" CreationDate="2011-05-02T19:33:27.583" UserId="2970" />
  <row Id="17489" PostId="10248" Score="1" Text="Income typically has right skewed distribution with few extreme high salaries. So median may be better in characterising the district as it is not sensitive to the extremes. However, I remember reading that inequalities were predicting crime so incorporating the extremes in the model may be valuable." CreationDate="2011-05-02T19:48:48.997" UserId="3911" />
  <row Id="17490" PostId="10244" Text="@Wookai thanks for the clarification.  I think I understand what you are going for... I added some more info to my answer (which was previously a comment to @Leo 's answer)" CreationDate="2011-05-02T19:56:01.797" UserId="4048" />
  <row Id="17491" PostId="10232" Score="1" Text="@nupul regarding the other ways to look at the problem, the issue with Monte-Carlo analysis is that it is prone to hide the fact that you don't understand the inputs very well under the blanket of sophisticated analysis. For this reason it is a good tool by those trying to 'hide' a lack of input data or understanding. For this reason I would recommend simple parametric analyses on key variables initially to understand how they affect the model and then report on those results rather than hiding it behind a complex Monte-Carlo analysis. " CreationDate="2011-05-02T22:02:18.887" UserId="210" />
  <row Id="17492" PostId="10232" Text="@nupul regarding the lack of input data you can do Monte-Carlo analysis with this lack of data but the problem is people who see your results won't necessarily understand how limited the inputs to the model are. Estimating a model fit based on the fundamental properties of the distributions is a feasible way to do this, however my preferred way of dealing with it is to try and find as much published data as possible and use that data. The problem with this approach is it is very time (and cost) consuming and with no guarantee of success so projects managers will rightly have some issues. " CreationDate="2011-05-02T22:07:11.833" UserId="210" />
  <row Id="17493" PostId="10249" Score="3" Text="@Cardinal Criticizing the statistical methods on that site would be like shooting turtles in a barrel: too easy, messy, time-consuming, and with little value.  Let them have their fun..." CreationDate="2011-05-02T22:45:33.193" UserId="919" />
  <row Id="17494" PostId="10249" Score="3" Text="@whuber: I suppose that's true. Only thing is, it appears they (actually) believe they're doing something real. Also, how many times have you shot turtles in a barrel? :)" CreationDate="2011-05-02T23:03:14.620" UserId="2970" />
  <row Id="17495" PostId="10253" Text="@Mike thanks for the suggestion I will look into it.  My biggest concern is the comparison though.  I think for most of the curves I could get a decent fit using a variety of approaches but I don't know how to then compare among lakes.  " CreationDate="2011-05-03T00:04:28.953" UserId="4048" />
  <row Id="17496" PostId="10239" Score="1" Text="I have to say this advice does not answer the question at all. 1) Their is nothing in the question that suggests the OP is dealing with spatial or temporal data. 2) How would aggregating the data help to identify any meaningful relationships (it uses less information than the original units!) " CreationDate="2011-05-03T00:21:59.000" UserId="1036" />
  <row Id="17497" PostId="10239" Score="1" Text="Also as a note, for any observed relationship to occur at an aggregated level it has to be present in the level of the original units, although a relationship at the aggregated level does not necessarily reflect what the relationship between the two variables is at the disaggregated level. See http://qmrg.org.uk/files/2008/11/38-maup-openshaw.pdf" CreationDate="2011-05-03T00:22:13.867" UserId="1036" />
  <row Id="17498" PostId="10196" Text="Great answer. I found this to be very helpful. A high level explanation is exactly what I wanted. @rossdavidh: Good point +1." CreationDate="2011-05-03T00:33:49.387" UserId="4403" />
  <row Id="17499" PostId="10228" Text="Thank you! I thought R questions fit here, there must many more R pros here than at StackOverflow.com&#xA;" CreationDate="2011-05-03T00:52:02.613" UserId="4432" />
  <row Id="17500" PostId="10244" Score="1" Text="@Kenny - you make many good points but I take exception with 2.  I would never recommend stepwise regression to someone new to regression.  The pitfalls are very numerous and extremely well documented.  Also, &quot;the explanatory variables of a multiple regression [...] need to be uncorrelated&quot; - if they were uncorrelated, then the relationships with Y would be identical to those obtained through bivariate correlation.  In other words, there would be little point in running the regression, which is largely to show how each predictor relates to Y while controlling for the others." CreationDate="2011-05-03T02:48:36.493" UserId="2669" />
  <row Id="17501" PostId="10211" Score="1" Text="@Behacad - assuming you can defend to your potential critics the representativeness of your sample, I would say unequivocally that trying to estimate relationships among 12 variables is asking too much of your 42 cases.  See if you can simplify your model to include just the 3 most interesting predictors.  Although I realize it's painful to part with data you may have worked hard to collect!" CreationDate="2011-05-03T03:00:24.923" UserId="2669" />
  <row Id="17502" PostId="9090" Text="@prob - I appreciate your giving it a shot.  I'm still unable to follow." CreationDate="2011-05-03T03:27:36.997" UserId="2669" />
  <row Id="17503" PostId="10236" Text="Thanks Bernd..." CreationDate="2011-05-03T05:55:26.517" UserId="1763" />
  <row Id="17504" PostId="10239" Text="agree with andy. " CreationDate="2011-05-03T05:55:53.303" UserId="1763" />
  <row Id="17505" PostId="10263" Text="@ocram: if we substitute your $P[X|z,\theta)$, we will end up $P(z|X,\theta)P(X|\theta)$ but not = $P(z|X,\theta)P(X,\theta) $." CreationDate="2011-05-03T06:57:04.883" UserId="4290" />
  <row Id="17506" PostId="10263" Score="1" Text="@Learner: Well... I have not read the document, but $\theta$ is the standard notation for a parameter. Now, a parameter is a fixed quantity, not a random variable, and it does not make sense to compute $P(\theta)$... Could it be a typo ?" CreationDate="2011-05-03T07:08:00.230" UserId="3019" />
  <row Id="17507" PostId="10247" Text="Very elucidating answer for us non-stats guys/gals." CreationDate="2011-05-03T07:45:51.270" UserId="144" />
  <row Id="17508" PostId="10241" Text="Thanks for the suggestion, I added one. It gives a better understanding of the &quot;inner&quot; structure." CreationDate="2011-05-03T08:16:24.337" UserId="3699" />
  <row Id="17509" PostId="10244" Text="It's true that as a beginner, I should try to first get a better understanding of the basic tools, before using the heavy machinery. I don't want to blindly apply some tests and guess some interpretation (as I did during my undergrad social psychology class ;)...)." CreationDate="2011-05-03T08:18:04.973" UserId="3699" />
  <row Id="17510" PostId="10241" Score="1" Text="sounds like a textbook case. " CreationDate="2011-05-03T08:19:52.893" UserId="1709" />
  <row Id="17511" PostId="10248" Text="Thanks for the suggestion, I tried with the mean. I don't have the raw income values however, but I have access to some detailed statistics (percentiles, etc.)." CreationDate="2011-05-03T08:22:30.697" UserId="3699" />
  <row Id="17512" PostId="10180" Text="@brett: I edited the question (we'll see if it pass peer review ;)). Please check that the edits do not contradict your intention. You misleadingly called the &quot;probability of being better&quot; of GWO &quot;confidence&quot;, which confused some statisticians here. Please correct me if you referred to another thing with the term &quot;confidence of probabilities&quot;. " CreationDate="2011-05-03T09:10:45.180" UserId="264" />
  <row Id="17513" PostId="10242" Text="@ltleung Do you picked the exponential distribution because its pdf decreases smoothly from 0 to infinity ? If it fits (i.e. the error is small enough) it feel that your approach is reasonable, although I do not know any standard recipes on converting probabilistic functions to fuzzy functions." CreationDate="2011-05-03T09:25:11.453" UserId="264" />
  <row Id="17518" PostId="10244" Text="@rolando2 - good point about stepwise regression, I agree it is often abused.  I am not sure I understand your point about the predictor variables.  If the predictors are correlated themselves then they cannot independently estimate the response as they are not independent themselves.  To quote Gotelli and Ellison 2004: &quot;collinearity [between the predictors] makes it difficult to estimate accurately regression parameters and to tease apart how much variation in the response variable is actually associated with each of the predictor variables&quot; (A Primer of Ecological Statistics) " CreationDate="2011-05-03T11:42:27.113" UserId="4048" />
  <row Id="17519" PostId="10242" Text="well, it can be any distribution. My doubt is that, since fuzzy set theory is based on possiblity, I wonder if it is appropriate to use a PDF to model a membership function." CreationDate="2011-05-03T12:00:57.797" UserId="4440" />
  <row Id="17520" PostId="10216" Text="@Charles You're a superstar! Thank you so much for the answer!" CreationDate="2011-05-03T12:06:41.283" UserId="4429" />
  <row Id="17521" PostId="10242" Text="@ltleung The question &quot;*it is appropriate to use a PDF to model a membership function*&quot; is awesome ! I suggest to generalize the question (i.e. reformulate it) with your case as example. " CreationDate="2011-05-03T12:38:23.683" UserId="264" />
  <row Id="17522" PostId="10186" Text="Thank you all for the insightful answers." CreationDate="2011-05-03T12:54:01.393" UserId="527" />
  <row Id="17524" PostId="9866" Text="@charles.y.zheng Sorry, I was wrong. I finally learned (among others [here](http://stats.stackexchange.com/questions/2356/are-there-any-examples-where-bayesian-credible-intervals-are-obviously-inferior-t)), that frequentists are not even allowed to calculate confidence intervals on empirical data. Hence my follow up ideas (which I did not reveal) about how a frequentist would answer my question were all wrong, too. I am little insecure, however, since the question got 4 but your answer not a single upvote :(." CreationDate="2011-05-03T13:18:18.003" UserId="264" />
  <row Id="17525" PostId="9866" Text="@charles.y.zheng Now I am not comfortable with the mixing of bayesian and frequentist ideas (e.g. when you say how frequentists deal with priors (which they don't, do they ?)). Maybe the answer is simply as you put in the comments: A frequentist cannot answer the question, since it is wrong in his world outlook (as Dikran wrote [here](http://stats.stackexchange.com/questions/2356/are-there-any-examples-where-bayesian-credible-intervals-are-obviously-inferior-t/6431#6431)) Sorry again for not believing you earlier." CreationDate="2011-05-03T13:32:31.140" UserId="264" />
  <row Id="17526" PostId="10274" Score="2" Text="From the help files, `lmrob` fits a regression model which is not the same as correlation.  I would expect `lm(A~B)` to be different from `lm(B~A)`.  In which package is `cor.mve` and `cor.mcd`?" CreationDate="2011-05-03T07:48:49.667" UserId="3556" />
  <row Id="17527" PostId="10274" Text="Did you mean `cov.mve` and `cov.mcd` in package `MASS`?" CreationDate="2011-05-03T08:01:35.150" UserId="3556" />
  <row Id="17528" PostId="10244" Text="@Kenny - it's a matter of degree.  *Excessive* correlation  (collinearity)  between predictors makes it difficult to disentangle relationships.  *Some* correlation is necessary for statistical control to have any effect.  If X and Z have an *r* of 0, the relationship between X and Y while controlling for Z (holding Z constant or partialling out Z) is still zero." CreationDate="2011-05-03T14:03:18.837" UserId="2669" />
  <row Id="17529" PostId="9995" Text="@Jeromy - question about your blog on scale construction:  You wrote about factor analysis, &quot;The more items per factor, the fewer participants required.&quot; Can you explain?  How far would you carry this principle?  btw I've read in 1 idiosyncratic source that what counts is the ratio not of items to participants but of obtained factors to participants (should be just 20:1) - any comment?  Thx." CreationDate="2011-05-03T14:20:01.567" UserId="2669" />
  <row Id="17530" PostId="10265" Text="Also found another paper by same other on [Tempered EM algorithm](http://www.cs.pitt.edu/~milos/courses/cs3750/Readings/Link-analysis/Hofmann-SIGIR99.pdf)" CreationDate="2011-05-03T15:13:32.000" UserId="4290" />
  <row Id="17531" PostId="9995" Text="@rolando2 The words you quote come from an a set of notes I wrote in 2006 http://web.psych.unimelb.edu.au/jkanglim/Anglim2006-SPSSIntroductionWorkshop.pdf. Thinking back, I took the idea from a lecture on factor analysis that summarised the findings of Velicer, W.F., and Fava J.L. (1987) An evaluation of the effects of variable sampling on component, image, and factor analysis. Multivariate Behavioral Research, 22, 193-209. " CreationDate="2011-05-03T15:34:12.707" UserId="183" />
  <row Id="17532" PostId="9995" Text="@rolando2 perhaps this site needs a question on sample size and factor analysis and role of various factors like loadings, number of items, etc." CreationDate="2011-05-03T15:36:25.230" UserId="183" />
  <row Id="17533" PostId="10258" Score="1" Text="Yes, rigorous cross-validation or bootstrapping will tell you how arbitrary the intervals are.  It is not reasonable to expect the world to operate discontinuously however.  Smooth fits are undoubtedly more realistic and will predict better at the margins.  I'm not clear on why tree-based methods are as popular as they are.  With absolutely huge databases the issues are less important." CreationDate="2011-05-03T15:38:39.473" UserId="4253" />
  <row Id="17534" PostId="10228" Score="2" Text="@Denis Gorodetski you'd be surprised at the number of R pros on stack overflow-- I often get more responses to programming oriented R questions there than here." CreationDate="2011-05-03T15:39:47.390" UserId="2817" />
  <row Id="17535" PostId="10106" Text="However if a later step assumes that lambda is estimated without error, then all confidence intervals will be too narrow and P-values too small (i.e., confidence coverages will be far from the claimed value such as 0.95).  A unified &quot;estimate lambda while estimating the betas&quot; approach is needed." CreationDate="2011-05-03T15:40:48.507" UserId="4253" />
  <row Id="17536" PostId="767" Text="@Dave You said that &quot; there are a lot of researchers who say BIC is better than AIC, using model recovery simulations as an argument. These simulations consist of generating data from models A and B, and then fitting both datasets with the two models.&quot; Would you be so kind as to point some references. I'm curious about them! :)" CreationDate="2011-05-03T16:21:43.600" UserId="2902" />
  <row Id="17538" PostId="10281" Text="Great suggestions. I like the response change idea. With this approach would I still need the week 0 score as a covariate?  Also what would be different if the week 36 value was week_36_score-week_18_score?" CreationDate="2011-05-03T16:42:06.827" UserId="2310" />
  <row Id="17540" PostId="10284" Text="@Zach: Any automation that ignores the possibilities of Level Shifts, Local Time Trends, Seasonal Pules and/or One-Time Pulses should be reviewed very carefully as these kinds of structures in the error process &quot;blind-side&quot; identification strategies." CreationDate="2011-05-03T19:03:29.357" UserId="3382" />
  <row Id="17541" PostId="10282" Text="@NRH. Thanks for the reference to Yann LeCun's talk. I look forward to watching it." CreationDate="2011-05-03T19:13:04.050" UserId="2798" />
  <row Id="17542" PostId="10283" Text="Unfortunately, I have no labeled data sets. There is only the assumption, that long tails or outliers indicate anomalies in the data set." CreationDate="2011-05-03T19:15:46.313" UserId="4446" />
  <row Id="17543" PostId="10244" Text="@rolando2 - thanks for the clarification.  That makes sense.  I edited the answer." CreationDate="2011-05-03T19:23:44.567" UserId="4048" />
  <row Id="17544" PostId="10258" Text="What I meant in the last sentence is that tree methods suffer less in terms of predictive accuracy when the sample size is exceedingly large.  Methods based on a single tree can not improve upon continuous modeling for any sample size, however.  And binning the data often results in gaming of the system at the margins." CreationDate="2011-05-03T19:36:43.027" UserId="4253" />
  <row Id="17549" PostId="10283" Text="@cryptron I see.  So what you need is a way to dynamically identify outliers." CreationDate="2011-05-03T20:15:08.090" UserId="2817" />
  <row Id="17550" PostId="10284" Text="@IrishStat: Abhishek asked &quot;is the excel solver best way to find the AR parameters?&quot;  I think using R in this case to find the AR parameters would be better than using excel." CreationDate="2011-05-03T20:17:01.757" UserId="2817" />
  <row Id="17551" PostId="10284" Text="@Zach: You are right R would be the best of two bad choices." CreationDate="2011-05-03T20:21:45.967" UserId="3382" />
  <row Id="17552" PostId="10251" Score="1" Text="Maybe I'm missing something so please correct me if I'm wrong, but it should be possible (at least in principle) to construct what is done in PCA using matrices as a (complicated) linear programming problem, but I don't know how you'd state all of the constraints required. Also I'm not sure that'd be very simple to do compared to just using PCA. Why are you trying to avoid matrices?" CreationDate="2011-05-03T20:45:54.897" UserId="4325" />
  <row Id="17553" PostId="9866" Score="1" Text="Perhaps my interpretation was not as mainstream as I believed, but there is nothing intrinsically wrong about putting frequentist and Bayesian methods on the same footing.  See Lehmann and Casella's Theory of Point Estimation, in which frequentist and Bayesian methods are compared via statistical decision theory." CreationDate="2011-05-03T21:10:30.877" UserId="3567" />
  <row Id="17554" PostId="9866" Text="sounds interesting, thank you !" CreationDate="2011-05-03T21:28:15.783" UserId="264" />
  <row Id="17555" PostId="10251" Text="@Chris I don't see how one can get to a linear programming problem. It wasn't my understanding either that matrices should be avoided in the *computation*. The question was what kind of problem is solved by PCA, and not the way it is done (by computing the SVD for instance). The solution by cardinal says that you find successive orthogonal directions of *maximal variance*. The solution I presented says that you find hyperplanes with minimal reconstruction error. " CreationDate="2011-05-03T21:30:04.800" UserId="4376" />
  <row Id="17556" PostId="10291" Text="Ok.  You're right.  It is best to explain my definitions.  And in thinking about it again, it's not the definitions that I need.  What I need is the appropriate method for creating 1 universal score.  Whether that be an Admission score or Traffic score.  How does one go about creating a universal metric that's a function of other variables, which were transformed to put them both on a similar scale?  And don't worry about the weights.  I understand that even just straight summing is weighting the metrics 1/1.  But that's less of a concern for me right now." CreationDate="2011-05-03T21:39:07.053" UserId="4455" />
  <row Id="17557" PostId="10283" Text="@Zach: That would solve a part of the problem, yes." CreationDate="2011-05-03T21:39:29.037" UserId="4446" />
  <row Id="17558" PostId="10292" Text="So, you're saying at least start with what I described as Standardization (studentizing), then adjust the weights to best fit the data/scenario?  That makes sense.  I just don't understand why I would divide by the SD.  And in researching I found something called the Standardized Mean Difference.... and I've just been confusing myself.  It seems like it should be simple.  You either put them both on Scale-A, or one on the same scale as the other, then sum.  But no.  Instead I'm confused and all Wiki'd out for the moment." CreationDate="2011-05-03T21:43:42.937" UserId="4455" />
  <row Id="17559" PostId="10251" Text="@chris I am hoping to find another way to view PCA, without the matrix algebra, in order to augment my understanding of it. " CreationDate="2011-05-03T21:52:46.890" UserId="74" />
  <row Id="17560" PostId="10251" Score="1" Text="@Chris, You have a quadratic objective function and an $\ell_2$ norm equality constraint. Alternatively, under the formulation in @NRH's answer, you have a matrix rank constraint. That's not going to beat itself down to a linear-programming problem. @NRH gives some good intuition, and, in fact, there is a very close connection between the two perspectives on PCA that have been given. Perhaps in collaboration with @NRH, we can add that to his/her post to make the full set of answers more complete." CreationDate="2011-05-03T21:59:50.473" UserId="2970" />
  <row Id="17561" PostId="10244" Text="To end my last comment I should have said not &quot;still zero&quot; but &quot;unchanged.&quot; " CreationDate="2011-05-03T22:14:34.293" UserId="2669" />
  <row Id="17562" PostId="10260" Text="(+1) Good points. Some suggestions: It would be good to define $\lambda_i$ and would be *really* nice to give a short proof of the result. Or, alternatively, it can be connected to the optimization problem involving Rayleight quotients. I think that would make the answers to this question very complete!" CreationDate="2011-05-03T22:14:39.503" UserId="2970" />
  <row Id="17563" PostId="10295" Text="This certainly is an interesting question, but I think you might be better off asking this on stats.SE." CreationDate="2011-05-03T22:05:24.150" />
  <row Id="17564" PostId="10295" Text="@Theo You may be right, actually. I'll flag for the mods to migrate if they think it's best." CreationDate="2011-05-03T22:08:59.920" />
  <row Id="17565" PostId="10295" Score="1" Text="Why would views not contribute to interesting-ness? (but worse, why would they contribute negatively?)  More interesting things tend to be viewed more often...  The fundamental problem here is what does _interesting_ even mean?  Does it means questions of general _interest_ or questions that are of interest to a more specific higher level audience?  For someone to answer this question with &quot;mathematical rigourousness&quot;, it needs to be posed rigorously first. " CreationDate="2011-05-03T22:11:27.433" />
  <row Id="17566" PostId="10295" Text="Views bias the questions because one question might, say, be link by a good site and receive tons of views--if you look at the [top rated questions](http://math.stackexchange.com/questions?sort=votes) they are all high view questions; by interesting I mean the questions that have more value as perceived by the users of the site. In any case, the question still stands: what is the correct way of combining views and votes to get the best predictor of quality?" CreationDate="2011-05-03T22:14:55.737" />
  <row Id="17567" PostId="10295" Text="**[Here](http://stats.stackexchange.com/questions/9885/application-of-machine-learning-methods-in-stackexchange-websites)** is a recent somewhat-related question to yours, though the answers won't be relevant." CreationDate="2011-05-03T22:22:27.917" UserId="2970" />
  <row Id="17568" PostId="10251" Text="@cardinal Yes, either way leads to the same thing of course. I really think this is very well explained in The Elements, and since the book is freely available online for download, it is possible to study the details there. I just might add that the data vectors in @cardinal's solution are assumed centered. " CreationDate="2011-05-03T22:25:02.033" UserId="4376" />
  <row Id="17569" PostId="10295" Score="2" Text="The math people asked good questions.  This question's logic seems circular: it appears to ask us for a formula to measure the &quot;quality&quot; of an SE question but it doesn't stipulate what &quot;quality&quot; means except to give non-operational synonyms like &quot;value as perceived by the users of the site.&quot;  You can't get something for nothing!  " CreationDate="2011-05-03T22:29:21.467" UserId="919" />
  <row Id="17570" PostId="10251" Text="@NRH, Actually, I like *ESL* a lot, but I think the treatment there of this topic is pretty superficial, as it is for many of the topics in the book. In particular, they don't prove (or even assign as an exercise) the important part of the solution for the optimization problem you give." CreationDate="2011-05-03T22:32:52.570" UserId="2970" />
  <row Id="17571" PostId="10251" Text="@NRH, Also, whether or not you choose to center the features is a decision outside of the scope of PCA. There are statistical situations where doing one or the other makes the most sense. Though, I'd say, centering does win out usually." CreationDate="2011-05-03T22:34:42.693" UserId="2970" />
  <row Id="17572" PostId="10289" Score="1" Text="The last part of the question sounds like you are trying to create a *valuation* out of *multiple attributes.*  For more on that see the question and replies at http://stats.stackexchange.com/q/9137 and http://stats.stackexchange.com/q/9358 .  In particular, note that neither normalization nor standardization have any direct relevance to the Dean's problem." CreationDate="2011-05-03T22:35:11.927" UserId="919" />
  <row Id="17573" PostId="10295" Text="@whuber: hopefully I've addressed your concerns in the question." CreationDate="2011-05-03T22:36:15.447" UserId="4456" />
  <row Id="17574" PostId="10295" Text="@Skl: Much better!" CreationDate="2011-05-03T22:38:00.850" UserId="919" />
  <row Id="17575" PostId="10295" Text="I don't really understand why views has anything to do with this, since a post can have an interesting title and be totally uninteresting; not to mention linking etc...For example if I look over my own questions that I have posted, I find that the best ones have the most upvotes, and that the view total is basically reflects other factors." CreationDate="2011-05-03T22:49:58.840" UserId="4457" />
  <row Id="17576" PostId="10251" Text="@cardinal, point taken on the possibility of choosing not to center, but without centering I would not call $\mathbf{S}$ the sample covariance." CreationDate="2011-05-03T22:50:57.140" UserId="4376" />
  <row Id="17577" PostId="10295" Text="@Matt: Well, first of all views limit the top score, $v&lt;n$, using the variables in the OP. Secondly, you only need to go [here](http://stats.stackexchange.com/questions?sort=votes) to see that votes totals are heavily biased by number of views. A question seen by fewer people is likely better than a question seen by more people, if the score is equal." CreationDate="2011-05-03T22:54:18.713" UserId="4456" />
  <row Id="17578" PostId="10251" Text="@NRH: Yes, I agree." CreationDate="2011-05-04T00:08:27.567" UserId="2970" />
  <row Id="17579" PostId="10293" Text="So you have 4 binary variables: cat1, cat2, cat3, and cat4. What do you mean when you say that you want to optimize cat4 as a function of cat1, cat2, cat3?" CreationDate="2011-05-03T22:50:46.673" UserId="4337" />
  <row Id="17580" PostId="10295" Text="I'd imagine that _many_ people are trying to do what you are trying to do: come up with a way to predict &quot;interestingness,&quot; including reddit, google, and yes, stackexchange...  It would be worth taking a look at what they have considered." CreationDate="2011-05-04T02:45:24.533" UserId="3567" />
  <row Id="17581" PostId="10291" Text="@Chris, I added my answer as an edit above." CreationDate="2011-05-04T03:31:17.220" UserId="2775" />
  <row Id="17582" PostId="10291" Score="1" Text="(+1) Good edit.  @Chris: you might be interested in the notes to a short set of PowerPoint slides [here](http://www.quantdec.com/Articles/openspace/Rank2.htm): this is a presentation on the subject I gave to non-technical people.  I mention it because it has some illustrations and guidance for how to &quot;create a universal metric.&quot;" CreationDate="2011-05-04T03:41:35.953" UserId="919" />
  <row Id="17583" PostId="10256" Text="If only the first few singular values/vectors are needed, Nash and Shlien give [an algorithm](http://dx.doi.org/10.1093/comjnl/30.3.268) reminiscent of the usual power method for computing dominant eigenvalues. This may be of interest to the OP." CreationDate="2011-05-04T03:44:56.077" UserId="830" />
  <row Id="17584" PostId="10285" Score="1" Text="You say that you have &quot;ran an experiment&quot; and you say &quot; how to go about computing sample size&quot;. The short answer is &quot;count your number of participants&quot;. But I assume you intended to ask &quot;what sample size is required in order to achieve a given (perhaps &quot;acceptable&quot;) statistical power. I'll edit your question to reflect this; feel free to modify if I have misconstrued." CreationDate="2011-05-04T04:01:12.163" UserId="183" />
  <row Id="17585" PostId="10293" Score="1" Text="@Leo One (speculative) interpretation is that the O.P. seeks a subset of $S$ of {cat1, cat2, cat3} such that $\max(S)$ and cat4 differ in as few cases as possible.  With $p$ independent categories a brute-force solution takes $O(2^p)$ effort, which is terrible.  Can one do better?  I hope @McP will correct any errors in this interpretation or confirm it is what he intended." CreationDate="2011-05-04T04:26:15.213" UserId="919" />
  <row Id="17586" PostId="10284" Text="Thanks @Zach and @IrishStat . I will be using R to get more reliable results." CreationDate="2011-05-04T05:16:34.093" UserId="4445" />
  <row Id="17587" PostId="10269" Text="@Irish I am removing the seasonality using time series decomposition by calculating the seasonla indexes for each of the 12 months. If you mean that seasonality may change over time, an adaptive model whose parameters are recalculated when new data is available should work. Currently I am using time series decomposition to identify randomness and working on data without seasonality and randomness. I add the seasonality back to the forecasts but don't know how to deal with adding randomness to the forecast. May be R will deal with this." CreationDate="2011-05-04T05:24:54.540" UserId="4445" />
  <row Id="17589" PostId="10260" Text="@cardinal, I believe I completed the missing steps in getting from the reconstruction formulation to the problem you solve. " CreationDate="2011-05-04T06:33:08.043" UserId="4376" />
  <row Id="17590" PostId="10303" Text="I have no idea if this title makes any sense. If someone can title this better, feel free." CreationDate="2011-05-04T06:46:21.963" UserId="726" />
  <row Id="17591" PostId="10271" Text="Just a note, the first graph does not look like anything normal for me." CreationDate="2011-05-04T08:06:14.713" UserId="2116" />
  <row Id="17592" PostId="10271" Score="1" Text="@cryptron, the key question is what is a **sound** threshold.  For example if each raised alarm and non-raised alarm incurs certain costs, the threshold can be chosen such that minimises total costs. For that we need cost data. Without the exact definition of **sound** it is impossible to measure how to evaluate the method chosen for picking the threshold." CreationDate="2011-05-04T08:16:28.320" UserId="2116" />
  <row Id="17593" PostId="10308" Text="@RockScience I've added two tags. Could you confirm they're correct?" CreationDate="2011-05-04T08:38:02.067" UserId="930" />
  <row Id="17594" PostId="10265" Text="@Learner Looks like the one that was suggested to you on [MetaOptimize](http://metaoptimize.com/qa/questions/5673/tempered-em), right?" CreationDate="2011-05-04T08:40:37.197" UserId="930" />
  <row Id="17595" PostId="10306" Text="@:mpiktas full fledged ARIMA software can include differencing factors ,a trend coefficient and an arbitrary ARMA structure . Full fledged Transfer Function software can include ( in lieu of the differencing operator and the trend coefficient ) one or more TIME VARIABLES ( E.G. 0,0,0,0,0,1,2,3,4... ) reflecting specific points in time where a time trend has been identified. Basic time series texts,as you suggested, do not cover this extension. " CreationDate="2011-05-04T08:40:58.850" UserId="3382" />
  <row Id="17596" PostId="10265" Text="@chi: yep. But im still not found the right material for it. " CreationDate="2011-05-04T08:48:10.290" UserId="4290" />
  <row Id="17597" PostId="10269" Text="@Abhishak.Unfortunately your seasonal indices assume a specific weighting structure i.e.the # of periods to use to compute the &quot;weights&quot; and that there are no Pulses or Level Shifts or Time Trends. In terms of seasonality changing over time you are not quite correct. Adapting to change requires a model form. For example if you had 6 years of data and thee was no June effect for the the first 3 years . The correct seasonal factor would reflect the June impact for the last three years. There is no adaptivity here in the sense you are using the word. Your approach has too many presumptive steps. " CreationDate="2011-05-04T08:50:16.423" UserId="3382" />
  <row Id="17598" PostId="10271" Text="@mpiktas: I have to admit, the word &quot;sound&quot; was unfortunate in this context because I have no way of rigorously evaluating the threshold (hence, I edited it away). Basically, the threshold is supposed to minimize false positives because they are way more costly than false negatives in network anomaly detection." CreationDate="2011-05-04T08:57:29.710" UserId="4446" />
  <row Id="17599" PostId="10306" Text="@IrishStat, good remark, but my main point is still valid, if you want to forecast trend, you need to go outside classical ARIMA framework. I think the features you mention fall into definition of ARIMAX, but the X part remains the obstacle,  you need to have an external model." CreationDate="2011-05-04T08:57:47.270" UserId="2116" />
  <row Id="17600" PostId="10271" Text="@cryptron, do you have any data on what is a false positive?" CreationDate="2011-05-04T09:05:45.810" UserId="2116" />
  <row Id="17601" PostId="10271" Text="@mpiktas: Unfortunately not. Since all available information is limited to a couple data points, I am operating merely with the assumption that points which are far away from the center of the distribution (see Fig.1) might be true positives, i.e. anomalies. I am aware of the fact, that it's not possible to &quot;minimize the false positive rate&quot; by having such limited information." CreationDate="2011-05-04T09:14:42.380" UserId="4446" />
  <row Id="17602" PostId="10306" Score="1" Text="@mpiktas I agree you need to find a specification for the X partof the ARMAX Model.Classical ARIMA modelling died in about 1979 when work was done to empirically identify via Intervention Detection Pulses,Level Shifts and Local Time Trends (see I.Chang or G.C. Tiao or Ruey Tsay's literature on this). Their work and in some rare cases my developments of their work have lead to much more usable ARIMA structures incorporating Deterministic Structure such as multiple time trends and/or multiple Level Shifts.Level Shifts are simply first differences of a time trend as Pulses are to Level Shifts. " CreationDate="2011-05-04T09:26:57.970" UserId="3382" />
  <row Id="17603" PostId="10308" Text="@chl sure, thanks" CreationDate="2011-05-04T09:31:10.707" UserId="1709" />
  <row Id="17605" PostId="10310" Text="I am not sure I fully understand the difference between &quot;Level Shifts ( i.e. intercept changes)&quot; and &quot;time trend changes&quot;. Can you provide with an example for each?" CreationDate="2011-05-04T09:37:02.560" UserId="1709" />
  <row Id="17606" PostId="10310" Text="@RockScience . Level Shift 1,2,3,4,5,8,9,10,11,12 .... ls at t=6 Another example of a ls change 1,1,1,1,1,2,2,2,2,2,... ls at t=6, Time trend change 1,2,3,4,5,7,9,11,13,15,17....tt change at t=5 " CreationDate="2011-05-04T09:52:34.420" UserId="3382" />
  <row Id="17607" PostId="10308" Text="@RockScience, what particular problem requires de-trending without look ahead bias? I personally worked with different de-trending schemes analyzing (not forecasting) the business cycles. So the common thing is to pay little attention to the borders (i.e. ignore the points at the begining and end of period). The same bias may be lowered for instance looking for an appropriate model that drives the trend (either deterministic or stochastic) at the sample without edges and simply forecasting values. Building such robust model with unbiased predictions would be an art anyway." CreationDate="2011-05-04T10:49:43.603" UserId="2645" />
  <row Id="17608" PostId="10308" Text="(+1) for the questions, if one knows the solution it would be nice to hear. Regarding HP, we may go for optimal filter (optimization in $\lambda$), but the bad edge behavior remains." CreationDate="2011-05-04T10:52:56.003" UserId="2645" />
  <row Id="17609" PostId="10310" Text="trends not necessarily are deterministic, so such _knowledge_ could be of no use in real time-series applications. Even in filtering you deal with unknown data generating process and some believes about what the frequencies should be passed through by the ideal filter, so you have huge trade-offs in this case. In deterministic systems though you could go for structural changes detection." CreationDate="2011-05-04T10:57:06.433" UserId="2645" />
  <row Id="17610" PostId="10308" Text="@Dmitrij Celov: You are lucky! I need to estimate the current trend. I need also to apply the same process in the past to evaluate the methodology. I need to do a backtest of this method, and that's why I am interested ONLY in the last datapoints. " CreationDate="2011-05-04T10:59:02.053" UserId="1709" />
  <row Id="17611" PostId="10308" Score="1" Text="Depending on the length of your data set, you could try to look for look-ahead value of bias first (say you have 100 points take 50 for evaluation of trend by say HP on the whole sample and only on the first 50 points, then move by one the window from 2 to 51 and so on, look at the accuracy of your edges). May be in your case it is not so huge as you imagine. " CreationDate="2011-05-04T11:05:46.970" UserId="2645" />
  <row Id="17612" PostId="10310" Text="@Dmitri: one possible model to deal with trend is y(t)=theta0 + y(t-1) + a(t) . This easily generalizes to y(t)=[theta0 + P(T1)] + y(t-1) + a(t) where the steady state ramp changes at period T1. In contrast y(t)=3*x1 + 2*x2 + a(t) where x1=1,1,2,3,4,...t and x2=0,0,0,0,0,1,2,3,4,...t-5 reflect points in time where the trend changed ( i.e. period 1 and period 6 ). Careful analytics will suggest without any  presumed knowledge which of these MODELS is most appropriate for a particular data set. The ARMA component must also be empirically found to complete the model along with any Level Shifts." CreationDate="2011-05-04T11:29:16.473" UserId="3382" />
  <row Id="17613" PostId="10302" Text="How do i calculate perplexity for pLSA. I have datamatrix $X$ which has the count and by TEM algorithm $p(d)$ and $p(w|d)$ are calculated. " CreationDate="2011-05-04T11:44:59.817" UserId="4290" />
  <row Id="17615" PostId="10311" Score="1" Text="@bgbg, I am tempted to say that this strange peak is the more related to poor binning algorithm of histogram than to the features of data. Besides that since no context is given there is a lot of possible answers, but writing them up would not be interesting." CreationDate="2011-05-04T12:14:21.737" UserId="2116" />
  <row Id="17616" PostId="10312" Text="Wolfgang, you were as close to the problem as possible without knowing the problem domain. Thank you" CreationDate="2011-05-04T12:28:18.123" UserId="1496" />
  <row Id="17617" PostId="10311" Text="@mpiktas: actually, the binning algorithm is not guilty here. Wolfgang was right, there was a strong bias in one category of measurements. Problem solved" CreationDate="2011-05-04T12:32:33.613" UserId="1496" />
  <row Id="17618" PostId="10311" Text="@bgbg, cheers, you got lucky :)" CreationDate="2011-05-04T13:38:27.373" UserId="2116" />
  <row Id="17619" PostId="10315" Text="Great, thanks for the idea Seb. Checking it out now." CreationDate="2011-05-04T14:30:42.793" UserId="4112" />
  <row Id="17620" PostId="10316" Score="6" Text="It is only significant at the 10% confidence level, but the confidence intervals are 5%." CreationDate="2011-05-04T15:08:44.733" UserId="4257" />
  <row Id="17621" PostId="10280" Text="I made this way harder than I needed to, didn't I?  haha.  Thanks for the input.  " CreationDate="2011-05-04T15:31:21.570" UserId="4450" />
  <row Id="17622" PostId="10309" Text="If you're referring to the function defined at http://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/ (@chl's link doesn't work for me): the function performs permutation tests on the pairwise comparisons by calling `symmetry_test()` from package `coin`. `symmetry_test()` lets you choose different test statistics which are explained in `vignette(&quot;coin_implementation&quot;)` - `Tmax` is one of them. Beware that this is a technical read about the implementation of permutation tests." CreationDate="2011-05-04T15:51:11.880" UserId="1909" />
  <row Id="17623" PostId="10293" Text="@Whuber.  Essentially exactly what you have propsed." CreationDate="2011-05-04T15:52:09.167" UserId="739" />
  <row Id="17624" PostId="10211" Text="Thank you for the reply. I have a &quot;feeling&quot; that estimating relationships amongst all these variables is hard to do with 42 data points, and I see where you are coming from. That being said, what would a statistical reason (preferably cited) for this problem be? How is this different from running a number of regressions/correlations on different dependent variables? The fit is good (and I'm actually running three different models for different experimental tasks) and results are consistent across models and in line with theory. Sorry if I'm coming off as defensive! " CreationDate="2011-05-04T15:54:26.547" UserId="3262" />
  <row Id="17625" PostId="10251" Text="@NRH and @Cardinal I think both of your posts and comments here are very insightful, but based on the OP and the comment @Neil made in these posts it seems like he's looking for something more along the lines of a way to view this as a systems of equations than in terms of linear algebra for this problem. Clearly it isn't a linear program, but I apparently had a brain fail and couldn't think of systems of equations yesterday. I think that's why he hasn't accepted either of your excellent answers yet." CreationDate="2011-05-04T16:05:46.187" UserId="4325" />
  <row Id="17626" PostId="10314" Text="Thanks! This looks like a really neat set of tools." CreationDate="2011-05-04T16:21:45.360" UserId="3988" />
  <row Id="17627" PostId="10314" Text="I agree, this looks like a great tool. I've bookmarked it for when I get home." CreationDate="2011-05-04T16:31:55.553" UserId="1764" />
  <row Id="17628" PostId="9041" Text="The graphviz tool you use makes a difference. Graphviz's sfdp or twopi might be good choices. You can also specify or override the locations of some nodes if you dive into the dot file." CreationDate="2011-05-04T16:35:30.013" UserId="1764" />
  <row Id="17629" PostId="10309" Text="@caracal Yep, that was supposed to point to the same post (but from Rbloggers). Thx for providing a working link." CreationDate="2011-05-04T16:44:38.353" UserId="930" />
  <row Id="17630" PostId="10301" Text="A BIP is the right idea, but how do you linearize the objective function?" CreationDate="2011-05-04T16:48:36.147" UserId="919" />
  <row Id="17631" PostId="10271" Score="1" Text="I'm confused by your plots. This is a univariate time series $\{x_t\}$ taking values in $0&lt;x_t \leq 5$? Or should it be $0 &lt;x_t \leq 0.5$? (from looking at the x axis in your first plot). A traceplot would be more helpful too. For example, do you get high scores for a sustained period of time or in short bursts (or both)? If both, is one more troubling than the other? If you can put down a reasonable model for the data you can use theoretical quantiles from the fitted distribution, which would solve the problem you've identified with the sample quantiles." CreationDate="2011-05-04T17:49:35.360" UserId="26" />
  <row Id="17632" PostId="10317" Text="Thanks for the link to the tutorial, which looks comprehensive. I did search here before posting my question. Links 1 and 3 appear not to be related to my question." CreationDate="2011-05-04T18:04:56.917" UserId="2824" />
  <row Id="17633" PostId="10316" Text="So confidence intervals for 10% would not include 1 then?" CreationDate="2011-05-04T18:06:02.627" UserId="2824" />
  <row Id="17634" PostId="10211" Text="(Not defensive--no worries!)  Having 42 cases exposes you to sampling error at the very least, even when estimating univariate statistics.  Now, in SEM, each variable gets used many times, because you're estimating the relationship of A to B while controlling for C, D, etc.  So the effects of sampling error will be propagated, which, to my understanding, is why one typically wants large samples.  In your case, you've got potentially more kinds of error than sampling error because you don't have a random sample.  So you need to draw a very large credible interval around any result you obtain." CreationDate="2011-05-04T18:07:16.597" UserId="2669" />
  <row Id="17635" PostId="10317" Text="@SabreWolfy, link 1 further elucidate how to interpret the coefficients in terms of the original units, link 3 describes the steps to interpret the effects in terms of probabilities (which is really applicable to your question, and the suggested plots in that question would be a reasonable response to me saying the size of the direct effect is difficult to interpret without knowing the variation in age). " CreationDate="2011-05-04T18:16:12.140" UserId="1036" />
  <row Id="17636" PostId="10323" Text="His GF could misunderstand Cauchy: &quot;or Cauchy as a reminder, that *not all distributions have finite moments*.&quot;. :P" CreationDate="2011-05-04T18:52:04.573" UserId="144" />
  <row Id="17637" PostId="10323" Text="@Roman, good catch :)" CreationDate="2011-05-04T18:53:20.300" UserId="2116" />
  <row Id="17638" PostId="10295" Text="Do you have access to the &quot;trajectory&quot; of every question (i.e. score and #views at many time instants), or do you just have the current state?" CreationDate="2011-05-04T19:07:21.467" UserId="3369" />
  <row Id="17639" PostId="10322" Text="Buy her a vacuum; chicks love gifts like that." CreationDate="2011-05-04T19:19:55.830" UserId="1657" />
  <row Id="17640" PostId="10322" Text="Sounds silly, but how about choosing on the basis of colour? (Otherwise, I would probably second the Cauchy choice -- although a positive distribution with infinite mean might be better if you want to indicate something about the depth of your affection ...)" CreationDate="2011-05-04T19:27:15.247" UserId="2126" />
  <row Id="17641" PostId="10320" Text="Strangely enough, for IMUs, sometimes noise can *improve* your measurements and inferences! (**Think**: dithering.) Can you give more details on the types of signals you're trying to measure, e.g., sampling rate, frequencies of interest, etc.?" CreationDate="2011-05-04T19:34:08.027" UserId="2970" />
  <row Id="17642" PostId="10313" Text="Could you elaborate on the relative merits/appropriateness of LR vs. Pearson $\chi^2$ (which is just an approximation of Fisher's exact test)? And why are these more accurate than Fisher's exact test?" CreationDate="2011-05-04T19:39:37.543" UserId="1720" />
  <row Id="17643" PostId="10313" Text="Edited my question to remove the rule mining context - I wasn't sure if that would be helpful, but I think it's distracting from my main question. BTW, it's not a decision tree algorithm - rule mining is a different problem, and it tries to output only the most significant rules, so over-interpretation is less of an issue. (I believe.)" CreationDate="2011-05-04T19:48:39.370" UserId="1720" />
  <row Id="17644" PostId="10328" Text="Any way to move it over automatically?" CreationDate="2011-05-04T15:23:22.390" UserId="3561" />
  <row Id="17645" PostId="10328" Text="With my privileges, I'm afraid, no. But I can (and will) flag it for admin attention." CreationDate="2011-05-04T18:52:18.897" UserId="1356" />
  <row Id="17646" PostId="10211" Text="Thank you again for your feedback. I understand where you are coming from, and I agree. Hopefully, given the strength of the relationships within the models, the consistency across the models, and convergence with theory, my critics will deem my methods appropriate for now. " CreationDate="2011-05-04T20:28:30.833" UserId="3262" />
  <row Id="17647" PostId="10328" Text="Did you check this pdf: http://www.nslij-genetics.org/wli/pub/ieee-embs06.pdf (especially page 2)?" CreationDate="2011-05-04T20:44:34.513" UserId="1909" />
  <row Id="17648" PostId="10112" Text="I'd recommend looking at http://en.wikipedia.org/wiki/Particle_filter and http://en.wikipedia.org/wiki/Kalman_filter and noting the similarity. If your noise is Gaussian, the DLM/Kalman approach will serve you well." CreationDate="2011-05-04T20:59:06.103" UserId="1764" />
  <row Id="17649" PostId="10295" Text="What do you mean by the &quot;trajectory&quot;? Something I could use to plot a view/vote scatter for each question?" CreationDate="2011-05-04T21:16:51.833" UserId="4456" />
  <row Id="17650" PostId="10324" Text="True, we don't have a vested interest and we may want to be funny (which may appear we are mean). Unless you want to cut us in a deal." CreationDate="2011-05-04T21:17:27.763" UserId="144" />
  <row Id="17651" PostId="10331" Text="Wikipedia provides (without explanation) an [acceptance-rejection sampling algorithm](http://en.wikipedia.org/wiki/Gamma_distribution#Generating_gamma-distributed_random_variables) for $\Gamma(\delta), 0 \lt \delta \lt 1$.  Using it you could generate (say) two $\Gamma(1/2)$ values and add them." CreationDate="2011-05-04T21:46:39.650" UserId="919" />
  <row Id="17653" PostId="10112" Text="Thanks. As I write in the question &quot;The noise of the amplitude may not be Gaussian and may be contaminated with outliers.&quot; Please give me some time to come up with more results, I am interested in closing this question by accepting an answer." CreationDate="2011-05-04T22:21:07.433" UserId="4240" />
  <row Id="17654" PostId="10331" Text="Thanks for the link, but I am supposed to use the $\Gamma(2,1)$ as the envelope function.  " CreationDate="2011-05-04T22:55:34.843" UserId="2984" />
  <row Id="17655" PostId="10320" Text="There were 5 IMUs that streamed data via a Bluetooth link. The maximum possible sampling frequency depended on this link. In the given setup, a sampling frequency of 50Hz was used." CreationDate="2011-05-04T23:01:50.770" UserId="3052" />
  <row Id="17656" PostId="10332" Score="1" Text="+1 You got it in just as I was writing the same thing... :-).  The expected number of $\Gamma(2)$ values needed per Exponential draw is about 3.407 with the optimal shift (of $\sqrt{1/2}$)." CreationDate="2011-05-04T23:18:42.690" UserId="919" />
  <row Id="17657" PostId="10333" Text="This is something *Mathematica* is good at already." CreationDate="2011-05-04T23:34:51.977" UserId="919" />
  <row Id="17658" PostId="10332" Text="Thanks, but I am not sure if I understand how to implement a location shift.  If I understand correctly, I just adjust the Gamma function by adding a constant, but I am not exactly sure where to put it in my code. " CreationDate="2011-05-04T23:36:58.397" UserId="2984" />
  <row Id="17659" PostId="10313" Score="1" Text="Nothing in what you wrote provides any indication that the method is safe from overinterpretation (a form of overfitting).  One reason I worry about the method is that it doesn't treat continuous variables continuously.   The problem with Fisher's test is that the P-values are too large.  See for example Crans GG Stat in Med 27:3598; 2008.  The word &quot;exact&quot; has misled practitioners for decades.  The ordinary Pearson $\chi^2$ is usually more accurate.&#xA;" CreationDate="2011-05-04T23:57:24.607" UserId="4253" />
  <row Id="17660" PostId="10327" Text="So, when is the divorce final?" CreationDate="2011-05-04T23:58:04.737" UserId="2775" />
  <row Id="17661" PostId="10295" Text="I meant how the number of views and the vote count changes with time. When it starts out, #views = 0 and #votes = 0. Then it goes to &quot;1 views, 0 votes&quot;, then, say, to &quot;2 views, 0 votes&quot;, then to &quot;3 views, 1 vote&quot;, etc., until it reaches the present state of &quot;100 views, 10 votes&quot;." CreationDate="2011-05-05T00:48:22.710" UserId="3369" />
  <row Id="17662" PostId="10313" Text="Interesting - sounds like a relatively recent result that directly contradicts I think several sources I've come across over time that recommend using Fisher over $\chi^2$. Unfortunately I don't have access to that paper. What about the likelihood ratio test, and what about comparing one dataset against the combined dataset - when should I use these?" CreationDate="2011-05-05T01:01:05.347" UserId="1720" />
  <row Id="17663" PostId="10327" Text="No such thing as too many tail puns." CreationDate="2011-05-05T01:36:38.503" UserId="71" />
  <row Id="17665" PostId="10332" Text="@Will Plot $x \to pdf(\Gamma)(x+1)$ to see what's going on." CreationDate="2011-05-05T02:16:37.737" UserId="919" />
  <row Id="17666" PostId="10112" Text="From what I've read, &quot;Without the assumption of normality, uncorrelatedness of the disturbances is sufficient for the Kalman filter to yield optimal linear foreasts.&quot; The particle filter is more general though..." CreationDate="2011-05-05T02:18:52.230" UserId="1764" />
  <row Id="17667" PostId="10337" Text="$Var[X] \stackrel{d}{=} \mathbb{E}[X^2] - (\mathbb{E}[X])^2 \to Var[X^2] = \mathbb{E}[X^4] - (\mathbb{E}[X^2])^2 $ or perhaps even more confusingly $Var[X^2] = \mathbb{E}[X^4] - (Var[X^2] + (\mathbb{E}[X])^2)$" CreationDate="2011-05-05T04:18:50.253" UserId="2168" />
  <row Id="17668" PostId="10337" Score="1" Text="In general there is no readily available formula. You can use delta method to get the approximation: here is [the relevant question](http://stats.stackexchange.com/questions/5782/variance-of-a-function-of-one-random-variable/5790#5790)." CreationDate="2011-05-05T05:59:48.613" UserId="2116" />
  <row Id="17669" PostId="10338" Text="@user4472, please provide some context and more details, otherwise the question will be closed. Also see faq on how to ask questions." CreationDate="2011-05-05T06:01:48.673" UserId="2116" />
  <row Id="17671" PostId="10316" Text="The p-value (last column first table) is the chance that the obtained result or worse would be attained if the null hypothesis were true. The confidence interval is a/the region that will hold the true value in e.g. 95% of the times. If it does not hold the hypothesized true value, then there is at most 5% chance that we would get the obtained result or worse, if the hypothesis is true. So this would imply your p-value to be lower than 5%. There is a very close relation between p-values and confidence intervals (statistics 101). But in short: yes, the CI for 10% will include 1." CreationDate="2011-05-05T06:51:52.730" UserId="4257" />
  <row Id="17672" PostId="10329" Text="There's a package dedicated to statistics, see R project for statistical computing (http://www.r-project.org/)." CreationDate="2011-05-05T07:15:22.040" UserId="144" />
  <row Id="17674" PostId="10339" Text="Another similar example: if +1 and -1 are equally likely then the variance is 1, but the variance of the squares is 0; if 0 and 2 are equally likely then the variance is again 1 but the variance of the squares is 4.  " CreationDate="2011-05-05T07:43:19.937" UserId="2958" />
  <row Id="17675" PostId="9810" Text="Sorry for asking again, but I really can not find a way...Does anyone know how I can compare these groups? Do I simply have to disregard that two of the groups are paired? (this does not seem correct to me...)" CreationDate="2011-05-05T07:59:41.990" UserId="4229" />
  <row Id="17676" PostId="10338" Text="@mpiktas I asked @user4472 to post their question on this site after I received an email from them. @user4472 I've endeavoured to convert your question into something more reasonable, feel free to edit, if I have misconstrued." CreationDate="2011-05-05T08:04:57.953" UserId="183" />
  <row Id="17677" PostId="10327" Text="Liked the suggestion about the note. Thanks for your help." CreationDate="2011-05-05T08:26:23.160" UserId="4468" />
  <row Id="17678" PostId="10338" Text="@Jeromy Anglim, thanks for explanation, I've upvoted the question." CreationDate="2011-05-05T08:38:26.637" UserId="2116" />
  <row Id="17679" PostId="10241" Text="What do you mean? It's not a homework if it's what you're implying, it's a project part of my PhD." CreationDate="2011-05-05T08:55:19.907" UserId="3699" />
  <row Id="17680" PostId="10271" Text="@JMS: You are right, I added a trace plot. And yes, it is an univariate time series taking values in $0 \le x_t \le 5$. Scores above 1 are just very rare. Usually, high scores happen in short bursts but both are troubling. I will think about your idea, thanks!" CreationDate="2011-05-05T09:02:14.173" UserId="4446" />
  <row Id="17681" PostId="10344" Score="4" Text="...where practice is large $n$." CreationDate="2011-05-05T09:07:17.417" UserId="88" />
  <row Id="17682" PostId="10347" Text="@Lars What do you want to modify: the heatmap or the dendrogram?" CreationDate="2011-05-05T09:50:08.093" UserId="930" />
  <row Id="17683" PostId="10347" Text="I would like to modify the heatmap in such a way that my columns get clustered according to the pre-computed distance matrix " CreationDate="2011-05-05T09:56:12.297" UserId="4473" />
  <row Id="17684" PostId="10346" Text="@ayush Could you edit your question clarifying what you mean by &quot;certain values&quot; and &quot;event rate&quot;? Could you explain what is making the situation &quot;not very comfortable&quot;?" CreationDate="2011-05-05T09:59:58.540" UserId="183" />
  <row Id="17685" PostId="10347" Text="@Lars I don't see any way to have different clustering scheme (distance + method) for rows and columns, after a quick look at the code. Some suggestions were provided on this [related question](http://stats.stackexchange.com/questions/6890/plotting-a-heatmap-given-a-dendrogram-and-a-distance-matrix-in-r); otherwise, the simplest solution is to play with the code and add your own modifications. I can show you how to do it if you want." CreationDate="2011-05-05T10:03:20.800" UserId="930" />
  <row Id="17687" PostId="10347" Text="If you could help me with this I would be thrilled !" CreationDate="2011-05-05T10:07:11.223" UserId="4473" />
  <row Id="17688" PostId="10347" Text="Would it be easier to use the mixOmics package instead ? I have it installed, however the looks are not as nice as in the pheatmap package" CreationDate="2011-05-05T10:09:07.783" UserId="4473" />
  <row Id="17689" PostId="10346" Text="The values are the value that categorical variable is taking..Lets say when it takes value C most, 60% of the times, the event happened(in historical data)and when it took value A 50% of time event didnt happen. I am saying its not very comfortable situation to be in because generally a model is termed bad when only 1 variable is making it predict good %age of events. Any addition to this variable is improving the prediction power by decimal values. Hope that makes the question more clear " CreationDate="2011-05-05T10:14:50.900" UserId="1763" />
  <row Id="17690" PostId="10346" Text="@ayush Thanks. That does clarify things. I'll update your question to make this clear." CreationDate="2011-05-05T10:26:46.650" UserId="183" />
  <row Id="17691" PostId="10346" Text="@jeromy-- Thanks!!" CreationDate="2011-05-05T10:32:58.843" UserId="1763" />
  <row Id="17692" PostId="10348" Text="@stefen -- thanks..Yes, I have done this check. This information is absolutely available in my model. So that gives me my answer. But there is something more which I wanted people to ponder upon..Removing this variable and keeping just others giving me a miserable prediction is indeed an indicator that when I dont keep this variable, all others fail in the combined way to give me anything fruitful..Going into detail..this itself means to business that if you want to recognize event guys just look for this indicator and you have got most of 'em. what say ??" CreationDate="2011-05-05T10:44:01.143" UserId="1763" />
  <row Id="17693" PostId="10347" Text="@Lars `pheatmap` relies on `grid` plotting functionalities, which is the reason why it looks so nice. " CreationDate="2011-05-05T11:05:00.440" UserId="930" />
  <row Id="17694" PostId="10344" Text="@Nick: So does that mean that the unbiased estimator is n-1 *also* under normality?" CreationDate="2011-05-05T11:07:45.757" UserId="230" />
  <row Id="17695" PostId="10344" Text="See e.g. [wikipedia](http://en.wikipedia.org/wiki/Variance#Population_variance_and_sample_variance) for a proof that this is the case regardless of distribution. So, in short (again): yes." CreationDate="2011-05-05T11:19:39.813" UserId="4257" />
  <row Id="17696" PostId="10342" Score="1" Text="@Wilduck, Wooldridge in his [book](http://books.google.com/books?id=cdBPOJUP4VsC&amp;printsec=frontcover&amp;dq=wooldridge+panel+data&amp;hl=fr&amp;ei=6JHCTcW0K4_tOaPW6J0I&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CC0Q6AEwAA#v=onepage&amp;q&amp;f=false) gives a very detailed explanation of this. Look in to chapters 10 and 11. In short the key is correlation of lags of independent variables with the error term, not with dependend variable. If the independent variables are strictly exogenous then there is no bias. If they are not the bias is of order 1/T, assuming of course that contemporaneous model holds." CreationDate="2011-05-05T12:05:25.997" UserId="2116" />
  <row Id="17697" PostId="10348" Text="@ayush Yes, as far as I see from here (if the predictor/predictor information is always available). Does the meaning of the predictor does not indicate this ? 1. As I said, I recommend to talk to someone with domain language, if it is not instantly clear. 2. Although you hit 60%, you still miss 40%. So your precision is solid, but not excellent. Depends if this is enough for your business.  " CreationDate="2011-05-05T12:07:12.647" UserId="264" />
  <row Id="17698" PostId="10348" Text="@stefen..Thanks..Yes, I understand your point. Yes,I am hitting 60% with this and some more variables..As it seems right now, I would not be able to take it to more than 65 %." CreationDate="2011-05-05T12:29:57.370" UserId="1763" />
  <row Id="17699" PostId="10353" Score="5" Text="@Sam Could you specify what your question is? Is this about the relevance or validity of using repeated-measures ANOVA with appropriate test for the variance-covariance structure, alternative ways for analyzing RCBD, or so?" CreationDate="2011-05-05T12:32:36.773" UserId="930" />
  <row Id="17700" PostId="10352" Text="@Nick -- Thank you !!! I would take note of what you wrote. I see predicting customer no of dependents as a dependent variable and taking out the list of predictors as one way." CreationDate="2011-05-05T12:39:45.363" UserId="1763" />
  <row Id="17701" PostId="10313" Text="The main result of the paper:`the test size of FET [when testing at the 0.05 level] was less than 0.035 for nearly all sample sizes before 50 and did not approach 0.05 even for sample sizes over 100'  I don't have a ready reference on the LR test but I expect it would perform similar to Pearson's test, and be better than FET.  Please elaborate on the original global purpose of the analysis." CreationDate="2011-05-05T13:05:43.360" UserId="4253" />
  <row Id="17702" PostId="10350" Text="In my opinion, if the variable is significant (p&lt;.0001) without imputation, and you have enough data, that is reason enough to leave it alone.  What are the confidence intervals telling you?" CreationDate="2011-05-05T13:30:45.720" UserId="3489" />
  <row Id="17703" PostId="10355" Score="1" Text="Do the arithmetic: (1) 100%-97% = 3% of 30,000 is 900.  Therefore you have to check 30,000 - 900 = 29,100 records to be *certain* of finding *at least* 97% of all error records.  (2) Since 6500/30000 is less than one-quarter of all records, then *if those you checked are representative of the whole group,* you have probably found only about a quarter of the errors, not 97% of them.  **Conclusions**: (1) to make significant progress you need to specify an acceptable chance of not finding 97% of the errors and (2) you need to characterize how you selected the 6500 for checking." CreationDate="2011-05-05T13:36:13.147" UserId="919" />
  <row Id="17704" PostId="10355" Text="@whuber - the 6500 were selected based on a descending financial value (i.e. the 6500 most valuable records) - what would be the best way to determine how representative of the data this is? I would like to be 99% confident I had found all of the 97% of cases if possble" CreationDate="2011-05-05T13:51:32.917" UserId="4475" />
  <row Id="17705" PostId="10355" Text="@Davin If it's possible the error rate varies with financial value, then you would be mistaken to project an overall error rate from the error rate you have observed in your sample.  For this reason usually one takes a *truly random* sample (controlled by random number generation) and extrapolates from it to the full set of records.  Even so, you're still going to have to check around 97% of *all* records to have any confidence of having found at least 97% of all errors.  *If* error rate varies with another variable--like financial value--you might do a little better than that." CreationDate="2011-05-05T13:59:29.070" UserId="919" />
  <row Id="17706" PostId="10336" Text="That is very clear. Thank you!" CreationDate="2011-05-05T14:07:25.127" UserId="3561" />
  <row Id="17707" PostId="10349" Text="I will try this at once and let you know how it went. Thanks for your huge effort- greatly appreciate it ." CreationDate="2011-05-05T14:15:55.497" UserId="4473" />
  <row Id="17708" PostId="10349" Text="+1 awesome answer and great reference for how to bend source code to suit your needs. This type of thing comes up on SO all the time, I wish there was a way to have this available in the R-tag on SO as a FAQ." CreationDate="2011-05-05T14:21:49.800" UserId="696" />
  <row Id="17709" PostId="10350" Score="1" Text="If you have 2 lakh (i.e. 2 hundred thousand) cases where the number of dependents is unknown, out of about 1 million, then you may be  able to include the fact that you don't know the number of dependents as a factor in the analysis.  " CreationDate="2011-05-05T14:21:59.207" UserId="2958" />
  <row Id="17710" PostId="10355" Text="@Whuber - I've added the data to the question - including the financial value and whether or not it has been checked and if so whether an error was found - any help in working out how to stratify / determine how the value affects error rate would be greatly appreciated!" CreationDate="2011-05-05T14:42:34.947" UserId="4475" />
  <row Id="17711" PostId="10357" Text="Actually, I don't think this is true for logistic regression: the odds-ratio parameter (which performs the discrimination) is invariant to stratification on the response variable. This is why it can be used for case-control studies." CreationDate="2011-05-05T15:05:11.153" UserId="495" />
  <row Id="17712" PostId="10362" Text="But is this pertinent to logistic regression? We don't need to describe the statistical distribution of either class, just the relative odds ratios (see my comment to @Henry)." CreationDate="2011-05-05T15:50:22.720" UserId="495" />
  <row Id="17713" PostId="10362" Text="In a univariate example perhaps, but if you have more than one explanatory variable then you need information about the distribution of patterns to correctly orient the &quot;discriminant&quot;." CreationDate="2011-05-05T16:02:13.777" UserId="887" />
  <row Id="17714" PostId="10361" Text="I think that the problem is that P(A_{+} B-{+} | c) should not be equal to P(A_{+} | c) x P(A_{+} | c); when they are equal they are called conditionally independent. Conditional independence is not the same as independence. " CreationDate="2011-05-05T16:08:46.037" UserId="3805" />
  <row Id="17715" PostId="10332" Text="Thanks, I plotted the PDF, and it looks like the inverse of the unit exponential distribution.  So if I take the negative of $\Gamma(2,1)(x+1)$ and add a constant (like 3), it should serve as an envelope for the unit exponential?" CreationDate="2011-05-05T16:10:59.637" UserId="2984" />
  <row Id="17716" PostId="10361" Text="Post this as an answer." CreationDate="2011-05-05T16:20:34.587" UserId="1146" />
  <row Id="17717" PostId="10357" Text="@Simon: I don't disagree with your comment on the odds ratio, but I have seen people fail to take this back to consequences for the population correctly after stratified sampling when they had done so in other cases.  For example, if you find that people for which factor A is true have twice the odds of being &quot;bad&quot; as those without factor A, this should not change with stratified sampling, but if you want to know what proportion of the population will be unnecessarily affected if you target those with factor A, then you need to carefully weight the information from your samples." CreationDate="2011-05-05T16:22:09.087" UserId="2958" />
  <row Id="17718" PostId="10361" Text="$P(c|B_{\text{positive}}) = 1$ is certainly correct, since you say  &quot;B ...  has the advantage of never giving a false positive&quot;.  " CreationDate="2011-05-05T16:37:38.953" UserId="2958" />
  <row Id="17719" PostId="10361" Text="@schenectady could you elaborate on that and maybe post it as an answer?" CreationDate="2011-05-05T17:00:27.330" UserId="2261" />
  <row Id="17720" PostId="10361" Text="@Henry yeah after starring at it for like 20 minutes I realize that you are right. Sigh, I feel really dumb now." CreationDate="2011-05-05T17:02:20.353" UserId="2261" />
  <row Id="17721" PostId="10357" Text="@Henry: sorry, that wasn't the bit I disagreed with. It was the first bit: a consequence of the invariance is that once you have the data, there is no point in stratifying, you're simply throwing away data. (the story is different when it comes to the cost of collecting data, hence the existence of case-control studies)." CreationDate="2011-05-05T17:04:54.607" UserId="495" />
  <row Id="17722" PostId="10362" Text="Wouldn't it be better to apply the multiplicative adjustment in log odds space before the logistic function is applied? " CreationDate="2011-05-05T17:10:48.767" UserId="2965" />
  <row Id="17723" PostId="10362" Text="IIRC, the assymptitically optimal adjustment is to multiply by the ratio of operational to training set class frequencies, this is based on Bayes rule, so it is applied to the probabilities rather than the log-odds ratio.  However, as we are just trying to correct for a mathematically intractable defficiency in the estimation, it probably doesn't matter too much how the adjustment is made, it is really just a &quot;fudge-factor&quot;." CreationDate="2011-05-05T17:20:42.177" UserId="887" />
  <row Id="17724" PostId="10332" Text="Thanks for the plot! I understand it now." CreationDate="2011-05-05T17:38:35.240" UserId="2984" />
  <row Id="17725" PostId="10361" Score="1" Text="+1 for a properly posted homework question." CreationDate="2011-05-05T17:51:40.883" UserId="582" />
  <row Id="17726" PostId="10329" Text="Yes, R is great for statistics. It tends to specialize in mostly numeric computations however. I'm interested in what we can do to assist researchers in symbolic manipulations. " CreationDate="2011-05-05T18:07:23.007" UserId="3830" />
  <row Id="17727" PostId="10359" Text="@Emre, the answer heavily depends on dependence structure. Independence property is unique, dependence is not. For the continuous distribution processes the most easiest to generate is linear processes (ARIMA) for discrete, I guess Markov." CreationDate="2011-05-05T18:09:43.310" UserId="2116" />
  <row Id="17728" PostId="10359" Text="Feel free to propose the dependence structure, as long as it allows you to set the correlation, mutual information, or something similar. I need to be able to dial in the level of dependence." CreationDate="2011-05-05T18:14:44.150" UserId="4479" />
  <row Id="17729" PostId="10359" Text="@Emre, by level of dependence you mean what?" CreationDate="2011-05-05T18:29:47.997" UserId="2116" />
  <row Id="17730" PostId="10359" Text="@mpiktas: To quote myself: &quot;The dependence parameter can be the correlation, the mutual information...&quot; between the latest sample and the preceding one(s)." CreationDate="2011-05-05T18:37:26.697" UserId="4479" />
  <row Id="17731" PostId="10357" Text="@Simon: When you say &quot;case-control study&quot;, do you mean that that you originally plan to take a sample of &quot;bad&quot; cases and a sample of &quot;good&quot; cases? With a higher proportion of &quot;bad&quot; cases than the small proportion in the population? If so, that is what I intended by &quot;stratified sampling, oversampling bad cases&quot; in my answer. " CreationDate="2011-05-05T18:51:49.270" UserId="2958" />
  <row Id="17732" PostId="10354" Text="+1 for the calling out the issue of large samples and Type I error" CreationDate="2011-05-05T19:43:27.397" UserId="1080" />
  <row Id="17733" PostId="10368" Text="@Pete As you said you could add quadratic,cubic,quartic ....and mixture terms but that would just be just nonsense. There is nonsense and there is non-sensical nonsense bot the most non-sensicle nonsense is &quot;statistical nonsense &quot;." CreationDate="2011-05-05T20:59:44.377" UserId="3382" />
  <row Id="17734" PostId="10331" Text="One small thing, rather than constantly resizing your vector, start off with: `x = numeric(100000)` This will be much faster." CreationDate="2011-05-05T21:11:17.380" UserId="8" />
  <row Id="17736" PostId="10369" Text="I removed the &quot;mixture&quot; tag because this is a question about a sum (or, equivalently, about correlated Normal variables), not about a mixture of them." CreationDate="2011-05-05T21:23:21.703" UserId="919" />
  <row Id="17737" PostId="10365" Text="Given your cost function, why might you expect to have a more concise way?  Personally I might write $F^{-1}(0.025|k, \theta)$ rather than $Q_{0.025|\text{parms}}$, but so long as you are clear what these mean, there is little difference.  " CreationDate="2011-05-05T22:09:00.527" UserId="2958" />
  <row Id="17738" PostId="10365" Text="@Henry The equation that I wrote is sufficiently concise, but I just 'made it up' and am not sure if there was a better or more accepted approach; where does your use of 'F' come from?" CreationDate="2011-05-05T22:11:39.317" UserId="1381" />
  <row Id="17739" PostId="10365" Score="1" Text="$F^{-1}$ is simply the inverse of the cumulative distribution function." CreationDate="2011-05-05T22:28:03.083" UserId="2958" />
  <row Id="17740" PostId="10368" Score="1" Text="@IrishStat it is *not* generally nonsense to add mixture and higher order terms; only bad when it is done without restraint and without regard to theory" CreationDate="2011-05-05T23:07:36.340" UserId="2260" />
  <row Id="17741" PostId="10366" Text="Not directly an answer to the question, but you might be interested in the pgfplots LaTeX package.  It allows you to embed plot code directly in a LaTeX document.  Thus, it can cope with arbitrarily complex LaTeX math expressions, and if you're going to wind up putting your gnuplot plots into a LaTeX document pgfplots would be one less step." CreationDate="2011-05-05T23:18:09.737" UserId="2975" />
  <row Id="17742" PostId="10368" Score="1" Text="@Pete . Correct! The absence of a pre-existing theory makes it silly." CreationDate="2011-05-06T00:32:40.887" UserId="3382" />
  <row Id="17744" PostId="10363" Text="@OP: IrishStat's comments below remind me that without some knowledge of how your independent variables are related to each other and/or the dependent variable, in theory, leaves you &quot;up the creek without a paddle&quot;.  For example, if `FF` was &quot;combustion yield&quot; and `AA` was amount of fuel, and `BB` was the amount of oxygen, you would look for an interacting term of `AA` and `BB` " CreationDate="2011-05-06T01:47:35.593" UserId="2260" />
  <row Id="17745" PostId="10370" Text="Is the categorical variable unordered? Approximately how many cases do you have? What is the frequency distribution across the categorical variable?" CreationDate="2011-05-06T03:32:08.343" UserId="183" />
  <row Id="17747" PostId="10333" Text="Excellent point. What could be added to a system that already expressed linear algebra well? Or is a solid expression of matrices sufficient? " CreationDate="2011-05-06T04:05:57.663" UserId="3830" />
  <row Id="17748" PostId="10385" Text="how about `stepAIC()` in package MASS?" CreationDate="2011-05-06T04:09:08.257" UserId="696" />
  <row Id="17749" PostId="10377" Text="Org-mode is incredibly powerful and flexible." CreationDate="2011-05-06T04:19:53.980" UserId="1381" />
  <row Id="17750" PostId="10329" Text="It's not evident that any of these symbolic systems need &quot;statistical constructs&quot; added to them.  *Mathematica* has done a lot of that in release 8, but it's still not likely to become a popular platform for applied statistics or students.  One reason is that good stats software supports the non-mathematical labor much better than the math software does: data input, management, transformation, querying, and documentation.  The stats software usually also supports a huge gallery of data visualization methods." CreationDate="2011-05-06T04:39:00.563" UserId="919" />
  <row Id="17751" PostId="10333" Text="@MR Rather than ask what *could be* added to these systems, think about why you're doing this.  For whom?  What uses to you want to support?  What are the compelling reasons (if any) to be adding statistical procedures to a symbolic system?  In other words, begin with a good functional specification." CreationDate="2011-05-06T04:42:54.477" UserId="919" />
  <row Id="17752" PostId="10379" Text="Hmm you have some very good points ... my data also contains a lot of outliers and is subject to significant variation on events like Christmas or New Year. But what do you mean by 'autoregressive or adaptive model structure'? I'll give ARMAX a shot though, thanks! &#xA;&#xA;" CreationDate="2011-05-06T04:54:27.290" UserId="4470" />
  <row Id="17753" PostId="10386" Text="Are the real data also sinusoidal? Anyway, I would look into using FFT... maybe finding the max frequency with a sliding time window." CreationDate="2011-05-06T05:43:05.433" UserId="582" />
  <row Id="17754" PostId="10313" Text="The application/analysis is just the binomial dataset comparison found in the Wikipedia articles on LR test, $\chi^2$ test, and Fisher test under their Examples sections - they all sound like the same problem (test the hypothesis that two samples have the same binomial distribution)." CreationDate="2011-05-06T06:03:54.107" UserId="1720" />
  <row Id="17755" PostId="10386" Text="@nico Yep, updated question to say so." CreationDate="2011-05-06T06:07:32.130" UserId="179" />
  <row Id="17756" PostId="10377" Text="I think you should have a separate tool for bibliography, as it is a very different task from note taking, TODO etc. My choice would go on Zotero, as it's FLOSS, very powerful, has a much easier workflow than some commercial tools (such as EndNote) and is compatible with both OpenOffice.org and Microsoft Word." CreationDate="2011-05-06T06:18:35.603" UserId="582" />
  <row Id="17757" PostId="10390" Text="so it means that we need to correct the distance estimated from  standard normal distribution hypotheses for non-standard data?" CreationDate="2011-05-06T07:32:37.663" UserId="4286" />
  <row Id="17758" PostId="10377" Text="You also might find [this question](http://stats.stackexchange.com/questions/2910/how-to-efficiently-manage-a-statistical-analysis-project) interesting." CreationDate="2011-05-06T07:33:55.030" UserId="264" />
  <row Id="17760" PostId="10382" Text="Maybe look at factor analysis or principal components analysis?" CreationDate="2011-05-06T08:28:03.920" UserId="179" />
  <row Id="17761" PostId="10388" Score="1" Text="What about changing of the starting values? Supply to `init` similar values to the output of SAS and see what happens." CreationDate="2011-05-06T08:31:50.037" UserId="2116" />
  <row Id="17762" PostId="10359" Text="If you have been given a distribution over sequences, how would you adjust the correlation between samples from it, since it is part of the distribution and thus not subject to change (since it's given)? Or do you mean distribution of the values at timesteps?" CreationDate="2011-05-06T08:35:54.853" UserId="2860" />
  <row Id="17763" PostId="10388" Text="@mpiktas: Thank you very much !!! It works ! I did not realize that the default initial value is zero for all variables in R. On the other hand, I have still to investigate what are the initial values in SAS. Again, Thank you very much !!!" CreationDate="2011-05-06T09:08:23.010" UserId="3019" />
  <row Id="17764" PostId="10385" Text="@chase thanks. I had thought about `polr`, but I wanted to use `clm` and polr is slow with my data size (and so is `clm`)" CreationDate="2011-05-06T09:09:44.853" UserId="1307" />
  <row Id="17765" PostId="10357" Text="@Henry: Yes, that is precisely what I meant as well. The question seemed to indicate that they already have data, hence there would be no point in stratifying." CreationDate="2011-05-06T09:18:46.250" UserId="495" />
  <row Id="17766" PostId="10362" Text="@Dikran: I don't understand what you mean about correctly orienting the &quot;discriminant&quot;. Even in the multivariate case, logistic regression is still just computing relative odds ratios." CreationDate="2011-05-06T09:21:35.937" UserId="495" />
  <row Id="17767" PostId="10388" Text="@ocram, glad to be helpful :) Note this is a dirty cheat, since you use other software package to provide starting values, which is not feasible in the long run. But when debugging this is a first thing to try." CreationDate="2011-05-06T09:23:43.797" UserId="2116" />
  <row Id="17768" PostId="10367" Text="@IrishStats What is &quot;G.E.P. Box&quot;?" CreationDate="2011-05-06T09:40:54.470" UserId="914" />
  <row Id="17769" PostId="10396" Text="@ayush, I suspect that you need answers in R, if so, please add tag r. More context on the problem, sample data, would go long way in helping to get appropriate answer. " CreationDate="2011-05-06T09:43:19.153" UserId="2116" />
  <row Id="17770" PostId="10395" Text="Looks to be a very good start, thank you.  Not sure if its going to be too computationally taxing but I guess I'll have to try it out and see!" CreationDate="2011-05-06T09:43:50.000" UserId="179" />
  <row Id="17771" PostId="10389" Text="@K-1, I assume that you asking about the maximum of absolute values of a similar **sequence** of random variables $\eta_j$ the difference being that $\xi_j\sim N(0,1)$ and $\eta_j\sim N(0,1)$? If so please correct the question. Now it asks about the absolute maximum of a **single** normal random variable. The answer to that question is $\infty$." CreationDate="2011-05-06T09:49:30.457" UserId="2116" />
  <row Id="17772" PostId="10396" Text="@mpiktas -- sorry,I need answers in SAS..Basically I had too many missing values in customer number of dependents so I am left with making a mini model with few relevant predictors to predict the number of dependents..These number of dependents can be 1,2,3 or 4. Therefore I need a regression method which has Dependent variable with more than 2 categories which happens in normal logistic regression. Hope that clarifies." CreationDate="2011-05-06T09:50:30.420" UserId="1763" />
  <row Id="17773" PostId="10394" Text="@K-1, please add more context. Reference where you found this would be very helpful." CreationDate="2011-05-06T09:51:19.987" UserId="2116" />
  <row Id="17774" PostId="10396" Text="@ayush, added sas tag, sorry I myself never used sas, so I cannot help, but maybe someone can give some reasonable answer." CreationDate="2011-05-06T09:52:19.283" UserId="2116" />
  <row Id="17775" PostId="10396" Text="anybody with sas experience please." CreationDate="2011-05-06T09:58:07.107" UserId="1763" />
  <row Id="17776" PostId="10395" Text="@fmark: the examples I posted computed pratically istantaneously." CreationDate="2011-05-06T10:03:23.500" UserId="582" />
  <row Id="17777" PostId="10367" Text="IrishStat: The data is not chronological. So the ordering of the observations is not of importance. The shift in the mean for observations #11-25 is merely a side-effect on how I retrieved the dataset." CreationDate="2011-05-06T10:07:21.997" UserId="914" />
  <row Id="17778" PostId="10368" Text="@Pete: What R^2 do you get for your model?" CreationDate="2011-05-06T10:08:04.350" UserId="914" />
  <row Id="17779" PostId="10362" Text="@Simon Byrne I think I see what you mean, I am too used to non-parametric methods where cutting down the size of the training set is useful!  For logistic regression it may be O.K. to just use the full dataset and then apply the fudge-factor to compensate for the estimation problem (as the multiplicative fudge factor is equivalent to differential weighting of the patterns from the positive and negative classes)." CreationDate="2011-05-06T10:10:20.500" UserId="887" />
  <row Id="17780" PostId="10362" Text="@Dikran: My point was that, if his data are already in the right proportions of bad-good, then you shouldn't need a fudge factor (I've given my own answer now)." CreationDate="2011-05-06T10:17:50.723" UserId="495" />
  <row Id="17781" PostId="10397" Text="This is only true though if you have enough data to adequately represent the minority class (which is usually the problem where there is a large class imbalance - the problem is the imbalance per se, but that you don't have enough samples from the minority class).  In that case cross-validation based differential weighting of the positive and negative classes, adjustment of alpha, or multiplicative adjustment (all equivalent IIRC) would be a good idea to boost the minority class.  Stratifying the data would be a cheap and cheerful equivalent." CreationDate="2011-05-06T10:21:14.613" UserId="887" />
  <row Id="17782" PostId="10397" Text="@Dikran: I don't see how any of this fails to be true in that case: you don't have to represent the minority class, you just need to represent the relative difference to the majority. If you don't have enough data for that, then throwing away observations from the majority isn't going to help." CreationDate="2011-05-06T10:29:49.217" UserId="495" />
  <row Id="17783" PostId="10395" Text="@nico I will be doing this on an embedded system with limited RAM and a slow processor, and coding it myself in C, so what is instantaneous on your desktop using an optimised implementation may not be a good indication :)" CreationDate="2011-05-06T10:33:39.227" UserId="179" />
  <row Id="17784" PostId="10367" Text="@knorv http://en.wikipedia.org/wiki/George_E._P._Box is a statistician/chemist who influenced many to study time series. He was one of my instructors in the subject." CreationDate="2011-05-06T10:34:39.353" UserId="3382" />
  <row Id="17785" PostId="10367" Text="@knorv: a side-effect ? Very interesting ! I can't wait to find out more about this." CreationDate="2011-05-06T10:36:48.710" UserId="3382" />
  <row Id="17786" PostId="10397" Text="@Simon, I have agreed with you that stratification isn't a good idea unless you don't want to adjust alpha, reweight the patterns or adjust the output of the model.  As I said the problem with class imbalance is not the imbalance per se, but that there is insufficient data for the minority class to adequately define the &quot;relative difference to the majority&quot;.  When that happens, on average it biases the output towards the majority class, and hence doing something to compensate for that bias is helpful.  Stratification is one way of doing that, but not nearly the best." CreationDate="2011-05-06T10:38:30.673" UserId="887" />
  <row Id="17787" PostId="10397" Text="@Simon, BTW thanks for the Prentice and Pyke reference, it looks useful." CreationDate="2011-05-06T10:39:11.470" UserId="887" />
  <row Id="17788" PostId="10203" Text="oh yes,this helped :) " CreationDate="2011-05-06T10:40:03.137" UserId="4402" />
  <row Id="17789" PostId="10379" Text="@Wern An adaptive/autoregressive component exists when the &quot;predictors&quot; includel either memory (previous values) of the endogenous (Y) series or exogenous series OR lagged values of the error term. It is important to identify and model , with possible  lead,contemporaneous and lag effects around EACH of the Holidays. Furthermore there may be assignable cause to specific days-of-the-month , dependent on the day-of-the-week. For example the 5th of the month may be important GIVEN that it was not a weekend. We have been able to successfully model/automatically identify this structure." CreationDate="2011-05-06T10:41:54.620" UserId="3382" />
  <row Id="17790" PostId="10362" Text="@Simon, as I said, that is only true if you have enough data for the minority class; when that is not the case, the model tends to be unduly biased towards the majority class.  In machine learning it is called the &quot;class imbalance problem&quot; and there has been quite a lot of work on it." CreationDate="2011-05-06T10:44:42.357" UserId="887" />
  <row Id="17791" PostId="10397" Text="@Dikran: It may well be biased for other models, but NOT for logistic regression, that's the whole point." CreationDate="2011-05-06T10:47:55.097" UserId="495" />
  <row Id="17792" PostId="9982" Text="Thanks, I have not ever worked with python. I will try to find an R solution" CreationDate="2011-05-06T10:53:41.863" UserId="4147" />
  <row Id="17793" PostId="10397" Text="@Simon, That is not the case, try doing some Monte Carlo simulation studies (as I have - I am working on a paper on this topic)." CreationDate="2011-05-06T10:57:08.433" UserId="887" />
  <row Id="17794" PostId="10362" Text="@Dikran: I've had a quick look, but are you aware of any references on class imbalance that mention logistic regression? (most ones i found were related to decision trees or SVM's). It's related to something I'm working on, so I'm keen to find out." CreationDate="2011-05-06T11:01:48.190" UserId="495" />
  <row Id="17795" PostId="10238" Text="@whuber Thank for this. I agree with you that mine is a discrete distribution but R don't know this and, for that, it is not correct the application of ks.test (for 1 sample) in this situation. I don't sure to undestand the second part.But my question was different and, perhaps, I don't explain very well. In others worlds:" CreationDate="2011-05-06T11:17:07.243" UserId="4345" />
  <row Id="17796" PostId="10362" Text="@Simon Not specifically, however the estimation problem looks to be pretty universal, it certainly affects simple Gaussian classifiers, which also have the same assymptotic/in expectation invariance properties, so I don't think LR is an exception." CreationDate="2011-05-06T11:19:03.493" UserId="887" />
  <row Id="17797" PostId="10387" Text="@Thomas: I suggest to change &quot;real values&quot; in the question header to &quot;scores&quot; or &quot;confidences&quot;, since &quot;real values&quot; is a rather arbitrary term (as explained in my response)." CreationDate="2011-05-06T11:21:04.587" UserId="264" />
  <row Id="17798" PostId="10355" Text="@whuber: i don't see the logic behind your use of the word &quot;certain&quot; in your first comment. We don't know that the errors are going to be in the exact same proportion (or a higher one) among the first 97% of records as among the last 3%." CreationDate="2011-05-06T11:21:26.613" UserId="2669" />
  <row Id="17799" PostId="10377" Text="what OS are you running?" CreationDate="2011-05-06T11:26:37.360" UserId="253" />
  <row Id="17800" PostId="10400" Text="@tal -- Thanks . With decision trees you get the rule based classification which gives you the probable indicators of the event. I already have indicators with me. I want a sas syntax so that I can use those classifiers for 4 categories and score my dataset. " CreationDate="2011-05-06T11:37:02.277" UserId="1763" />
  <row Id="17801" PostId="10238" Text="... In the first 5 hours I see 15 birds. In the first hour, nothing. In the second and third only 1, in the fourth 9 birds and in the fifth 4 birds. Then the frequency of the birds is (0,1,1,9,4) and the expected is (3,3,3,3,3). In the other question suggested the others samples: obs=c(2,3,4,4,4,4,4,4,4,4,4,5,5,5,5): number 1 there is not because in the first hour no birds. Number 2 and 3 only one time because in the second and third hour there is only one bird...." CreationDate="2011-05-06T11:38:26.060" UserId="4345" />
  <row Id="17802" PostId="10386" Score="1" Text="For the constant, known frequency case, it probably still breaks down further into whether you assume coherent detection or not (i.e., do you also know the *phase*)? At any rate, something like the empirical signal power should be close to a sufficient statistic when integrated over one period. Have you looked in, e.g., H. V. Poor, *[An Introduction to Signal Detection and Estimation](http://www.amazon.com/Introduction-Detection-Estimation-Electrical-Engineering/dp/0387941738)*, 2nd. ed., Springer? It almost surely does the entire development for *very* similar problems." CreationDate="2011-05-06T11:45:40.800" UserId="2970" />
  <row Id="17803" PostId="10238" Text="@whuber Then I need to know (and I ask) if there is a way for use the ks.test for one sample and get the same result. R know the uniform discrete distribution?..... Thank" CreationDate="2011-05-06T11:54:54.607" UserId="4345" />
  <row Id="17804" PostId="10386" Text="To follow up, in the Poor book, see the example on page 166. Of course, that particular example can be viewed in a more (statistically) &quot;traditional&quot; regression context." CreationDate="2011-05-06T11:55:28.157" UserId="2970" />
  <row Id="17805" PostId="10395" Text="@fmark: sure, but at least you know it's not slow on a desktop!!! :D Anyway, I'm sure there's plenty of fast implementation of FFT (it's called &quot;fast&quot; Fourier transform for a reason). I never implemented one, though, so I cannot help in this case." CreationDate="2011-05-06T11:58:39.467" UserId="582" />
  <row Id="17806" PostId="10388" Text="@mpiktas Now, *please*, put this as an answer so that @ocram can accept it :-)" CreationDate="2011-05-06T12:02:44.533" UserId="930" />
  <row Id="17807" PostId="10394" Score="3" Text="**My guess**: If the sample follows a Gaussian distribution, then the *[asymptotic relative efficiency](http://en.wikipedia.org/wiki/Efficiency_(statistics))* of the robust estimator in 95%." CreationDate="2011-05-06T12:08:38.043" UserId="2970" />
  <row Id="17808" PostId="10389" Score="4" Text="I don't think that is an equality. I think that is an asymptotic equivalence. It definitely doesn't hold for $n = 1$." CreationDate="2011-05-06T12:12:58.103" UserId="2970" />
  <row Id="17809" PostId="10395" Text="@nico Absolutely, I'll definitely give it a try :)" CreationDate="2011-05-06T12:18:47.927" UserId="179" />
  <row Id="17810" PostId="10392" Text="The modulus got dropped in your answer. Including it destroys the linearity, but not the scale equivariance." CreationDate="2011-05-06T12:23:39.673" UserId="2970" />
  <row Id="17811" PostId="10376" Text="@Henry +1 for Baltic letter, how do you know it? Note that `ū` has a smaller bar compared to $\bar{u}$, so the end result will depend on the font, it might not look good." CreationDate="2011-05-06T12:25:08.360" UserId="2116" />
  <row Id="17812" PostId="10355" Text="@rolando2 You are correct: I should have compute the 3% on the basis of *the number of errors.*  That changes the value of 29,100 to an (unknown) higher value, perhaps around 29,800 (if the initial results are representative)." CreationDate="2011-05-06T12:31:58.237" UserId="919" />
  <row Id="17813" PostId="10368" Text="@knorv: I don't quite remember but it was &gt; 0.90.  When plotted about the regression line the points appeared to have a little bit of an &quot;S&quot;/cubic shape so I'm guessing the function &quot;f&quot; was a mathematical creation where someone typed 100A + 50(B+C) + higher order terms involving D &amp; E. " CreationDate="2011-05-06T12:39:00.743" UserId="2260" />
  <row Id="17814" PostId="10397" Text="@Dikran: I ran some simulations, but splitting and averaging does worse. R code &amp; results here: http://www.statslab.cam.ac.uk/~spjb4/logistic.r" CreationDate="2011-05-06T12:39:48.743" UserId="495" />
  <row Id="17815" PostId="10387" Text="I think I should keep &quot;real values&quot; because I might not have had the question had the documentation used less arbitrary terms." CreationDate="2011-05-06T12:57:32.860" UserId="3874" />
  <row Id="17816" PostId="10396" Score="1" Text="@ayush When you say &quot;dependents&quot;, do you mean like &quot;this person has five dependent children&quot;.  If so, what happens if someone has 0 or 5 dependents?" CreationDate="2011-05-06T13:03:39.893" UserId="183" />
  <row Id="17818" PostId="10399" Score="1" Text="With an ordered outcome you porobaly don't want to do nominal regression." CreationDate="2011-05-06T13:14:31.847" UserId="279" />
  <row Id="17819" PostId="10380" Text="It seems to me that the SE values computed in the first paragraph must be the standard errors of the logarithm of the odds ratio, not the standard errors of the odds ratio itself." CreationDate="2011-05-06T13:35:47.533" UserId="25" />
  <row Id="17820" PostId="10389" Text="It seems that something like $E(\max_j |\xi_j|) = \sqrt{2 \ln n} + C_n$ is correct with $C_n \sim -.3...$ as $n \to \infty$." CreationDate="2011-05-06T13:45:20.233" UserId="4062" />
  <row Id="17822" PostId="10392" Text="@cardinal: Thanks - it was only half visible in the question, so I read it as a conditional sign." CreationDate="2011-05-06T13:48:15.007" UserId="2958" />
  <row Id="17823" PostId="10387" Text="@Thomas good point" CreationDate="2011-05-06T14:31:43.447" UserId="264" />
  <row Id="17824" PostId="10359" Text="@bayer: The latter; the marginal." CreationDate="2011-05-06T14:41:45.433" UserId="4479" />
  <row Id="17825" PostId="10376" Text="@mpiktas: [Wikipedia](http://en.wikipedia.org/wiki/Macron) lists ū for Latvian and Lithuanian." CreationDate="2011-05-06T14:42:55.263" UserId="2958" />
  <row Id="17827" PostId="10370" Text="The levels are not ordered. I have around 10,000 observations. The frequency distribution is as follows: level A appears in around 11% of the observations. Level B appears in 8%. Level c appears in 5%. About 15 of these levels cover 50% of the observations in the dataset." CreationDate="2011-05-06T14:56:02.507" UserId="616" />
  <row Id="17828" PostId="10413" Text="My data is related to protein interaction data. We can talk about it more in detail (privately) if you are interested. I will look into MCL—thanks!" CreationDate="2011-05-06T15:13:50.583" UserId="3561" />
  <row Id="17829" PostId="10407" Score="1" Text="With large `n` the independent Poisson approximation is probably fine. Did you try simulation studies of how well the formula is working?" CreationDate="2011-05-06T15:33:20.120" UserId="279" />
  <row Id="17830" PostId="10413" Text="Nice simulation on your website" CreationDate="2011-05-06T16:06:59.480" UserId="3362" />
  <row Id="17831" PostId="10367" Text="@IrishStat: I meant that I just happened to retrieve the records in a certain order (think `ORDER BY`). The rows have no inherent special order. So you can safely re-arrange them without losing any information. Sorry if I confused you :-)" CreationDate="2011-05-06T16:35:13.727" UserId="914" />
  <row Id="17832" PostId="10368" Text="@Pete: Ah, ok! Given the small error term in the dataset an R^2 of 0.99 is possible in this case. So keep searching to get the bonus point! :-)" CreationDate="2011-05-06T16:38:10.840" UserId="914" />
  <row Id="17833" PostId="10363" Text="@Pete: Interacting terms are absolutely possible. I hope I didn't rule out that by framing my question incorrectly." CreationDate="2011-05-06T16:39:43.537" UserId="914" />
  <row Id="17834" PostId="10424" Score="3" Text="Why not use robust, non-parametric summaries like median, IQR, MAD, letter statistics, and so on?  They are easy to compute, easy to understand, are defined for all distributions (power laws might not even have SDs), resist the effects of outliers, and often are more efficient at estimating distributions when parametric assumptions (like Gaussianity) break down." CreationDate="2011-05-06T17:19:37.903" UserId="919" />
  <row Id="17835" PostId="10424" Text="I'll have to read up before I can properly answer that question, but I am looking at confidence intervals because I would like to create a running model of what is normal and what kind of traffic is significantly interesting. I feel like a simple expected range like &quot;10,000 to 20,000 pageviews&quot; would be really easy to digest." CreationDate="2011-05-06T17:34:12.327" UserId="3198" />
  <row Id="17836" PostId="10427" Text="*omitted because of collinearity" CreationDate="2011-05-06T17:44:21.417" UserId="4394" />
  <row Id="17837" PostId="10424" Score="3" Text="A confidence interval would not even answer the question of what trafic is &quot;normal&quot;. You need a prediction interval for that. And the use of percentiles is the most robust way of doing that, especially if you have loads of data." CreationDate="2011-05-06T17:55:18.057" UserId="279" />
  <row Id="17838" PostId="10392" Text="@Henry, (+1) No problem." CreationDate="2011-05-06T18:00:23.123" UserId="2970" />
  <row Id="17839" PostId="10389" Text="@Hans, Hmm. Can you explain? Leting $(X_n)$ be a sequence of iid standard normals and $b_n = \sqrt{2\log n}$, it is well-known that $b_n(M_n - b_n) \to G$ in distribution where $M_n = \max_{1\leq k \leq n}X_k$ and $G$ is a standard Gumbel random variable. Arguments from extreme-value theory indicate that a dominating random variable exists, so that $\mathbb{E} b_n(M_n-b_n) \to \mathbb{E} G = \gamma \approx 0.57721$. This would seem to contradict your claim. But, perhaps I'm missing something. Maybe the modulus affects things, though this seems doubtful in light of simple symmetry arguments." CreationDate="2011-05-06T18:05:09.813" UserId="2970" />
  <row Id="17840" PostId="10367" Text="@norv Precisely what was that &quot;order&quot; . It doesn't appear to be based upon anything in the matrix e.g. ascending order according to a particular variable. Please explain ?" CreationDate="2011-05-06T18:31:14.770" UserId="3382" />
  <row Id="17841" PostId="10429" Score="1" Text="@MYaseen208 Unfortunately, your question cannot motivate any decent answer as it stands. You might consider adding information about your design (crossed effects--random or not--vs. nested ones), what you qualify as multivariate (is this on the response variable(s) or the number of predictors), etc." CreationDate="2011-05-06T19:37:09.407" UserId="930" />
  <row Id="17842" PostId="10434" Text="@Filip, Perhaps you want to just plot the data you have vs the data that is predicted by the models, and look for the error(norm) between the two sets of data. The one with the least error is the &quot;best&quot; model. http://en.wikipedia.org/wiki/Norm_(mathematics)" CreationDate="2011-05-05T21:39:58.513" />
  <row Id="17843" PostId="10434" Text="Is your model linear or nonlinear? For linear models (where the free parameters are coefficients multiplying functions like $y=a+bx$ or $y=c+d\exp(x)+f\exp(2x)$), there is the coefficient of determination $R^2$ that you can use as the starting point, which has a similar purpose to Pearson's correlation coefficient for general linear models (the nearer to 1, the more likely the fit is good). Alternatively, you can take the sum of squares of residuals $\chi^2=\sum_i(y_i-f(x_i))^2$ (i.e. picakhu's suggestion)." CreationDate="2011-05-05T23:19:36.877" UserId="830" />
  <row Id="17844" PostId="10434" Text="@Picakhu: Thank you. I have implemented that for the time being.&#xA;@J.M.: It's a mix of both linear models and exponential models. And I don't have one X-value for every Y-value along the spectra (for example i don't have any 32 year old testperson). Any other method you can recommend? &#xA;@JasonMond:Aah. Is there any way to move my post there, or is it best practice to create a new one?" CreationDate="2011-05-06T17:44:29.543" />
  <row Id="17845" PostId="10434" Text="@Filip, I'm migrating this for you." CreationDate="2011-05-06T19:43:57.187" />
  <row Id="17846" PostId="10437" Text="I tried to add latex sintax, but it failed. Any help here on how to do this?" CreationDate="2011-05-06T20:04:30.853" UserId="3058" />
  <row Id="17847" PostId="10380" Text="Correct. We need the SE of the log odds ratios, not the odds ratios. The meta-analysis is conducted using the log odds ratios, as these are symmetric around 0 (as opposed to the odds ratios, which are not symmetric around 1) and whose distribution is much closer to normality." CreationDate="2011-05-06T20:06:46.403" UserId="1934" />
  <row Id="17848" PostId="10329" Text="I'm not sure why you think CAS is rarely used in statistics; personally I use Mathematica all the time. [Casella &amp; Berger](http://www.amazon.com/Statistical-Inference-George-Casella/dp/0534243126) even has an appendix devoted to Mathematica. But as a statistician I generally spend more time with data than doing math, like @whuber points out." CreationDate="2011-05-06T20:12:27.757" UserId="26" />
  <row Id="17850" PostId="10437" Text="@Mancel Don't indent your formulaes; otherwise, they will be rendered as verbatim (`&lt;pre&gt;...&lt;/pre&gt;` in HTML). See [here](http://stats.stackexchange.com/editing-help) for further information on Markdown formatting." CreationDate="2011-05-06T20:12:54.353" UserId="930" />
  <row Id="17851" PostId="10329" Text="Also recognize that many (most?) academic researchers and students have free access to mature packages like Maple and Mathematica through uni machines and/or site licenses. " CreationDate="2011-05-06T20:17:36.503" UserId="26" />
  <row Id="17852" PostId="10440" Text="Okay thanks. In your opinion then, if someone identified 2 as the midpoint of the age range 0-4, is this answer completely wrong? Or is it the case that 2.5 is more accurate but 2 is a valid answer?" CreationDate="2011-05-06T20:18:07.937" UserId="4498" />
  <row Id="17855" PostId="10440" Text="@Anne Using 2, 7, etc., is likely to be biased within most age groups, depending on the pattern of the age distribution in your population.  This would result in a net bias of around -1/2 year in many calculations.  In lots of applications that's an unimportant error; in other applications it might matter.  It wouldn't change the estimated standard deviation at all.  The biggest problem with using 2, 7, etc., is it indicates a lack of understanding of what the data mean, which would raise red flags in the mind of a critical reviewer of the work." CreationDate="2011-05-06T20:25:46.807" UserId="919" />
  <row Id="17856" PostId="10427" Score="3" Text="@BEF, The error generated by Stata means some of your independent variables are perfectly collinear. The likely culprit is within the dummy variables. Either you forgot to exclude at least one of the dummies, or some combination of the other independent variables is perfectly collinear (or one of the dummy variables has no variation). @chl , Stata will automatically drop a variable when perfect collinearity occurs in a regression model (I'm fairly confident this is the error message the OP is talking about)." CreationDate="2011-05-06T20:26:35.963" UserId="1036" />
  <row Id="17857" PostId="10440" Text="Yes, true. thanks." CreationDate="2011-05-06T20:33:28.173" UserId="4498" />
  <row Id="17858" PostId="10441" Text="Is this the same as my question here:&#xA;http://mathoverflow.net/questions/64120/how-to-estimate-sample-mean-and-variance-from-derived-data" CreationDate="2011-05-06T20:29:15.847" UserId="4500" />
  <row Id="17859" PostId="10441" Text="What does that last bracket mean?  And what do you mean by &quot;total variance&quot;?  Is it anything other than the variance of the combined dataset?" CreationDate="2011-05-06T20:38:28.527" UserId="919" />
  <row Id="17860" PostId="10442" Text="thanks so far! This is good help... But: How can I interpret high or (low) values of theta?  In McCaullaughs book generalized linear models there is a link to this paper from anscombe to make an interpretation of k. But unfortunately I don't really get it. The paper is http://www.claremontmckenna.edu/facultysites/math/FacMember/MOneill/Math152/Handouts/Anscombe2.pdf" CreationDate="2011-05-06T21:20:09.323" UserId="4496" />
  <row Id="17861" PostId="10442" Text="You just have to read the first page. So theta (or k in anscombe) is the shape parameter of the negbin distribution and it manages, if the distribution is closer to gamma (k -&gt; 0) or poisson (k -&gt; infinity). But what does it mean to the fit? How can I interpret theta for example for the cars estimation?&#xA;" CreationDate="2011-05-06T21:28:57.377" UserId="4496" />
  <row Id="17863" PostId="10434" Text="@Filip, think not only of identifying the least wrong model, but characterizing where your preferred model fits best, and where it fits worst.  I strongly recommend plotting the data and function as others have recommended." CreationDate="2011-05-06T21:44:12.217" UserId="3748" />
  <row Id="17864" PostId="10439" Score="2" Text="Because the errors are assumed normal, then the parameter estimates--being linear functions of the data, whence of the errors too--must themselves be normal, implying a normal distribution for $\hat{v}$." CreationDate="2011-05-06T21:52:31.023" UserId="919" />
  <row Id="17865" PostId="10320" Text="Sorry for the delay in responding. Is that 50Hz per channel? Can you describe the error further? Is this primarily vibration/thermal-noise error or are you concerned about drift in an estimate of position, say, due to nonidealities in numerical double integration? What are your frequencies of interest? What is your dynamic range, e.g., in LSBs? What is your error standard deviation?" CreationDate="2011-05-06T22:57:52.517" UserId="2970" />
  <row Id="17866" PostId="10256" Text="@NRH, Thanks for catching (and correcting) my typos before I managed to see them!" CreationDate="2011-05-06T22:59:29.707" UserId="2970" />
  <row Id="17867" PostId="10260" Text="Nice work. I believe the only remaining gap is in your last statement. It is not immediately apparent that optimizing the sum is the same as performing the sequence of optimizations in my answer. In fact, I don't think it follows directly, in general. But, it needn't be addressed here, either." CreationDate="2011-05-06T23:01:39.500" UserId="2970" />
  <row Id="17868" PostId="4255" Text="@whuber @nico A zip file for XML can be found for example at http://www.stats.ox.ac.uk/pub/RWin/bin/windows/contrib/2.13/ (and similarly for some other versions).  There are other issues with it, but eventually it seems to work" CreationDate="2011-05-06T23:15:16.307" UserId="2958" />
  <row Id="17869" PostId="10399" Text="Good point, Aniko, I missed that it was ordinal.  @ayush - a quick google search on 'SAS &quot;ordinal regression&quot;' turns up a lot of info. " CreationDate="2011-05-06T23:39:57.310" UserId="2669" />
  <row Id="17870" PostId="10451" Text="Thanks for your answer.  I have a question.  First how to generate a random correlation matrix.  Secondly, after reassign some numbers, the new matrix might not be a correlation matrix any more.  How do I make sure it is still a correlation matrix?" CreationDate="2011-05-07T00:50:19.740" UserId="4383" />
  <row Id="17871" PostId="10451" Text="It is still positive (semi)definite if you do the reassignment in the way I describe, unless I'm terribly mistaken. There are all sorts of ways that you might generate a correlation matrix; see some of the links in the sidebar. A simple way is to draw a random matrix $A$ which is $n\times p$ with $n&lt;p$, compute $A^TA$ and then normalize it to have ones on the diagonal." CreationDate="2011-05-07T01:02:12.653" UserId="26" />
  <row Id="17872" PostId="10451" Score="1" Text="I am very interested in seeing this. Say I generate a random corr matrix that happens to have 20% of them between -0.3 and 0.3.  Then I need to resample 70% of the entries unfiormly between -0.3 to 0.3.  How do we see the new matrix is still psd?   " CreationDate="2011-05-07T01:20:30.280" UserId="4383" />
  <row Id="17873" PostId="10451" Score="2" Text="@JMS, Why would the matrix still be positive semidefinite necessarily? I don't believe your statement is true." CreationDate="2011-05-07T01:21:16.563" UserId="2970" />
  <row Id="17874" PostId="10450" Text="You have to be aware that you can't get *arbitrary* negative correlations between variables, for one thing." CreationDate="2011-05-07T01:23:42.490" UserId="2970" />
  <row Id="17875" PostId="10451" Text="@cardinal, thanks for your comment.  We had some discussion about the normally distributed random corr matrix. I don't think i can get the number large enough.  So I want to give up normal and see if there is a general way to generate such correlation matrix ." CreationDate="2011-05-07T01:29:26.727" UserId="4383" />
  <row Id="17876" PostId="10450" Text="Can you define what you mean by &quot;random&quot;? This seems related to your [previous question](http://stats.stackexchange.com/questions/10121/generate-normally-distributed-correlation-matrix). " CreationDate="2011-05-07T01:29:55.743" UserId="2970" />
  <row Id="17877" PostId="10451" Text="@JMS This matrix has a negative eigenvalue:$$\left(&#xA;\begin{array}{cccccc}&#xA; 1 &amp; -\frac{1}{5} &amp; 0 &amp; \frac{1}{5} &amp; \frac{3}{10} &amp; \frac{3}{10} \\&#xA; -\frac{1}{5} &amp; 1 &amp; \frac{1}{10} &amp; \frac{3}{10} &amp; \frac{3}{10} &amp; \frac{1}{5} \\&#xA; 0 &amp; \frac{1}{10} &amp; 1 &amp; \frac{3}{10} &amp; -\frac{1}{5} &amp; -\frac{1}{5} \\&#xA; \frac{1}{5} &amp; \frac{3}{10} &amp; \frac{3}{10} &amp; 1 &amp; -\frac{3}{10} &amp; -\frac{3}{10} \\&#xA; \frac{3}{10} &amp; \frac{3}{10} &amp; -\frac{1}{5} &amp; -\frac{3}{10} &amp; 1 &amp; -\frac{1}{10} \\&#xA; \frac{3}{10} &amp; \frac{1}{5} &amp; -\frac{1}{5} &amp; -\frac{3}{10} &amp; -\frac{1}{10} &amp; 1&#xA;\end{array}&#xA;\right)$$" CreationDate="2011-05-07T01:38:25.837" UserId="919" />
  <row Id="17878" PostId="10451" Text="...and I am terribly mistaken :) Not sure where my head was at." CreationDate="2011-05-07T01:39:39.313" UserId="26" />
  <row Id="17879" PostId="10451" Text="Thanks for all the comments.  I try to generate the corr matrix I described.  It is not hard to get a small matrix by rejecting the ones that are not psd. But for biger matrix, it's become very difficult to get the psd with the property I want. " CreationDate="2011-05-07T01:45:09.437" UserId="4383" />
  <row Id="17880" PostId="10121" Text="@cardinal The downvote is because my request for a clarification got no response, but I should have made a further comment.  As far as I can tell, this still is not a well-formulated question because it is based on a false assumption ($AA^T$ will not have normally distributed entries) and is nonsensical (correlation is not standard deviation).  The downvote does not mean the question is *uninteresting*; it means it remains ill-posed." CreationDate="2011-05-07T01:46:02.770" UserId="919" />
  <row Id="17881" PostId="10121" Text="@whuber, agreed on all counts. I have been trying to extract what problem the OP is trying to solve. But, have failed to get a clear picture (yet). Perhaps I have not yet asked the right question." CreationDate="2011-05-07T01:49:31.007" UserId="2970" />
  <row Id="17882" PostId="10450" Score="1" Text="Is the 90/10 requirement &quot;hard&quot;? In lower dimensions you might be able to get close by drawing from a Wishart centered at $I$, computing the correlation matrix, and rejecting samples that aren't within some tolerance. Though I suspect this won't scale well at all..." CreationDate="2011-05-07T01:52:50.363" UserId="26" />
  <row Id="17883" PostId="10451" Text="If we use t(A)*A to construct a correlation matrix and construct the row of A from mulitvariate normal with different correlations, can we control the size of the entries in the resulting correlation matrix?" CreationDate="2011-05-07T01:53:31.203" UserId="4383" />
  <row Id="17884" PostId="10451" Text="Just *checking* the positive-definiteness of a big matrix after you've fiddled with individual entries is going to be computationally intensive. My comments in your [other question](http://stats.stackexchange.com/questions/10121/generate-normally-distributed-correlation-matrix) give a couple indications as to why your construction may be difficult to obtain." CreationDate="2011-05-07T01:54:29.427" UserId="2970" />
  <row Id="17885" PostId="10451" Text="@Richard This is a Wishart distribution; see the comment I just added, and http://en.wikipedia.org/wiki/Wishart_distribution" CreationDate="2011-05-07T01:55:23.947" UserId="26" />
  <row Id="17886" PostId="10121" Text="thanks for all the comments. Can I try to clarity the question? " CreationDate="2011-05-07T02:06:05.807" UserId="4383" />
  <row Id="17887" PostId="10449" Score="1" Text="This will make the final correlations difficult to determine. What you describe is essentially generating from a [multivariate Gaussian copula](http://en.wikipedia.org/wiki/Copula_%28statistics%29#Gaussian_copula)." CreationDate="2011-05-07T02:06:37.680" UserId="2970" />
  <row Id="17888" PostId="10121" Text="The motivation is this.  For a set of n time series data, the correlation distribution often look quite close to normal.  I would like to generate many &quot;normal&quot; correlation matrices to represent the general siutation and use them to calculate risk number. Do you think I answer your question?" CreationDate="2011-05-07T02:11:49.530" UserId="4383" />
  <row Id="17889" PostId="10451" Text="@JMS, thanks.  That's very interesting.  I want to look into that." CreationDate="2011-05-07T02:12:47.750" UserId="4383" />
  <row Id="17890" PostId="10451" Text="@cardinal/whuber I explain further of the other question.  Do you think you understand that question better?" CreationDate="2011-05-07T02:14:16.297" UserId="4383" />
  <row Id="17891" PostId="10121" Text="@Richard Thanks, that's much better.  I begin to see what you are looking for. Perhaps we should clean up this thread: would you mind putting your clarifications as edits in the original question?  Then we can delete many of these comments, remove the downvote (which cannot be done until the question itself is modified), and make some progress.  Consider pursuing @Rick Wicklin's suggestion below, too: it looks like he is on to something you might find helpful." CreationDate="2011-05-07T02:15:25.320" UserId="919" />
  <row Id="17892" PostId="10450" Text="@JMS We're starting to get some clarification in new comments to the preceding question linked to by @Cardinal: you might want to check there." CreationDate="2011-05-07T02:19:01.073" UserId="919" />
  <row Id="17893" PostId="10121" Text="@whuber, I guess I should not say normally distributed.  I should say &quot;approximately&quot; normally distributed in the sense that for large n, if you plot the qqnorm of the entries, the are almost on the same straight line as normal." CreationDate="2011-05-07T02:21:24.400" UserId="4383" />
  <row Id="17894" PostId="10121" Text="@cardinal, I am very interested in your comments on April 28 23:38.  Would you recommend a reference that discusses this kind of analysis?" CreationDate="2011-05-07T02:29:54.873" UserId="4383" />
  <row Id="17895" PostId="10121" Text="@whuber, I have clarify the original question.  Thanks for the suggestion." CreationDate="2011-05-07T02:39:00.837" UserId="4383" />
  <row Id="17896" PostId="10450" Text="@whuber Good, thanks. Might we close this then? It seems &quot;duplicate in intention&quot; as it were and doesn't contain much beyond my foolishness." CreationDate="2011-05-07T03:25:01.313" UserId="26" />
  <row Id="17897" PostId="10439" Text="So are you saying that the normal confidence interval is correct? If I understand correctly, by that logic we would also use normal confidence intervals for the parameters. But we use intervals based on the t distribution." CreationDate="2011-05-07T03:38:54.587" UserId="3835" />
  <row Id="17898" PostId="10394" Text="the reference is:     Rousseeuw P.J., and Croux, C. (1993). “Alternatives to median absolute deviation.” J. American Statistical Assoc., 88, 1273-1283." CreationDate="2011-05-07T04:51:43.573" UserId="4286" />
  <row Id="17899" PostId="10398" Text="Generally, in terms of an R/Sweave workflow, RStudio is perfectly functional now and rapidly maturing. No bib support but I've always preferred using a citation manager that syncs to a .bib anyway. It is IMO a better choice for these tasks if you don't already know (or want to *really* learn) Emacs. Of course really knowing Emacs pays other dividends..." CreationDate="2011-05-07T05:12:26.773" UserId="26" />
  <row Id="17900" PostId="10439" Text="The t distribution is used because you're estimating the error variance; if that were known then you would have a normal distribution like @whuber says." CreationDate="2011-05-07T05:39:44.737" UserId="26" />
  <row Id="17901" PostId="10096" Score="1" Text="This would fall under the umbrella of intervention analysis; the common reference is Box &amp; Tao 1975's JASA paper, &quot;Invervention analysis with applications&quot; to something or other. There's been a lot of work since then too, obviously. " CreationDate="2011-05-07T05:47:02.330" UserId="26" />
  <row Id="17902" PostId="10439" Text="Thanks for your comment. What I'm asking is, can the t distribution also be used for a confidence interval for v as defined in the question, and if so, with how many degrees of freedom? " CreationDate="2011-05-07T06:05:32.147" UserId="3835" />
  <row Id="17903" PostId="10260" Text="@cardinal, it follows by induction. You provide the induction start, and in the induction step choose orthonormal vectors $w_1, \ldots, w_q$ that maximize the sum and arrange it so that $w_q$ is a unit vector orthogonal to $u_1, \ldots, u_{q-1}$. Then by your results $w_q^T \mathbf{S} w_q \leq u_q^T \mathbf{S} u_q$ and by the induction assumption $\sum_{i=1}^{q-1} w_i^T \mathbf{S} w_i \leq \sum_{i=1}^{q-1}u_i^T \mathbf{S} u_i$. Of course, the basis is not a unique basis for the $q$-dimensional space. You can also generalize the &quot;convex combination argument&quot; you use to give a direct prove." CreationDate="2011-05-07T07:10:12.693" UserId="4376" />
  <row Id="17904" PostId="10398" Text="i totally agree about RStudio, but given that i am learning bits of pieces of many different languages along with all the stats and referencing, emacs suits me perfectly." CreationDate="2011-05-07T07:24:53.187" UserId="656" />
  <row Id="17905" PostId="10420" Score="2" Text="@melon. Your question is quite vague. What do you mean by parametric and non parametric? (i know of a few different ways in which these terms are used)." CreationDate="2011-05-07T07:32:44.057" UserId="656" />
  <row Id="17907" PostId="10453" Score="1" Text="@NRH Thank you, that is exactly the sort of thing I was looking for. But is there a mistake in the formula? The dimensions don't seem to match in $a^T(\mathbf{X}^T\mathbf{X})^{-1}a$. Should $\mathbf{X}$ be the $n \times (p+1)$ matrix having ones in the first column?" CreationDate="2011-05-07T09:39:21.743" UserId="3835" />
  <row Id="17908" PostId="10450" Text="@cardinal, sorry for being ignorance.  I missed the part of the thread yesterday and did not answer your question.  Regard &quot;can't get abitrary negativ correlations&quot;, I hope my clarification of previous question answers your question." CreationDate="2011-05-07T09:56:05.660" UserId="4383" />
  <row Id="17909" PostId="10456" Text="hmm ok so there is any convenient way, no? " CreationDate="2011-05-07T10:28:24.667" UserId="4496" />
  <row Id="17910" PostId="10457" Score="4" Text="this is a better fit for cross validated." CreationDate="2011-05-06T15:17:58.817" UserId="696" />
  <row Id="17911" PostId="10456" Text="@MarkDollar No one that I can think of. And in fact, I don't know if there exists a dedicated function for Cook distance in Tobit or censored regression. A first start would be to plot the outcome variable against each explanatory variable, to check whether there's any evidence that the errors are not identically and normally distributed. I also found this paper, [Influence diagnostics in the tobit censored response model](http://www.springerlink.com/content/r88515367r5078k4/) which detailed influence measures for Tobit models. (...)" CreationDate="2011-05-07T11:21:46.333" UserId="930" />
  <row Id="17912" PostId="10456" Text="@MarkDollar (...) You might also take a look at UCLA resources on [Tobit regression with R](http://www.ats.ucla.edu/stat/R/dae/tobit.htm) and check whether the [VGAM](http://cran.r-project.org/web/packages/VGAM/index.html) package they used contains additional diagnostic tools. " CreationDate="2011-05-07T11:23:40.403" UserId="930" />
  <row Id="17913" PostId="10459" Score="1" Text="nice idea for a collection ! @mods: Beside cw maybe renaming to &quot;movies every statistician should have seen&quot; or something like that ?" CreationDate="2011-05-07T12:28:31.583" UserId="264" />
  <row Id="17914" PostId="10389" Text="@cardinal - Just a numerical observation with $n \le 2000$. In that range it appears that $E(M_n) \sim \sqrt{2 \log n} - .27...$, but of course that gap could be closing.  " CreationDate="2011-05-07T12:35:58.850" UserId="4062" />
  <row Id="17915" PostId="10459" Text="@steffen Thx. Better to flag the question for mods attention in the future (we aren't notified with @mods)." CreationDate="2011-05-07T12:37:52.090" UserId="930" />
  <row Id="17916" PostId="10456" Text="thank you! This is a good way to find out what to do!" CreationDate="2011-05-07T12:51:58.733" UserId="4496" />
  <row Id="17917" PostId="10260" Text="@NRH, Sorry I've only had a chance to glance at this haphazardly. The statement &quot;*...and arrange it so that $w_q$ is a unit vector orthogonal to $u_1,\ldots,u_{q−1}$...*&quot; seems to me to be the same gap I was referring to. In the induction step, when going from $n=1$ to $n=2$, you are forcing a nesting of the linear subspaces whereas the problem itself doesn't directly make that requirement. (Of course, it still turns out to be the case.) But, it's quite likely I'm just missing a subtle point you've (already) made." CreationDate="2011-05-07T13:01:36.087" UserId="2970" />
  <row Id="17918" PostId="10389" Text="@Hans, I think it is closing. The convergence in distribution is slow. Also, doing a proper simulation in this case is pretty hard since we're looking at the maximum modulus. So, algorithmic nonidealities in the random number generation can come into play. For example, though nobody uses the inversion method for normal random variates, if you did, due to asymmetries in IEEE floating point, the smallest possible normal random variate you can generate is about -37.52, while the largest is only 8.21." CreationDate="2011-05-07T13:08:08.790" UserId="2970" />
  <row Id="17919" PostId="10121" Text="@Richard, this is a fairly straightforward application of the delta method. I don't have a reference immediately handy, but Chapters 12 and/or 13 (I believe) of Lehmann and Romano, *Testing Statistical Hypotheses*, 3rd. ed. should cover this. If I recall, it may even have this example." CreationDate="2011-05-07T13:14:02.123" UserId="2970" />
  <row Id="17920" PostId="10121" Text="@Richard, in the previous comment of mine that you reference, note that there is a multiplication by $\sqrt{n}$ in order to get this convergence of distribution. Note also that using a sequence of independent normals results in the largest asymptotic variance. So, crudely, the off-diagonals will have mean correlation of zero and variance of about $n^{-1/2}$." CreationDate="2011-05-07T13:17:20.590" UserId="2970" />
  <row Id="17921" PostId="10459" Text="By the way, I found this page, which might add some movies to the collection: http://world.std.com/~reinhold/mathmovies.html" CreationDate="2011-05-07T14:41:51.013" UserId="4504" />
  <row Id="17922" PostId="10464" Score="1" Text="Can the reason that the 10% are binned be ascribed to a random process or is there some underlying factor influencing the binning?" CreationDate="2011-05-07T15:03:36.103" UserId="3805" />
  <row Id="17923" PostId="10459" Score="2" Text="Nothing that will inspire anybody to take up mathematics. Stick to books." CreationDate="2011-05-07T15:21:49.440" UserId="4479" />
  <row Id="17924" PostId="10467" Score="1" Text="That's clever.  But it raises interesting questions.  How would you code the categorical ages?  What do you do if some of those categorical age intervals overlap each other?  Does it even make sense to develop a model in which intervals of age are separate categories yet actual ages are supposed to be *linearly* related to the outcome?" CreationDate="2011-05-07T15:30:05.167" UserId="919" />
  <row Id="17925" PostId="10464" Text="Do the age bins have any chance of overlapping each other?" CreationDate="2011-05-07T15:30:51.530" UserId="919" />
  <row Id="17926" PostId="10463" Text="Thanks for the link to a possible duplicate.  The questions are related but they are not duplicates: here, you are asking for how to analyze the data, not for a sample size." CreationDate="2011-05-07T15:33:10.263" UserId="919" />
  <row Id="17927" PostId="10453" Text="@mark999, yes, $\mathbf{X}$ has $p+1$ columns. I have corrected that in the answer. Thanks." CreationDate="2011-05-07T16:12:30.993" UserId="4376" />
  <row Id="17928" PostId="10260" Score="1" Text="@cardinal, I am not forcing a nesting, merely using a dimension consideration. If we have a $q$-dimensional subspace you can always choose $w_q$ in that space such that it is orthogonal to a $(q-1)$-dimensional subspace. Then you fill up the $w$-basis in any way you like. " CreationDate="2011-05-07T16:21:59.617" UserId="4376" />
  <row Id="17929" PostId="10431" Text="@deps_stats: These are univariate BLUP. But I'm looking for multivariate BLUP." CreationDate="2011-05-07T16:32:57.737" UserId="3903" />
  <row Id="17930" PostId="10460" Text="one can also use this to generate discussion on spurious correlation" CreationDate="2011-05-07T16:56:36.450" UserId="4048" />
  <row Id="17931" PostId="10462" Text="@Frank Harrell - good points about the problems associated with the conditions you outline but if one has all continuous variables with different scales then isn't standardization the only way to compare slopes? " CreationDate="2011-05-07T17:07:22.547" UserId="4048" />
  <row Id="17932" PostId="10439" Text="@mark The variances and covariances all ultimately depend on the estimated variance of the residuals.  Thus the DF to use is the DF in this estimate, equal to the number of data values minus the number of parameters (including the constant)." CreationDate="2011-05-07T17:44:12.920" UserId="919" />
  <row Id="17933" PostId="10459" Text="@Emre Maybe. But nevertheless it is entertaining for those who already deal with math (even more if the movie is presenting things wrong or with exaggeration)." CreationDate="2011-05-07T18:38:36.787" UserId="264" />
  <row Id="17934" PostId="10376" Text="@Henry I think you need to use a double backslash. Otherwise, it should work with a `latex`, `epslatex`, or `pstricks` terminal." CreationDate="2011-05-07T18:41:36.627" UserId="930" />
  <row Id="17935" PostId="10462" Text="@Frank, I suppose it depends on what type of models you are running, but standardization of predictor variables is often useful.  Centering them means that the intercept becomes interpretable as the mean predicted outcome and the relative importance of different predictors becomes more obvious.  I usually leave binary predictors alone, but sometimes other scaling options are worth considering.  Finally, in some cases having predictors with wildly different standard deviations can lead to computational/convergence problems." CreationDate="2011-05-07T19:01:32.493" UserId="3748" />
  <row Id="17936" PostId="10464" Text="@schenectady I don't think the binning can be ascribed to a random process. My understanding is that survey participants who refused to give their age were asked to give an age range. @whuber Yes, there is overlap. For example, the &gt;18 category is not exclusive of the 34 -- 60 category." CreationDate="2011-05-07T19:45:27.993" UserId="4506" />
  <row Id="17937" PostId="10467" Text="@crayola Nice, I'll give it a go." CreationDate="2011-05-07T19:47:46.807" UserId="4506" />
  <row Id="17938" PostId="10407" Text="This question is closely related to the Generalized Birthday Problem.&#xA;http://en.wikipedia.org/wiki/Birthday_problem" CreationDate="2011-05-07T20:40:31.580" UserId="3567" />
  <row Id="17939" PostId="10398" Text="Most certainly - those are the sort of dividends I was referring to :)" CreationDate="2011-05-07T20:45:14.947" UserId="26" />
  <row Id="17940" PostId="7045" Text="Actually, the glm function is for generalized linear models (logistic regression, Poisson regression, etc.). Of course, if you use glm with family=gaussian, then it is a general linear model, but in that case, one would just use the lm function." CreationDate="2011-05-07T21:11:17.343" UserId="1934" />
  <row Id="17941" PostId="10453" Text="@NRH Thanks, you've been very helpful." CreationDate="2011-05-07T21:22:37.010" UserId="3835" />
  <row Id="17942" PostId="7694" Text="Do you mean an exponential smoothing state space model? What R packages are you using?" CreationDate="2011-05-07T22:27:27.823" UserId="2817" />
  <row Id="17943" PostId="10477" Text="It's not particularly geeky - it is a Spanish thriller (and like all thrillers has some implausible plot lines to move the story along and resolve it at the end) with a few logic puzzles using numbers.  If you want to solve the puzzles yourself, a pause button might be useful." CreationDate="2011-05-07T22:30:31.657" UserId="2958" />
  <row Id="17944" PostId="10465" Text="An absorbing story and one I'd recommend, but it referred to math rather than truly involding math.  I'd say the same thing about the tv show Numb3rs." CreationDate="2011-05-07T22:43:12.617" UserId="2669" />
  <row Id="17945" PostId="10478" Score="2" Text="Are you aware of papers establishing consistency results for the lasso? Knight &amp; Fu (2000), Yu &amp; Zhao (2006), and various papers by Meinshausen." CreationDate="2011-05-07T23:27:17.467" UserId="2970" />
  <row Id="17946" PostId="10260" Text="@NRH, now I see where you were going, and, yes, I think that works. Nice. There are a couple of other ways to go about solving this that I know of, but they're fundamentally a bit different." CreationDate="2011-05-07T23:42:13.990" UserId="2970" />
  <row Id="17947" PostId="10480" Text="Presumably $n$ is supposed to be a positive integer. One minor point is that is possible to achieve the value $n$, if $U=1$, but that the probability of doing so is $0$. " CreationDate="2011-05-08T00:03:02.987" UserId="2958" />
  <row Id="17948" PostId="10369" Text="$X_i$ is also assumed independent of $Y_i$, yes?" CreationDate="2011-05-08T00:14:07.770" UserId="2970" />
  <row Id="17949" PostId="9804" Text="It may turn out that R is better at fitting ARIMA models than rapidminer, in which case you might consider learning a new piece of software." CreationDate="2011-05-08T02:14:33.117" UserId="2817" />
  <row Id="17950" PostId="9892" Text="@Simon Byrne +1 for R and glmnet.  Glmnnet can handle logistic regression with both the lasso and the elastic net. It's also an extremely fast implementation of the algorithm, and I suggest trying it out if you know any R." CreationDate="2011-05-08T02:18:16.327" UserId="2817" />
  <row Id="17951" PostId="818" Text="You may also want to check out kernlab and caret for SVMs.  They're interesting (thought not necessarily better) alternatives." CreationDate="2011-05-08T02:27:21.267" UserId="2817" />
  <row Id="17952" PostId="10489" Text="That seems to work. I am mostly looking for a quick and dirty hack to use for demonstrations in a class I'm teaching, but if I can post it on a webpage so much the better." CreationDate="2011-05-08T02:28:41.013" UserId="98" />
  <row Id="17953" PostId="10485" Text="+1, excellent explanation." CreationDate="2011-05-08T02:30:36.773" UserId="2116" />
  <row Id="17954" PostId="10488" Text="Thanks.  This is basically what I'm looking for." CreationDate="2011-05-08T02:32:23.130" UserId="98" />
  <row Id="17955" PostId="10369" Text="@cardinal: yes, they are independent." CreationDate="2011-05-08T02:33:24.340" UserId="795" />
  <row Id="17956" PostId="7045" Text="Interesting comment, now I understand the difference is that one can use different distribussions and the other not, by the way I have been reading the material below and it might interest you. Thanks for the aclaration between the differences in glm and lm" CreationDate="2011-05-08T04:58:08.900" UserId="1808" />
  <row Id="17957" PostId="10475" Text="Your answer is excellent, now I understand so much more, I can not believe it. Let me tell you that I use to perform .s3 procedure transformation in all my analysis (at least I intuitively understood that. The process to take care of the variables before 5&amp; or after 95% was called by a friend something like winzorize, but I tend to use it with a 2 percent margin (I guess it all depends on the variables you have).I am eager to  the other transformations using functions and else to see how it works." CreationDate="2011-05-08T05:07:47.830" UserId="1808" />
  <row Id="17958" PostId="10475" Text="Regarding the second package it seemed amazing to me, explaining al that methods for what is the best option for modifying the variables, let me tell you that at least I intuited several stuff of the things that are there, for example I use to eliminate dependent variables when some of them where highly correlated, which to eliminate was decided by my with different criteria such as first eliminating the variable less correlated with the dependent variable.   I really appreciate this, hope we keep in touch. " CreationDate="2011-05-08T05:11:29.103" UserId="1808" />
  <row Id="17961" PostId="5115" Text="Thank you very much for doing that, but I am sorry to tell you that I can not see anything i the first link." CreationDate="2011-05-08T05:28:34.270" UserId="1808" />
  <row Id="17962" PostId="6950" Text="Thanks guys for the tips they where helpfull" CreationDate="2011-05-08T05:37:53.490" UserId="1808" />
  <row Id="17963" PostId="8807" Text="I found really interesting the question. Is there a way I can mark a question to follow it?" CreationDate="2011-05-08T05:48:35.697" UserId="1808" />
  <row Id="17964" PostId="10371" Text="just for reference, _Eureqa_'s using [Genetic Programming](http://en.wikipedia.org/wiki/Genetic_programming) to find solutions to the symbolic regression problem." CreationDate="2011-05-08T07:47:33.510" UserId="4360" />
  <row Id="17965" PostId="10502" Text="This minet looked nice, but it needs to have some external adapter to make appropriate input form general data :-(" CreationDate="2011-05-08T09:09:27.770" UserId="88" />
  <row Id="17966" PostId="10478" Text="Yes, but my question in not about asymptotic consistency, which is the subject of the papers you mentioned." CreationDate="2011-05-08T11:09:33.770" UserId="30" />
  <row Id="17967" PostId="9653" Text="Be sure also to talk about hypothesis testing (Neyman-Pearson tests) and not significance testing (Fisher tests). These approaches are commonly mixed even if there is no notion of error in the second one, and proper usages should be different because they lead to different kinds of conclusion." CreationDate="2011-05-08T11:25:32.707" UserId="4443" />
  <row Id="17968" PostId="9720" Text="Sorry for the late reply, but I know unfortunately too little on the topic anyway to answer that you are using the right technique. My guess is that SNA contains other tools that will help (e.g. model different centrality measures when you suppress any combination of your four nodes)." CreationDate="2011-05-08T12:08:50.377" UserId="3582" />
  <row Id="17969" PostId="10505" Text="The [Forbes article](http://blogs.forbes.com/alexknapp/2011/04/30/a-computer-that-knows-when-to-say-thats-what-she-said/)" CreationDate="2011-05-08T12:18:01.137" UserId="2958" />
  <row Id="17970" PostId="10478" Text="Those papers are (primarily) about *model selection* consistency, which I'd say is *very* related to the question you asked. :)" CreationDate="2011-05-08T13:05:01.993" UserId="2970" />
  <row Id="17971" PostId="10499" Text="(+1) All three of those papers are worth a careful reading for those interested in this subject. The Dantzig selector paper has some very nice math; however, I have not seen it get much traction in applications, nor do I expect it to. I think, among other things, the very noisy regularization paths make people nervous and so, with no obvious benefit over the lasso, make it a hard sell." CreationDate="2011-05-08T13:08:05.933" UserId="2970" />
  <row Id="17972" PostId="10480" Text="@Henry The question makes sense for any real $n$, not just positive integers!" CreationDate="2011-05-08T13:59:33.710" UserId="919" />
  <row Id="17973" PostId="10509" Text="Please do not simultaneously [cross-post](http://math.stackexchange.com/questions/37837/pearson-correlation-problem) questions on multiple SE sites." CreationDate="2011-05-08T14:00:00.600" UserId="2970" />
  <row Id="17974" PostId="10509" Text="@mmm We're glad you considered posting this question on the stats.se site.  Unfortunately, it needs to be closed until you can (a) decide on one site where you want to post it and (b) explain what the numbers in the table mean and what &quot;weighted averages&quot; and &quot;neighbourhood&quot; would mean in this context." CreationDate="2011-05-08T14:08:20.083" UserId="919" />
  <row Id="17975" PostId="10475" Text="Elimination of variables in this way will result in major biases.  The approach is not driven by any statistical principles I can think of.  Use a principled approach, i.e., either pre-specify the model and use only methods blinded to Y (the response variable) to select the Xs, or use something like glmnet.  I don't see how Winsorization applies here either.  Variable selection without shrinkage (direct penalty for selection and estimation) is a disaster.  The original set of functions you specified for the predictor X's doesn't make as much sense as using spline bases, usually." CreationDate="2011-05-08T14:24:01.447" UserId="4253" />
  <row Id="17976" PostId="10467" Score="1" Text="Using the &quot;continuous age is available&quot; as a predictor will result in major problems of interpretation and inference.  The problem as originally specified is an ideal candidate for multiple imputation.  For those subjects for whom continuous age is known you also know the bins.  That allows you to develop a multiple imputation model for continuous age given the bin.  Then you can form a model with only continuous age (90% real, 10% imputed)." CreationDate="2011-05-08T14:26:50.727" UserId="4253" />
  <row Id="17977" PostId="10462" Text="I'm not clear on how such standardization adds clarity rather than subtracts from it.  Also, the mean is not the obvious choice for centering (median? mode? 43rd percentile? choice of dispersion measure is even more problematic)  Relative importance can be judged in many ways, e.g., partial $R^2$, partial $\chi^2$, inter-quartile-range covariate effects, ...  Also I have not found standardization to be helpful computationally when using modern matrix math routines such as the ones R uses under the hood.  Kenny note that standardization is not the way to compare slopes." CreationDate="2011-05-08T14:33:28.240" UserId="4253" />
  <row Id="17978" PostId="9877" Text="In the link above see the course handout.  Regarding choice of number of knots, the simplest guiding principle is to choose the number you can afford to fit related to what we know about causes of overfitting (events per predictor rules of thumb, etc.).  This is based on the assumption that everything is nonlinear (which is true except for very special cases)." CreationDate="2011-05-08T14:35:51.670" UserId="4253" />
  <row Id="17979" PostId="10475" Text="@Frank Harrell caret's feature selection routines include a cross-validation step which can be use to greatly reduce potential bias.  Also the preProcess step only removes highly correlated variables or variables with extremely low variance (which could cause problems during cross validation) " CreationDate="2011-05-08T15:01:22.737" UserId="2817" />
  <row Id="17980" PostId="10475" Text="@mariana soffer I'm glad I could help you out! If you like my answer, feel free to 'accept' it by clicking the green check mark!" CreationDate="2011-05-08T15:02:12.197" UserId="2817" />
  <row Id="17981" PostId="10462" Text="If you have binary variables, don't standardize them, only continuous one. See this article by Gelman (&lt;http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf&gt;, suggesting dividing variables by two standard deviations. In any case, it helps to achive convergence if you are fitting a Bayesian model." CreationDate="2011-05-08T15:04:15.683" UserId="3058" />
  <row Id="17982" PostId="10420" Score="1" Text="@richie Out of curiosity--given that there is a standard meaning of [parametric statistics](http://en.wikipedia.org/wiki/Parametric_statistics)--what different kinds of usage do you have in mind?  It seems to me this question is sufficiently clear and well posed, but what you (and @Fr in a reply) may be objecting to is that it is too broad and would benefit from being narrowed.  @Melon Do you have a particular context, problem, or example in mind that would help focus this question?" CreationDate="2011-05-08T15:05:30.897" UserId="919" />
  <row Id="17983" PostId="10462" Text="And I don't see why it would be better to interpret effects on the original scale. The effect of GDP on a logistic regresssion is hard to interpret looking only to coefficients, because GDP is much bigger than probability scale. Interactions are better understood with standardized coefficients or centered variables. Finally, if you have $x$ and $x^{2}$, in this case don't standardize." CreationDate="2011-05-08T15:17:09.037" UserId="3058" />
  <row Id="17984" PostId="10467" Text="I agree that your solution would ultimately result in a more interpretable output. However, if &#xA;- the goal is just to control for age in a regression where the interest is on another variable (e.g. find the effect of some kind of treatment controlling for age), and&#xA;- the individuals for whom continuous age is missing are an as-if-random sample of the population (conditional on the other covariates),&#xA;I fail to see where the &quot;major problems of interpretation and inference&quot; come from?" CreationDate="2011-05-08T15:26:20.270" UserId="2044" />
  <row Id="17985" PostId="10480" Text="@whuber:You are correct in general, but I still suspect my &quot;presumably&quot; is what was intended. " CreationDate="2011-05-08T15:32:46.347" UserId="2958" />
  <row Id="17986" PostId="10503" Text="Some sort of plot would be sensible to explore the data.  It is difficult to comment on your other statements without more information such as what &quot;more important&quot; might mean." CreationDate="2011-05-08T15:36:09.743" UserId="2958" />
  <row Id="17987" PostId="10510" Text="If a mod could make this a community wiki, that would be great!" CreationDate="2011-05-08T16:10:07.440" UserId="364" />
  <row Id="17988" PostId="10510" Score="1" Text="It's not 100% relevant, so I'm not posting this as an answer. But, J. P. A. Ioannidis, [Why most published research findings are false](http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124), *PLoS Med* **2** (8): e124, 2005. is worth a read." CreationDate="2011-05-08T16:30:42.017" UserId="2970" />
  <row Id="17989" PostId="10407" Text="@Aniko: I haven't run extensive simulations yet. But the examples I tried are roughly correct." CreationDate="2011-05-08T16:51:31.203" UserId="565" />
  <row Id="17990" PostId="10407" Text="@Charles: In which sense are they related? Can you please elaborate on it? I see only a very vague connection and would be happy to understand it better." CreationDate="2011-05-08T16:55:33.027" UserId="565" />
  <row Id="17991" PostId="10510" Text="If I had more reputation I might add the following tags: bibliography, recommended_reference" CreationDate="2011-05-08T17:06:25.933" UserId="3748" />
  <row Id="17992" PostId="10420" Text="@whuber in psychology (and possibly elsewhere) the term non-parametric means not using the normal distribution, and I was attempting to see if that was the definition the asker was using. " CreationDate="2011-05-08T17:47:44.057" UserId="656" />
  <row Id="17993" PostId="10462" Text="There are too many exceptions then for standardization to be a recommended procedure in general.  And standardization does not make interpretation of odds ratios easier.  In fact the reverse is true.  Instead, by computing an inter-quartile-range odds ratio (as done automatically in the R rms package) you can say that the effect of moving GDP from a to b absolute units is to increase odds of Y=1 by x.  And there are too many choices of standardization (IQR, Gini's mean difference, SD, inter-decile range, etc.) to make interpretation of interactions straightforward." CreationDate="2011-05-08T17:49:44.257" UserId="4253" />
  <row Id="17994" PostId="10467" Score="1" Text="Creating indicator variables as outlined creates major difficulties even if one is only wanting to control for age.  See @Article{kno10unp,&#xA;  author = 		 {Knol, Mirjam J. and Janssen, Kristel J. M. and Donders, Rogier T. and Egberts, Antoine C. G. and Heerding, E. Rob and Grobbee, Diederick E. and Moons, Karel G. M. and Geerlings, Mirjam I.},&#xA;  title = 		 {Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example},&#xA;  journal = 	 J Clinical Epidemiology,&#xA;  year = 		 2010,&#xA;  volume = 	 63,&#xA;  pages = 	 {728-736}&#xA;}" CreationDate="2011-05-08T17:55:06.110" UserId="4253" />
  <row Id="17995" PostId="10420" Text="@Richie I'm pretty sure that even in psychology &quot;non-parametric&quot; means more than just not using the normal distribution.  However, different fields do co-opt technical terms and use them in their own ways.  I would be interested to see an authoritative reference where the term is actually defined in the sense you give." CreationDate="2011-05-08T19:08:27.687" UserId="919" />
  <row Id="17996" PostId="10511" Score="1" Text="+1.  It is interesting--and quite telling--that the readings start and end with Meehl." CreationDate="2011-05-08T19:13:20.710" UserId="919" />
  <row Id="17997" PostId="10517" Score="2" Text="This question is perhaps overly broad.  Do you have a specific problem you want to address?  If so, please describe it and how you are generating the data, because those are crucial pieces of information for selecting probability distributions." CreationDate="2011-05-08T19:20:24.620" UserId="919" />
  <row Id="17998" PostId="10462" Text="One further comment.  It is almost an accident that a relationship is linear.  This only happens in general in Newtonian mechanics or when relating one variable to the same variable measured at a different time.  Nature is nonlinear the vast majority of the time.  With sufficient sample sizes, the nonlinearity (e.g., significance of nonlinear spline basis functions) becomes apparent." CreationDate="2011-05-08T19:54:03.747" UserId="4253" />
  <row Id="17999" PostId="10505" Text="I am actually using Twitter at the moment.  It is super noisy though." CreationDate="2011-05-08T19:59:45.497" UserId="4513" />
  <row Id="18000" PostId="10511" Text="@Michael - You've done a real service in sharing this." CreationDate="2011-05-08T20:22:47.040" UserId="2669" />
  <row Id="18001" PostId="10485" Text="I didn't realize that square brackets represent the floor function. Doesn't the [0,1] represent the interval? I'm still confused as to how to answer. The distribution for U is between 0 and 1 and nothing outside of that. So does that mean the distribution for X is between 1 and 2? How do I answer this?" CreationDate="2011-05-08T20:54:27.607" UserId="4401" />
  <row Id="18002" PostId="10407" Score="1" Text="The central difficulty in your problem (and in the birthday problem) is the difficulty of determining the distribution of the maximum (supremum norm) of a multinomial random variable, which involves summing over partitions." CreationDate="2011-05-08T20:57:54.440" UserId="3567" />
  <row Id="18003" PostId="10407" Text="@pul, Some questions: When you say &quot;more often than any other one&quot; do you mean there are *strictly* more counts for the double-probability outcome than all others or *at least as many* as the maximum number of counts over the other outcomes? Is the approximation you're using that you have a sequence $X_1, X_2, \ldots, X_{n+1}$ of independent Poisson r.v.'s where $\lambda_1 = 2t/(n+1)$ and $\lambda_i = t/(n+1)$ for $2 \leq i \leq n+1$? Or some other model? I just want to make sure I understand what you're asking." CreationDate="2011-05-08T21:00:46.233" UserId="2970" />
  <row Id="18004" PostId="10524" Score="2" Text="@IrishStat: Wow, thank you for your answer!  I'm a real novice at statistics, and I do not entirely understand what you've done here. I wish I knew how to reproduce this in R so that I could examine the steps in the process, and look up the various functions being used." CreationDate="2011-05-08T21:18:10.923" UserId="4521" />
  <row Id="18005" PostId="10307" Text="Thank you Jeromy. &#xA;I looked up G power 3, ANOVA repeated measures within-between interaction:&#xA;Only the total sample size is reported assuming equal sample size for the two groups.&#xA;1. How would it work if the sample sizes are slightly different, for example: N1/ N2 = 1.16. &#xA; (2) I have to input correlation between repeated measures. Is this the correlation between repeats after merging the data from the two groups?" CreationDate="2011-05-08T21:28:01.940" UserId="4453" />
  <row Id="18006" PostId="10485" Text="This is as far as I got: When the random variable $X=x, x/n \le U \lt (x+1)/n$. Is that the answer?" CreationDate="2011-05-08T21:41:55.933" UserId="4401" />
  <row Id="18008" PostId="10485" Text="@styfle - pretty much - now you just use my hint to get the probability that $X=x$" CreationDate="2011-05-08T21:48:15.313" UserId="2392" />
  <row Id="18009" PostId="10529" Score="5" Text="If it's possible to quote the passage in question that might help: sometimes nuances get lost in the retelling." CreationDate="2011-05-08T22:27:27.147" UserId="919" />
  <row Id="18010" PostId="10487" Text="@Greg: I must seriously take issue with your dismissive remarks about the role of time series analysis. Unfortunately ignoring the &quot;time series complication&quot; has serious side effects that may either falsely reject or falsely accept hypothesis for models that are under-specified i.e. those that ignore the ARMA COMPONENT IN THE ARMAX MODEL. There is no such thing as a free lunch when it comes to modelling correctly." CreationDate="2011-05-08T22:27:37.503" UserId="3382" />
  <row Id="18011" PostId="10528" Text="How does one adjust the p-value to account for identifying T1 *post hoc* (*i.e.*, noticing it after reviewing the data)?" CreationDate="2011-05-08T22:29:35.337" UserId="919" />
  <row Id="18012" PostId="10485" Text="@probabilityislogic Sorry I don't follow. What do you mean &quot;get the probability that X = x&quot;?" CreationDate="2011-05-08T22:37:51.870" UserId="4401" />
  <row Id="18014" PostId="10524" Score="3" Text="@IrishStat I removed your comment replying to Frank because (1) it is counterproductive, from the point of view of this site, to split discussions off to some other location and (2) it's a bad idea to post your email addresses in such a public, searchable spot." CreationDate="2011-05-08T22:38:49.073" UserId="919" />
  <row Id="18015" PostId="10524" Text="@whuber ok I agree . I will edit my post and eliminate those two things." CreationDate="2011-05-08T22:42:07.977" UserId="3382" />
  <row Id="18016" PostId="10524" Score="2" Text="@Frank: Thanks for the Kudos !" CreationDate="2011-05-08T22:44:45.167" UserId="3382" />
  <row Id="18017" PostId="10307" Text="Also, I would like to know how the non-centrality parameter works here." CreationDate="2011-05-08T23:31:40.313" UserId="4453" />
  <row Id="18018" PostId="10524" Text="@IrishStat: Thanks so much for taking the time to help me out and to further my understanding!  It was quite productive for me!  " CreationDate="2011-05-09T01:05:49.167" UserId="4521" />
  <row Id="18019" PostId="10520" Text="Good work!  So far this seems like the best answer." CreationDate="2011-05-09T01:06:02.383" UserId="2817" />
  <row Id="18021" PostId="10489" Text="For quartz, it's faster to draw a white rectangle over the existing plot and then just redraw the lines.  See `tourr:animate_dist` for an example of this approach." CreationDate="2011-05-09T01:34:31.070" UserId="46" />
  <row Id="18023" PostId="10420" Score="1" Text="&quot;Nonparametric&quot; is a pretty big umbrella, covering everything from rank-sum tests to kernel regression to infinite mixture models, etc. A little clarification/context would buy better answers, I think." CreationDate="2011-05-09T01:57:33.000" UserId="26" />
  <row Id="18024" PostId="10528" Text="all refinements i.e. diagnostic checking leading to augmentation are treated the same as structure suggested while the data was in the drawer. Adding a lag to a predictor variable based upon diagnostic checking simply adds another null hypothesis to the list. This doesn't differ from deleting a non-significant coefficent." CreationDate="2011-05-09T02:20:07.193" UserId="3382" />
  <row Id="18025" PostId="10528" Text="@Irish I don't follow that.  T1 doesn't seem to be a lag: it stipulates a changepoint.  Isn't it easy to find &quot;significant&quot; changes if you're allowed first to inspect the data to choose where the changepoint is?" CreationDate="2011-05-09T03:43:51.063" UserId="919" />
  <row Id="18026" PostId="10538" Text="Thanks. I had problems using improper priors (as you anticipated), but I got the results above using an inverse-gamma(1,1) prior on both the variance parameters." CreationDate="2011-05-09T04:37:44.723" UserId="4528" />
  <row Id="18027" PostId="10409" Text="@aniko -- thanks. Though I fear that Proc Mi has notoriety for not dealing well with big datasets. My dataset has 12 lakh rows. Any views ?" CreationDate="2011-05-09T05:30:29.250" UserId="1763" />
  <row Id="18028" PostId="10399" Text="@rolando2 -- thanks." CreationDate="2011-05-09T05:31:25.053" UserId="1763" />
  <row Id="18029" PostId="10396" Text="@jeromy -- dependents could be any one. Not specified anywhere that children could be the only dependents." CreationDate="2011-05-09T05:32:12.570" UserId="1763" />
  <row Id="18030" PostId="10422" Text="How might I go about calculating covariates? " CreationDate="2011-05-09T05:41:06.293" UserId="4470" />
  <row Id="18031" PostId="10396" Text="@ayush I didn't mean to say that dependents could only be children. I understand that dependents can be a spouse, a parent, a grandparent, or anyone. My point is, what happens if someone has 0 or 5 dependents? At present your dependent variable sounds like a count variable as opposed to a categorical variable with four levels. This may have implications for modelling." CreationDate="2011-05-09T05:41:40.460" UserId="183" />
  <row Id="18032" PostId="10541" Text="Please fix the question by giving more details and clarifying, so I can reopen it." CreationDate="2011-05-09T07:26:56.397" UserId="88" />
  <row Id="18033" PostId="10541" Text="@mbq   for a question there are 5 answer options .There are 5 number of trials .in each trial the answer is choosen from the question. What will be the probability of each answer option at the last. " CreationDate="2011-05-09T07:50:13.307" UserId="4444" />
  <row Id="18034" PostId="10420" Text="@whuber i'm not suggesting that this confusion exists everywhere, but from a number of books and all of my undergraduate lectures, its what I thought until i actually started reading books from outside the field. The one book i remember seeing it in was the SPSS survival guide, but i wouldn't regard that as an authoritive reference. " CreationDate="2011-05-09T09:01:33.283" UserId="656" />
  <row Id="18035" PostId="10532" Text="@MYaseen208 Would you mind clarifying what you mean by &quot;simulation of experimental design data&quot;? Are you looking after some function or R code that possibly generate random data for e.g., a RCBD or Latin square?" CreationDate="2011-05-09T09:31:13.823" UserId="930" />
  <row Id="18036" PostId="10520" Text="@whuber: Nice work -- you're getting close! :-) It is true that the data was artifically generated with a formula plus an error term. But the formula is not exactly the one you've found - you're missing out on a couple of terms. But you're close and you're currently in the lead :-)" CreationDate="2011-05-09T09:46:31.290" UserId="914" />
  <row Id="18037" PostId="10520" Text="Clarification: you're missing out on a couple of terms should have been &quot;you're missing out on more than one term&quot;." CreationDate="2011-05-09T09:53:09.997" UserId="914" />
  <row Id="18038" PostId="10528" Text="@whuber Yes you are right. But think of a Stepwise Forward Process in Regression where you examine the alternative (omitted) variables for &quot;potential incorporation&quot; . This is no different in that sense that you are deciding which &quot;omitted trend&quot; should be added to the model in order to render the resultant error process Gaussian. " CreationDate="2011-05-09T10:56:05.637" UserId="3382" />
  <row Id="18040" PostId="10485" Text="$Pr(X=x)=Pr(\frac{x}{n}\leq U &lt; \frac{x+1}{n})$.  But from the hint, $Pr(a&lt;U&lt;b)=b-a$ so we get $Pr(X=x)=\frac{x+1}{n}-\frac{x}{n}=\frac{1}{n}$." CreationDate="2011-05-09T11:23:43.803" UserId="2392" />
  <row Id="18041" PostId="9546" Text="The Wikipedia page does not really provide guidance on how cross-over designs should be analyzed.  It states that most cross-over designs are fully crossed and that beyond that the analysis is guided by the clinical trial protocol.  Mixed linear models with random effects are also introduced as an option (which may be viable).  I'll take a look at the books. " CreationDate="2011-05-09T11:57:44.167" UserId="196" />
  <row Id="18042" PostId="10407" Text="@cardinal: With &quot;more often&quot; I meant &quot;strictly&quot;, but I would be content also with &quot;as many or more&quot;. I have only $n$ random variables in total, but otherwise your model is correct." CreationDate="2011-05-09T11:57:49.920" UserId="565" />
  <row Id="18043" PostId="10546" Score="2" Text="@user862, the proofs I know off-hand aren't particularly direct. Durrett has a derivation in his probability book which is pretty clean. It takes 3-4 pages, I think; which, if you've read any of his books, is a long proof by his standards. Resnick takes a Little bit more abstract approach in his stochastic processes text. Constructing and wielding bigger hammers lets him get more general results, though. Ross undoubtedly has a treatment in his stochastic processes book, but I'm not that familiar with it. " CreationDate="2011-05-09T11:58:01.667" UserId="2970" />
  <row Id="18044" PostId="10407" Text="@charles: The birthday paradox as I know it looks only for the probability that max &gt;= 2. But I'll take a look, if I can find something more general using this term." CreationDate="2011-05-09T11:59:46.003" UserId="565" />
  <row Id="18045" PostId="10407" Text="Yes, sorry. Thanks for catching that typo. With the other $n+1$'s running around, it naturally slipped out as I was typing. The other reason that could have happened was due to the way I was thinking of the model. An equivalent one is to consider $n+1$ outcomes of equal probability and ask what the chance is that the sum of the first two is greater than each of the remaining individual counts. From there, I was hoping I could take advantage of some symmetry considerations. " CreationDate="2011-05-09T12:04:30.413" UserId="2970" />
  <row Id="18046" PostId="10407" Text="The Poisson model is a good start and I can give you an immediate exponential bound in that case. Since the problem you consider is a conditional form of the Poisson model, then we should be able to connect the two that way. " CreationDate="2011-05-09T12:06:24.437" UserId="2970" />
  <row Id="18047" PostId="10539" Score="2" Text="Do you know anything specific about the distributional properties of your stream? For example, are they, say, positive? Bounded? Any other details you can provide will be helpful. Moments are pretty easy to calculate and store for a stream. There are also previous questions here about directly estimating *quantiles* from a stream, which sounds like what you *really* are trying to do. You might search for, and look through, those. " CreationDate="2011-05-09T12:23:17.573" UserId="2970" />
  <row Id="18048" PostId="5856" Score="1" Text="@gd047 Just found an ungated PDF version of [Displaying Uncertainty With Shading](http://www.mrc-bsu.cam.ac.uk/personal/chris/papers/denstrip.pdf)." CreationDate="2011-05-09T12:24:09.047" UserId="930" />
  <row Id="18049" PostId="10475" Text="I have not used caret but it would have to incorporate penalization (shrinkage), not just cross-validation, to do the job." CreationDate="2011-05-09T12:34:04.680" UserId="4253" />
  <row Id="18050" PostId="10441" Text="@whuber which last bracket? &quot;total variance&quot; means the variance of the total dataset." CreationDate="2011-05-09T12:37:39.560" UserId="4499" />
  <row Id="18052" PostId="10409" Text="I would go ahead and try it before worrying." CreationDate="2011-05-09T12:41:34.333" UserId="279" />
  <row Id="18054" PostId="10445" Text="@Aniko thanks. Unfortunately, I can't guarantee that my partitions are all the same size. I'm running a massively parallel process where I need to calculate the variances of each partition in parallel then combine in the end, but the results/samples from each parallel process are not equal (it's a Monte Carlo simulation of received photons)." CreationDate="2011-05-09T12:43:32.603" UserId="4499" />
  <row Id="18057" PostId="10475" Text="@Frank Harrell the feature selection routines in caret are aimed at algorithms that do not have &quot;built-in&quot; feature selection. At the end of a feature selection run, you get a cross-validated variable importance measure, which you can then use yourself for shrinkage or selection.  You can read more about the process here: http://cran.r-project.org/web/packages/caret/vignettes/caretSelection.pdf" CreationDate="2011-05-09T13:35:17.677" UserId="2817" />
  <row Id="18058" PostId="10422" Text="@Wern: It sounds like you need AT LEAST 2 sets of covariates: a set of dummy variables for hour-of-day, and a variable representing weekend vs. weekday.  In other words, you need a matrix of the following 0/1 variables: H1,H2,H3,H4...H23,Weekday, where H1=1 if hour==1, and is 0 otherwise.  H2=2 if hour==2, and is 0 otherwise, etc.  Weekday= 1 on weekdays and 0 on weekends.  Additionally, you might include: an interaction between weekday and hour, day of week, month of year, holiday, and other interactions. Let me know if you need help doing this in R.  Start with hour and weekday dummies." CreationDate="2011-05-09T13:41:17.440" UserId="2817" />
  <row Id="18059" PostId="10422" Text="@Wern: After re-reading your question, it sounds like you have 15-minute data, so rather than calculating H1-H23, you should calculate I1-I95, where &quot;interval&quot; counts the 15-minute intervals since the beginning of the day.  Also, be sure to omit 1 variable from this set of dummies, e.g. H24 or I96.  What kind of data are you looking at?  Web traffic?  Electricity load?" CreationDate="2011-05-09T13:44:36.940" UserId="2817" />
  <row Id="18060" PostId="10532" Text="@chl: Yes, I'm looking some R codes to generate random data for experimental designs like CRD, RCBD, Latin and other design situations. Any idea. Thanks" CreationDate="2011-05-09T13:59:38.567" UserId="3903" />
  <row Id="18061" PostId="10441" Text="@Gall The expression $[X_1, X_2, \dots, X_n]$ could mean many things (although conventionally it would be a vector): I was looking for a clarification." CreationDate="2011-05-09T13:59:39.973" UserId="919" />
  <row Id="18062" PostId="10528" Text="@Irish That's an illuminating analogy.  If I understand, in effect you contemplate having one potential variable for every period (designating a possible change at that period) and are invoking a systematic process to determine which of these should be included in the model.  This suggests that some standard p-level adjustment procedures, such as the Bonferroni, might reasonably be applied.  Would that be valid?" CreationDate="2011-05-09T14:02:10.187" UserId="919" />
  <row Id="18063" PostId="10363" Text="The problem is you didn't rule *anything* out in your question.  There are infinitely many functions that will fit your data with R=0.99 or whatever metric you choose to use.  You seem to be asking if anyone can guess the particular one you've used.  A useful statistical analysis will take the system that generated the data into account.  " CreationDate="2011-05-09T14:04:45.693" UserId="2260" />
  <row Id="18064" PostId="10520" Text="@knorv You're right; one can do better." CreationDate="2011-05-09T14:13:17.313" UserId="919" />
  <row Id="18065" PostId="10538" Text="Try to use a Uniform with a good range, instead of inverse-gamma. That's what Gelman suggests in his book with Hill. Besides that, if you want to chek your results, check if it makes sense. Do you expect, based on you data and knowledge, that variation among individuals is roughly the same as variation within individuals?" CreationDate="2011-05-09T14:15:12.087" UserId="3058" />
  <row Id="18066" PostId="10552" Text="Is the cross validation giving me the best frontier fitting my dataset ? I mean there not different parameters to decide the best kernels ?" CreationDate="2011-05-09T14:26:09.917" UserId="4531" />
  <row Id="18067" PostId="10528" Text="@whuber: That is precisely what I am suggesting and have implemented in order to find trend/level shift/seasonal pulse breakpoints. " CreationDate="2011-05-09T14:44:37.643" UserId="3382" />
  <row Id="18068" PostId="10528" Text="@Irish I missed that: where is this indicated in your reply?" CreationDate="2011-05-09T14:45:50.653" UserId="919" />
  <row Id="18069" PostId="10420" Score="1" Text="@richie I am grateful for that insight.  I wasn't aware that using &quot;non-parametric&quot; with certain audiences could potentially cause miscommunication.  That's something I'll watch out for in the future." CreationDate="2011-05-09T14:47:28.933" UserId="919" />
  <row Id="18070" PostId="10432" Text="@Colin It would be nice to have a look at the pictures. You've explained your problem with detail, it would be a nice complement. :)" CreationDate="2011-05-09T15:03:19.427" UserId="2902" />
  <row Id="18071" PostId="10467" Text="@Frank Harrell: Of couse, imputing the values makes perfect sense. So much so that I'm now slightly embarassed to have asked the question. Thank you!" CreationDate="2011-05-09T15:22:35.293" UserId="4506" />
  <row Id="18072" PostId="10552" Text="Each will also need some tuning to select the best parameters for that kernel.  This tuning should also be done via cross-validation, WITHIN the overall cross-validation.  SVMs are tricky." CreationDate="2011-05-09T15:30:11.363" UserId="2817" />
  <row Id="18073" PostId="10552" Text="Thx :) Very Helpful" CreationDate="2011-05-09T15:39:52.557" UserId="4531" />
  <row Id="18074" PostId="10487" Text="Sorry, I did not mean to sound dismissive (and have edited to clarify).  My thoughts were more to the point that if you model using only time series without covariates then I would expect certain lags to be very significant (7 days could be likely due to weekend effects), but if those covariates are included, then those lags may become completely unimportant (or not).  I would focus on the covariates first." CreationDate="2011-05-09T15:52:48.600" UserId="4505" />
  <row Id="18075" PostId="10552" Score="1" Text="@404Dreamer_ML I added some example code.  There are other, better ways to do this (i.e. use a separate test set to compare your models after you've fit them, or better yet cross-validation), but this code will at least give you a framework to work from." CreationDate="2011-05-09T15:57:24.697" UserId="2817" />
  <row Id="18076" PostId="10538" Text="As @Manoel indicates, the conjugate prior isn't necessarily a good choice. Try another prior and see what happens. The paper above has BUGS/R code for the half-t prior IIRC. " CreationDate="2011-05-09T16:04:18.917" UserId="26" />
  <row Id="18077" PostId="10538" Score="3" Text="And another suggestion: In general, if you're wondering whether your model/code can perform like you hope it helps to generate some data from your prior and fit it. So here you might take $\sigma^2$ and $\sigma^2_\theta$ orders of magnitude different, generate a bunch of data and see what happens." CreationDate="2011-05-09T16:08:10.580" UserId="26" />
  <row Id="18078" PostId="10539" Text="They represent processing times, so they are positive, and mostly tightly clustered unless there is some sort of technical problem or overload in the system. I'll look for the quantile questions; they might be good enough. Still I'm curious how to go from moments to computing the value associated with an arbitrary percentile. I know storing moments is easy, it's how to use them that I don't know." CreationDate="2011-05-09T16:14:19.393" UserId="4530" />
  <row Id="18079" PostId="10555" Text="That's what I am doing at the moment. However, this quickly becomes unwieldy as you increase the order of the chain. Does anyone know any good sources on the practical side of things? Some approximation methods perhaps?" CreationDate="2011-05-09T16:14:46.140" UserId="4479" />
  <row Id="18080" PostId="10549" Text="You have misunderstood the question. I am looking for the distribution of $X_{i_j}$ as a function of $j, n, \sigma_x, \sigma_y$. I do not actually observe the $X_i$ and $Y_i$, and cannot condition on them. One may assume, w.l.o.g. that $\sigma_x = 1$, and thus consider only the parameters $j, n, \sigma_y$." CreationDate="2011-05-09T16:24:09.640" UserId="795" />
  <row Id="18082" PostId="10561" Text="@mbq, == does partial matching for strings?" CreationDate="2011-05-09T16:50:53.490" UserId="2116" />
  <row Id="18083" PostId="10432" Text="@deps_stats: Thank you for the encouragement, I'm glad to see that people think I've posed my question at least somewhat decently. Unfortunately, my data is probably not something that should be on a public website. It isn't exactly classified, but its definitely not for public consumption. I will try to generate some artificial data that will serve as a good example for the purpose of this question, but I may not have time to do that for a day or so." CreationDate="2011-05-09T16:56:33.570" UserId="2426" />
  <row Id="18084" PostId="10542" Text="(+1) for the links." CreationDate="2011-05-09T16:58:41.827" UserId="2645" />
  <row Id="18085" PostId="10445" Text="Awesome! Thanks. Also, I believe the summation limit on $\bar{X}$ should be 1 to g, but I can't make 1 letter edits." CreationDate="2011-05-09T17:04:01.360" UserId="4499" />
  <row Id="18086" PostId="10558" Text="Well it means I have to learn the tricks beyond X12-ARIMA first, because most of the diagnostic tools are usually hidden in statistical packages. From practical point of view, when I tried monkey style press-the-bottom-get-the-result, I have found that Tramo/Seats works better (judging purely visually by laughing-monkey test) than X12-ARIMA, for stl I usually do the same monkey style work, so what I want is to learn the art of seasonal decomposition. (+1) for the general guides! " CreationDate="2011-05-09T17:08:06.417" UserId="2645" />
  <row Id="18087" PostId="10561" Text="@mpiktas No, but only exact matching is needed here. `grepl` would be OK to do the more general job." CreationDate="2011-05-09T17:13:25.210" UserId="88" />
  <row Id="18088" PostId="10546" Text="Found the proof in Durrett's book. It is explained really clearly. Thanks for the pointers." CreationDate="2011-05-09T17:14:27.080" UserId="862" />
  <row Id="18089" PostId="10445" Text="You are right, I fixed the formula." CreationDate="2011-05-09T17:17:49.307" UserId="279" />
  <row Id="18090" PostId="10537" Text="It help to mention how much data you have. " CreationDate="2011-05-09T17:20:28.230" UserId="1146" />
  <row Id="18092" PostId="10565" Text="Thanks. Although either way I am not able to report, based on the Wilcoxon/Mann-Whitney, that there is a difference between the means of the two groups." CreationDate="2011-05-09T17:43:27.590" UserId="2824" />
  <row Id="18093" PostId="10566" Text="Thanks Andy! That's really helpful :]" CreationDate="2011-05-09T17:50:16.903" UserId="22" />
  <row Id="18098" PostId="10570" Text="@Greg Snow: Thanks for the detailed reply." CreationDate="2011-05-09T18:10:33.393" UserId="2824" />
  <row Id="18099" PostId="10432" Text="@Colin No problem. I'm currently working on market segmentation for a pharmacy wholesaler. The problem is that their data look a lot like signals. I see a lot of similarities between my problem and your problem. I'll try to post some research that might help you." CreationDate="2011-05-09T18:34:55.277" UserId="2902" />
  <row Id="18100" PostId="10432" Text="@deps_stats: I *wish* my data were more like signals. I'm much more comfortable with signal processing than with this sort of statistical modeling/analysis." CreationDate="2011-05-09T18:38:08.847" UserId="2426" />
  <row Id="18101" PostId="10569" Score="1" Text="This really cannot be answered unless you post the actual code and output so we can see how SAS is interpreting the problem." CreationDate="2011-05-09T18:48:44.047" UserId="3489" />
  <row Id="18102" PostId="10528" Text="@whuber: Apologetically I decided not to &quot;avvertise this automatic feature&quot; in AUTOBOX but just allude to it's non-automatic implementation using the &quot;expert statistician&quot; at the controls. " CreationDate="2011-05-09T18:51:30.610" UserId="3382" />
  <row Id="18103" PostId="10571" Text="Please, ask a question." CreationDate="2011-05-09T18:59:45.890" UserId="88" />
  <row Id="18104" PostId="10562" Score="1" Text="What does it mean &quot;most sensitive to finding natural clusters&quot;??" CreationDate="2011-05-09T18:59:49.557" UserId="253" />
  <row Id="18105" PostId="10571" Text="To get an answer you would probably need to give more information about what you want to know. Is it the concept of cointegration, the parametrization and / or the tests used or is it the actual output that you don't understand. In the former case you might want to read up on these things in a textbook such as Johansens own book http://www.math.ku.dk/~sjo/book/book.html  " CreationDate="2011-05-09T19:00:47.393" UserId="4376" />
  <row Id="18106" PostId="10363" Text="@Pete: This is no problem (and I would even call it realistic in a real life setting), just see my answer below." CreationDate="2011-05-09T19:02:29.753" UserId="230" />
  <row Id="18107" PostId="10569" Score="1" Text="Did you add a `class` statement with your categorical variable? If not, SAS is interpreting it as continuous." CreationDate="2011-05-09T19:16:40.563" UserId="279" />
  <row Id="18108" PostId="10576" Text="I have created a function (LRT.POIS) with parameters, nSim, Lambda1, Lambda2, but from here on I am kind of lost." CreationDate="2011-05-09T19:26:21.857" UserId="4098" />
  <row Id="18109" PostId="10567" Text="The angle can be considered unbounded, you just have to consider that, for instance, 520° = 160°. (Note that this is a &quot;quick comment&quot; from another non-statistician, so please take it with care)" CreationDate="2011-05-09T19:50:06.227" UserId="582" />
  <row Id="18110" PostId="9577" Text="@N26 - Because Model 2 is not really considered &quot;nested&quot; from Model 1, I don't know how valid using QIC to assess comparative fit would be. One thought I had was to use multiple imputation techniques to equalize the number of observations, and then one could arguably compare the QIC values for those models. However, some literature, e.g. &quot;Applied Longitudinal Data Analysis for Epidemiology&quot; by Twisk, showed really inconsistent results by using MI techniques on models that have dichotomous response variables. I wish I could help more." CreationDate="2011-05-09T20:01:54.127" UserId="3309" />
  <row Id="18111" PostId="10576" Score="2" Text="I added some example code to show the basic process." CreationDate="2011-05-09T20:19:18.200" UserId="4505" />
  <row Id="18112" PostId="10543" Text="Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation?" CreationDate="2011-05-09T19:07:23.040" UserId="4496" />
  <row Id="18113" PostId="10578" Score="1" Text="(+1) great question @infinity!" CreationDate="2011-05-09T20:51:27.650" UserId="1307" />
  <row Id="18114" PostId="10520" Score="2" Text="@whuber I already gave my +1, but I'd like to add that this is very instructive to read one's approach to such a problem. You're worth the bounty in any manner whatsoever." CreationDate="2011-05-09T20:52:13.500" UserId="930" />
  <row Id="18115" PostId="10578" Text="Is there anything in the [Wikipedia article](http://en.wikipedia.org/wiki/Fisher_information) which is causing problems?  It measures the amount of information that an observable random variable $X$ carries about an unknown parameter $\theta$ upon which the probability of $X$ depends, and its inverse is the Cramer-Rao lower bound on the variance of an unbiased estimator of $\theta$." CreationDate="2011-05-09T20:55:49.177" UserId="2958" />
  <row Id="18116" PostId="10432" Text="@Colin Have you tried any clustering analysis by now? (k-means, hierarchical clustering, partitioning around medoids...)" CreationDate="2011-05-09T21:13:36.740" UserId="2902" />
  <row Id="18117" PostId="10432" Text="@deps_stats: The only one of those I am familiar with is k-means, which I have used in the past for auto-thresholding images. In this case, looking at scatter plots of my data has convinced me that the data does not cluster into well defined populations. At least, not in a way that well reflects the human-defined classification. I've already left my office for the day, but I think scatter plots would be acceptable for me to share, soI'll make some edits tomorrow." CreationDate="2011-05-09T21:22:13.903" UserId="2426" />
  <row Id="18118" PostId="10573" Score="1" Text="+1 for discussing the the real question: how to proper model the relationship: is it linear? angular? etc. Theory might help here." CreationDate="2011-05-09T21:37:18.723" UserId="3058" />
  <row Id="18119" PostId="10564" Text="What's the sample size? It's hard to base a decision of significance only based on p-value." CreationDate="2011-05-09T21:43:08.283" UserId="3058" />
  <row Id="18120" PostId="10564" Text="Another concern here is about the Wilcoxon test (or the Mann-Whitney). I'm no experte on these tests, but I think they assume you are comparing two independent samples and test if their mean differ. Is that assumption valid in your case?" CreationDate="2011-05-09T21:45:04.447" UserId="3058" />
  <row Id="18121" PostId="10573" Score="1" Text="@Manoel You're right about the theory.  The problem is related to [steric hindrance](http://en.wikipedia.org/wiki/Steric_effects#Steric_hindrance): not only does the angle matter, but (fixing an orientation of one of the molecules), the orientation of the other molecule likely matters as much or more.  The approach angle and the relative orientation closely interact: it's a question of geometry.  This simple analysis already provides guidance concerning how the angle of approach might be modeled and indicates it's not going to enter as a simple variable." CreationDate="2011-05-09T22:02:01.393" UserId="919" />
  <row Id="18122" PostId="10371" Score="3" Text="+1 Impressive performance for a mindless, automatic tool!" CreationDate="2011-05-09T22:08:52.917" UserId="919" />
  <row Id="18123" PostId="10549" Text="ok - so basically you need to have $y$ removed from this equation? (integrated out)" CreationDate="2011-05-09T22:11:52.227" UserId="2392" />
  <row Id="18124" PostId="10578" Text="I understand that but I am not really comfortable with it. Like, what exactly does &quot;amount of information&quot; means here. Why does negative expectation of the square of the partial derivative of the density measures this information? Where does the expression come from etc. That's why I am hoping to get some intuition about it." CreationDate="2011-05-09T22:18:29.743" UserId="4101" />
  <row Id="18125" PostId="10549" Text="yes; and it is not independent of Z..." CreationDate="2011-05-09T22:21:19.120" UserId="795" />
  <row Id="18126" PostId="10577" Score="1" Text="(+1) Appealing to conditional densities is not strictly necessary here. Note that $\renewcommand{\Pr}{\mathbb{P}}\Pr(N_t = n) = \Pr(S_n \leq t, S_{n+1} &gt; t) = \Pr(S_n \leq t, S_n + T_{n+1} &gt; t)$ and we need only integrate the joint density over the region $\{(s_n, t_{n+1}): 0 \leq s_n \leq t, t_{n+1} &gt; t - s_n\} \subset \mathbb{R}^2$. Since $S_n$ and $T_{n+1}$ are independent, this is a straightforward undertaking." CreationDate="2011-05-09T22:37:46.090" UserId="2970" />
  <row Id="18127" PostId="10578" Text="@Infinity: The [score](http://en.wikipedia.org/wiki/Score_%28statistics%29) is the proportional rate of change in the likelihood of the observed data as the parameter changes, and so useful for inference.  The Fisher information the variance of the (zero-meaned) score. So mathematically it is the expectation of the square of the first partial derivative of the logarithm of the density and so is the negative of the expectation of the second partial derivative of the logarithm of the density." CreationDate="2011-05-09T23:07:39.263" UserId="2958" />
  <row Id="18128" PostId="10422" Text="Hmm, I think I know what you mean. I'll try it out and let you know - thanks! The data is web traffic. " CreationDate="2011-05-09T23:31:50.450" UserId="4470" />
  <row Id="18129" PostId="10568" Text="As I understand it, AIC and cross validation (at least, leave-one-out cross validation) are asymptotically equivalent, therefore your suggesting that the OP employ cross validation boils down to ignoring the uncross-validated SE and go with the AIC instead." CreationDate="2011-05-10T02:33:52.357" UserId="364" />
  <row Id="18130" PostId="10568" Text="I'd also suggest that your statement that BIC is more parsimonious than AIC needs to be expanded to clarify that BIC will typically choose simpler models than AIC, but that this by no means implies that the models that BIC chooses are &quot;better&quot;. BIC fundamentally assumes that the &quot;true&quot; model is amongst the models under consideration, which is probably a stretch for most circumstances. AIC on the other hand simply attempts to minimize future error prediction (hence the connection to cross validation) and makes no claims regarding the &quot;truth&quot; of the selected model." CreationDate="2011-05-10T02:36:08.643" UserId="364" />
  <row Id="18131" PostId="10588" Text="@Emre Thanks so much for the speedy response!  I am terrible at reading math notation but it looks straightforward enough :) Also - with Matlab I get the best fit when I do regression with this formula: a*x^b+c (99% fit)&#xA;Would there happen to be a more general way to do this that might for that formula? " CreationDate="2011-05-10T02:58:01.447" UserId="4542" />
  <row Id="18132" PostId="10588" Text="@Emre Is there a name for what I am trying to do, in case I need to find out how to do it for another distribution? Is it &quot;confidence interval,&quot; or is that something else?" CreationDate="2011-05-10T03:03:11.850" UserId="4542" />
  <row Id="18133" PostId="10588" Text="Read the linked article to learn how to estimate the rate. What you want to do is called &quot;Transformation of a random variable&quot;." CreationDate="2011-05-10T03:07:29.610" UserId="4479" />
  <row Id="18134" PostId="10583" Score="3" Text="The Heckman procedure has not been extended to any models besides OLS, see http://dx.doi.org/10.1007/s10940-007-9024-4. While I agree with your comment, given the elementary nature of the question it would probably be better to further elaborate on your statement. Also the OP gives no evidence they have a suitable instrument to estimate the selection bias, which would make the Heckman procedure a moot point (although self selection into treatment is still relevant)." CreationDate="2011-05-10T03:20:56.297" UserId="1036" />
  <row Id="18135" PostId="10580" Text="Best. Answer. Ever. (bazinga!)" CreationDate="2011-05-10T03:34:25.827" UserId="2426" />
  <row Id="18136" PostId="10569" Text="Thanks Aniko..I am getting the estimates for class variables categorically separately now." CreationDate="2011-05-10T05:18:24.007" UserId="1763" />
  <row Id="18137" PostId="10569" Text="Sorry to spam here..can somebody please get the profiles &quot;ayush biyani&quot; and &quot;ayush&quot; merged into one ayush biyani and get me the user id , password to me." CreationDate="2011-05-10T05:33:57.800" UserId="1763" />
  <row Id="18138" PostId="10564" Text="@Manoel: Outcome 1 has 209 data points; outcome 2 has 72 data points. Yes, the samples are independent as they are either predictors for outcome 1 or outcome 2 (mutually exclusive)." CreationDate="2011-05-10T06:22:50.803" UserId="2824" />
  <row Id="18139" PostId="10561" Text="@mbq, yes but that assumes that there are multiple columns with the same name. R allows that, but I try to avoid using this feature." CreationDate="2011-05-10T07:50:40.327" UserId="2116" />
  <row Id="18140" PostId="10363" Score="1" Text="Pete: Out of the infinite number of functions that will fit the data with `R^2 &gt;= 0.99` one would like to find the one with the best performance to complexity ratio (and out of sample fit of course). Sorry for not writing that explicity, I thought that was obvious :-)" CreationDate="2011-05-10T08:12:49.690" UserId="914" />
  <row Id="18141" PostId="10570" Text="@Greg Snow: Yes, I included all variables in the model and looked at the p-values of each variable." CreationDate="2011-05-10T08:37:20.340" UserId="2824" />
  <row Id="18142" PostId="10561" Text="@mpiktas Apparently so does the OP, thus this question. " CreationDate="2011-05-10T08:57:56.297" UserId="88" />
  <row Id="18143" PostId="10183" Text="For another approach, see Sizintsev et al., [Histogram-based search: A comparative study](http://www.cse.yorku.ca/~sizints/histogram_search/histogram_search_main.html) (2009, 4p)." CreationDate="2011-05-10T10:03:57.503" UserId="557" />
  <row Id="18145" PostId="10594" Text="**Cross-posted**: (simultaneously) http://math.stackexchange.com/questions/38221/arbitrary-distribution-uniform-distribution" CreationDate="2011-05-10T11:55:49.807" UserId="2970" />
  <row Id="18146" PostId="10592" Text="Do the columns have a natural ordering to them? Or do you want to find a tridiagonal approximation under some (&quot;optimal&quot;) permutation of the columns? I'm assuming that when you say $A = \mathrm{Cov}(X)$ you're speaking of the covariance structure of the features. Can you confirm this?" CreationDate="2011-05-10T12:01:24.277" UserId="2970" />
  <row Id="18147" PostId="10591" Score="2" Text="**Hint**: Delta method. (Also, look up *variance-stabilizing transforms*, which is part of the motivation.)" CreationDate="2011-05-10T12:03:42.743" UserId="2970" />
  <row Id="18148" PostId="10592" Text="No, there's no natural ordering, and yes, covariance of the 100 features. Methods that add up a full covariance matrix, then approximate it, would be &gt;&gt; O(size X); I'm looking for a fast simple approximation, which will necessarily be crude." CreationDate="2011-05-10T12:07:47.473" UserId="557" />
  <row Id="18149" PostId="10592" Text="So, you want a tridiagonal approximation under some (to be determined by the data) permutation, yes?" CreationDate="2011-05-10T12:09:03.283" UserId="2970" />
  <row Id="18150" PostId="10539" Text="Did you see [this question](http://stats.stackexchange.com/questions/7959/algorithm-to-dynamically-monitor-quantiles)?" CreationDate="2011-05-10T12:20:59.133" UserId="2970" />
  <row Id="18151" PostId="10593" Text="@mpiktas Excellent, thanks for your answer! Am I correct in assuming that I could use Matlab's regression toolbox in estimating F?" CreationDate="2011-05-10T12:23:35.883" UserId="4544" />
  <row Id="18152" PostId="10593" Text="@Mike, I am not that familiar with Matlab, so I cannot say. Judging from the toolbox name (regression) probably not. Do not hurry to accept this answer as I do not feel that it is a complete one." CreationDate="2011-05-10T12:27:25.573" UserId="2116" />
  <row Id="18153" PostId="10593" Text="@Mike, and please do not cross-post. Though in this case you got lucky, from the answer here and at math.SE, you got something." CreationDate="2011-05-10T12:31:21.627" UserId="2116" />
  <row Id="18154" PostId="10594" Text="@chl, @whuber, @mbq, I've flagged this question for merge both here and in math.SE, but forgot to inform you in the flag message." CreationDate="2011-05-10T12:34:52.423" UserId="2116" />
  <row Id="18155" PostId="10593" Text="@mpiktas Ah, ok I did not realize that was not allowed... probably should have checked." CreationDate="2011-05-10T12:41:03.810" UserId="4544" />
  <row Id="18156" PostId="10593" Score="2" Text="@Mike, not exactly disallowed, frown upon is the better word." CreationDate="2011-05-10T13:03:25.493" UserId="2116" />
  <row Id="18157" PostId="10422" Text="@Wern if you like my answer (or IrishStat's) feel free to accept one by clicking the green check mark." CreationDate="2011-05-10T13:04:10.673" UserId="2817" />
  <row Id="18158" PostId="10560" Text="Hello Mpiktas, your solution works excellently. many thanks. and thanks to all as well" CreationDate="2011-05-10T13:17:29.283" UserId="4340" />
  <row Id="18159" PostId="10594" Text="Do you have any idea what probability distribution your variable follows? The financial sector has this unfortunate tendency to assume a normal distribution even when it's not appropriate..." CreationDate="2011-05-10T09:48:09.793" UserId="830" />
  <row Id="18160" PostId="10594" Text="@J.M If I use the distribution fitting tool in Matlab I get the best fit with normal distribution. It is less peaked at the center than a normal distribution though." CreationDate="2011-05-10T12:25:51.657" UserId="4544" />
  <row Id="18161" PostId="10594" Text="A QQ plot will give you a quick and intuitive sense of how close the underlying data is to being normally distributed." CreationDate="2011-05-10T12:30:38.297" UserId="2425" />
  <row Id="18162" PostId="10594" Score="1" Text="Mods: If you would like to migrate this to stats.SE we would merge it with the version posted there.  Alternatively, we would be happy to migrate our version here, but the question looks more statistic-y than math-y." CreationDate="2011-05-10T13:31:03.860" UserId="919" />
  <row Id="18163" PostId="10595" Text="@Chris Thanks so much! That is extremely helpful. I am going to try to program it in Matlab in a few hours." CreationDate="2011-05-10T12:13:44.927" UserId="4544" />
  <row Id="18164" PostId="10595" Text="@Chris Can you please explain the difference between the larger `X` and the smaller `x` in F(x)=P(X&lt;x) ? I have a shameful lack of knowledge of mathematical notation." CreationDate="2011-05-10T12:35:53.190" UserId="4544" />
  <row Id="18165" PostId="10595" Text="@Mike The capital $X$ is the random variable, then lowercase $x$ is a fixed real number, so $P(X&lt;x)$ means &quot;the probability that the random variable $X$ takes a value less than the real number $x$&quot;" CreationDate="2011-05-10T13:46:20.603" UserId="2425" />
  <row Id="18166" PostId="10592" Text="added, tried to clarify. If a good (satisficing) permutation could be found in O(Nfeatures), yes, that would do." CreationDate="2011-05-10T14:09:55.670" UserId="557" />
  <row Id="18167" PostId="10597" Text="It sounds to me like you might be interested in geographically weighted regression. http://ncg.nuim.ie/ncg/GWR/whatis.htm" CreationDate="2011-05-10T14:26:57.167" UserId="1036" />
  <row Id="18169" PostId="10560" Text="@Owusu, the expression of gratitude in this site is upvoting of the answers and their acceptance. See the [faq](http://stats.stackexchange.com/faq). Either way, glad to be helpful." CreationDate="2011-05-10T14:51:30.617" UserId="2116" />
  <row Id="18170" PostId="10595" Text="@Chris Could you please take a look at the pictures I added to my question? Am I on the right track?" CreationDate="2011-05-10T15:02:13.657" UserId="4544" />
  <row Id="18171" PostId="10589" Text="&quot;One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is.&quot;  I'm sure you could derive analogues for Fisher information in terms of other transformations of the likelihood, but then you wouldn't get as neat of an expression for the Cramer-Rao lower bound." CreationDate="2011-05-10T15:11:42.110" UserId="3567" />
  <row Id="18172" PostId="10578" Text="It's been closed over there." CreationDate="2011-05-10T15:18:21.257" UserId="26" />
  <row Id="18173" PostId="10598" Text="The &quot;similarity&quot; indexed by correlation refers to the degree to which high/low values in one distribution (relative to that distribution's mean and variance) are paired with high/low values from a second distribution (relative to that second distribution's mean and variance). Since high/low are relative, the two distributions can be very dissimilar in both mean and variance and still obtain a high correlation, so I don't think you can use the correlation as a means of comparing distributions." CreationDate="2011-05-10T15:22:05.763" UserId="364" />
  <row Id="18174" PostId="10598" Text="@Mike, What if I let the scaling vary. Suppose that $x$ and $y$ are multivariate Gaussians, with a simple diagonal covariance. And then, I just test for the two means (which are vectors) are linearly related. i.e., $\mu_1 = s \mu_2$ for any scalar $s$?" CreationDate="2011-05-10T15:24:47.843" UserId="2728" />
  <row Id="18175" PostId="10595" Text="The pictures look identical to ones I generated in R using your data, so I'd say you're on the right track." CreationDate="2011-05-10T15:25:47.990" UserId="2425" />
  <row Id="18177" PostId="10592" Text="@Denis There are approximations when the variables have additional structure, such as when they form a time series or realizations of a spatial stochastic process at various locations.  These effectively rely on assumptions that let us relate the covariance between one pair of variables to that between other pairs of variables, such as between pairs separated by the same time lags.  Calculations can be $O(Nf \log(Nf)$ in such cases.  Absent such a model, I don't see how you can avoid computing all pairwise covariances." CreationDate="2011-05-10T16:10:58.233" UserId="919" />
  <row Id="18178" PostId="10562" Text="i mean that the clustering method should be able to find near optimal clusters. Those with high intra similarity and low external similarity (wrt other clusters)" CreationDate="2011-05-10T16:12:38.637" UserId="4534" />
  <row Id="18179" PostId="10564" Text="@SabreWolfy You have a good sample, so your p-value seems ok. Take a look at my answer below to see if it helps you." CreationDate="2011-05-10T16:13:27.903" UserId="3058" />
  <row Id="18181" PostId="8738" Text="I took this information and have tried to build a model in SAS; you may find the question I've asked about it relevant: &quot;[How do I ensure PROC ARIMA is performing the correct parameterization of input variables?](http://stats.stackexchange.com/q/10604/1583)&quot;" CreationDate="2011-05-10T17:07:25.850" UserId="1583" />
  <row Id="18182" PostId="10604" Text="This is a continuation of a problem I asked a question about; which question is, &quot;[What is the term for a time series regression having more than one predictor?](http://stats.stackexchange.com/q/8738/1583)&quot;" CreationDate="2011-05-10T17:09:25.773" UserId="1583" />
  <row Id="18183" PostId="10515" Text="I have a large simulated loss data (from catastrophic models developed at my school) to calculate some extreme quantiles. previously they used non-parametric methods to do this (find the point estimate for these extreme quantiles and CIs). I am using parametric models (extreme value theory ,fat tail distributions etc) to do it. I have been thinking about the pros and cons for these two methods. I appreciate all your help" CreationDate="2011-05-10T17:18:15.067" UserId="4497" />
  <row Id="18184" PostId="10601" Text="@Manoel: Thanks for the explanation (typo: p-valor). How can education be a good predictor if &quot;their means don't differ&quot; and &quot;on average [...] managers and non-managers have the same level of education&quot;. Surely this means that the distribution of education is the same in the two groups. If that is the case, how can education be a good predictor if it is the same in the two groups?" CreationDate="2011-05-10T17:34:44.797" UserId="2824" />
  <row Id="18185" PostId="10608" Text="I forgot to mention, I am currently exploring the caret package in R." CreationDate="2011-05-10T17:44:04.613" UserId="2252" />
  <row Id="18186" PostId="10515" Text="@ Melon if you add your comment to your question above, you are far more likely to get better responses." CreationDate="2011-05-10T17:49:57.540" UserId="656" />
  <row Id="18187" PostId="10605" Text="Hi Hammer, can you please elaborate why it is not meaningful for discrete random variable? An example is highly appreciated. Thanks," CreationDate="2011-05-10T18:20:53.147" UserId="4551" />
  <row Id="18188" PostId="10613" Score="1" Text="@bezalel, can you give a reference to this paper?" CreationDate="2011-05-10T18:38:52.683" UserId="2116" />
  <row Id="18189" PostId="10607" Score="1" Text="@ayush -1, please consider improving the way you ask questions. I am asking solely because you have asked several questions already which were poorly written. They got edited to make them better, but you cannot expect others to fix your mistakes forever. " CreationDate="2011-05-10T18:44:03.910" UserId="2116" />
  <row Id="18190" PostId="10605" Text="Symmetry is not meaningful for a *categorical* random variable, because there is no natural ordering. Even if there is ordering, usually we don't know anything about spacing, so it is unclear which category should correspond to which &quot;symmetric&quot; one. But there are discrete distributions where symmetry makes sense, eg binomial or poisson distributions." CreationDate="2011-05-10T18:45:36.077" UserId="279" />
  <row Id="18191" PostId="10613" Score="3" Text="This is immediate from the definition of the p-value as the [probability integral transform](http://en.wikipedia.org/wiki/Probability_integral_transform) of the test statistic using the distribution under the null hypothesis.  The conclusion requires that the distribution be continuous.  When the distribution is discrete (or has atoms), the distribution of p-values is discrete, too, and therefore can only approximately be uniform." CreationDate="2011-05-10T18:46:20.550" UserId="919" />
  <row Id="18192" PostId="10613" Text="@mpiktas, the paper is rather specific as it deals with proteome bioinformatics. [Here it is.](http://dx.doi.org/10.1021/pr8011107). However, the statement can be easily separated from the context; see the very beginning of section 3.1." CreationDate="2011-05-10T18:48:04.623" UserId="4552" />
  <row Id="18193" PostId="10613" Score="1" Text="@bezalel, @whuber gave the answer which was something I suspected. I asked the original reference just to be sure that something was not lost in translation. Usually it does not matter whether the article is specific or not, statistical content always shows through :)" CreationDate="2011-05-10T18:56:26.233" UserId="2116" />
  <row Id="18194" PostId="10614" Text="That's very interesting (and so is the related Geary C). This might be close to what I need. " CreationDate="2011-05-10T19:16:59.583" UserId="139" />
  <row Id="18195" PostId="10610" Text="Thank you for your detailed answer. I am not familiar with support vector machines, but I am eager to try it. After a quick read on wikipedia, I am trying linear SVM with the default settings in R on a random subset of my data (it was taking a while with the whole dataset). Do you recommend any good articles or readings for me to familiarize myself with SVMs?" CreationDate="2011-05-10T19:20:58.617" UserId="2252" />
  <row Id="18196" PostId="10611" Text="Alas, PCA was the first thing I attempted to reduce the dimensionality of the data. It did not provide meaningful results, and I was told to be wary of creating transformations that may only be applicable to the training set and not to the future data. Thank you for the book recommendation - though I am familiar with LDA, there are other methods that may prove to be useful. I will see if there is a copy in my building, or at the university library." CreationDate="2011-05-10T19:23:56.087" UserId="2252" />
  <row Id="18197" PostId="10608" Text="@veol caret has a feature selection routine called rfe that you might find useful.  You can read more here, but be careful not to use it on algorythms (such as glmnet) that have built-in feature selection. http://cran.r-project.org/web/packages/caret/vignettes/caretSelection.pdf" CreationDate="2011-05-10T20:09:07.910" UserId="2817" />
  <row Id="18198" PostId="10608" Text="Do you have much of a budget to explore this problem? Is your dataset one that you can publicly share?" CreationDate="2011-05-10T20:10:57.363" UserId="2817" />
  <row Id="18199" PostId="10618" Text="This question definitely does not belong on the stats site, but it's so unclearly stated it is hard to determine where it should be posted.  Consider cleaning it up and posting on SO." CreationDate="2011-05-10T20:21:30.513" UserId="919" />
  <row Id="18200" PostId="10619" Text="This appears really to be a question about fitting a distribution, not a regression question: see http://stats.stackexchange.com/questions/10594/converting-arbitrary-distribution-to-uniform-one ." CreationDate="2011-05-10T20:23:43.180" UserId="919" />
  <row Id="18201" PostId="10619" Text="Well, I need to be able to get F(x) given x through a direct formula (without summation or anything fancy)  so isn't that regression?" CreationDate="2011-05-10T20:38:32.840" UserId="4544" />
  <row Id="18202" PostId="10601" Text="One possibility is because of outliers (skewed distribution). In my example, imagine that there are a few outliers, with a lot of education, but that aren't managers (they are CEO, whatever). This would push up the average educational level of non-managers, but most of non-managers would have low educational level." CreationDate="2011-05-10T20:42:08.733" UserId="3058" />
  <row Id="18203" PostId="10601" Text="By the way, I guess that in the context of one single predictor, logistic regression and ANOVA would produce the same result." CreationDate="2011-05-10T20:43:11.713" UserId="3058" />
  <row Id="18204" PostId="10607" Score="3" Text="@ayush I'll second @mpiktas's comment, and I would recommend you take some time to think about your question(s) and how to improve them (I'm not speaking about english typos, but about their clarity). I also noticed that you only accepted few answers so far. Any reason why? It you feel users missed important points when answering your past questions, you should notify them; otherwise, if one of those responses looks acceptable from your point of view, you should mark it as is." CreationDate="2011-05-10T20:55:36.873" UserId="930" />
  <row Id="18205" PostId="10614" Text="Geary C helps you see how values close to one another cluster, even values in the middle of the distribution. Moran's I helps you see how very high values cluster with very high values, and very low values cluster around very low values. So perhaps you are correct and the simplest and best method is Geary's C. Remember the Geary's C approach is exploratory and will not let you condition your results on other factors. Look at this python module for code to run Geary's C: http://pysal.org/1.1/library/esda/geary.html." CreationDate="2011-05-10T20:57:43.540" UserId="4329" />
  <row Id="18206" PostId="10600" Text="@IrishStat: Thanks for your reply!  I think you're right.  My desire for using CCA comes mostly from the idea that I have two variable sets, and I want to see what relations, if any, exist between them.  And my understanding is that CCA is often used for such purposes, but I have not come across any examples of CCA applied to time series data, like I have.  I've also read multivariate normality is important for CCA, and my variables are most likely poisson distributed.  &#xA;&#xA;I did a regularized CCA, all the canonical correlations came out as nearly 1, which I interpret to mean something is wrong." CreationDate="2011-05-10T21:10:01.887" UserId="4521" />
  <row Id="18207" PostId="10619" Text="@Mike Yes, but this is an extremely specialized application: the curve has to go from 0 to 1 monotonically; the original data are extremely highly correlated; and the meaning of &quot;good&quot; depends on how you are going to use the fit.  (E.g., you might be better off fitting the derivative of this curve rather than the fit itself, or you might need to get a much better fit near $F(x)= 1$ than elsewhere: have you read Taleb's Black Swan tirade?)  If you don't disclose all those things, even the best experts could unknowingly give you formulas that perform poorly." CreationDate="2011-05-10T21:10:06.293" UserId="919" />
  <row Id="18208" PostId="10614" Text="Let me play with these some more. If it seems to do what I need (and I think it will), this sounds like the best answer. " CreationDate="2011-05-10T21:13:09.977" UserId="139" />
  <row Id="18209" PostId="10462" Text="I never interpret logitic regression coefficients in odds-ratio, I find them confusing! In any case, interactions without standardization is pretty annoing to interpret in logistic regressions. My regresssions always have interactions terms, sometimes more than one." CreationDate="2011-05-10T21:20:20.367" UserId="3058" />
  <row Id="18210" PostId="10462" Text="@Frank Harrel, thanks for your anwer. I voted it up, because it's a good answer, with a valid point of view, which enhanced my knowledge of the subject. However, I still desagree with you about standardization, even though it is not always the case the best solution to help interpreting coefficients. And because I desagree with you and I think my question was not properly answered, I'll not mark your answer as an accepted answer. " CreationDate="2011-05-10T21:23:58.183" UserId="3058" />
  <row Id="18212" PostId="1684" Text="@Dirk: Good typography should not draw attention to its self and getting different text from LaTeX and R graphics [is very visible](http://i.imgur.com/4MLQq.png) (EPS output on the left, [Sweave source code](https://gist.github.com/965459)). Admittingly the example is a bit biased as no effort is made to customize the font used by `eps`. The point is that `tikz`  should match the LaTeX document font, font size, letterspacing, kerning, ligaturing, etc. *by default*. If it doesn't, I consider it a bug. " CreationDate="2011-05-10T21:54:52.830" UserId="13" />
  <row Id="18214" PostId="10608" Text="@Zach Thanks for the link. As for the second comment, I was actually hired to work on this project (I'm a university co-op student), so I have ample time to work on this problem. As for the dataset, I am not allowed to share it, but I may be able to link to some articles that are publicly available concerning the type of analysis we do on the data if you want to read about it." CreationDate="2011-05-10T22:09:18.863" UserId="2252" />
  <row Id="18215" PostId="10625" Score="1" Text="@Mike This is one case where Eureqa will likely fail unless some clever pre-processing is done.  The problem is that CDFs like this typically have some error function-like tails and Eureqa has no function sufficiently like Erf.  If you first smooth this &quot;curve&quot; (it's really the EDF of 400K+ points) and ask Eureqa to fit its *derivative* in an appropriate manner it might do a good job.  (I suggest a Box-Cox transformed normal pdf for starters.)  That presumes the OP needs a good fit to the PDF itself, of course." CreationDate="2011-05-10T22:53:17.733" UserId="919" />
  <row Id="18216" PostId="10578" Text="Yes, I flagged it over there. I deleted the cross-posting comment." CreationDate="2011-05-10T23:02:59.517" UserId="2970" />
  <row Id="18217" PostId="10456" Text="@MarkDollar, you should consider accepting this answer." CreationDate="2011-05-10T23:05:05.367" UserId="2970" />
  <row Id="18218" PostId="10626" Text="+1 for the really nice idea of checking against quantiles.  A good way to fit tails is to fit the 1/4, 1/8, 1/16, ... quantiles.  Note that this is not really half-normal: the mode is slightly positive.  Also, it's short tailed compared to the normal.  You can correct that with a Box-Cox transformation or something similar and get a much closer fit overall." CreationDate="2011-05-11T00:27:32.033" UserId="919" />
  <row Id="18219" PostId="10600" Score="1" Text="There are time-series uses for the canonical correlation, although the best one requires that you use a multiple-taper method of spectrum estimation. If all of your correlations are coming out as 1, that could be not that something is wrong, but that your series are highly correlated. This can happen easily if they are all seasonal and in-phase and you haven't prewhitened them." CreationDate="2011-05-11T01:00:04.653" UserId="781" />
  <row Id="18220" PostId="10615" Score="1" Text="look up the AGGREGATE command in the help menu" CreationDate="2011-05-11T01:01:31.923" UserId="1036" />
  <row Id="18221" PostId="10631" Text="Did you not answer your own question?" CreationDate="2011-05-11T01:02:35.340" UserId="1036" />
  <row Id="18222" PostId="10631" Text="@Andy: How come? I don't think so, I mentioned that all of the linear models I know are a very bad fit, and that I am wondering if there is another transformation that can be performed on variables to find a better linear fit." CreationDate="2011-05-11T01:04:58.647" UserId="4560" />
  <row Id="18223" PostId="10631" Score="1" Text="A cubic looking curve would suggest a cubic type transformation! Try fitting an OLS model with higher order polynomial X terms (specifically X^3). There are other potential solutions as well (splines, breaking the X variable into different categories and using dummy variables). A recent post also details exploratory data analysis in examining such relationships, http://stats.stackexchange.com/questions/10363/data-mining-how-should-i-go-about-finding-the-functional-form/10520#10520" CreationDate="2011-05-11T01:15:29.690" UserId="1036" />
  <row Id="18224" PostId="10621" Text="Is this the same as zero inflated Poisson regression or no?" CreationDate="2011-05-11T01:41:19.397" UserId="2310" />
  <row Id="18225" PostId="10621" Text="Probably not, try searching for conditional logistic regression in SAS. There should be info on that, and then I don't think it would be too hard to modify it for Poisson regression.&#xA;&#xA;In addition to GENMOD there is a new procedure called COUNTREG that handles categorical outcomes (of course GLIMMIX does too)." CreationDate="2011-05-11T02:01:27.547" UserId="2310" />
  <row Id="18226" PostId="10631" Score="4" Text="It might be worth remarking that &quot;rises to some point and then falls to a steady level without a consequent rise&quot; is distinctly *non-cubic* behavior.  Cubics don't have horizontal asymptotes.  More (quantitative) details would be helpful." CreationDate="2011-05-11T03:15:39.600" UserId="919" />
  <row Id="18227" PostId="10628" Score="4" Text="This question is loaded with typographical errors.  (For example: the first two equations are identical but shouldn't be; the expressions for the probabilities of the vectors $x$ and $y$ are incorrect; the pdfs are not normalized to unity; the indicator in the integral makes no sense.)  Since you're being quite technical in expression, it would behoove you to check over the text and fix the mistakes so that you are not misunderstood.  Perhaps, though, it would be easier and clearer to express the problem in plainer English and provide some motivation?" CreationDate="2011-05-11T03:22:33.073" UserId="919" />
  <row Id="18228" PostId="10604" Text="if you want to post the SAS output or email it to me I might be able to decipher the output to discover what exactly they are doing with your specification." CreationDate="2011-05-10T18:03:00.307" UserId="3382" />
  <row Id="18230" PostId="10607" Text="@chl-- I have written earlier that I was not able to get user id and password for my earlier id which was required for accepting the answers..but you would notice (if you ) that I consistently say Thanks or any other happy gestures to express my satisfaction..About the clarity of questions..i really didnt understand if I was very vague or not expressing the question..As far as I see, somebody comes and edits the way to phrase the question. Thats english then..aint it ? still I will consider asking the questions the best way I can." CreationDate="2011-05-11T05:01:03.847" UserId="1763" />
  <row Id="18231" PostId="10610" Score="1" Text="@veol A linear SVM is a good place to start, but don't just use the default settings, the key to getting good performance from an SVM lies in choosing a good kernel and in carefully optimising the hyper-parameters (the regularisation parameter and any kernel parameters).  This can be done by minimising the cross-validation error or a bound on generalisation performance (e.g. the &quot;radius-margin&quot; or &quot;span&quot; bounds.  I don't use R, but any decent SVM package should automate this procedure via gradient descent or grid search." CreationDate="2011-05-11T06:49:10.003" UserId="887" />
  <row Id="18232" PostId="10610" Score="1" Text="@veol As to sources of information, as well as the two key papers on SVMs (Boser Guyon and Vapnik and Cortes and Vapnik), there are many good textbooks that cover this sort of thing, such as &quot;learning with kernels&quot; by Scholkopf and Smola, &quot;kernel methods for pattern recognition&quot; by Shawe-Taylor and Christianini or &quot;pattern recognition and machine learning&quot; by Bishop.  Best of luck with your project!" CreationDate="2011-05-11T06:51:52.463" UserId="887" />
  <row Id="18233" PostId="10607" Text="@ayush I have merged your two accounts (and deleted our recent comments). I leave the rest as a reminder. Marking a response as accepted is different from thanking the answerer, because then it means the problem has been solved and the thread might be &quot;freezed&quot; in some sense (at least temporarily, because sometimes there's no definitive answer). (b) Yes, some users take some time to try to improve or reword questions to make them more likely to get a good response. If you're not happy with changes that were suggested, you can roll back to the previous version, at any time." CreationDate="2011-05-11T07:19:45.313" UserId="930" />
  <row Id="18234" PostId="10617" Text="@Greg Sorry for the typo I introduced (should read `\leq` in TeX)!" CreationDate="2011-05-11T07:25:44.110" UserId="930" />
  <row Id="18235" PostId="10610" Text="(+1) nice explanation of the basics. Again where a moment where I'd like to add an answer to my favorites (instead of the whole question)." CreationDate="2011-05-11T07:36:39.890" UserId="264" />
  <row Id="18236" PostId="10591" Text="Thanks Mpikts! I'll be quite honest, I don't really get how to start. What are the main tools and the &quot;start&quot; that I'll need to proof this? " CreationDate="2011-05-11T07:43:59.023" UserId="4496" />
  <row Id="18237" PostId="10561" Text="@mbq thanks for clarifying the confusion with _names(mydat)[c(name)]_ Since the OP has written _I know that the column/variable name &quot;name&quot;_ I was like &quot;wow that is possible ?&quot; and tried to access the names by name ... stupid me ;). May I humbly suggest to edit the question ?" CreationDate="2011-05-11T07:49:44.997" UserId="264" />
  <row Id="18238" PostId="10562" Text="Hi Chet, I would imagine that if there was only one such algorithm, everyone would simply use that.  I suspect that the reason that many different algorithms are still in practice would be because  different algorithms will performance differently on different settings of the problem/datasets.  Hence, I am not sure anyone can help you choose the &quot;best&quot; algorithm for your problem with the information you've provided so far.  Cheers, Tal" CreationDate="2011-05-11T08:16:43.040" UserId="253" />
  <row Id="18239" PostId="10611" Score="1" Text="There's a PDF copy available for download at the link above if that helps." CreationDate="2011-05-11T08:44:26.817" UserId="656" />
  <row Id="18240" PostId="10561" Text="@steffen I'm afraid it would be confusing; but a comment under question is ok. We still don't really know the whole context of OP's problem... :-( It seems the `name` variable is set previously." CreationDate="2011-05-11T08:50:14.423" UserId="88" />
  <row Id="18242" PostId="10642" Text="I need to know essentially which factors have the biggest influence on the response variable. Am I right in thinking that with the first design, I won't be able to distinguish main effects from first-order interactions; whereas with the second design, the main effects from A and E will be free from first-order interactions, while B, C, D will not?" CreationDate="2011-05-11T09:09:52.947" UserId="4370" />
  <row Id="18243" PostId="10641" Score="4" Text="With 2 groups, all common multivariate test statistics (Wilks' $\Lambda$, Roy's largest root, Pillai-Bartlett-trace, Hotelling-Lawley-trace, Hotelling's $T^{2}$) are equivalent: they are functions of the largest eigenvalue of $A^{-1} B$, the only eigenvalue $\neq 0$ with 2 groups. They are strictly monotic functions of each other, and give the same $p$-value. Example: http://pastebin.com/Dq9jczE1" CreationDate="2011-05-11T09:37:10.537" UserId="1909" />
  <row Id="18244" PostId="10640" Text="Is your response binary as well? or is it continuous? what model or functional relationship between the measured variable and the $5$ factors are you interested in estimating?  If you specify the goal of your experiment and what prior information you have, you are much more likely to get a more helpful answer." CreationDate="2011-05-11T09:42:29.300" UserId="2392" />
  <row Id="18245" PostId="1686" Text="+1 Package demands heavily depend on each individual and I think Task Views are the way to start anyone." CreationDate="2011-05-11T11:32:47.637" UserId="144" />
  <row Id="18246" PostId="10640" Text="@probabilityislogic I'm trying to optimize the response time for a website, and I've identified 5 factors believed to have an influence on that response time (such as thread pool size, connection pool size, etc). For each experimental run, I do a load-test and measure the average response times." CreationDate="2011-05-11T13:56:13.500" UserId="4370" />
  <row Id="18247" PostId="10657" Text="You may want to be a bit more specific about your data and what you want to do with it?" CreationDate="2011-05-11T14:30:52.550" UserId="2044" />
  <row Id="18249" PostId="10657" Text="Specificity provided :)" CreationDate="2011-05-11T14:54:56.017" UserId="4402" />
  <row Id="18250" PostId="10659" Text="With a sufficient size of data set, a model isn't needed. Take all the people of a given age (say, 8 years old) and take the percentile within that group. You might find the 30th percentile is .8 meters, the 60th percentile is 1.2 meters, etc. " CreationDate="2011-05-11T15:02:12.527" UserId="3919" />
  <row Id="18251" PostId="10617" Text="Nice reference, +1" CreationDate="2011-05-11T15:38:36.443" UserId="26" />
  <row Id="18252" PostId="10656" Text="Of course you *can* :) Are you asking how to do it in the software, or whether you can draw a particular conclusion, etc?" CreationDate="2011-05-11T15:40:30.730" UserId="26" />
  <row Id="18253" PostId="10608" Text="@veol Oh, I see.  It should be a good learning experience for you then.  It sounded like an interesting analysis, I'd love to see one of the articles." CreationDate="2011-05-11T15:47:26.770" UserId="2817" />
  <row Id="18254" PostId="10663" Text="Thank you so much. I was drowning in statistical tests when I didn't even need them due to the nature of the research. The confusing thing (to me) was that some variables like birth weight can be turned into a nominal variable (low birth weight, normal birth weight). Is it better to keep these variables as values on continuous ranges or as nominal variables? Sorry if its a dumb question, I'm quite new to PCA." CreationDate="2011-05-11T16:11:33.133" UserId="4566" />
  <row Id="18255" PostId="10611" Text="Ah, hadn't noticed that. I went straight to the table of contents. Reading this feels like a good place to start." CreationDate="2011-05-11T16:27:26.423" UserId="2252" />
  <row Id="18256" PostId="10663" Text="@JMS - I liked your 1st paragraph.  The 2nd:  PCA with 27 cases and 6 variables?  Better to simply look at scatterplots, as you said earlier.  (Even correlations can hardly be trusted because with this small sample outliers can have a dramatic distorting effect.)" CreationDate="2011-05-11T16:34:20.417" UserId="2669" />
  <row Id="18257" PostId="10663" Text="@rolando2, what number of variables is more sensible to use for PCA with 27 cases? It may be possible to leave some variables out or combine them into a ratio." CreationDate="2011-05-11T16:50:52.537" UserId="4566" />
  <row Id="18259" PostId="10636" Text="@jkd -- No, we can use different functions under proc logistic. Yes,the interpretation can be different. Like logit says it is log(odds)..probit is related to pth quantile probability distribution. CLL has a different formula. we have link= option in proc logistic which allows us to do so." CreationDate="2011-05-11T18:28:26.770" UserId="1763" />
  <row Id="18260" PostId="7231" Text="Plain KNN is not very good for faces. Faces have been shown to lie on a ~25 dimensional nonlinear manifold of the pictures. " CreationDate="2011-05-11T19:05:08.840" UserId="2860" />
  <row Id="18261" PostId="10568" Text="Thanks! This is a good hint! I'll keep myself busy in studying cross validation! I know the BIC criteria, but unfortunately it doesn't work with glms (I am using R) and I've heard it should not be used if you are not using a OLS method. Might be a rumor... At last a personal question: What would you recommend to use? AIC or Cross Validation? " CreationDate="2011-05-11T19:18:42.820" UserId="4496" />
  <row Id="18262" PostId="10671" Text="Well that is the thing, the placenta is not clearly the cause or effect of any one thing, it is the middle-man between mother and baby. So I am searching for hypotheses to potentially investigate in the future using more samples." CreationDate="2011-05-11T19:27:05.627" UserId="4566" />
  <row Id="18264" PostId="10671" Text="Hi Sarah,If I were you I will try to do &quot;data mining&quot;, which means, I will do the cluster analysis using all possible continuous variables. Then you might find the (dis)similar characteristic (between)within clusters. After that, you might have better idea what you want to do for the next step. Good Luck " CreationDate="2011-05-11T19:43:01.520" UserId="4559" />
  <row Id="18265" PostId="10663" Text="@rolando2 I agree to a point. But though it'll be noisy I still think that one PC will probably be a reasonable composite of area/perimeter/number of vessels and might be useful. I wouldn't include the &quot;responses&quot; in the PCA." CreationDate="2011-05-11T19:47:46.650" UserId="26" />
  <row Id="18266" PostId="10663" Text="@Sarah See my comment to rolando; my thinking was to reduce the measurements on the blood vessels to a single &quot;variable&quot;, the first PC score. Rolando rightly points out that this isn't necessarily that helpful since there are only 3 variables, and the estimate of the scores will be noisy." CreationDate="2011-05-11T19:50:39.530" UserId="26" />
  <row Id="18267" PostId="10663" Score="1" Text="But I should also mention that your responses should be transformed to look roughly normally distributed in PCA, so you wouldn't want to discretize any continuous variables in general. Anyway, the takehome is to plot your data every which way and see what you find and whether it is interesting - it sounds like this isn't the phase of your research where you need to start plowing through models and tests, and your dataset is small enough that you can basically do the exploration by hand." CreationDate="2011-05-11T19:55:46.717" UserId="26" />
  <row Id="18268" PostId="10663" Text="Hello to you both... well said, JMS.  What I said about correlation goes doubly, more so, for PCA, which in effect tries to estimate large numbers of correlations crisscrossing the dataset.  One outlier can have effects on many of the parameters estimated.  Usually I'm considered the one with the lax criteria, but honestly I wouldn't try PCA at all with just 27 cases.  I'd stick with bivariate relationships, or possibly correlating an A with a B while controlling for a C. That said, with the aid of theory you still might want to create an a priori index or two out of your original predictors." CreationDate="2011-05-11T19:58:11.427" UserId="2669" />
  <row Id="18269" PostId="10672" Text="Presumably there is not a recognised optimal solution, as otherwise the [Wikipedia page](http://en.wikipedia.org/wiki/Multi-armed_bandit) would say so and there would not be an experimental [Sourceforge page](http://sourceforge.net/projects/bandit/)" CreationDate="2011-05-11T20:17:38.280" UserId="2958" />
  <row Id="18270" PostId="10674" Score="2" Text="It might help if you showed us some data (invented if necessary) which illustrates your question." CreationDate="2011-05-11T20:18:59.450" UserId="2958" />
  <row Id="18272" PostId="10674" Text="@Henry that is actually exactly what the data looks like: (1, 3), (1, 4), (2.4, 4), ... where x is time in seconds and the y coordinate is [0,200]" CreationDate="2011-05-11T20:25:22.280" UserId="4571" />
  <row Id="18273" PostId="10642" Text="@lindelof Yes, you're right. In the first $2_{\text{III}}^{5-2}$ 8-run design, all main effects will be aliased with a least one first order interaction. I'll try to elaborate more in a response if I find time." CreationDate="2011-05-11T20:29:11.360" UserId="930" />
  <row Id="18275" PostId="10636" Text="The same applies with R, Stata, and any decent statistical software. Logistic regression belongs to GLMs (where G=generalized), whatever the link function (which relates the outcome to a linear combination of explanatory variables) you choose." CreationDate="2011-05-11T21:00:20.077" UserId="930" />
  <row Id="18276" PostId="10629" Text="@Greg Can you add the link to Li &amp; Duan's paper?" CreationDate="2011-05-11T21:02:02.370" UserId="930" />
  <row Id="18277" PostId="10677" Text="@chl: thanks for the suggestions.  I've looked into the caret package, and it seems that rfe is limited to using linear models, random forests, bagged trees, naive bayes, or ROC curves for feature selection.  Is there any kind of variable importance measure that can be computed directly from the SVM?" CreationDate="2011-05-11T21:07:18.737" UserId="2817" />
  <row Id="18278" PostId="10631" Text="Hi,&#xA;You might try to fit a &quot;nonlinear&quot; model. Unfortunately, I can't see the graphic of your variable but I do believe that nonlinear regression will be the best choice for your question. Sincerely," CreationDate="2011-05-11T01:27:16.763" UserId="4559" />
  <row Id="18279" PostId="9425" Text="@Parbury:  First one brief point:  Power is directly related to the Type II error rate, not the Type I. Now forgive me but I'm going to spout off some opinions.  Might what you're doing be seen as &quot;gaming the system,&quot; trying to rig it so that more results get classified as sig. or nonsig.?  These binary judgments are so much less informative, and potentially so much more misleading, than reports of actual effect sizes--in your case, regarding group differences in means.  I like to see people use *p*-values to garnish results rather than to structure them.  End of editorial--argue away!" CreationDate="2011-05-11T22:32:50.897" UserId="2669" />
  <row Id="18280" PostId="10680" Score="2" Text="This is only a very partial answer, so I won't actually post it as an answer. You ARE correct that complex statistics need larger populations; you're referring to the concept of &quot;degrees of freedom&quot;, which is simply the number of independent variables minus one. Also, when doing something like a p-test, your rejection threshold depends on the number of degrees of freedom in addition to the p-value you chose (typically .05)." CreationDate="2011-05-11T17:55:58.897" />
  <row Id="18281" PostId="10680" Score="2" Text="If more people read [Hooke's *How to Tell the Liars from the Statisticians*,](http://www.amazon.com/dp/0824718178) maybe there won't be as many &quot;statistical suckers&quot; as we now have in the world." CreationDate="2011-05-11T18:11:16.577" UserId="830" />
  <row Id="18282" PostId="10680" Score="1" Text="I think you might benefit from asking this question on stats stackexchange -- I flagged the question so maybe it will be moved over there." CreationDate="2011-05-11T18:27:40.307" UserId="4561" />
  <row Id="18283" PostId="10680" Score="1" Text="this should be CW." CreationDate="2011-05-11T18:52:38.817" UserId="3797" />
  <row Id="18284" PostId="10680" Text="I didn't even know we had a forum dedicated to statistical analysis. I'd move the question, If I knew how..." CreationDate="2011-05-11T19:00:16.363" />
  <row Id="18285" PostId="10680" Score="2" Text="@J.M., or [Huff's *How to Lie With Statistics*](http://www.amazon.co.uk/dp/0140136290)" CreationDate="2011-05-11T21:14:19.220" />
  <row Id="18286" PostId="10682" Score="2" Text="The variation of the &quot;Chinese baby&quot; joke I've heard had the expectant mother being afraid that her baby might be considered as an illegal alien and thus deported..." CreationDate="2011-05-11T18:13:40.367" UserId="830" />
  <row Id="18288" PostId="10355" Text="@whuber Thanks for your reply and G'day." CreationDate="2011-05-11T23:08:49.807" UserId="2669" />
  <row Id="18289" PostId="10687" Score="2" Text="What do you mean by &quot;direction&quot;? Have you read the answers to similar questions http://stats.stackexchange.com/search?q=causal ? The short answer is no!" CreationDate="2011-05-11T23:41:56.017" UserId="4376" />
  <row Id="18290" PostId="10687" Score="1" Text="Neither of your suggestions imply causation (or direction).  " CreationDate="2011-05-11T23:43:12.390" UserId="2958" />
  <row Id="18291" PostId="10689" Text="I meant positive correlation or negative by direction. Thankyou for your response and link to similar questions." CreationDate="2011-05-12T00:49:19.733" UserId="4572" />
  <row Id="18292" PostId="10665" Text="Why would you want confidence intervals (or prediction intervals) for the historic period?" CreationDate="2011-05-12T00:59:50.307" UserId="159" />
  <row Id="18293" PostId="10689" Text="Thats quite a crazy X axis in that picture! (But good example!)" CreationDate="2011-05-12T01:47:45.950" UserId="1036" />
  <row Id="18294" PostId="10689" Text="Another.....Cheese, Butter, and Sheep in Bangladesh, versus the S&amp;P500 (R^2=0.99) ......   http://nerdsonwallstreet.typepad.com/my_weblog/files/dataminejune_2000.pdf  .... " CreationDate="2011-05-12T02:02:24.293" UserId="2775" />
  <row Id="18295" PostId="10663" Text="Thank you all so much, this has been extremely helpful, especially the debates!" CreationDate="2011-05-12T02:10:57.133" UserId="4566" />
  <row Id="18296" PostId="10693" Text="-3.15 of what?  By itself, that number is meaningless." CreationDate="2011-05-12T02:25:00.650" UserId="919" />
  <row Id="18297" PostId="10665" Text="@Rob Hyndman: I don't know, but I wasn't 100% sure what Sharma D was asking for, so I answered it both ways.  Out of curiosity, how exactly does your forecast package calculate confidence intervals for arima models?  I recall reading somewhere that is some cases a simulation is used, but it's been a while since I've read the documentation." CreationDate="2011-05-12T02:36:06.170" UserId="2817" />
  <row Id="18298" PostId="10665" Text="Actually, the `forecast()` function just calls the `predict.Arima()` function which uses the psi method as explained by @IrishStat. For linear ARIMA models, there is no need to resort to simulation. For some nonlinear ETS models, simulation is used." CreationDate="2011-05-12T02:41:56.770" UserId="159" />
  <row Id="18299" PostId="6652" Text="Let's analyze this obtuse concept of confidence interval one small step further. If it is true that the 95% CI means that if I were to conduct an experiment 100 times, 95 of these 100 experiments would yield a mean somewhere within that confidence interval given that the null hypothesis is true. I can either reject a null hypothesis or not based on my one experiment and confidence interval. Isn't it essentially saying that, well the true population mean will be within that confidence interval 95% of the time I conduct this experiment given that H0 is true. The 5% of the time it doesn't fall .." CreationDate="2011-05-12T02:58:02.370" UserId="4574" />
  <row Id="18300" PostId="10693" Score="2" Text="@tom I've changed the question to make it less of a homework question. " CreationDate="2011-05-12T03:59:36.887" UserId="183" />
  <row Id="18301" PostId="6652" Text="(continuation) ... within that confidence interval is due to either sampling error or the null hypothesis is simply not true? Doesn't it all refer back to the TRUE POPULATION mean? After all, that is the reason we perform statistical analysis. Someone, please explicate! I'm losing sleep over this. Thanks.  [This is the remainder of Mario's comment, which was converted from a reply format.]" CreationDate="2011-05-12T04:00:34.750" UserId="919" />
  <row Id="18302" PostId="6652" Text="@Mario Your assumption is not true!  Out of 100 repetitions of the experiment, we expect 95 *of the CIs* (not the means) to *contain* the true (but unknown) mean.  The CI is random but the true population mean is not." CreationDate="2011-05-12T04:01:09.007" UserId="919" />
  <row Id="18304" PostId="10691" Text="That's was a very intresting read." CreationDate="2011-05-12T05:56:56.490" UserId="4576" />
  <row Id="18305" PostId="10656" Text="Can you please explain both? :)  I saw this study where they use FA to group people's personalities. After they derive the groups, it seems they use Multiple Response Analysis and crosstab it with gender (males and females). As if they change the factor scores into zeros and ones (just a hunch). Thanks in advance!" CreationDate="2011-05-12T06:15:55.913" UserId="4565" />
  <row Id="18306" PostId="10672" Text="Shouldn't this be on Theoretical Computer Science SE?" CreationDate="2011-05-12T06:31:29.310" UserId="88" />
  <row Id="18307" PostId="10677" Text="@Zach I thought there was an example of using `rfe` with SVM. I have to check then." CreationDate="2011-05-12T06:42:55.990" UserId="930" />
  <row Id="18308" PostId="10685" Text="@rolando2 -- thanks..Yes,I did that..only that improved my accuracy from 12- 15%..But what I am unable to fathom is how is decision tree getting it to 65%..It seems like however good interaction I may try, I would not cross 17%-18%..Can we do only so much as learning the interactions from the Decision Tree ?? No other usage ?" CreationDate="2011-05-12T07:09:55.450" UserId="1763" />
  <row Id="18310" PostId="10629" Text="@all -- can we take this post as an opportunity to discuss the interpretation of probit and CLL in some what detail..I'll state the CLL formula here and start it. log(-log(1-pi)).&#xA; &#xA;&#xA;" CreationDate="2011-05-12T07:13:26.070" UserId="1763" />
  <row Id="18311" PostId="10661" Text="impressive! Thank you!" CreationDate="2011-05-12T08:38:52.610" UserId="4496" />
  <row Id="18312" PostId="10663" Text="@JMS Why is normality important for PCA? I can understand wanting to avoid doing PCA on highly skewed variables, but what if the variables were symmetric but non-normal (e.g. approximately uniformly distributed)? Would that be a problem for PCA?" CreationDate="2011-05-12T09:04:09.317" UserId="3835" />
  <row Id="18313" PostId="10700" Text="does the following link help ? http://archives.math.utk.edu/ICTCM/VOL13/C013/paper.html" CreationDate="2011-05-12T09:04:55.520" UserId="1763" />
  <row Id="18314" PostId="326" Text="are there any sas related good blogs as well ?" CreationDate="2011-05-12T09:07:06.923" UserId="1763" />
  <row Id="18315" PostId="10693" Score="1" Text="@tom- the t test is a way to measure the difference between two groups..specifically the mean of two groups. You can perform this test and see whether the t value is large enough in the t table of significance.In your case the variable to measure would be productivity. " CreationDate="2011-05-12T09:33:30.300" UserId="1763" />
  <row Id="18316" PostId="10700" Text="yeah... I was looking at that earlier... It's good tutorial on how to use solver.. but it's less specific when it comes to ins and outs of ordianl logistic regression" CreationDate="2011-05-12T10:25:17.700" UserId="333" />
  <row Id="18317" PostId="10619" Text="@whuber: I did tell Mike @ m.SE that he needed to have *some* idea about how things are distributed in his application, and that assuming things are &quot;normal&quot; doesn't always work... oh well." CreationDate="2011-05-12T11:16:20.837" UserId="830" />
  <row Id="18318" PostId="9555" Score="1" Text="Apart from Björck, the other neat reference for solving least-squares problems is the (classic) [Lawson/Hanson](http://books.google.com/books?id=ROw4hU85nz8C&amp;pg=PA174) book." CreationDate="2011-05-12T11:49:51.213" UserId="830" />
  <row Id="18319" PostId="9555" Score="1" Text="@J. M., thanks! This was first book I read on this topic, but I forgot the authors and the exact title. Very good book. It also has Fortran code examples, which was very useful for me too. " CreationDate="2011-05-12T12:00:05.197" UserId="2116" />
  <row Id="18321" PostId="10708" Text="Agree. But I need it in excel for demostration purpose. Excel is very transparent.. shows every estimation step." CreationDate="2011-05-12T12:29:51.913" UserId="333" />
  <row Id="18322" PostId="10705" Text="Indeed, the subdiagonal entries aren't even ***stored*** in any of the standard implementations of a $QR$ decomposition." CreationDate="2011-05-12T13:06:17.360" UserId="2970" />
  <row Id="18323" PostId="10709" Text="That was my understanding too, but how should I choose one design over another?" CreationDate="2011-05-12T13:10:47.353" UserId="4370" />
  <row Id="18325" PostId="10665" Text="@Rob Hyndman I see, thanks for the clarification." CreationDate="2011-05-12T13:15:31.900" UserId="2817" />
  <row Id="18326" PostId="10677" Text="@chl♦: Thank you!" CreationDate="2011-05-12T13:16:40.150" UserId="2817" />
  <row Id="18327" PostId="10703" Score="2" Text="This is a little off-topic, but I would urge you to consider whether you really want to be doing this in the first place. If you are implementing this QR  factorization, e.g., as a project in a course on numerical methods, that is fine and good. For (almost) *any* other reason, you should strongly consider using a standard numerical linear algebra package. For example, `numpy`, $R$ and MATLAB are all using LAPACK under the hood. " CreationDate="2011-05-12T13:19:56.203" UserId="2970" />
  <row Id="18328" PostId="10709" Text="If you are interested in only studying main effects, believe that two factor interactions are negligible, and you have no other criteria to meet; then either design will do. It is VERY important that you randomize your experiment; or you could introduce hidden effects. Good luck!" CreationDate="2011-05-12T13:21:52.583" UserId="3805" />
  <row Id="18329" PostId="10689" Score="1" Text="That graph is *obviously* outdated. Either that or there is bias due to lack of surveyors available to sample in the [Gulf of Aden](http://en.wikipedia.org/wiki/Gulf_of_Aden)" CreationDate="2011-05-12T13:23:18.743" UserId="2970" />
  <row Id="18330" PostId="10705" Text="@cardinal: right, for purposes of compactness, one often stores the (scaled) Householder transformations for forming $\mathbf Q$ where the zeroes of the triangular matrix ought to be." CreationDate="2011-05-12T13:29:16.510" UserId="830" />
  <row Id="18331" PostId="10703" Text="To echo @cardinal, much intensive research has been put into optimally implementing QR decomposition for various computing environments via LAPACK; unless you already are competent in numerical linear algebra and believe you can do better than LAPACK, you should stick to a canned routine." CreationDate="2011-05-12T13:31:50.657" UserId="830" />
  <row Id="18332" PostId="9563" Text="However, as a general rule of thumb, downdates are more prone to numerical instability than updates." CreationDate="2011-05-12T13:33:32.077" UserId="830" />
  <row Id="18333" PostId="10711" Text="How would you define &quot;well-roundedness&quot; in the SE context, first of all?" CreationDate="2011-05-12T13:34:41.717" UserId="830" />
  <row Id="18334" PostId="10711" Text="@J.M, I don't know, that's why I would hope that the community would help to flesh out the concepts a little bit more. " CreationDate="2011-05-12T13:53:25.180" UserId="175" />
  <row Id="18335" PostId="10663" Text="Not necessarily. Strictly speaking, Gaussian random variables aren't a requirement for PCA; PCA is just a transformation of the data. PCA does assume that the important relationships are linear, an assumption which is met by Gaussian random variables. The results will be easier to interpret if the data are approximately multivariate Gaussian though. I'd encourage you to ask this as a separate question, if a similar one doesn't already exist; it's a good one &amp; other users will likely have more perspective to add." CreationDate="2011-05-12T14:13:49.997" UserId="26" />
  <row Id="18336" PostId="10672" Text="@mbq since reinforcement learning is a branch of machine learning, I don't think so ;)" CreationDate="2011-05-12T14:18:24.113" UserId="264" />
  <row Id="18337" PostId="10665" Text="Thanks, Information provided was very much helpful. its very much easy in R than in SPSS. Regards" CreationDate="2011-05-12T14:18:43.533" UserId="4427" />
  <row Id="18338" PostId="10656" Text="I think @rolando2 has you covered on the software front. I'm not familiar with multiple response analysis, but certainly it's reasonable to look at the distribution of factor scores overall, across different groups, etc, however you do it." CreationDate="2011-05-12T14:21:21.787" UserId="26" />
  <row Id="18339" PostId="10672" Text="@steffen Sure, the name seemed &quot;tcsy&quot;." CreationDate="2011-05-12T14:48:06.567" UserId="88" />
  <row Id="18340" PostId="10703" Text="This is both a learning exercise and a stepping stone towards building some highly-specialized tools for which no suitable pre-canned solutions exist." CreationDate="2011-05-12T14:56:26.827" UserId="439" />
  <row Id="18341" PostId="10689" Text="That data was before Al Gore became a pirate." CreationDate="2011-05-12T15:01:13.007" UserId="2775" />
  <row Id="18342" PostId="10607" Text="@chl - fair enough. Lemme see to the guidelines now. Thanks " CreationDate="2011-05-12T15:02:07.517" UserId="1763" />
  <row Id="18343" PostId="10628" Text="Many of these expressions don't even render in a readable manner, unfortunately. The last display formula is an example." CreationDate="2011-05-12T15:06:34.340" UserId="2970" />
  <row Id="18345" PostId="10706" Text="Aha - it's Rq not Qr() [I think].  Works brilliantly!  Upvote from me.  " CreationDate="2011-05-12T15:19:12.770" UserId="1991" />
  <row Id="18346" PostId="10703" Text="@aix Even if your routine is highly specialized, you should be able to link to LAPACK. That is unless there are licensing restrictions of some sort you have to deal with." CreationDate="2011-05-12T15:35:52.230" UserId="3265" />
  <row Id="18347" PostId="10685" Score="1" Text="65% vs. 17% is indeed a perplexing difference.  Maybe one of us could figure out more if you posted that tree, or at least a list of the main and interaction effects involved.  Also, is there an element of crossvalidation to your project?" CreationDate="2011-05-12T15:42:18.870" UserId="2669" />
  <row Id="18348" PostId="10714" Score="2" Text="I would say the opposite - it has special meaning (the S3 dispatch you mention), but some old naming conventions caused function names that are not S3 generics to have names that include a `.`. That pertains to names of functions. As for names of (non-function) objects, then yes, there is no special meaning." CreationDate="2011-05-12T15:54:31.907" UserId="1390" />
  <row Id="18349" PostId="10703" Text="@JWM: While we're on the subject, please could you point me to a standard LAPACK routine that would update a QR factorization with new rows? I need to be able to call it from `numpy` and can't compile anything (i.e. it has to be callable from pure Python/numpy). I can't for the life of me find anything... Thanks!" CreationDate="2011-05-12T16:10:42.003" UserId="439" />
  <row Id="18350" PostId="10703" Text="@aix I have used `numpy` very little, but check out this link: [http://www.scipy.org/NumPy_for_Matlab_Users](http://www.scipy.org/NumPy_for_Matlab_Users). Near the bottom of the table there is an example of a QR decomposition. I believe `numpy` uses LAPACK." CreationDate="2011-05-12T16:16:09.643" UserId="3265" />
  <row Id="18351" PostId="10703" Text="@JWM: Thanks for the link. Sadly, there's nothing on that page about *updating* a factorization with new rows." CreationDate="2011-05-12T16:22:03.397" UserId="439" />
  <row Id="18352" PostId="10725" Score="2" Text="@Melon Can you add some references and/or discuss the above claim. As it stands it looks more like a comment than a response." CreationDate="2011-05-12T17:43:55.370" UserId="930" />
  <row Id="18353" PostId="10628" Text="@whuber, Ive corrected two typos so hopefully it is less confusing. Please let me know if anyone notices any other." CreationDate="2011-05-12T17:44:02.390" UserId="2728" />
  <row Id="18354" PostId="10653" Text="Thank you! Your answer is very helpful for solving my overall problem. However, my question isn't fully answered; I'm seeking to learn where the _p_, _d_, and _q_ values of ARIMA(_p_, _d_, _q_) are controlled for the dependent and input variables. If I'm interpreting your answer right, the **AR** component is set for all three variables at once by the `p=12` portion of the `estimate` statement. Also, it appears that you are saying that the **I** component has been assumed to be the same for all three variables (but you haven't said where this is set)." CreationDate="2011-05-12T18:47:20.333" UserId="1583" />
  <row Id="18355" PostId="10721" Score="1" Text="@Matt Parker, good points - If rep is part of the metric, I also think you need to look at how the rep was gained.  A person that gained their rep from small gains on a lot of questions/answers would likely be more well rounded than a person who gained the same rep on a single question that garnered a ton of up-votes." CreationDate="2011-05-12T18:49:51.113" UserId="4048" />
  <row Id="18356" PostId="10653" Text="I'm confused when you say that the SAS procedure assumes the differencing operators are the same for the transfer function and the original series. I'm not sure, but my best guess from reading the SAS documentation is that the differencing operators are set by changing the value in each instance of `(1)` in `identify var=VariableX(1) nlag=24 crosscorr=( VariableY(1) VariableZ(1) );`. Is that not the case?" CreationDate="2011-05-12T18:52:33.610" UserId="1583" />
  <row Id="18357" PostId="10721" Score="3" Text="@Kenny That's a good point and I think the way you've stated it is correct - but I feel like many of the most highly-voted questions and answers are often least indicative of expertise, whereas good answers to really gritty technical questions often have very low votes." CreationDate="2011-05-12T19:14:04.153" UserId="71" />
  <row Id="18358" PostId="10663" Text="@JMS Thanks for the explanation. I don't think it warrants a separate question (for me)." CreationDate="2011-05-12T19:15:05.240" UserId="3835" />
  <row Id="18359" PostId="10721" Text="Check out whuber's answers, for example.  His answers are uniformly excellent, so what differentiates between his highly-rated answers and his 1-voters?  When I look at the former, I see superb answers to questions that I understand; at the latter, superb answers to questions I don't even begin to grok.  So low-voted answers can actually be indicative of deep specialization (but note: if you're curious about what a well-rounded SE user looks like, [this is it](http://stats.stackexchange.com/users/919/whuber?tab=accounts))." CreationDate="2011-05-12T19:16:18.003" UserId="71" />
  <row Id="18361" PostId="10721" Score="1" Text="In fact, @Graviton, it might be wise to seek out multi-site users that you'd consider to be well-rounded and see how they perform in your various metrics." CreationDate="2011-05-12T19:18:55.960" UserId="71" />
  <row Id="18362" PostId="10707" Text="Thanks for your quick reply! Although it didn't contain the solution, it did provide the needed insight. " CreationDate="2011-05-12T19:24:00.383" UserId="4575" />
  <row Id="18363" PostId="10653" Text="@Firefeather It appears to me that the default is to use a Numerator Polynomial with 1 coefficient and with no delay. Since I am not a SAS expert but am reasonably comfortable with time series input and output I can't precisely tell you where to specify that explicitely." CreationDate="2011-05-12T19:33:26.430" UserId="3382" />
  <row Id="18364" PostId="10653" Text="@Firefeather It is well known that a possible model is Y(t)=constant + coefficient*X(t) + a(t) where the ARIMA model for both contain differencing operators. In other words stationarity conditions for the ARIMA structure are not necessarily present in the Transfer Function Model. Again I can't tell you how to explicitely specify both of these components." CreationDate="2011-05-12T19:37:40.483" UserId="3382" />
  <row Id="18365" PostId="10615" Text="@Andy W: Thank you for the hint. As far as I understand - I'm no SPSS pro - AGGREGATE allows to aggregate certain cases and compute statistics over these grouped cases. The actual problem is how to use frequency data as the base of a SPSS computation. Each of the above records is itself somehow aggregated data. I could write a program in another program and prepare - based of the above data - individual cases. But how do I use the aggregates as a basis of an SPSS computation?" CreationDate="2011-05-12T19:38:57.770" UserId="4554" />
  <row Id="18366" PostId="10653" Text="@Firefeather Post your data and we can really get down to business otherwise we are talking about this from 500 miles high." CreationDate="2011-05-12T19:51:40.983" UserId="3382" />
  <row Id="18367" PostId="10712" Text="The dot you see with the is_spam~. command means that there are no explanatory variables. Typically with model formulas, you will see y~x, but if you have no x variable, y~. says to guess at the value of y using no other variables. This is the same as the model $y=\beta_0$" CreationDate="2011-05-12T20:06:05.607" UserId="1118" />
  <row Id="18368" PostId="10663" Text="Glad to help. I'd still encourage you to ask if you et a chance though, so that others can find the answer easier. But no pressure :)" CreationDate="2011-05-12T20:07:41.907" UserId="26" />
  <row Id="18370" PostId="10615" Text="@Stefan, It is still not clear what you are asking to do. If you could provide an example of what your data matrix looks like and what statistics you would like to compute that would help." CreationDate="2011-05-12T20:12:22.090" UserId="1036" />
  <row Id="18371" PostId="10615" Text="@Andy W: Thanks for pointing out! Tried to describe the problem better. Sorry, non-native EN speaker - and non-SPSS user." CreationDate="2011-05-12T20:22:51.600" UserId="4554" />
  <row Id="18372" PostId="10712" Score="3" Text="@Christopher On the contrary, the `.` in the formula tells R to use all variables in the dataframe `spambase.training` (except `is_spam`) as predictors. The model $y = \beta_{0}$ is fit with `y ~ 1`." CreationDate="2011-05-12T20:25:07.533" UserId="1909" />
  <row Id="18373" PostId="10615" Score="1" Text="You can compute the variance directly without reshaping your data, but if you simply want to reshape the data into format c and then use the in-house functions me and Bruce Weaver recently answered a question on the google group SPSS forum, http://groups.google.com/group/comp.soft-sys.stat.spss/browse_thread/thread/3e8944a721133c3d#, that explains how you would do so. (the only difference is that you would need to do three seperate loop and xsave commands and then add the data files back together." CreationDate="2011-05-12T20:36:03.443" UserId="1036" />
  <row Id="18374" PostId="10615" Text="@Andy W: Ah, I see, the other solution unfolds FORMAT F data records into multiple cases of FORMAT C data. Since SPSS is well suited to process FORMAT C data, but not FORMAT F data? That's what I wanted to figure out. Thank you! Could you post an answer instead of a comment? I'm happy to accept it." CreationDate="2011-05-12T20:44:27.157" UserId="4554" />
  <row Id="18375" PostId="10714" Text="At the beginning of a name, a `.` makes the object invisible to `ls()` though." CreationDate="2011-05-12T20:48:19.293" UserId="1909" />
  <row Id="18376" PostId="10708" Score="1" Text="Not a good idea to demonstrate an untrustworthy platform, in my humble opinion.  Plus you will limit yourself quit a bit in terms of features implemented." CreationDate="2011-05-12T21:07:50.950" UserId="4253" />
  <row Id="18377" PostId="10706" Text="Sorry that should have been Rq.  I'll correct the above." CreationDate="2011-05-12T21:09:53.490" UserId="4253" />
  <row Id="18378" PostId="10720" Text="Yes, this is indeed the way to go. in fact it's a little easier to get estimates of the high quantiles, especially if you're willing to tolerate error in the rank of the form $\epsilon n$ where $n$ is the total number of items, and \epsilon &gt; 0$ is some user defined error term" CreationDate="2011-05-12T22:33:15.727" UserId="139" />
  <row Id="18379" PostId="10743" Score="1" Text="I don't see the point of doing these alternating projects. It seems like a waste. Let $\newcommand{\Xm}{\mathbf{X}}\newcommand{\Dm}{\mathbf{D}}\Xm$ be your initial matrix and $\widetilde{\Xm}$ the first projection onto the PSD cone. Let $\Dm = \mathrm{diag}(\widetilde{\Xm})$, i.e., the diagonal matrix of the diagonal entries of $\widetilde{\Xm}$. Then $\hat{\Xm} = \Dm^{-1/2} \widetilde{\Xm} \Dm^{-1/2}$ is a positive semidefinite correlation matrix. No iteration needed." CreationDate="2011-05-12T23:14:18.507" UserId="2970" />
  <row Id="18380" PostId="10743" Score="1" Text="Also, a note on `R`. You can use `pmax(eig$values,0)` instead of the more cumbersome `sapply` call." CreationDate="2011-05-12T23:16:07.210" UserId="2970" />
  <row Id="18381" PostId="10712" Text="I didn't check sources beforehand. Thank you for the correction!" CreationDate="2011-05-12T23:31:13.197" UserId="1118" />
  <row Id="18382" PostId="10740" Text="Cleared a bit; I hope you didn't meant some other GP method." CreationDate="2011-05-12T23:38:11.450" UserId="88" />
  <row Id="18383" PostId="10703" Text="Unfortunately, LAPACK does not support QR updating/downdating; Daniel Kressner however has written (FORTRAN) routines for doing this, see [here](http://www.math.ethz.ch/~kressner/qrupdate.php)." CreationDate="2011-05-13T00:03:30.480" UserId="830" />
  <row Id="18384" PostId="10743" Text="@cardinal Right on both counts.  Thanks for the suggestions." CreationDate="2011-05-13T00:17:00.747" UserId="1815" />
  <row Id="18385" PostId="10732" Score="3" Text="The Wikipedia page at http://en.wikipedia.org/wiki/Quantile#Estimating_the_quantiles_of_a_population provides a nice table of the various definitions." CreationDate="2011-05-13T00:52:15.733" UserId="159" />
  <row Id="18386" PostId="10743" Text="(+1) For the basic idea. But, I don't think the distribution is going to be at all uniform on $[-3,3]$, especially for larger dimensions." CreationDate="2011-05-13T01:14:12.173" UserId="2970" />
  <row Id="18387" PostId="10721" Text="@Matt, I understand the point your put forward, but I'll appreciate if you can provide a nice starting point/framework for us to begin with. " CreationDate="2011-05-13T02:31:42.057" UserId="175" />
  <row Id="18388" PostId="10744" Text="Simple: it's &quot;grading on a curve.&quot;  :-)  I give an algorithm at http://www.quantdec.com/envstats/notes/class_03/probability.htm (&quot;Reading Q-Q plots&quot;)." CreationDate="2011-05-13T03:41:50.213" UserId="919" />
  <row Id="18389" PostId="10709" Text="If he thinks interactions are negligible, should he include them an all? Also, will he have to be careful of his choice of design in order that he has replication? (I'm mainly curious for myself; I only had one lecture on fractional factorial designs.)" CreationDate="2011-05-13T04:32:47.537" UserId="3874" />
  <row Id="18390" PostId="10712" Text="@caracal (+1) Wow I was just wondering how to do this. Thanks!" CreationDate="2011-05-13T04:37:10.597" UserId="3874" />
  <row Id="18391" PostId="10671" Text="@Tu, do you have any suggestion on what kind of clustering (hierarchical, partitional, density, subspace), and what distance metric (Euclidean, mahalanobis, k-means, etc) to use? I am not familiar with cluster analysis. Mahalanobis seems appealing since it corrects for scale in data but I'm not sure if that is important." CreationDate="2011-05-13T05:09:17.253" UserId="4566" />
  <row Id="18392" PostId="10704" Text="Ok, I'll look for DBSCAN and give it a try. Thanks" CreationDate="2011-05-13T06:43:10.880" UserId="4147" />
  <row Id="18393" PostId="10727" Text="@Melon, this [question](http://stats.stackexchange.com/questions/5782/variance-of-a-function-of-one-random-variable) and its answers can be of interest." CreationDate="2011-05-13T07:23:24.060" UserId="2116" />
  <row Id="18394" PostId="10672" Text="@mbq I don't get it. What does &quot;tscy&quot; mean ?" CreationDate="2011-05-13T08:28:12.977" UserId="264" />
  <row Id="18395" PostId="10752" Text="Hi, even I was thinking along similar lines! I have some questions -" CreationDate="2011-05-13T09:16:00.073" UserId="4596" />
  <row Id="18396" PostId="10752" Text="How did you choose order = c(0,1,1) and seasonal = c(0,1,1) for ## Refit the model with lowest BIC?" CreationDate="2011-05-13T09:20:26.363" UserId="4596" />
  <row Id="18397" PostId="10708" Text="Thanks Frank. I will be not demostrating the platform but the ordianl logistic regression concept. Transparent simple example in excel will do that very efficiently... I'd say." CreationDate="2011-05-13T09:21:03.123" UserId="333" />
  <row Id="18398" PostId="10752" Text="@rOOkie if you run the code, the line after `## identify the model....` shows the parameters for the model with minimal BIC. That was the model I then refitted." CreationDate="2011-05-13T09:22:58.723" UserId="1390" />
  <row Id="18399" PostId="10746" Text="Thanks Thomas. When I say transparent. I mean something like excel which allows me to see every step of the calculation." CreationDate="2011-05-13T09:23:10.530" UserId="333" />
  <row Id="18400" PostId="10746" Text="I don't need to see/know/understand source code of the platform I'm using..." CreationDate="2011-05-13T09:24:49.357" UserId="333" />
  <row Id="18401" PostId="10758" Score="3" Text="Mark there is an answer http://stats.stackexchange.com/questions/577/is-there-any-reason-to-prefer-the-aic-or-bic-over-the-other" CreationDate="2011-05-13T09:51:24.567" UserId="2645" />
  <row Id="18402" PostId="10752" Text="Ah okay, got it :)" CreationDate="2011-05-13T10:10:03.363" UserId="4596" />
  <row Id="18403" PostId="10751" Text="I will try this too :)" CreationDate="2011-05-13T10:16:30.690" UserId="4596" />
  <row Id="18404" PostId="10751" Text="@mpiktas By changing The number of values &quot;in the out of sample&quot; don't you change the &quot;best model&quot; ?. Isn't this like the tail wagging the dog ? Also what about tests of necessity and sufficiency for the parameters in the &quot;best model&quot;" CreationDate="2011-05-13T10:59:05.940" UserId="3382" />
  <row Id="18405" PostId="582" Text="@amanda - note that AIC is also equivalent to ML when dimension doesn't change.  Your answer makes it seem like this is only for BIC." CreationDate="2011-05-13T12:10:48.610" UserId="2392" />
  <row Id="18406" PostId="10389" Text="@cardinal - I actually computed the expected values of the maxima as numerical integrals, using Mathematica. But of course there could be other systematic errors in the integration routines.  " CreationDate="2011-05-13T12:20:45.593" UserId="4062" />
  <row Id="18407" PostId="8968" Text="I try to re-word your question: - You have 3 observed vectors: $M$, $N$, and $R$. - You know that these are linear combinations of 4 unobserved quantities: $a$, $b$, $c$ and $err$, and that these are normally distributed and uncorrelated. You also know some of the weights: $$M =1 \cdot a + 1 \cdot b + 0 \cdot c + 0 \cdot err$$ $$N =1 \cdot a + 0 \cdot b + 1 \cdot c + 0 \cdot err$$ $$R = \beta_a \cdot a + \beta_b \cdot b + \beta_c \cdot c + 1 \cdot err$$ where $\beta$s are unkown. - You want to estimate $a$, $b$ and $c$. Is this what you say?" CreationDate="2011-03-30T15:56:40.507" UserId="3911" />
  <row Id="18409" PostId="10751" Text="@IrishStat, what I had in mind for this example, is dividing the data into train and test part, fitting different ARMA models for the train set and using the forecasting performance on the test set to select the &quot;best&quot; model. I do not claim that this will be the **best** model, only best to this criteria. " CreationDate="2011-05-13T12:41:05.177" UserId="2116" />
  <row Id="18410" PostId="7816" Text="+1 I am intrigued by the possibility of using data mining (especially CHAID) for exploring potential effects.  It would be interesting to see an application, such as with the artificial (and small) dataset at http://stats.stackexchange.com/q/10363/919" CreationDate="2011-05-13T12:43:23.043" UserId="919" />
  <row Id="18411" PostId="10369" Text="A recent and related question that popped up on math.SE: http://math.stackexchange.com/questions/38873/distribution-of-the-maximum-of-noisy-random-variables" CreationDate="2011-05-13T12:55:51.290" UserId="2970" />
  <row Id="18412" PostId="10659" Text="@ Nick &amp; @Sabbe Thanks - we'd have to collect a lot more data though, so I don't think this would work.  I should have said this in the question - apologies.  " CreationDate="2011-05-13T13:23:08.993" UserId="1991" />
  <row Id="18413" PostId="10717" Text="Thank you very much for your answer! I updated the entry with a plot of the distributions (wasn't allowed to do so in the first place). The reason for my question was actually the visualisation of the distributions: Although using the 0.05 cutoff three distributions would be considered significantly different, the boxplots still show some considerable overlap between them. I removed the datapoints from each &quot;Group&quot; from &quot;Full&quot; before doing the calculation. The random sampling approach sounds like a very interesting idea!" CreationDate="2011-05-13T14:07:37.197" UserId="4579" />
  <row Id="18414" PostId="579" Text="@mbq - I don't see how cross validation overcomes the &quot;un-representativeness&quot; problem.  If your training data is un-representative of the data you will receive in the future, you can cross-validate all you want, but it will be unrepresentative of the &quot;generalisation error&quot; that you are actually going to be facing (as &quot;the true&quot; new data is not represented by the non-modeled part of the training data).  Getting a representative data set is vital if you are to make good predictions." CreationDate="2011-05-13T14:21:44.363" UserId="2392" />
  <row Id="18415" PostId="583" Text="@john - nice answer.  a possible alternative take on AIC and BIC is that AIC says that &quot;spurious effects&quot; do not become easier to detect as the sample size increases (or that we don't care if spurious effects enter the model),  BIC says that they do.  Can see from OLS perspective as in Raftery's 1994 paper, effect becomes approximately &quot;significant&quot; (i.e. larger model preferred) in AIC if its t-statistic is greater than $|t|&gt;\sqrt{2}$, BIC if its t-statistic is greater than $|t|&gt;\sqrt{log(n)}$" CreationDate="2011-05-13T14:33:54.803" UserId="2392" />
  <row Id="18416" PostId="10735" Text="I see what the intent is and it seems corect to me; however, look in my next answer what happens (I enter a new answer as I cannot write code here)." CreationDate="2011-05-13T15:09:16.180" UserId="892" />
  <row Id="18417" PostId="10768" Score="2" Text="Yes, that's a pretty well-known result and is easy to generalize. (How?) You can also find $g$, at least implicitly. It's essentially an application of the inverse probably transform technique commonly used to generate random variates of an arbitrary distribution. " CreationDate="2011-05-13T15:28:14.540" UserId="2970" />
  <row Id="18418" PostId="10520" Text="@whuber: Because of the way the question was worded (looking for some &quot;aha&quot; thing...a real simple solution), isn't it more likely that the answer is just f = a^2 + (b*c)? Without the 50? Try it. The R^2 just tastes better." CreationDate="2011-05-13T15:57:36.967" UserId="2775" />
  <row Id="18419" PostId="10717" Text="It looks like this comes down to the question of practical vs statistical significance.  Visually you are not seeing much difference between the groups, but the tests are significant.  You need to ask yourself if they are close enough for what you want to do and worry less about tests of very narrow hypotheses.  Consider the 2 distributions, one is uniform from 0 to 0.99, the other is uniform from 0.01 to 1, are they different, the tests can say yes given enough data, but for most practical considerations the differences won't matter." CreationDate="2011-05-13T16:11:26.203" UserId="4505" />
  <row Id="18420" PostId="579" Text="@probabilityislogic Sure; I tried here to explain that *IC based selection may become invalidated by looking from CV perspective; of course CV may be equally easy broken by bad sample selection. However this won't help with selecting better model. " CreationDate="2011-05-13T16:16:08.747" UserId="88" />
  <row Id="18421" PostId="10768" Score="1" Text="@cardinal Please, answer. The team is later complaining that we are not fighting with low answered ratio." CreationDate="2011-05-13T16:25:34.477" UserId="88" />
  <row Id="18422" PostId="10767" Score="1" Text="Is your problem to model the pct% of hours contributed by each position? You might want to try a logratio transform--see http://en.wikipedia.org/wiki/Compositional_data" CreationDate="2011-05-13T16:41:02.423" UserId="3567" />
  <row Id="18424" PostId="10770" Text="Yes studies shows that at some ranges of lambda parameter for poisson LRT may not be a method of choice. I have tried simulations for likelihood ratio test (for both poisson and binomial) but have not tried permutation tests yet. Based on simulation and looking at my real data, I am pretty sure likelihood ratio test will not be valid all the times. Therefore, I would like to try some other tests such as bayesian or classical (Fisher exact test McNamara) for the comparison purpose. I will also give some more thoughts to yours recommendations and will try if i could do it in R. thanks" CreationDate="2011-05-13T16:44:11.620" UserId="4098" />
  <row Id="18425" PostId="10721" Text="I don't have anything specific in mind, unfortunately." CreationDate="2011-05-13T16:49:00.930" UserId="71" />
  <row Id="18426" PostId="10171" Text="Didn't manage to get back in touch with him (Donald Geman)" CreationDate="2011-05-13T17:01:13.233" UserId="3896" />
  <row Id="18427" PostId="579" Text="@mbq - my point is that you seem to &quot;gently reject&quot; IC based selection based on an alternative which doesn't fix the problem.  Cross-validation is good (although computation worth it?), but un-representative data can't be dealt with using a data driven process.  At least not reliably.  You need to have prior information which tells you how it is un-representative (or more generally, what logical connections the &quot;un-representative&quot; data has to the actual future data you will observe)." CreationDate="2011-05-13T17:12:27.210" UserId="2392" />
  <row Id="18428" PostId="10751" Text="@mpkitas Agreed ! It will be the best model to predict the out-of-sample from that 1 origin , given the criteria. That is an awfully &quot;soft&quot; highly qualified determination of what might be the &quot;best model&quot; ." CreationDate="2011-05-13T17:28:44.030" UserId="3382" />
  <row Id="18429" PostId="10767" Text="What variables will you know when you do the forecasting?  Will you know the total hours? project type? contribution from position A?  i.e. what part of the above table will be blank when you do your forecast?" CreationDate="2011-05-13T17:38:58.420" UserId="2392" />
  <row Id="18430" PostId="10753" Text="O yes, I was thinking same, sort of putting into a mixture model. But still, I am wondering how can I choose a proper covariance matrix and set its hyper-parameter. I am playing with a code, and changing parameter without having an idea is embarrassing !" CreationDate="2011-05-13T17:39:05.747" UserId="4581" />
  <row Id="18431" PostId="10520" Text="@bill I did try it, early on.  I trust my explanation provides room for your proposal as well as the two I have included.  There's more than one right answer.  I continued the analysis and included those extra terms because it was clear there are patterns in the residuals and that accounting for them materially reduces the residual variance.  (I will confess that I have spent very little time and attention on this, though: the total time for the initial analysis, including writing the answer, was 17 minutes.  More time often translates to more insight...)" CreationDate="2011-05-13T17:47:47.710" UserId="919" />
  <row Id="18432" PostId="10171" Score="2" Text="@Ugo, there are unclear points in your question that you might want to clarify. In the linked paper there is a pretty clear description of the algorithm. Do you want to ask something specific about this algorithm? Is it the idea of doing feature selection by computing marginal $p$-values that you want an explanation of? Moreover, you should question **Definition 2** in the paper. It is an unsupported claim, which may be a working assumption, but small marginal $p$-values do not in general imply relevance. LAR is, by the way, doing linear regression and is not really for binary responses." CreationDate="2011-05-13T18:08:12.700" UserId="4376" />
  <row Id="18433" PostId="5729" Text="@ steffen -- can u please elaborate on the weight thing..I didnt get it..I dint get the meaning of weights here..does it have any thing to do with weights option in proc logistic..can you please xplain this a bit more ?? thanks in advance." CreationDate="2011-05-13T18:13:41.277" UserId="1763" />
  <row Id="18434" PostId="10767" Text="Most likely total hours will be known (or estimated, anyway). I'm reading up on compositional data analysis now." CreationDate="2011-05-13T18:28:28.890" UserId="4600" />
  <row Id="18435" PostId="10636" Score="1" Text="I think that is what I am trying to say:  that logistic regression belongs to GLMs, rather than GLMs belonging to logistic regression.&#xA;&#xA;Just because SAS has those options for its proc logistic does not mean that it is logistic regression." CreationDate="2011-05-13T18:32:24.133" UserId="2312" />
  <row Id="18436" PostId="579" Text="@probabilityislogic Well, I show that IC sux in comparison to CV, so the fact that CV sux too only makes IC sux even more. But you are right that I've abused the word &quot;representative&quot; in the answer -- I'll try to fix it. And in fact I'm a general denier of model selection =)" CreationDate="2011-05-13T18:45:40.870" UserId="88" />
  <row Id="18437" PostId="579" Text="@mbq - model average ftw!" CreationDate="2011-05-13T19:08:33.370" UserId="2392" />
  <row Id="18438" PostId="10695" Text="I'm not sure I understand your question. The spectra will of course be different if you smooth in the time-domain (before transforming), or in the frequency-domain (after transforming). They're different operations. The only way you would get the same shape of spectra would be if the smoother used in the time-domain was a Fourier transform pair with the frequency-domain smoother (as a transfer function). This is not, by default, the case, and especially not the case when you're using two different smoothers.&#xA;&#xA;Perhaps you can update this with a more explicit statement of your objective?" CreationDate="2011-05-13T19:18:20.323" UserId="781" />
  <row Id="18439" PostId="10762" Text="Thanks for the input.  I ran the model, did not notice any change on the fixed effects, but the variance and std.dev of the wave random effect were 0.  Should I drop the wave random effect?" CreationDate="2011-05-13T20:10:01.613" UserId="4594" />
  <row Id="18440" PostId="10671" Text="You might want to try &quot;Model-based clustering&quot;. If you use R, you could install this package by install.packages(&quot;mclust&quot;). This method will provide the best cluster numbers, which are determined by BIC value. Of course, you could use other clustering methods but I believe this model-based clustering method is the best. " CreationDate="2011-05-13T20:59:39.843" UserId="4559" />
  <row Id="18441" PostId="10747" Text="@Andy W: Thx! I'll pass the information along to my colleague. In fact, my PHP solution does just this ;-)" CreationDate="2011-05-13T21:31:25.043" UserId="4554" />
  <row Id="18442" PostId="8968" Text="Could you slap some dimensions on these? Your notation is really unclear. What do you mean when you say you get two vectors from PCA and not 3? What are you putting *in* to the PCA routine? What sort of estimate are you hoping to get out? The regression you give us might be the best unbiased estimate of R (for squared error), though again I suspect I'm not following your notation..." CreationDate="2011-05-13T21:34:37.783" UserId="26" />
  <row Id="18443" PostId="10687" Text="I think the OP meant &quot;direction&quot; in the sense of positive vs negative correlation, not the direction of any causal relationship between X and Y." CreationDate="2011-05-13T21:39:29.340" UserId="26" />
  <row Id="18444" PostId="8514" Text="Why use uniform breaks? Why not a log scale? Or perhaps in your application you could choose breakpoints that have some meaning (e.g. rural/suburban/urban)." CreationDate="2011-05-13T21:49:10.173" UserId="26" />
  <row Id="18445" PostId="10608" Text="@Zach Unfortunately, I could not find free versions of the papers I was given to read. Here is a paper concerning the imaging of the cells and calculating numerical features, it is close to the type of data I am dealing with: [Link](http://www.ncbi.nlm.nih.gov/pubmed/8570289). " CreationDate="2011-05-13T21:56:51.010" UserId="2252" />
  <row Id="18446" PostId="10786" Text="@Frank thanks for your answer. Could you explain what you mean by 'assigning 3 degrees of freedom to treatment using 3 dummy variables'? " CreationDate="2011-05-13T22:03:10.667" UserId="1381" />
  <row Id="18447" PostId="10784" Text="I don't think the standard deviation of {2,5.33,4.33,3.66} is 1.21; I make it nearer 1.4 (or over 1.6 if you use a bias adjustment)." CreationDate="2011-05-13T22:19:19.227" UserId="2958" />
  <row Id="18449" PostId="10775" Text="Thank you so much for your help!&#xA;&#xA;As I'm not so familiar with the R syntax, this works only for fixed group sizes, doesn't it?&#xA;&#xA;Is there also an existing function for comparing the permuted values with the observed one and giving back the p-value?&#xA;&#xA;" CreationDate="2011-05-13T22:32:06.827" UserId="4602" />
  <row Id="18450" PostId="10777" Text="Thanks a lot for your help! That's a very interesting idea, I didn't know that this is also allowed. Just by chance, do you have a reference in mind who did such a double between/within permutation?" CreationDate="2011-05-13T22:33:09.120" UserId="4602" />
  <row Id="18451" PostId="9838" Text="Consult the King. He will not disappoint.  King, G., Tomz, M., &amp; Wittenberg., J. (2000). Making the Most of Statistical Analyses: Improving Interpretation and Presentation. Am. J. Pol. Sci, 44(2), 347-361.&#xA;" CreationDate="2011-05-13T22:45:16.310" UserId="2347" />
  <row Id="18452" PostId="10784" Text="The general flavor of the replies is that any set of column statistics that are invariant under permutations cannot supply information to deduce anything specific about *individual* rows (although certain bounds can be obtained).  All conventional statistics are permutation invariant, including all moments and order statistics.  If you were to view columns as if they were *time series* you could obtain some permutation-dependent statistics and then be able to recover information about individual rows." CreationDate="2011-05-13T22:53:54.173" UserId="919" />
  <row Id="18454" PostId="10791" Text="Thanks! I'll check that book out. However, since the $\Omega$ is still arbitrary (it could just as well been $[0,2)$ in your example, is the unit interval $[0,1]$ or $[0,1)$ the 'prefered' space which will work in all circumstances? Or are there situations where a more complicated $\Omega$, like $R^2$ would be beneficial?" CreationDate="2011-05-13T22:57:51.880" UserId="4608" />
  <row Id="18455" PostId="10768" Score="1" Text="@mbq: Thanks for your comments, which I understand and respect greatly. Please understand that sometimes considerations of time and/or place do not allow me to post an answer, but do permit a quick comment that can get the OP or other participants off to a start. Be assured that, going forward, if I am able to post an answer, I will do so. Hopefully my continued participation through comments will be ok as well." CreationDate="2011-05-13T23:18:42.517" UserId="2970" />
  <row Id="18456" PostId="10793" Text="Indeed, in the case of a random sample, the empirical CDF converges *uniformly*. [Massart's refinement](http://projecteuclid.org/euclid.aop/1176990746) of the [Dvoretzky-Kiefer-Wolfowitz inequality](http://en.wikipedia.org/wiki/Dvoretzky%E2%80%93Kiefer%E2%80%93Wolfowitz_inequality) shows that this occurs *exponentially* fast in probability." CreationDate="2011-05-13T23:26:41.300" UserId="2970" />
  <row Id="18457" PostId="10794" Score="1" Text="Running an initial look at the relationship outside of the formal analysis will get the d.f. wrong, resulting in inaccurate confidence interval coverage.  A good regression text will describe how dummy variables are constructed.  Good software packages do this automatically if you make them sense that treatment is a categorical variable." CreationDate="2011-05-13T23:30:15.380" UserId="4253" />
  <row Id="18458" PostId="10776" Text="That was a great help. Loved the derivation part. I am wondering if there is any R implementation of this test? since eventually I am going try them in R to see how they work." CreationDate="2011-05-13T23:58:04.337" UserId="4098" />
  <row Id="18459" PostId="10793" Text="@cardinal Wouldn't it also be the case that any reasonable interpolator of the EDF would also have the same convergence properties?" CreationDate="2011-05-14T00:08:23.347" UserId="919" />
  <row Id="18460" PostId="10768" Score="1" Text="@cardinal Some of us are also guilty of the same, for the same reasons..." CreationDate="2011-05-14T00:16:03.197" UserId="919" />
  <row Id="18461" PostId="10793" Text="@whuber, I suppose one could quibble with the word *reasonable*. But, I think my answer in any event would be &quot;no&quot;. Consider any discrete, non-constant random variable and a linear interpolant. Certainly, the uniform convergence would fail in this case." CreationDate="2011-05-14T00:33:46.317" UserId="2970" />
  <row Id="18462" PostId="10794" Text="@Frank Harrell, d.f. is density function? I liked @rolando2 answer, and would add one question to to you, @David: Did you choose the treatment values (for theoretical reasons, for instance) or somehow the treatment is produced by some process beyond your control? " CreationDate="2011-05-14T01:12:43.207" UserId="3058" />
  <row Id="18463" PostId="10793" Text="@cardinal--oh, of course.  I was thinking only of continuous CDFs.  It wouldn't make much sense to interpolate discrete distributions." CreationDate="2011-05-14T01:30:49.203" UserId="919" />
  <row Id="18464" PostId="10793" Text="@whuber: Yes, I figured that was likely the case. (Funny, I didn't get pinged for this. Bug due to the dashes?)" CreationDate="2011-05-14T02:12:45.257" UserId="2970" />
  <row Id="18465" PostId="10793" Text="@whuber: I suspect that for continuous cdfs it must be the case. I'll see if I can come up with a complete and simple argument tomorrow. Intuitively, I would expect that the left-continuous with right-hand limits modification of the empirical cdf could also be shown to satisfy the DKW inequality in the continuous case. That result would then seem to imply your statement." CreationDate="2011-05-14T02:15:32.427" UserId="2970" />
  <row Id="18466" PostId="10794" Text="@Frank - Wouldn't concern for accurate coefficient confidence intervals come after model-building and thus after determining whether a particular X-Y relationship is linear?" CreationDate="2011-05-14T02:28:40.167" UserId="2669" />
  <row Id="18467" PostId="10793" Score="2" Text="I wonder then if there's an interpolated estimator with better finite sample properties, besides trivial cases like $N=2$ or something. Though I imagine it'd require extra assumptions about the true distribution. The efficiency of the ecdf is quite remarkable, really." CreationDate="2011-05-14T02:40:23.463" UserId="26" />
  <row Id="18468" PostId="10776" Text="These are not complicated functions, so they do not require an &quot;implementation&quot;, but merely evaluation of certain functions (such as hyper-geometric sampling pdf and gamma function)." CreationDate="2011-05-14T04:37:32.113" UserId="2392" />
  <row Id="18469" PostId="10671" Text="Thank you for your insight, I will try cluster analysis :]" CreationDate="2011-05-14T06:16:09.833" UserId="4566" />
  <row Id="18470" PostId="10795" Score="2" Text="@Vinh Nguyen, did you read the entire Wikipedia page? There is a section on geometry and on conditional independence for the normal distribution. You can find more in [this book](http://www.amazon.com/Graphical-Models-Oxford-Statistical-Science/dp/0198522193). " CreationDate="2011-05-14T07:18:48.210" UserId="4376" />
  <row Id="18471" PostId="10369" Text="The solution posted on math.SE is conceptually identical to the solution I give below - but formulated using a slightly different terminology. " CreationDate="2011-05-14T07:28:08.807" UserId="4376" />
  <row Id="18472" PostId="10087" Text="Sorry, you're right, I made a typo. Please disregard my previous comment. `qplot(dist, locationAdjustedTonnes, data = df)` does show the data with the linear effect of dist_fr_middle removed. " CreationDate="2011-05-14T08:10:37.397" UserId="827" />
  <row Id="18473" PostId="10752" Score="1" Text="Hmm, interesting: downvote on my answer without courtesy of indicating what is wrong. And the other answer here is effectively the same as mine, but hasn't been downvoted." CreationDate="2011-05-14T10:06:52.347" UserId="1390" />
  <row Id="18474" PostId="10802" Text="@leopino Generally, we would expect a more detailed overview of the suggested toolbox in order to consider it as a *response* rather than a *comment*. You should provide link and explain how the aforementioned TB really answers the question." CreationDate="2011-05-14T10:18:54.443" UserId="930" />
  <row Id="18475" PostId="10789" Score="3" Text="Note that the random variable is the function from $\Omega$ to $\mathbb{R}$, not from $\mathcal{B}$ to $R$. The requirement is that random variable is measurable with respect to $\mathcal{B}$. " CreationDate="2011-05-14T12:03:17.687" UserId="2116" />
  <row Id="18476" PostId="10768" Text="@cardinal OK; I don't want to argue with you, thus I'm only asking. " CreationDate="2011-05-14T12:40:35.547" UserId="88" />
  <row Id="18477" PostId="10768" Text="@mbq: Do not worry. I'm not arguing and I don't think you are either. I understand clearly why you were asking and will be sure to post answers whenever my situation allows. I was just trying to explain that situation and my constraints. :)" CreationDate="2011-05-14T13:19:39.350" UserId="2970" />
  <row Id="18478" PostId="10804" Text="Note that `lprice` and `lcarat` are *log transformed*." CreationDate="2011-05-14T13:22:46.340" UserId="2970" />
  <row Id="18479" PostId="10798" Text="You are sampling to get the posterior distribution of the variable, right? Maybe I'm missing something here, but it doesn't matter if the variable is categorical or not, since the autocorrelation of the MCMC gives to you how fast it will sample the whole space of the distribution, right? Btw, are you using it in regression?" CreationDate="2011-05-14T13:23:02.300" UserId="3058" />
  <row Id="18480" PostId="10791" Score="1" Text="@Leo: Yes. Continuous-time stochastic processes provide an example. The canonical example is Brownian motion, where the sample space $\Omega$ is taken to be $\mathcal{C}$, the space of all continuous real-valued functions." CreationDate="2011-05-14T13:35:25.387" UserId="2970" />
  <row Id="18481" PostId="10776" Text="I am going to give it a try by incorporating above equations in R and see how successful I am, since I am not an expert with R yet :)" CreationDate="2011-05-14T13:42:38.350" UserId="4098" />
  <row Id="18482" PostId="10800" Text="I think it's reasonable - I do it fairly frequently. Don't know of a name for it" CreationDate="2011-05-14T14:19:32.517" UserId="46" />
  <row Id="18483" PostId="10653" Text="Curious why I received a &quot;down vote&quot; without the common courtesy of explaining why I was wrong or in error or what WIKI policy I was unknowingly violating. " CreationDate="2011-05-14T14:35:02.410" UserId="3382" />
  <row Id="18484" PostId="10736" Text="thanks for your answer. I now have a better understanding. Could you elaborate on your last sentence: &quot;the analysis of extremes there is a qualitative difference between distributions with finite moment generating function on a positive interval and those with infinite moment generating function on (0,∞). &quot; ?" CreationDate="2011-05-14T15:29:14.433" UserId="4497" />
  <row Id="18485" PostId="10729" Text="could you elaborate more on your last point?" CreationDate="2011-05-14T15:39:12.860" UserId="4497" />
  <row Id="18486" PostId="10804" Text="(The downvote is *not* mine.)" CreationDate="2011-05-14T15:39:41.647" UserId="2970" />
  <row Id="18487" PostId="10792" Text="@JMS: What about $Z \sim N(0,1)$ ?" CreationDate="2011-05-14T15:45:03.130" UserId="3577" />
  <row Id="18488" PostId="10780" Text="@whuber: Nice pic! Q: What was that drawn in? TikZ?" CreationDate="2011-05-14T15:47:32.147" UserId="3577" />
  <row Id="18489" PostId="10800" Score="1" Text="What exactly are you trying to accomplish? This strikes me as a little odd and, in particular, (highly) **prone to misinterpretation**. How would you appropriately label your axes of the last plot to make it easier to interpret?" CreationDate="2011-05-14T15:49:26.547" UserId="2970" />
  <row Id="18490" PostId="10780" Text="@brianjd: If I recall, @whuber does many of his plots using Mathematica." CreationDate="2011-05-14T15:52:41.210" UserId="2970" />
  <row Id="18491" PostId="10794" Text="Not at all.  It needs to be an explicit part of model building.  Otherwise a simple bootstrap or Monte Carlo simulation would show that when you claim 0.95 confidence interval coverage (or compute P-values) the actual coverage is much less than that.  A classic paper on this topic is @ARTICLE{gra91,&#xA;  author = {Grambsch, P. M. and {O'Brien}, P. C.},&#xA;  year = 1991,&#xA;  title = {The effects of transformations and preliminary tests for&#xA;          non-linearity in regression},&#xA;  journal = Stat in Med,&#xA;  volume = 10,&#xA;  pages = {697-709}&#xA;}&#xA;d.f. = degrees of freedom.  A few stat courses would pay off." CreationDate="2011-05-14T15:58:27.843" UserId="4253" />
  <row Id="18492" PostId="10462" Text="I maintain that such standardization (which is arbitrary - why not standardize by the inter-quintile range?) makes interpretation worse.  Staying close to the original scale usually helps because that's the scale of the experiment.  It is absolutely necessary to understand odds ratios to understand logistic regression." CreationDate="2011-05-14T16:01:40.557" UserId="4253" />
  <row Id="18493" PostId="10026" Text="For sure.  The causal chain has to be understood before modeling even commences." CreationDate="2011-05-14T16:02:55.923" UserId="4253" />
  <row Id="18494" PostId="10792" Score="2" Text="@brianjd: I don't believe so. Let $g$ be a continuous strictly monotonic function (hence, having a well-defined inverse $g^{-1}$) that satisfies your conditions. Then, it must be that $\renewcommand{\Pr}{\mathbb{P}}\Phi^{\alpha}(u) = \Pr(g(Z) \leq u) = \Pr( Z \leq g^{-1}(u)) = \Phi(g^{-1}(u))$ and so $g^{-1}(u) = \Phi^{-1}(\Phi^{\alpha}(u))$. So the inverse is identified fairly explicitly, but not $g$ itself. This is what I meant in my previous comment about $g$ being found *implicitly*." CreationDate="2011-05-14T16:17:46.527" UserId="2970" />
  <row Id="18495" PostId="10804" Text="@cardinal Good point -- I mindlessly copied the variable names. However with log-log axes this makes even less sense ;-)" CreationDate="2011-05-14T16:18:45.790" UserId="88" />
  <row Id="18496" PostId="10805" Score="1" Text="Depends on why it has an upper bound: is it like a binomial?Describe your outcome!" CreationDate="2011-05-14T16:25:53.093" UserId="279" />
  <row Id="18497" PostId="10800" Text="A side note -- if you plot a log-log graph, it is better to use [log axes](http://had.co.nz/ggplot2/coord_trans.html). Otherwise you suggest a linear relation between price and weight, which is in fact exponential." CreationDate="2011-05-14T16:38:17.233" UserId="88" />
  <row Id="18498" PostId="10772" Text="@cb Please note that some angles are not cyclic: the approach angle in the original question is a good example.  If you have a specific follow-up question, why not post it?" CreationDate="2011-05-14T18:03:39.517" UserId="919" />
  <row Id="18499" PostId="10772" Text="@cb Concerning your answer: the dependent variable is not continuous; it is either 0 or 1.  Logistic regression is not the same as applying the logistic transformation to the independent variable and then regressing the {0,1} dependent variable on it." CreationDate="2011-05-14T18:05:16.817" UserId="919" />
  <row Id="18500" PostId="10780" Score="1" Text="@cardinal You're right.  Actually, I use whatever is handy and seems like it will do a good job quickly.  FWIW, here's the code: Module[&#xA;{y, w, a = 0.1, z = 3.24, f = ChiDistribution[7.6], xmin=0, xmax=5},&#xA;y = CDF[f,z];&#xA;w = InverseCDF[f, y^(1/a)];&#xA;Show[&#xA;Plot[{CDF[f, x],CDF[f,x]^a} , {x, xmin, xmax}, Filling-&gt;{1-&gt;{2}}],&#xA;Graphics[{&#xA;Dashed,&#xA;Arrow[{{z,0}, {z,y}}], &#xA;Arrow[{{z,y}, {w,y}}],&#xA;Arrow[{{w,y}, {w,0}}]&#xA;}]&#xA;]&#xA;]" CreationDate="2011-05-14T18:10:32.753" UserId="919" />
  <row Id="18501" PostId="10793" Text="@cardinal This is just a test to confirm that you are getting notified via the &quot;@&quot; mechanism.  If you do get pinged with this, you must be correct about the dashes (and I'll be careful about that in the future)." CreationDate="2011-05-14T18:13:55.717" UserId="919" />
  <row Id="18502" PostId="10736" Score="1" Text="@Melon, sure. First, I have edited &quot;extremes&quot; to &quot;rare events&quot;, which is more appropriate I believe. What I was referring to, in particular, is that you can use the *exponential change of measure* technique if you have a light tail (that is, not a heavy tail) and you need other tools, and get different kinds of results, if the tail is heavy. A  reference is Chapter XIII in [Applied Probability and Queues](http://www.amazon.com/Applied-Probability-Queues-Stochastic-Modelling/dp/0387002111)." CreationDate="2011-05-14T18:25:43.307" UserId="4376" />
  <row Id="18503" PostId="10806" Text="Just the other day someone somewhere on one of the related StackExchange sites was commenting on this scheme as &quot;poor man's Kalman filter&quot;.  If I manage to unearth the link I will add it here." CreationDate="2011-05-14T18:26:52.963" UserId="334" />
  <row Id="18504" PostId="10793" Text="@whuber I got this ping. I'm assuming you got mine that included a colon after your name. So, I'm guessing the dashes confused things. Should we report as a possible bug via meta?" CreationDate="2011-05-14T18:39:18.543" UserId="2970" />
  <row Id="18505" PostId="10772" Text="@whuber: I did not post it as own question as the original question made me curious - it is very closely related to the original question. However, if the policy here is that questions have to be their own thread, regardless how closely they relate to previous questions, I'll do that from now on. OK thanks. should I do that now, or should we leave this here where it once is. Being a chemist and no statistician, I do have follow up questions to your comment..." CreationDate="2011-05-14T18:42:05.380" UserId="4598" />
  <row Id="18506" PostId="10793" Score="1" Text="@JMS, I would guess that even restricting to continuous distributions may not yield a *uniformly* better interpolated estimator. I say this because we can approximate a discrete distribution by a continuous one to arbitrary precision. (Even with continuously differentiable $F$ we may not get such a statement.) It's a good question." CreationDate="2011-05-14T18:42:56.763" UserId="2970" />
  <row Id="18507" PostId="10791" Text="@cardinal, the *sample space* is (usually) $\mathcal{C}$. The *background space* $\Omega$ can be $\mathcal{C}$ with $X$ the identity map. This is the *canonical background space*. The [Kahunen-Loève expansion](http://en.wikipedia.org/wiki/Karhunen%E2%80%93Lo%C3%A8ve_theorem) gives a series expansion of the sample path of Brownian motion based on a sequence of i.i.d. normal random variables, which, in turn, could be obtained as transformations of the Lebesgue measure on $[0, 1]$. Thus it is possible to get Brownian motion using $\Omega = [0,1]$. " CreationDate="2011-05-14T18:43:13.287" UserId="4376" />
  <row Id="18508" PostId="10772" Text="@cb Curiosity is a great reason to post a question!  You can add a link to this question if you want to give a context to your question." CreationDate="2011-05-14T18:43:27.843" UserId="919" />
  <row Id="18509" PostId="10791" Score="1" Text="@NRH, Yes, I should have said *can be taken* instead of *is taken*. I was (somewhat purposefully) trying to brush that under the rug." CreationDate="2011-05-14T18:48:06.327" UserId="2970" />
  <row Id="18510" PostId="10791" Text="My point with the comment above is that there may be many choices of $\Omega$ but none are preferred. My experience is that any specific requirements on the structure of $\Omega$ is more of a nuisance than a benefit. " CreationDate="2011-05-14T18:50:08.770" UserId="4376" />
  <row Id="18511" PostId="10791" Text="@NRH, I'm not quite sure what you mean by *preferred*. In some cases, certain constructions have mathematical and/or pedagogical benefits. For example, by taking $\Omega = \mathcal{C}$ for the construction of Brownian motion, the &quot;proof&quot; that the sample paths are almost surely continuous is a triviality (a *tautology* even!). " CreationDate="2011-05-14T19:10:52.330" UserId="2970" />
  <row Id="18512" PostId="10772" Text="(I'm posting still here, as most of this is again more related to the chemistry of the initial question).&#xA;@whuber: For me steric hindrance doesn't seem the same kind of bound as the limits that concentrations cannot be positive or that a probability must be in [0, 1]. The angle in an actual reaction may never take certain values, but those values are still part of the concept of an angle. Just like the height of a person may never take the value of 10 m, yet I wouldn't say modeling the height of persons needs a bound on the outcome. &#xA;" CreationDate="2011-05-14T19:21:01.460" UserId="4598" />
  <row Id="18513" PostId="10772" Text="Can you explain to me why the steric hindrance makes the angle non-cyclic? If the 0° direction happens to be open for the approach of the educt, then 359° are at least almost as good. If it isn't, 359° won't be much better, either. &quot;Cylcic&quot; seems to me a property of an angle, which angles are allowed is a property of the considered reaction: that would change if another reaction is modeled, the cylcic behaviour not." CreationDate="2011-05-14T19:21:26.657" UserId="4598" />
  <row Id="18514" PostId="10772" Text="Your other comment leads me to a &quot;vocabulary&quot; question. If &quot;logistic regression&quot; implies {0, 1} outcome even if the name has the regression inside, what is the name for modeling posterior probabilities using a logistic function? (I know there's more to logistic regression than my answer - that's why I marked it is a line of thoughts) &#xA;" CreationDate="2011-05-14T19:24:29.820" UserId="4598" />
  <row Id="18515" PostId="10791" Text="@cardinal, in @Leo's comment it was asked if $[0,1]$ is 'preferred' in all circumstances. I am just saying that IMO there is no such $\Omega$ and that it is beneficial not to require anything about $\Omega$ in general. When you want to work with a specific example, there may be reasons to choose one specific $\Omega$. Note, however, that the 'tautology' is sweeping under the rug that the *existence* of Brownian motion as a probability measure on $\mathcal{C}$ needs to be established.      " CreationDate="2011-05-14T19:27:49.267" UserId="4376" />
  <row Id="18516" PostId="10772" Text="@cb The angle describes the approach of one molecule relative to the other.  In the frame of one molecule, the approach direction can be described in spherical coordinates.  The approach direction is the *latitude*, which ranges from an extreme of $-\pi/2$ to $+\pi/2$.  This is a fixed bound and there is no cyclicality to it.  Steric hindrance has nothing to do with this, but it's an additional factor showing that reactivity *must* depend on the internal orientation of the approaching molecule as well as its angle of approach." CreationDate="2011-05-14T19:31:37.270" UserId="919" />
  <row Id="18517" PostId="10791" Text="@NRH, sorry for my slowness of mind today. I had failed to connect the *preferred* reference to @Leo's previous comment. Thanks. Regarding the &quot;tautology&quot; remark, my point was that in other constructions, the a.s. continuity of sample paths is a *theorem*, whereas, under the $\mathcal{C}$-based construction with identity map, it is tautological. Of course, the fact that B.M. can be constructed this way must first be shown. But, that is a little bit beside the point." CreationDate="2011-05-14T19:45:23.670" UserId="2970" />
  <row Id="18518" PostId="10610" Text="Thanks for the great answer, I learned a lot.  How do I do kernel logistic regression in R?" CreationDate="2011-05-14T19:48:40.270" UserId="2817" />
  <row Id="18519" PostId="10809" Text="I agree on the hand-waving part... I am fine with simplifying assumptions on the time-varying form of the regression parameters, as long as they are clearly stated. Of course feel free to reference existing literature." CreationDate="2011-05-14T20:40:24.827" UserId="30" />
  <row Id="18520" PostId="10811" Text="(+1) Let me just add the amazingly detailed an accurate [A History of Probability and Statistics and Their Applications before 1750](http://www.amazon.com/History-Probability-Statistics-Applications-before/dp/0471471291/ref=ntt_at_ep_dpi_2) by Anders Hald as a possible reference." CreationDate="2011-05-14T22:22:27.890" UserId="4376" />
  <row Id="18521" PostId="10791" Text="@cardinal, you are right. It is completely besides the point." CreationDate="2011-05-14T22:24:34.043" UserId="4376" />
  <row Id="18522" PostId="10811" Text="@NRH Thanks for the reminder: I was going to cite that for Bernoulli but forgot.  See the second chapter.  (I was unimpressed with some of the characterizations in the overview chapter, Chapter 1, however, so I do have some misgivings about this book.)" CreationDate="2011-05-14T22:29:40.130" UserId="919" />
  <row Id="18523" PostId="10800" Text="@cardinal, what sort of misinterpretation do you mean? As for y-axis labels, the 3rd graph to me is &quot;log(price) detrended&quot; while the 2nd graph is simply &quot;residuals&quot;.&#xA;" CreationDate="2011-05-14T23:07:58.013" UserId="827" />
  <row Id="18524" PostId="10800" Text="@mbq, yes, fair point." CreationDate="2011-05-14T23:08:27.200" UserId="827" />
  <row Id="18525" PostId="10806" Text="Thanks. I would like to see how this can be reframed as a Kalman filter." CreationDate="2011-05-14T23:10:57.513" UserId="30" />
  <row Id="18526" PostId="10806" Text="I doubt there is a formal derivation, hence the quotes around poor man's version of adaptive parameters." CreationDate="2011-05-14T23:20:20.307" UserId="334" />
  <row Id="18527" PostId="10792" Text="@brianjd - What @cardinal said :) I couldn't even think of a special case for $F_Z$ where you'd get a closed form (not to say there isn't one of course)." CreationDate="2011-05-14T23:28:57.723" UserId="26" />
  <row Id="18528" PostId="10800" Text="@TMOD, let's say, on your last plot, there is a point corresponding to $(-0.4, 3.3)$. Using just the plot, how would one interpret the meaning of that point? (The question is a bit rhetorical, with the point being to think about the information that the plot is, and is not, conveying.)" CreationDate="2011-05-14T23:30:41.753" UserId="2970" />
  <row Id="18530" PostId="10797" Text="(+1) Good description. I think the other reason the former notation is so popular is that it more naturally translates to intuitive notions in applications. (Voted up several hours ago.)" CreationDate="2011-05-14T23:39:16.263" UserId="2970" />
  <row Id="18531" PostId="3542" Text="@Jeromy, after starting _The Grammar of Graphics_ it also occurred to me also that there should exist a _Grammar of Tables_ (and some kind of `gtable` package). I'm also curious about whether or not someone could extend Hadley Wickham's ggplot2 to feature controls and interactivity from the GoG. I know there are some packages out there that add interactivity, but don't know if they have a consistent grammar." CreationDate="2011-05-14T23:49:37.807" UserId="827" />
  <row Id="18532" PostId="10804" Score="1" Text="if you take the difference, then you get log(price/carat)" CreationDate="2011-05-14T23:51:18.127" UserId="2392" />
  <row Id="18533" PostId="10089" Text="What type of data do you have? is it categorical/classification data?  such as &quot;patient did/didn't respond&quot;?  or do you have continuous data? such as &quot;patient has blood pressure of xx&quot;? or a mixture or the two?  also I have never heard of the term &quot;three-arm&quot; before.  What does this mean?" CreationDate="2011-05-15T00:00:56.340" UserId="2392" />
  <row Id="18534" PostId="10797" Text="@cardinal, thanks for making that point more clear. It seems more natural to think and argue in terms of a sum of variables, not a convolution of probability measures, and we would like the mathematics to reflect that. " CreationDate="2011-05-15T00:06:47.567" UserId="4376" />
  <row Id="18535" PostId="10792" Text="@JMS: $Z \sim \mathcal{U}[0,1]$ would be one positive example." CreationDate="2011-05-15T02:38:14.113" UserId="2970" />
  <row Id="18536" PostId="10792" Text="@cardinal I never would have thought of such a rare distribution... but now that you mention it a $Beta(a, 1)$ should work in general, giving you back a $Beta(a\alpha, 1)$." CreationDate="2011-05-15T03:11:26.123" UserId="26" />
  <row Id="18537" PostId="10645" Text="I thought about such an approach, but this would mean that we assume a-priory that 5% of the existing observations are &quot;atypical&quot;, &quot;faulty&quot; etc. Which might not be the case" CreationDate="2011-05-15T07:57:34.420" UserId="1496" />
  <row Id="18538" PostId="10686" Text="From what I read (e.g. http://www.chem.uoa.gr/applets/AppletQtest/Appl_Qtest2.html), Q-Test assumes normal distribution of the data. In my case this assumption is most probably wrong (see the asymmetric shape of the histogram). " CreationDate="2011-05-15T08:04:02.430" UserId="1496" />
  <row Id="18539" PostId="10407" Text="I've found out that the probability for a specific $t$ is a rational polynomial in $n$. The first few are: $2/(n+1)$, $4/(n+1)^2$, $4(3n-1)/(n+1)^3$, $8(3n^2-5n+4)/(n+1)^4$, $8(5n^3-20n^2+45n-26)/(n+1)^5$, $4(15n^4-110n^3+465n^2-742n+388)/(n+1)^6$, $4(21n^5-245n^4+1645n^3-4837n^2+6440n-2992)/(n+1)^7$, $16(7n^6-119n^5+1155n^4-5313n^3+12544n^2-14230n+5972)/(n+1)^8$. Haven't found a way to generalise this for arbitrary $t$, though." CreationDate="2011-05-15T11:28:39.943" UserId="4360" />
  <row Id="18540" PostId="10407" Text="@Thies: Thanks for the effort. Could you please post your comment with proof idea as an answer. If nothing better shows up in the next 5 hours ...." CreationDate="2011-05-15T11:49:10.010" UserId="565" />
  <row Id="18541" PostId="10792" Text="@JMS, Yes, but $\mathrm{Beta}(a,1)$ essentially *are* the transformations of $\mathcal{U}[0,1]$ of the type described in the question. :)" CreationDate="2011-05-15T12:05:55.343" UserId="2970" />
  <row Id="18543" PostId="10407" Text="@Thies, note that the $p_i$ here are all rational with least common denominator of $n+1$. So, of course, in a sample of size $t$, *any* probability of a multinomial of this type will be rational with denominator $(n+1)^t$." CreationDate="2011-05-15T13:15:47.157" UserId="2970" />
  <row Id="18544" PostId="10711" Text="At the risk of sounding like one of the knee-jerk PC police - Let's be inclusive of the women who visit our site.  @Graviton, All of your references are to males." CreationDate="2011-05-15T13:29:17.627" UserId="2669" />
  <row Id="18545" PostId="10407" Text="@cardinal: sure, the $/(n+1)^t$ part is kind of a given. But i didn't expect to find a polynomial of $(t-2)$th order in $n$ for the nominator part that is consistent with every $n$ i checked so far. Perhaps someone can shed some light on why it should look like this?" CreationDate="2011-05-15T13:30:36.817" UserId="4360" />
  <row Id="18546" PostId="10407" Text="@Thies, it's not of order $t-2$ for $t=1$. :) But, more seriously, your point about the numerator is a bit more interesting. There is a lot of symmetry in this problem, so it doesn't strike me a shocking on the surface." CreationDate="2011-05-15T13:38:34.307" UserId="2970" />
  <row Id="18547" PostId="10819" Score="1" Text="(+1) Someone who is contributing to Python tags on both SO and cooking has some exotic tastes in food :-)  I hear it tastes like chicken." CreationDate="2011-05-15T14:01:53.430" UserId="919" />
  <row Id="18548" PostId="10716" Text="Re the edit: It's hard to know whether you're correct or not because there's nothing in your reply that describes what role $f$ plays: you never use it nor do you say how it would be applied in this problem." CreationDate="2011-05-15T14:04:27.870" UserId="919" />
  <row Id="18549" PostId="10721" Score="1" Text="@KennyPeanuts, to complement @Matt's point above, there seem to also be an appreciable number of users on SE sites that respond to a tremendous number of questions, yet yielding few upvotes. That's not usually indicative of expertise or well-roundedness, either. (Though, it *is* one of the easiest strategies to garner &quot;reputation&quot;)." CreationDate="2011-05-15T14:09:44.130" UserId="2970" />
  <row Id="18550" PostId="10407" Text="@cardinal: Why don't you post your exponential bound. Within the next 2 hours I'll pick the best available answer..." CreationDate="2011-05-15T14:13:25.970" UserId="565" />
  <row Id="18551" PostId="10769" Text="I can reproduce the problem. The reason seems really to be numerical instability, as changing C0=matrix(0) to C0=matrix(1E-12) allowed me to get results very close to yours." CreationDate="2011-05-15T14:14:57.700" UserId="4575" />
  <row Id="18552" PostId="10407" Text="@pul: It really only works well for [$t = \Omega(n \log n)$](http://en.wikipedia.org/wiki/Big_O_notation#Family_of_Bachmann.E2.80.93Landau_notations) and I'm not sure that case is particularly interesting to you." CreationDate="2011-05-15T14:23:24.153" UserId="2970" />
  <row Id="18553" PostId="10721" Text="@cardinal, good point.  I think this is especially true since there is a general reluctance to down-vote answers that are simply mediocre (and I am not saying this is actually a bad thing). I am not sure how you would effectively and easily distinguish between a user that answered a lot of questions with so-so answers and thus gained their rep in small steps but is not truly well rounded and a truly well rounded user that gained their rep via great answers to a diversity of very specialized questions that don't generate many up votes (as @Matt points out).   " CreationDate="2011-05-15T14:39:14.897" UserId="4048" />
  <row Id="18554" PostId="10823" Text="Could you please tell me how to find $\sigma^2$?" CreationDate="2011-05-15T15:10:04.113" UserId="565" />
  <row Id="18555" PostId="10407" Text="@cardinal: What probabilities for the more likely one to be first do get for such $t$? (And what would you comment to leonbloy's answer?)" CreationDate="2011-05-15T15:11:48.280" UserId="565" />
  <row Id="18556" PostId="10825" Text="Thanks for the effort! I'm not sure what your results imply for $n$ and $t$ in the hundreds." CreationDate="2011-05-15T15:14:16.077" UserId="565" />
  <row Id="18557" PostId="10407" Score="1" Text="A hard bound on the Poisson model is as follows. Let $Z_1 \sim \mathrm{Poi}(2t/(n+1))$ and $Z_i \sim \mathrm{Poi}(t/(n+1))$ for $2 \leq i \leq n$. All the $Z_i$ are mutually independent. Then $\mathbb{P}(Z_1 &gt; \max_{i\geq 2} Z_i) \geq 1 - (n-1) \exp(-c t / (n+1))$ where $c = (\sqrt{2}-1)^2$. As you can see, it only works well for $t \geq 6 n \log n$ or so." CreationDate="2011-05-15T15:22:39.833" UserId="2970" />
  <row Id="18558" PostId="10645" Text="Well, if you're not willing to make other assumptions (like normality), it's all you really have... How else would you define &quot;atypical&quot;?" CreationDate="2011-05-15T15:36:20.503" UserId="4257" />
  <row Id="18559" PostId="10823" Text="$\sigma^2$ is the variance of $s$, which is the sum of $n$ independent poisson" CreationDate="2011-05-15T15:39:56.847" UserId="2546" />
  <row Id="18560" PostId="10407" Text="@cardinal: Are you sure about your approximate formula? I just looked at some value (with t big enough) and didn't get reasonable results." CreationDate="2011-05-15T15:47:28.230" UserId="565" />
  <row Id="18561" PostId="10407" Text="Yes, I'm sure of the bound in the Poisson case. Can you give me a couple of pairs $(n,t)$ of test values?" CreationDate="2011-05-15T15:53:35.670" UserId="2970" />
  <row Id="18562" PostId="10825" Text="I actually tried to approximate the probability for bigger $n,t$ by just taking the $x^2$ part of the series into account, but that only works for small probabilities, for the probabilities that you're interested in, the approximation is way off." CreationDate="2011-05-15T15:55:45.900" UserId="4360" />
  <row Id="18563" PostId="10823" Text="@leonbloy: OK, in our case we have therefore $\sigma^2 = t$ (thanks!). And how do I get h&quot;?" CreationDate="2011-05-15T15:56:45.653" UserId="565" />
  <row Id="18564" PostId="10407" Text="@cardinal: Sorry, I had a bug. The results are now reasonable, but not very good. For $n = 199$ and $t = 8115$ I get 99% success with my approximation and your estimate gives &gt;=81.7% (where $6n \log n = 6320$). " CreationDate="2011-05-15T16:16:55.503" UserId="565" />
  <row Id="18565" PostId="10823" Text="I'd approximate the second derivative by the second difference : $A_{t+1}-2A_t+A{t-1}$, the probability of  your 'success' event evaluated at different values of $t$" CreationDate="2011-05-15T16:30:18.960" UserId="2546" />
  <row Id="18566" PostId="10825" Text="@Thies: I'm not convinced of neither your answer (as small $n$ don't seem to help for the $n$'s I need) nor the other (as I don't understand (yet??) its correctness/helpfulness). As I have more hope to get something out of the other answer and as I don't want to let the 100 points vanish, I'll probably accept the other answer. Sorry for not picking yours!" CreationDate="2011-05-15T16:55:54.877" UserId="565" />
  <row Id="18567" PostId="10823" Text="@leonbloy: I'm not really convinced of your answer (yet???), but before letting the bounty points vanish into nowhere, I'll accept your answer." CreationDate="2011-05-15T17:00:42.380" UserId="565" />
  <row Id="18568" PostId="10808" Score="1" Text="+1; as an addition, it is nice to take a look at `any()` and `all()` in this context." CreationDate="2011-05-15T17:01:27.207" UserId="88" />
  <row Id="18569" PostId="10823" Text="Well, I'm not fully convinced myself, and I'm not after bounty points. You can very well accept this answer only when (if) you confirm it's correct-helpful. No problem." CreationDate="2011-05-15T17:05:11.503" UserId="2546" />
  <row Id="18570" PostId="10827" Text="In the search for outliers, you should first regress your dependent variable against both $x_{1}$ and $x_{2}$ and look for outliers. " CreationDate="2011-05-15T17:09:22.920" UserId="3805" />
  <row Id="18571" PostId="10827" Text="Is finding outliers the pupose of your investigation? If so, then you should first regress your dependent variable against both $x_{1}$ and $x_{2}$ and then perform outlier tests. If finding possible causation, then you should consider performing a designed experiment. If the purpose of your experiment is to find a relationship between your two independent variables, looking at a happenstance of collected data will not do the trick.  " CreationDate="2011-05-15T17:15:21.577" UserId="3805" />
  <row Id="18572" PostId="10824" Text="@sesqu The visualization is helpful, but if you compare the lower right to the upper right of the figures, it appears to me that location has a stronger effect than level. (p.s I think l[i]=response should be r[i]=...) " CreationDate="2011-05-15T18:48:30.273" UserId="1381" />
  <row Id="18573" PostId="10583" Text="Andy, the Heckman procedure is extended to a variety of models including panel data estimation. Just google search provides many links." CreationDate="2011-05-15T19:05:28.043" UserId="4540" />
  <row Id="18574" PostId="10716" Text="(+1) Thanks for the edit: that makes the idea much clearer.  You might want to make the example match the counts, though: it seems the example concerns *three* users and *three* sites rather than four of each.  BTW, how do you determine which sites to use?  Nobody is active on all, or even a majority, of the sites.  This is a problem because some people get 1 rep point to start with and others get 101.  If we insist $f$ be constant in the interval $[1,101]$ to deal with this, then it's impossible for $f$ to be concave and increasing for any larger values!" CreationDate="2011-05-15T19:19:50.823" UserId="919" />
  <row Id="18575" PostId="10827" Text="@George - It isn't clear to me what you mean by outliers. If there are outliers in your data then they will affect the calculation of the regression line.  Why are you looking for outliers in both $x_1$ and $x_2$ simultaneously? " CreationDate="2011-05-15T19:28:26.423" UserId="4048" />
  <row Id="18576" PostId="10818" Text="Yes, I agree. This is count data and the model you suggest is appropriate. My only concern is that I intepret zero callers as those who chose not to call for some unknown and unobserved reasons. Thus, I was thinking more to apply Heckman's sample selection procedure, i.e., to predict probability to call, obtain inverse Mills ratio and then use the OLS with the ratio included as an independent variable to get the predicted values estimates. Thus, it seems I am restricted in this approach to use OLS. Do not you think so?" CreationDate="2011-05-15T19:32:23.450" UserId="4617" />
  <row Id="18577" PostId="10820" Text="Certainly, I agree that with the time series nature of the data as you suggested. I have done a univariate ARIMA modeling on this dataset, but the purpose is to model data using the OLS procedure in order to account for self-selection bias. The problem is that having so many zeros in the dependent variable and the time series nature of the data introduces violations of OLS assumptions. I attempt to correct for autocorrelation and heteroscedasticity. But, the OLS fit is very sensitive to many zeros in the dependent variable which probably bias parameter estimates. " CreationDate="2011-05-15T19:38:05.520" UserId="4617" />
  <row Id="18578" PostId="10686" Text="@bgbg - You're right, for p-values to be exactly correct, you need a normal distribution. With your case and its &quot;outliers,&quot; the distribution is slightly-to-moderately skewed. I think you could make a convincing argument that if p is nowhere near your alpha and is, say, .5, then it would not fall below your alpha under a normal distribution either. I was trying to hint earlier that you really don't need to run a test, since the lack of outliers is so apparent." CreationDate="2011-05-15T19:39:02.530" UserId="2669" />
  <row Id="18579" PostId="10818" Text="Well, OLS is a very simple generalised linear model. The wiki page on Heckman's procedure (ref 6) indicates that there are some nonparametric approaches that can be taken, but I don't know a huge amount about these. In addition, I think @IrishStat's suggestion of incorporating time structure into the model is definitely something which would be useful." CreationDate="2011-05-15T19:44:59.193" UserId="656" />
  <row Id="18580" PostId="10824" Text="Yes. The location effect is remarkable, and so you'd really want a good model for that before you start estimating treatment effects. Unfortunately, there is a lot of missing data so it's difficult to say what it should be - the best I can come up with would be a modelling location effect as an average of the neighbours response + random component, and then trying the treatment on that. That's very suspect, so any additional domain knowledge would be valuable. typo fixed." CreationDate="2011-05-15T20:14:42.780" UserId="2456" />
  <row Id="18581" PostId="10827" Text="@schenectady Use $$ for LaTeX in comments, please." CreationDate="2011-05-15T20:28:27.890" UserId="88" />
  <row Id="18582" PostId="10820" Text="@Mermex . I have &quot;seen&quot; data like this but it might help (me) if you actually posted one of these difficult series along with any indicators that you might think are important. " CreationDate="2011-05-15T20:33:25.587" UserId="3382" />
  <row Id="18583" PostId="10583" Text="@EEE, I am not aware of any work in which the Heckman estimator has been extended to situations in which the model is not OLS (panel data can still be OLS). Feel free to provide examples, particularly the OP is asking about a binary outcome, so any recommendation should take that into mind." CreationDate="2011-05-15T20:48:51.720" UserId="1036" />
  <row Id="18584" PostId="10747" Text="The mathematics behind variance or standard deviation isn't a problem at all. I listened to a few math lectures ;-)" CreationDate="2011-05-15T22:30:21.773" UserId="4554" />
  <row Id="18585" PostId="10747" Text="@Stefan, It is still totally unclear what you want, or why this needs to be accomplished in SPSS as opposed to whatever language. Feel free to clarify your question if you are in search of something different." CreationDate="2011-05-16T00:18:00.850" UserId="1036" />
  <row Id="18586" PostId="10843" Text="@JMS. I see. CV performance is pretty good in the example I mentioned above. Does the shape of the response really matter if the residuals look fairly normal?" CreationDate="2011-05-16T04:45:14.630" UserId="845" />
  <row Id="18587" PostId="10843" Text="Probably not, but histograms are cheap :) Comparing the transformed/untransformed response might give you some insight into the data &amp; anticipate some questions of your analysis." CreationDate="2011-05-16T04:53:44.470" UserId="26" />
  <row Id="18588" PostId="10834" Text="Unfortunately TreeView has given me a wonderful buffer overflow error :(." CreationDate="2011-05-16T04:56:02.327" UserId="4623" />
  <row Id="18589" PostId="10844" Text="Thanks. I realised this group is far more advanced than my level of stats, but I appreciate your simple answer!" CreationDate="2011-05-16T05:27:49.673" UserId="4627" />
  <row Id="18590" PostId="10813" Text="Think there's a typo in your 1st eq (you've got $y_{T-t}$ on both sides of the conditional)" CreationDate="2011-05-16T05:31:43.130" UserId="26" />
  <row Id="18592" PostId="10666" Text="Thank you so much @rolando2! Great help! I have another question, i tried the descriptive statistics you taught me (factor scores DV and gender for &quot;factor&quot;), but the output is VERY long and i couldn't explain it. Sorry for being a bit stupid here. I was looking for a breakdown in gender (i.e 30% men Factor 1; 80% men in Factor 2; 45% men in Factor 3; etc.). Many thanks in advance!!!" CreationDate="2011-05-16T07:01:19.683" UserId="4565" />
  <row Id="18593" PostId="10834" Text="@Hans Strange; unless you're trying to plot something huge, it should have worked. Are you sure the format is OK?" CreationDate="2011-05-16T07:23:45.123" UserId="88" />
  <row Id="18594" PostId="10847" Score="3" Text="Maybe helps: if Y is distributed exponentially, then X=Y^2 is distributed with f_X. You can have a look [here](http://en.wikipedia.org/wiki/Exponential_distribution) for the MLE of Y..." CreationDate="2011-05-16T07:57:54.890" UserId="1443" />
  <row Id="18595" PostId="10847" Text="@teucer, sorry did not see your comment, so I posted practically the same answer." CreationDate="2011-05-16T08:19:03.723" UserId="2116" />
  <row Id="18596" PostId="10813" Text="@JMS - thanks for that. have edited it accordingly" CreationDate="2011-05-16T08:35:42.113" UserId="2392" />
  <row Id="18597" PostId="10809" Text="@whuber - I would say that the exponentially weighted regression is a very crude approximation *for the particular model you have described*.  But it could well be an exact solution to a different model.  For the model you describe it would be much better to include the heteroscedastic component due to variation in $\alpha(t)$ (or assume that it has no variation, and you are dealing with random intercept).  You are making it seem as though geometric weighting is always sub-optimal, which is isn't.  It depends on your prior information." CreationDate="2011-05-16T08:47:18.967" UserId="2392" />
  <row Id="18598" PostId="10844" Score="1" Text="There could be trouble if X can be zero." CreationDate="2011-05-16T09:01:58.300" UserId="3835" />
  <row Id="18599" PostId="10838" Score="2" Text="I doubt you you have to do this -- it seems you're making something in a very wrong way. " CreationDate="2011-05-16T09:36:33.513" UserId="88" />
  <row Id="18600" PostId="10838" Text="@mbq, in [Eviews](http://www.eviews.com) for example this is pretty normal coding practice. Not that I am advocating it, Eviews rates only a bit lower than Excel in my top evil software list :)" CreationDate="2011-05-16T10:24:53.960" UserId="2116" />
  <row Id="18601" PostId="10838" Score="2" Text="@mpiktas In R, it is more natural to make a list, set its `names` parameter and later either just use it, `attach` it or convert it into an environment with `list2env` and `eval` inside it. With no loops, parse or other ugly stuff." CreationDate="2011-05-16T10:38:52.900" UserId="88" />
  <row Id="18603" PostId="10838" Text="@mbq, hm, `list2env` is a relatively new function. And still it will produce the variables in the some environment, when the OP wants to get the variables in the top environment. So the ugliness still remains :)" CreationDate="2011-05-16T10:49:20.860" UserId="2116" />
  <row Id="18604" PostId="10836" Text="You have luck with probabilityislogic's answer, but I close it to show that in general we don't accept chaotic pleas like this." CreationDate="2011-05-16T10:50:12.000" UserId="88" />
  <row Id="18605" PostId="10500" Text="@NRH, good one. It has a not so good sequel called Hypercube. " CreationDate="2011-05-16T11:07:19.853" UserId="2116" />
  <row Id="18606" PostId="10216" Text="@charles, hm interesting, I always though that Gibbs sampling is when you sample $p(x|y)$ and $p(y|x)$ to get the sample of $(x,y)$. The sampling scheme described here I thought is called Metropolis-Hastings. Am I wrong?" CreationDate="2011-05-16T11:17:34.403" UserId="2116" />
  <row Id="18607" PostId="10837" Text="I was wondering if I could clarify a few things with you...&#xA;Doc 1&#xA;Page 2/4 – when you have a moment please can you help me there? Its simpler then it looks and my questions are very short. &#xA;Page 3 – yes. To stay in state ‘a’ its 1-Pa.(t). But when you rearrange the minus sign comes from nowhere still. Try it and you will see what i mean. Doc2&#xA;Q1 – yes its a continuous life annuity. Rather than the answer I wanted to know how you can show a function is decreasing if you know mu is increasing... &#xA;Thanks!&#xA;p.s. this is not homework. My exam is on Thursday so its prep for it! :)" CreationDate="2011-05-16T11:34:55.227" UserId="4624" />
  <row Id="18608" PostId="10838" Text="Variables in `.GlobalEnv` is precisely what I try to omit." CreationDate="2011-05-16T11:52:41.963" UserId="88" />
  <row Id="18609" PostId="10838" Text="@mbq, why? I can understand that there are specific scenarios when this is not desirable, but in general why not? " CreationDate="2011-05-16T12:01:41.637" UserId="2116" />
  <row Id="18610" PostId="10845" Score="2" Text="`library(fortunes)` `fortune(106)`" CreationDate="2011-05-16T12:18:22.847" UserId="144" />
  <row Id="18611" PostId="10845" Text="@Roman, strange, I've started using `parse` after reading R help pages. I agree that sometimes it is an overkill, for example in `formula` management, but I found it very useful. Note that I cannot rethink the question as suggested in the fortune, since I did not ask it." CreationDate="2011-05-16T12:28:23.340" UserId="2116" />
  <row Id="18612" PostId="10845" Text="@Roman, I see that there are at least 3 people which do not like to use `parse`, yet nobody gave clear reason why. Is it slow? Prone to errors? Too sophisticated? I usually use it as analogue of `source`. " CreationDate="2011-05-16T12:31:35.057" UserId="2116" />
  <row Id="18613" PostId="10845" Score="1" Text="@mpiktas : it has to do with the fact that the underlying scoping rules can result in inpredictable results when used within a function. Also (as mentioned in the help files), R and S can give a different result due to the difference in scoping rules. It is also slower than other solutions. This will matter when you have to do this many times. And last but not least, in most cases there is a more elegant and easier solution than using eval(parse()). In this case that's working with lists or using assign." CreationDate="2011-05-16T12:40:04.230" UserId="1124" />
  <row Id="18614" PostId="10837" Text="Minus sign does not &quot;come from nowhere&quot; we have the differential eq. $\frac{\partial}{\partial t}p_{ab}(s,t)=\sum_{j\neq b}[p_{aj}(s,t)\mu_{jb}(s+t)-p_{ab}(s,t)\mu_{bj}(s+t)]$.  For &quot;no leaves&quot; we take $a=b$ and must also have first part of sum be zero (otherwise would have left state $a$).  So we get $\frac{\partial}{\partial t}p_{\overline{aa}}(s,t)=-p_{\overline{aa}}(s,t)\sum_{j\neq a}\mu_{aj}(s+t)$  This is a standard exponential differential equation, with solution $\exp\left(-\int_{0}^{t}\sum_{j\neq a}\mu_{aj}(s+r)dr\right)$." CreationDate="2011-05-16T12:41:17.927" UserId="2392" />
  <row Id="18615" PostId="10845" Text="@Joris, ok this is clearly R and the code is not in a function :) In this case I used `parse` as a substitute for running a code which I created using string manipulation instead of writing the code by hand. I fail to see the deficiencies of `parse` in this scenario, since this is the way R is used, you write some code and then you run it. " CreationDate="2011-05-16T12:53:07.693" UserId="2116" />
  <row Id="18616" PostId="10752" Text="@Gavin, sh*t happens :) For the record, I've upvoted your question, I liked the code.   " CreationDate="2011-05-16T13:01:10.053" UserId="2116" />
  <row Id="18617" PostId="10809" Text="@prob I agree, but I haven't been able to find a model that exactly justifies this approach, so I had to settle for pointing out some of the the things such a model might entail.  I notice your reply doesn't make any progress in this direction, either ;-)." CreationDate="2011-05-16T13:01:16.723" UserId="919" />
  <row Id="18618" PostId="10845" Score="1" Text="@mpiktas : I never said it's deficient. I just gave you the reason why in general a eval(parse()) construct is advised against by eg Thomas Lumley, member of the R core development team. (cfr the refernce of @Roman Lustrik)" CreationDate="2011-05-16T13:06:20.167" UserId="1124" />
  <row Id="18619" PostId="10812" Text="The K-W test (like the Wilcoxon-Mann-Whitney test) does not test medians.  It tests concordance probabilities or Hodges-Lehmann locations (like pseudomedians).  See http://en.wikipedia.org/wiki/Mann-Whitney_U.  These tests are special cases of the proportional odds ordinal logistic model, which allows for various generalizations, including multiway layouts and covariate adjustment." CreationDate="2011-05-16T13:07:17.110" UserId="4253" />
  <row Id="18620" PostId="10845" Text="@Joris, thanks for your explanations. I just felt a bit stupid, since I use `parse` extensively mainly for reasons given in this [example](http://stats.stackexchange.com/questions/8243/program-to-compute-partial-derivatives/8253#8253). I do not think that besides automatic differentiation (which is not yet implemented fully) you can produce more elegant code. I would however be glad to be proven wrong." CreationDate="2011-05-16T13:12:29.560" UserId="2116" />
  <row Id="18621" PostId="10845" Text="@mpiktas : Your link is an example where the use of parse is the right choice. Note that you never used `eval()` there, you use parse (correctly) to convert a string to an expression object needed by the `deriv()` function. That's a whole other beast than the problem of OP. Nice piece of code by the way." CreationDate="2011-05-16T13:25:22.757" UserId="1124" />
  <row Id="18622" PostId="10845" Text="elegant workspaces are nice too  n &lt;- 3; vals &lt;- structure(rnorm(n), names = paste(&quot;a&quot;, 1:n, sep = &quot;&quot;))&#xA;" CreationDate="2011-05-16T13:25:34.657" UserId="3343" />
  <row Id="18623" PostId="10845" Text="@Joris, I use `eval` later in the code to evaluate the function :) Though I take care to use the correct environment." CreationDate="2011-05-16T13:36:37.303" UserId="2116" />
  <row Id="18624" PostId="10845" Text="@mdsumner, your code produces named vector, when the OP wants variables with certain names in the global environment." CreationDate="2011-05-16T13:37:21.170" UserId="2116" />
  <row Id="18625" PostId="10792" Text="Continuing @cardinal's thought - in the case of a strictly increasing CDF (for example) we could just solve for $g$ directly to get $g = F^{-1}(F^{1/\alpha})$.  The probability integral transform gives that the inner quantity is $U^{1/\alpha}$, that is, a Beta.  The inverse CDF is closed form sometimes (exponential, pareto....) but even when it isn't we can calculate it pretty easily in many cases.  Exponential works out nicely." CreationDate="2011-05-16T13:37:41.017" UserId="1108" />
  <row Id="18626" PostId="10809" Text="@whuber - and where do I make an approximation in my equation for it not to be exact?" CreationDate="2011-05-16T13:39:11.590" UserId="2392" />
  <row Id="18627" PostId="10814" Text="See my comment in @JMS answer.  For $Z\sim N(0,1)$ it looks like the answer is $g(z) = \Phi^{-1}(\Phi^{1/\alpha}(z))$ which isn't closed form but can be calculated easily enough.  And you can make it easier by recognizing that the input to the inverse CDF is a suitably chosen Beta distribution.  The answer will be nice in cases where the inverse CDF is nice, and there are some of those running around." CreationDate="2011-05-16T13:43:19.967" UserId="1108" />
  <row Id="18628" PostId="10858" Text="The full recipe, is that you can control the input. You should do so. Validate responses or request numeric responses. Adding a comment field could provide some valuable qualification to prices that you can review afterwards. " CreationDate="2011-05-16T14:07:04.470" UserId="776" />
  <row Id="18630" PostId="10858" Text="Really looking for a stastical solution, but thanks, anyway." CreationDate="2011-05-16T14:14:19.127" UserId="4631" />
  <row Id="18631" PostId="10857" Text="@mpiktas: Dumb question. If the t-distribution with n-1 df can be derived from the sum of n indepent normal distributions (see wikipedia) and given df high enough so that the t-distribution approximates the normal distribution, doesn't derive from that that the sum of t-distributions is again a t-distribution ?" CreationDate="2011-05-16T14:15:26.320" UserId="264" />
  <row Id="18632" PostId="10830" Text="Thanks for the answer! Looks very good, but I have to check the details. What do you mean with &quot;I ... corrected the typos by reproducing DasGupta's Table 4, ...&quot;? (By the way, if you had answered 2-3 hours earlier, you'd save me some headaches about what to do with my bounty.)" CreationDate="2011-05-16T14:15:58.487" UserId="565" />
  <row Id="18633" PostId="10794" Text="@rolando If I understand ANOVA correctly, it requires that the four treatments are independent factors, whereas in this case the treatments are points along a continuous variable." CreationDate="2011-05-16T14:16:13.697" UserId="1381" />
  <row Id="18634" PostId="10859" Text="The data set I'm working with represents the most recent 6 months of data, so that much is already in place. And thanks for the help, but I'm really looking for a solution stated in statistical terms." CreationDate="2011-05-16T14:17:19.920" UserId="4631" />
  <row Id="18635" PostId="10786" Text="@Frank is that the same as performing an ANOVA?" CreationDate="2011-05-16T14:17:29.140" UserId="1381" />
  <row Id="18636" PostId="10857" Text="@mpiktas: What about the test-statistic of the t-test, which seems to be derived from the difference of two t-distributions ?" CreationDate="2011-05-16T14:17:57.253" UserId="264" />
  <row Id="18637" PostId="10857" Text="@steffen, no. It will be approximately normal, since you will add two approximately normal distributed normal variables. t-distribution with high df is approximately normal, but approximately normal is not necessarily t-distribution with high df." CreationDate="2011-05-16T14:19:06.097" UserId="2116" />
  <row Id="18638" PostId="10824" Text="@sesqu... there is no missing data, data from all 24 randomly located plots is there." CreationDate="2011-05-16T14:19:51.760" UserId="1381" />
  <row Id="18639" PostId="10857" Text="@steffen, t-test statistic is derived from the difference of two normals not two t-distributions. Note that definition of t distribution is a fraction of normal and square root of chi-square. " CreationDate="2011-05-16T14:21:28.250" UserId="2116" />
  <row Id="18640" PostId="10775" Text="The above would work for unequal numbers of measurements per subject.  In the above code it would weight each observation equally, but it could be rewritten to weight individuals equally (replace unlist with sapply and mean).  As long as you can write a function to compute your complex aggregate measure using unequal sizes, this idea will work." CreationDate="2011-05-16T14:23:41.210" UserId="4505" />
  <row Id="18641" PostId="10857" Text="@mpiktas thank you for the response ! I often make the mistake to mix &quot;precisely equivalent&quot; with &quot;close enough for practical purpose&quot; which was the case here. I am going to reformulate this question in a few minutes since it is, well, rather stupid :)." CreationDate="2011-05-16T14:31:39.487" UserId="264" />
  <row Id="18642" PostId="10729" Text="You said that you estimated the parameters from the data, so there is uncertainty about the parameters (you could fit a CI to show this).  If you only generate data at the values you estimated then it does not take into accont this uncertainty.  You can see this by increasing the number of data points you generate, as the number increases your interval estimate will converge towards having 0 width around the value if you knew the exact parameters.  To get a realistic estimate you need to include uncertainty about the parameters, bootstrapping to estimate the parameters is one way to do this." CreationDate="2011-05-16T14:31:57.620" UserId="4505" />
  <row Id="18643" PostId="10857" Text="@steffen, I often say to my students there are no stupid questions, only stupid people who do not ask any questions. I am not a very popular teacher I should add :)" CreationDate="2011-05-16T14:35:22.313" UserId="2116" />
  <row Id="18644" PostId="10814" Text="It would be good to double-check your arithmetic." CreationDate="2011-05-16T14:40:53.957" UserId="2970" />
  <row Id="18645" PostId="10792" Text="@G. Jay Kerns, perhaps I'm having a &quot;slow&quot; day, but I don't believe your candidate $g$ works as you intended." CreationDate="2011-05-16T14:41:45.517" UserId="2970" />
  <row Id="18646" PostId="10856" Text="@steffen, this is [related question](http://stats.stackexchange.com/questions/6304/what-is-the-sum-of-squared-t-variates) which might be of interest." CreationDate="2011-05-16T14:46:24.977" UserId="2116" />
  <row Id="18647" PostId="10855" Score="2" Text="To get a useful answer, you'll need to provide more information. What does your experiment consist of?" CreationDate="2011-05-16T15:10:44.107" UserId="2970" />
  <row Id="18648" PostId="10830" Text="@pul His inequalities are in the wrong direction: what he claims are $p(n,K,x)$ are really $1 - p(n,K,x-1)$.  Sorry about the bounty problem: I knew how to answer this one when it first appeared but needed to check the results first and had no time to do anything about it until the weekend." CreationDate="2011-05-16T15:41:59.553" UserId="919" />
  <row Id="18649" PostId="10809" Text="@probability You provide no justification: You simply announce the result I had already posted.  In other words, you observe that when OLS minimizes such an expression it's really doing weighted least squares.  OK, but isn't that perfectly obvious?  What justifies this choice of weights?  Where do they come from?" CreationDate="2011-05-16T15:46:21.920" UserId="919" />
  <row Id="18650" PostId="10786" Text="Yes, this is ANOVA if there are no covariates to adjust for and if one is willing to assume normality and equal group variances.  Had there been more levels it would have been better to model the profile over levels using a smooth fit (e.g., quadratic)." CreationDate="2011-05-16T15:48:22.063" UserId="4253" />
  <row Id="18651" PostId="10659" Text="Smooth modeling (e.g., using quantile regression) requires a much lower sample size than required by binning the data.  However the sample sizes needed are not trivial." CreationDate="2011-05-16T15:49:58.097" UserId="4253" />
  <row Id="18652" PostId="10856" Score="1" Text="Google the Satterthwaite t-test, the CABF t-test (Cochran's approximation to the Behrens-Fisher), and the Behrens-Fisher problem." CreationDate="2011-05-16T15:51:46.923" UserId="919" />
  <row Id="18653" PostId="10752" Text="@mpiktas - Thanks for that :-) I don't care about the rep or the downvote really - I'm more interested in *why* someone thinks the idea isn't useful. I'm not a statistician so am quite happy to be taught a thing or two, esp if providing incorrect advice." CreationDate="2011-05-16T15:54:08.560" UserId="1390" />
  <row Id="18654" PostId="10774" Text="Plots 37 and 39 are both at x=18, y=10.  Typo?" CreationDate="2011-05-16T15:54:11.180" UserId="3601" />
  <row Id="18655" PostId="10860" Score="2" Text="This is the right approach.  The criterion usually is 1.5 IQRs away from the *quartiles*, not the median.  See Tukey's *EDA* text, for example.  (I believe he originated this rule, which he analyzes rigorously in a paper buried somewhere in his collected works.)  It becomes appropriate when the data are first re-expressed in a way that makes the distribution approximately symmetric about its median.  For simple ways to determine a re-expression, see *EDA* *op. cit.*" CreationDate="2011-05-16T15:54:36.967" UserId="919" />
  <row Id="18656" PostId="10786" Text="@Frank that is a good point. instead of using 6 reps at each of four levels, 3 reps at each of eight levels... will keep that in mind for future. Is it necessarily inappropriate to fit a quadratic or other two parameter non-linear model through four points?" CreationDate="2011-05-16T15:55:45.037" UserId="1381" />
  <row Id="18657" PostId="10774" Text="@Aaron thank you for pointing that out. y= [0,10]. Fixed." CreationDate="2011-05-16T15:59:52.223" UserId="1381" />
  <row Id="18658" PostId="10863" Text="so user4633 is actually Mitch right?" CreationDate="2011-05-16T16:06:21.600" UserId="2310" />
  <row Id="18659" PostId="10824" Text="There is missing data in the sense that not every x,y pair has data." CreationDate="2011-05-16T16:25:24.693" UserId="3601" />
  <row Id="18660" PostId="10500" Text="@mpiktas, I never did see Hypercube, and it does not sound like I should :-) I saw The Cube some ten years ago and found it pretty entertaining. " CreationDate="2011-05-16T16:47:42.630" UserId="4376" />
  <row Id="18661" PostId="10865" Text="@Aaron thanks for your helpful answer; it is not clear why in part 2 you updated `model` instead of `m0`, eg.  `m2 &lt;- update(m0, .~.+x*y)` so that all three models can be compared using `anova(m0,m1,m2)`; after doing this, `m2` is a big looser (AIC = 64) &#xA;it seems that your part " CreationDate="2011-05-16T16:51:45.703" UserId="1381" />
  <row Id="18662" PostId="10865" Text="p.s. the last line should be 'after changing y-value of plot 37 to 5'; the actual value is 0, but the results are equivalent." CreationDate="2011-05-16T16:52:52.990" UserId="1381" />
  <row Id="18663" PostId="10863" Text="Voted to close. I also flagged the question." CreationDate="2011-05-16T17:04:33.760" UserId="2970" />
  <row Id="18665" PostId="10856" Score="1" Text="For the special case where the degrees of freedom is 1 (the Cauchy distribution) the answer to the original question is 1. The sum (or difference) of two independent Cauchy distributed random variables is Cauchy with scale parameter $2$, but then again, the Cauchy distribution does not even have a mean value." CreationDate="2011-05-16T17:20:05.647" UserId="4376" />
  <row Id="18666" PostId="10794" Text="@David - 0, 1, 4, and 9 are each a level of the single treatment variable.  In ANOVA you would test whether the mean is the same across each of these 4 levels.  Your comment seems to apply to a situation that doesn't apply here, in which you had 4 different predictor variables (e.g., sex, ethnicity, income, and region)." CreationDate="2011-05-16T17:24:08.377" UserId="2669" />
  <row Id="18668" PostId="10794" Text="@rolando, I guess my question was whether it is appropriate to compare the levels using ANOVA instead of regression " CreationDate="2011-05-16T17:47:46.057" UserId="1381" />
  <row Id="18669" PostId="4051" Text="+1. Very good explanation. I also think that some &quot;robust&quot; methods are rather ad hoc (truncated means), and that &quot;robust&quot; is tied to a particular aspect of a method and is not a general quality but many people interpret &quot;robust&quot; to mean &quot;I don't have to worry about my data, since my method is robust&quot;." CreationDate="2011-05-16T17:54:21.560" UserId="1764" />
  <row Id="18670" PostId="10860" Text="Okay, I'll look into this, thanks much!" CreationDate="2011-05-16T18:03:13.553" UserId="4631" />
  <row Id="18671" PostId="10864" Text="@John Salvatier: the thing that bogles me is whether the assumption about independence of year halves is valid. If not, $Cov(Year)=Cov(A/2+B/2)=Cov(A/2)+Cov(B/2)+\rho_{A/2, B/2}w_{A/2}w_{B/2}$, where $A/2$ and $B/2$ stand for different half years and $\rho_{A/2, B/2}=\sigma_{A/2}\sigma_{B/2}w_{A/2}w_{B/2}$ where $w$ stands for the proportions." CreationDate="2011-05-16T18:34:51.013" UserId="2914" />
  <row Id="18672" PostId="10216" Text="@mpiktas.  In the second example, $x$ is the 'attack' and $y$ is the buff.  So indeed, I present an algorithm to sample $(x,y)$ given $p(x|y)$ and $p(y|x)$." CreationDate="2011-05-16T18:38:44.777" UserId="3567" />
  <row Id="18674" PostId="10801" Text="Where is this explained in the video?  It's an hour long.  Thanks!" CreationDate="2011-05-16T19:07:00.767" UserId="4610" />
  <row Id="18675" PostId="10795" Text="@NRH The geometry is explained in the partial correlation page, which I'm not even sure how it relates to the concentration matrix yet.  Does that graphical models book have an explanation of the elements of the concentration matrix?  Thanks!" CreationDate="2011-05-16T19:09:08.877" UserId="4610" />
  <row Id="18676" PostId="10216" Text="@charles, thanks, I missed it. For some strange reason I always find strict mathematical texts more easily understandable than the non-strict examples. Maybe in this case the fact that I have never played D&amp;D is working against me. +1 for the answer though." CreationDate="2011-05-16T19:09:47.770" UserId="2116" />
  <row Id="18677" PostId="10835" Text="Thanks for the pointers B_Dev. It's not exactly MAUP problem in this case since my areas are not mutually exclusive. And for more pointers about regionalization methods see my other question: http://gis.stackexchange.com/questions/3189/what-tools-for-regionalization-redistricting-can-you-recommend" CreationDate="2011-05-16T19:40:31.533" UserId="22" />
  <row Id="18678" PostId="10865" Text="If you compare `m0`,`m1`, and `m2` as you suggest you get the warning: `Fitted objects with different fixed effects. REML comparisons are not meaningful.`  To compare fixed effects you have to use regular maximum likelihood instead of REML.  See edit." CreationDate="2011-05-16T19:49:27.213" UserId="3601" />
  <row Id="18679" PostId="10786" Text="No it's not inappropriate, it just assumes a quadratic shape.  It requires 3 parameters whereas the most flexible mean model with 4 treatments requires the same number." CreationDate="2011-05-16T20:10:45.557" UserId="4253" />
  <row Id="18680" PostId="10795" Text="@Vinh, see answer below." CreationDate="2011-05-16T20:26:36.157" UserId="4376" />
  <row Id="18681" PostId="10871" Text="Very similar in Belgium." CreationDate="2011-05-16T20:33:10.667" UserId="4257" />
  <row Id="18682" PostId="10867" Text="Do you have any reason to believe your values should be normally distributed? Dixon's Q-test is based on normality, while real data is often non-normal." CreationDate="2011-05-16T20:37:10.447" UserId="279" />
  <row Id="18683" PostId="10872" Text="Ok, assume I have 20 biological replicas with 20 measurements in each, and I'm asking the same questions. Can I work at the level of 20 averaged measurements corresponding to 20 replicas as if this was my original sample? Or is it better to stay at the level of 400 measurements?" CreationDate="2011-05-16T20:38:32.787" UserId="4337" />
  <row Id="18684" PostId="10871" Text="@Tal Could you indicate school grade? " CreationDate="2011-05-16T20:41:47.090" UserId="930" />
  <row Id="18685" PostId="10874" Text="@Tal Should we set this as CW?" CreationDate="2011-05-16T20:43:07.247" UserId="930" />
  <row Id="18686" PostId="10867" Text="Yes, I believe they are normally distributed." CreationDate="2011-05-16T20:43:17.337" UserId="4337" />
  <row Id="18687" PostId="10864" Text="It would be surprising if the two year halves were non-negligibly correlated since asset returns are [martingales](http://en.wikipedia.org/wiki/Martingale_(probability_theory))." CreationDate="2011-05-16T21:03:17.097" UserId="1146" />
  <row Id="18688" PostId="10872" Score="1" Text="From what I quickly gather from [The R outliers package description](http://cran.cnr.berkeley.edu/web/packages/outliers/outliers.pdf), I tend to say you may be able to say something about 20 observations, yes. But don't quote me on that." CreationDate="2011-05-16T21:05:06.860" UserId="4257" />
  <row Id="18689" PostId="10872" Score="1" Text="The problem with doing a separate outlier detection for each replica is multiple testing, i.e. with enough tests you start getting false positives." CreationDate="2011-05-16T21:08:04.633" UserId="279" />
  <row Id="18690" PostId="10871" Text="chl - I'm speaking of all grades until the 12th grade (age 18)" CreationDate="2011-05-16T21:19:38.030" UserId="253" />
  <row Id="18691" PostId="10874" Text="Yes chl - that would be great." CreationDate="2011-05-16T21:20:13.227" UserId="253" />
  <row Id="18695" PostId="10872" Text="@Aniko: I'm not even going there :-) I guess if there are 3 biological replicates with each 20 technical ones, the impact of multiple (3) testing will not be insurmountable (even a FWER correction only implies multiplying p-values by 3, so really strong deviations should get picked up easily). But you're right of course: @Leo did ask for 20 biological replicas :-(" CreationDate="2011-05-16T21:56:46.237" UserId="4257" />
  <row Id="18696" PostId="10864" Text="@hhh dividing the covariance like this is incorrect.  Think in terms of scatterplots: behind each year's paired data is a set of about 250 pairs of daily returns.  The question is trying to recover the scatter of those daily returns from observing the scatter of these averaged-out points.  There is no fixed relationship, or even useful bounds available, unless one makes strong assumptions.  In particular, dividing by two--even assuming no temporal correlation--will not generally produce the correct answer." CreationDate="2011-05-16T22:00:53.813" UserId="919" />
  <row Id="18697" PostId="10870" Text="Trimmed means are a good idea, in keeping with the need for a robust statistic.  But should the OP be comfortable with just 10% trimming?  Maybe 20% trimming would be better?  This thinking brings us pretty quickly to one extreme of just reporting the median (a 50% trimmed mean).  The other extreme is some analysis of outliers, rejection of the outliers, and re-estimation of whatever statistics one wants." CreationDate="2011-05-16T22:05:27.320" UserId="919" />
  <row Id="18698" PostId="10874" Score="8" Text="Chapman Hall / CRC and Wiley also publish quality statistics texts.  But one shouldn't judge a book by its publisher..." CreationDate="2011-05-16T22:09:48.507" UserId="3567" />
  <row Id="18699" PostId="6061" Text="@Raskolnikov I suggest migrating this question to MathSE" CreationDate="2011-05-16T22:24:30.970" UserId="2902" />
  <row Id="18700" PostId="10865" Text="@Aaron thank you for all of your help. I am not sure why, but I am getting errors when I try to fit other correlation structures, e.g. using corExp as in the Pinheiro and Bates example. I have [opened a question on SO](http://stackoverflow.com/q/6024223/199217) about this error, but your input would be appreciated." CreationDate="2011-05-16T22:39:12.653" UserId="1381" />
  <row Id="18701" PostId="10363" Text="@knorv: So, was whuber's answer the right answer?  Almost the right answer? " CreationDate="2011-05-16T23:16:47.270" UserId="2775" />
  <row Id="18702" PostId="479" Text="I agree that the background option is a good idea.  But the red / green is not - my office mate is colour blind, and can't tell the difference.  So even though it's good from the &quot;traffic lights&quot; point of view, it can cause problems!" CreationDate="2011-05-16T23:20:33.240" UserId="1991" />
  <row Id="18703" PostId="10790" Text="+1 too, thanks for your help." CreationDate="2011-05-16T23:23:32.020" UserId="4607" />
  <row Id="18704" PostId="10844" Text="@mark999 I don't think so; $f$ is continuous from his notation so $Pr(X=0)=0$. If the integral diverges then the expectation is still well defined, it's just infinite. Depends on what troubles you :)" CreationDate="2011-05-16T23:54:37.250" UserId="26" />
  <row Id="18705" PostId="10870" Text="I'm not exactly an expert, but just tossing out the highest and lowest 10% sounds very different from tossing out values that are clearly anomalous. If I have a hundred values that are around where I would expect them, why would I want to trim any of them? It's only the one or two out in left field I want to catch and discard..." CreationDate="2011-05-16T23:57:40.347" UserId="4631" />
  <row Id="18706" PostId="10792" Text="@cardinal It's fantastic that I wrote that comment whilst oblivious to that fact..." CreationDate="2011-05-16T23:59:20.303" UserId="26" />
  <row Id="18707" PostId="10855" Text="From the question, I am guessing that the interest is in the distribution of the &quot;goodness&quot; value, that the runs are independent simulations and that we need to repeat the runs $n$ times where $n$ is large enough to estimate the mean and standard variation sufficiently well. Is that the question?" CreationDate="2011-05-17T00:04:30.533" UserId="4376" />
  <row Id="18708" PostId="10874" Text="I find that even within a publisher there's often a pretty wide range of offerings. Authorship is usually a much better indicator; I bet I could pull down any book I love and find one published by the same house that I hate." CreationDate="2011-05-17T00:12:02.867" UserId="26" />
  <row Id="18709" PostId="10666" Text="@joms There is no such thing as being &quot;in&quot; Factor 1:  each person has a score on Factor 1.  Perhaps the men have scores shifted lower compared to women's--that's the sort of thing you could show with the Explore command.  See if you can generate a single boxplot at the end of your output that shows all the Factors split by gender." CreationDate="2011-05-17T00:40:58.397" UserId="2669" />
  <row Id="18710" PostId="10794" Text="Yes, in your case I believe it amounts to the same thing.  Regression with dummy variables can be equivalent to ANOVA.  They're both subsumed under what's called the General Linear Model." CreationDate="2011-05-17T00:44:21.170" UserId="2669" />
  <row Id="18711" PostId="10881" Text="Part of your answer seems to have been deleted." CreationDate="2011-05-17T00:46:50.387" UserId="919" />
  <row Id="18712" PostId="10881" Text="Thanks. I'm not really sure what happened." CreationDate="2011-05-17T01:04:11.573" UserId="4637" />
  <row Id="18713" PostId="10883" Score="2" Text="This interpretation is sort of on the right track, at least at the beginning. I'm not sure who Gaines is or how it relates to this problem. But, see my hint, which provides you the answer. If needed, I can expand on it a bit. I would *definitely* not equate asymptotic relative efficiency to resources used, as you have tried to do in your last paragraph." CreationDate="2011-05-17T02:17:21.617" UserId="2970" />
  <row Id="18714" PostId="10882" Text="Note I suggested using the _upper_ bound from the same formula." CreationDate="2011-05-17T02:47:28.910" UserId="279" />
  <row Id="18715" PostId="10394" Text="@cardinal Your interpretation is almost always what is intended, especially in discussions of robust estimators.  I would elevate your comment from &quot;guess&quot; to &quot;near certainty.&quot;" CreationDate="2011-05-17T02:48:37.427" UserId="919" />
  <row Id="18716" PostId="10882" Text="@metrobalderas, the questions can be migrated between sites. I'll flag the questions, maybe moderators will be able to merge." CreationDate="2011-05-17T03:09:39.360" UserId="2116" />
  <row Id="18717" PostId="10847" Score="1" Text="@Mitch, always, always indicate that the question is related to homework. The homework is for you to learn, just getting the correct answer will not help you, and may even hurt you in the long run. I am surmising that this is a homework from the question of another user. " CreationDate="2011-05-17T03:55:12.423" UserId="2116" />
  <row Id="18718" PostId="10834" Text="Perhaps not. My example.tre file: (A,B,(C,D));" CreationDate="2011-05-17T04:18:51.350" UserId="4623" />
  <row Id="18719" PostId="10874" Score="1" Text="I'm not the biggest fan of O'Reilly when it comes to statistics. O'Reilly's book on R (the owl book) is designed for a much less technical audience. The Springer texts are really pretty fantastic, and one of the more reliable sources. Wiley (Clinical Trials, Categorical Data Analaysis) and ChapmanHall/CRC (Gelman's Bayesian book) too. Between those three, you've got a fair sample of statistics." CreationDate="2011-05-17T04:52:07.477" UserId="1118" />
  <row Id="18720" PostId="10882" Text="@mp Reading over the SO version and the article it references makes it clear the question belongs here: I think we would have quite a few things to say about the idea of ranking data by means of LCLs!  I do not believe that CV mods can migrate a question *from* another site, but if the SO mods migrate it here we can do the merge easily.  It might be best for @metro to flag the SO question and request the merge." CreationDate="2011-05-17T05:18:28.750" UserId="919" />
  <row Id="18721" PostId="10809" Text="@whuber - I do say what justifies the weights - your prior information that the noise or error in the observations becomes large as you move backward in time from $T$.  How is this not any more justified than your claim that &quot;the coefficients vary with time&quot;?  Your solution talks about variability in the co-efficients, and mine talks about the variability in the noise level.  I can't see how your approach can be justified any more than mine - you make no explanation of what justifies the choice of $k$ in your example either (and neither do I)." CreationDate="2011-05-17T05:21:55.047" UserId="2392" />
  <row Id="18722" PostId="10882" Text="I've flagged. I'm sorry about the duplicates, but it also covers both topics." CreationDate="2011-05-17T05:24:25.530" UserId="4638" />
  <row Id="18723" PostId="10884" Text="do we have to answer this ?? " CreationDate="2011-05-17T05:28:42.627" UserId="1763" />
  <row Id="18724" PostId="10853" Text="Thanks for the rapid &amp; clear reply!" CreationDate="2011-05-17T05:29:10.877" UserId="3773" />
  <row Id="18725" PostId="10809" Text="@Prob the question is, why?  What conceivable conditions would have to apply for the noise to increase backwards in time?  My analysis, non-rigorous as it is, nevertheless indicates the kinds of assumptions and gross simplification that may be needed." CreationDate="2011-05-17T05:29:26.823" UserId="919" />
  <row Id="18726" PostId="10884" Text="@Jeromy, by beta weights you mean the linear regression coefficients?" CreationDate="2011-05-17T05:29:34.063" UserId="2116" />
  <row Id="18727" PostId="10884" Score="1" Text="@mp Conventionally, betas are the coefficients when all variables have been standardized.  (That should instantly make them recognizable as partial correlations, answering the question... :-)" CreationDate="2011-05-17T05:30:51.860" UserId="919" />
  <row Id="18728" PostId="10853" Text="(yes, the gene expression values are more or less normally distributed)" CreationDate="2011-05-17T05:30:56.537" UserId="3773" />
  <row Id="18729" PostId="10884" Score="1" Text="@ayush I realise it is an elementary question, so feel free to not answer it yourself. However, I think the site can benefit from having questions at various levels of difficulty; and I'd like to add my own answer after giving others a chance to answer that picks up on a few general issues." CreationDate="2011-05-17T05:41:30.053" UserId="183" />
  <row Id="18730" PostId="10885" Text="...assuming the behavior is truly linear across the entire range of $X$ values!  (A more cautious answer might couch the same idea in terms of *average* changes and also avoid any hint of suggesting the relationship is causal.)" CreationDate="2011-05-17T05:44:19.400" UserId="919" />
  <row Id="18731" PostId="10884" Text="Good point, @Jeromy.  I'm sure @ayush would not have provided such a comment (which could easily be misinterpreted as rude or worse) were the same question to be posed by a new user.  Let's take this as testimony to your high reputation here and see whether any of the replies help enlighten your correspondent." CreationDate="2011-05-17T05:48:01.703" UserId="919" />
  <row Id="18732" PostId="10885" Text="@whuber, I knew that putting the word best was not a wise choice:) Thanks for your comment, I'll try to rephrase the answer. " CreationDate="2011-05-17T05:49:18.297" UserId="2116" />
  <row Id="18733" PostId="10885" Text="@mp &quot;Best&quot; isn't necessarily a problem.  I'm just trying to give you a hard time :-)  (But &quot;induce&quot; did get my attention...)  If you're truly after the &quot;best&quot; explanation, recall that a common point of confusion among the uninitiated is how to interpret interaction coefficients: after all, you can't independently vary (say) $X Y$; you do so by varying either $X$ or $Y$ or both.  So an explanation that handles that situation would be most welcome." CreationDate="2011-05-17T05:52:20.453" UserId="919" />
  <row Id="18734" PostId="10884" Score="2" Text="@whuber. good point. Being a stats consultant in psychology, I sometimes get questions by email that are fairly elementary. My ideal situation is to encourage such students to post directly here. In general, I prefer to answer these questions on this site rather than send an email response to the student. That way, my response can be an ongoing resource for the internet, and others may come up with an even better response." CreationDate="2011-05-17T06:05:28.880" UserId="183" />
  <row Id="18735" PostId="10885" Text="@whuber, yes induce was a poor choice. I'll leave the explanation of interaction terms for somebody else :)" CreationDate="2011-05-17T06:06:29.770" UserId="2116" />
  <row Id="18736" PostId="10353" Text="@sam I liked where you were going with this question. After 12 days with no response from you, I have updated your question, so that there is an actual question. Feel free to modify if I have misconstrued." CreationDate="2011-05-17T06:40:18.880" UserId="183" />
  <row Id="18737" PostId="10870" Text="@whuber I just used 10% as an example. I don't know what the best choice would be, or even how &quot;best&quot; would be defined. @Max I don't know that trimmed means are definitely good for your situation, but at least you would be consistent and avoid subjective decisions on what to discard." CreationDate="2011-05-17T07:04:09.780" UserId="3835" />
  <row Id="18738" PostId="10856" Text="@whuber thank you, apparently I didn't know that" CreationDate="2011-05-17T07:24:22.543" UserId="264" />
  <row Id="18739" PostId="10881" Text="&gt;a combination of abilities can be more powerful than if the two abilities were on separate cards.&#xA;&#xA;Yeah, I was hoping for a method that could yield answers like:&#xA;v(X) = 1.6&#xA;&#xA;v(X,Y)=0.2&#xA;&#xA;...&#xA;&#xA;Or even functions where the result depends on another variable." CreationDate="2011-05-17T07:30:10.073" UserId="4636" />
  <row Id="18741" PostId="10830" Text="@whuber: No need to be sorry ;-). Thanks again." CreationDate="2011-05-17T08:10:35.097" UserId="565" />
  <row Id="18742" PostId="10846" Text="Perfect, thanks, that's what i thought, just needed to confirm." CreationDate="2011-05-17T08:14:34.080" UserId="4617" />
  <row Id="18743" PostId="10818" Text="yes, agreed, thanks" CreationDate="2011-05-17T08:15:15.703" UserId="4617" />
  <row Id="18744" PostId="10846" Text="@Mermex, you can accept the answer if you think it answered your question. See [faq](http://stats.stackexchange.com/faq)." CreationDate="2011-05-17T08:26:38.380" UserId="2116" />
  <row Id="18745" PostId="10880" Text="@Steve nice idea (a general problem I wondered about when I was playing Magic the Gathering). a) I checked out the excel-file. It seems that the price-column is missing. Or did I overlook something ? b) the prices are trade-prices, aren't they ? In this case the prices are not related to the value of the card attributes (but number of prints) ? " CreationDate="2011-05-17T08:32:35.090" UserId="264" />
  <row Id="18746" PostId="10880" Text="@Steve Sorry for misused terms. With &quot;value of card attributes&quot; I meant something like &quot;attack has the value 3&quot; not the estimated money-value of &quot;attack=3&quot;." CreationDate="2011-05-17T08:41:20.590" UserId="264" />
  <row Id="18747" PostId="10809" Text="@whuber - if you apply the same argument to your own line of reasoning, you will find the same response &quot;because my prior information indicates this is the case&quot;.  This is the same answer as you would give for &quot;slowly varying&quot; co-efficients.  One more concrete example I can give is if you are dealing with data which is of varying quality due to continuous improvement in the data collection process, thus leading you to expect less noise as time progresses.  Another is when the measurement process improved over time (you upgrade your measuring equipment each year)" CreationDate="2011-05-17T09:45:14.200" UserId="2392" />
  <row Id="18748" PostId="10874" Text="Does the question relate to content, or things like typesetting and paper quality?" CreationDate="2011-05-17T10:02:43.013" UserId="3835" />
  <row Id="18749" PostId="10890" Text="@MarkDollar, unobserved heterogeneity can have different interpretations (google does not give a unique definition for example), can you please cite some reference, or give the precise definition you want to be explained. " CreationDate="2011-05-17T10:21:50.930" UserId="2116" />
  <row Id="18750" PostId="6061" Text="Hey @deps_stats. It's actually already there. So you can close this one if you want." CreationDate="2011-05-17T11:22:41.183" UserId="2036" />
  <row Id="18751" PostId="10853" Text="You're welcome Yannick.  I suspect what I suggest will work.  Cheers, Tal" CreationDate="2011-05-17T11:30:22.173" UserId="253" />
  <row Id="18752" PostId="10855" Text="@cardinal: I gave the info in the question. @NRH: Even though I am not good enough with the language of statistics, I believe your guess is right to the point." CreationDate="2011-05-17T12:34:05.563" UserId="4629" />
  <row Id="18753" PostId="10879" Text="Hi Whuber - wonderful references - thank you!" CreationDate="2011-05-17T13:06:18.597" UserId="253" />
  <row Id="18754" PostId="10792" Text="@JMS: It was a good observation, nonetheless. :)" CreationDate="2011-05-17T13:21:14.340" UserId="2970" />
  <row Id="18755" PostId="10864" Text="@John Salvatier: I think it is better to say that asset returns can be modeled by stochastic processes such as martingales and markov chains. Sometimes, it is useful to analyse things with path-depended models but then again you cannot beat state-depended models in some point-wise analysis. Sorry I am a bit lost what you are referring to with this comment. Can you write it mathematically to your answer?" CreationDate="2011-05-17T13:46:04.193" UserId="2914" />
  <row Id="18756" PostId="10809" Text="@prob But why a geometric sequence of weights?  Where does that come from?" CreationDate="2011-05-17T14:00:58.253" UserId="919" />
  <row Id="18757" PostId="10870" Text="@Max The idea is that removing a few values at both ends of the data is unlikely to affect your estimate of where most of the values lie, whether or not what you throw out is an outlier.  I'm just pushing that point a little, knowing how bad Web data can be, and suggesting an even simpler approach of using the middle value to estimate your typical price: that's the same as &quot;tossing out&quot; all but a single value!  You could also look into a closely related statistic, the [Winsorized mean](http://en.wikipedia.org/wiki/Winsorized_mean)" CreationDate="2011-05-17T14:05:03.540" UserId="919" />
  <row Id="18758" PostId="10885" Text="@mp re Note 2: Ah, Taylor's Theorem!  But real data aren't even continuous, much less differentiable.  The *model* might enjoy these mathematical properties.  Especially in explanations for the uninitiated, it may be worthwhile to distinguish the model's behavior from the behavior we expect of the data.  Also, Taylor's Theorem says little about the range of $X$ values over which near-linearity holds.  The regression model says this range is infinite!" CreationDate="2011-05-17T14:13:11.800" UserId="919" />
  <row Id="18759" PostId="10882" Text="Oh, thank! I had no idea there was a SE fully dedicated to it." CreationDate="2011-05-17T01:07:50.457" UserId="4638" />
  <row Id="18760" PostId="10894" Text="I'm sorry, I'm not mathematician. How can I determine this lower bound? Also, could you expand this low _upper_ bound concept? It seems a little too much for me hehe. Thanks for your response." CreationDate="2011-05-16T19:09:34.150" UserId="4638" />
  <row Id="18761" PostId="10894" Text="The formula you linked to has a +/- in it. Using the - gives you the lower bound which you are using to sort the items if I understand you correctly. I am suggesting using the +, which gives you the upper bound of the interval, and throw away items that have a low value for it." CreationDate="2011-05-16T19:37:41.627" UserId="279" />
  <row Id="18762" PostId="10894" Text="Uh, maybe I'm missing something, but yes, the original formula has a +/- sign, however the Ruby implementation only has a minus sign, and, with further examination, there's some other things missing from the original formula; for instance, the `alpha/2` is not present in any form in the whole formula." CreationDate="2011-05-17T01:07:05.433" UserId="4638" />
  <row Id="18763" PostId="10894" Text="`alpha` is there, it is just called `power` for some reason." CreationDate="2011-05-17T02:41:54.543" UserId="279" />
  <row Id="18764" PostId="10894" Score="1" Text="The formula defines a confidence _interval_: it has two ends. One is obtained by using the + and one is by using the -. The lower end (the lower bound of your title) is obtained by using the -." CreationDate="2011-05-17T02:43:34.403" UserId="279" />
  <row Id="18765" PostId="10891" Text="@crypton How does DBSCAN deal with autocorrelated series. For example a time series 1,9,1,9,1,9,5 would have an inlier/unusual value. Does it deal with Level Shifts in the series or a number of possible trends ? How does it deal with seasonal autoregressive data where other than say June of each month the series is &quot;regular&quot; while every June the series is &quot;large&quot; . Do the &quot;large June values&quot; get detected as outliers ? Just curious !" CreationDate="2011-05-17T15:44:27.277" UserId="3382" />
  <row Id="18766" PostId="10837" Text="Ah - I see what I did wrong now! Thanks! :)" CreationDate="2011-05-17T15:53:49.087" UserId="4624" />
  <row Id="18767" PostId="10895" Text="This is a copy of question that was closed for clearly stated reasons.  Please correct the problems before reposting it." CreationDate="2011-05-17T16:13:03.227" UserId="919" />
  <row Id="18768" PostId="10898" Score="1" Text="@user4624 There is a problem with your series of questions because they do not stand alone: they cannot be read directly, but instead rely on a cumbersome mechanism to download pdf files. Please restate the questions here so (1) they will have a permanent record on this site and (2) we don't need to go through that download process. So that they don't accumulate downvotes or irrelevant comments while you're doing that, they will be deleted. You should be able to undelete them yourself once they are fixed up, but if you encounter any trouble, just flag one for moderator help" CreationDate="2011-05-17T16:14:41.353" UserId="919" />
  <row Id="18769" PostId="10896" Text="Please see the comment at http://stats.stackexchange.com/questions/10898/kolmogorov-differential-equations#question" CreationDate="2011-05-17T16:19:59.290" UserId="919" />
  <row Id="18770" PostId="10895" Text="Please see the comment at http://stats.stackexchange.com/questions/10898/kolmogorov-differential-equations#question .  I am concerned that you could not see earlier comments and apologize if that is the case." CreationDate="2011-05-17T16:20:33.627" UserId="919" />
  <row Id="18771" PostId="10882" Text="@metro No problem.  It took some research to figure out where it best belonged.  Usually a single posting is best; if the question doesn't fit your first choice site, it will get migrated to a better site :-)." CreationDate="2011-05-17T16:22:05.823" UserId="919" />
  <row Id="18772" PostId="10881" Text="You can do that sort of thing with regression analysis, it just starts to get really complicated. Doing a full regression analysis would require a package of statistical software as well as some sort of digital file containing all of the card info." CreationDate="2011-05-17T16:23:03.510" UserId="4637" />
  <row Id="18773" PostId="10899" Score="2" Text="@Tauf Yes: First close the rod into a circle.  Then break it (uniformly, randomly, independently) at $n+1$ breakpoints.  This introduces a helpful symmetry :-)." CreationDate="2011-05-17T16:24:01.017" UserId="919" />
  <row Id="18774" PostId="10897" Score="2" Text="You can choose your units of measurement--meters, yards, parsecs, whatever--without materially changing the problem.  So, choose a unit in which the rod has length 1.  Done! :-)" CreationDate="2011-05-17T16:25:30.953" UserId="919" />
  <row Id="18775" PostId="10834" Text="@Hans All I can say is that it works for me :-( " CreationDate="2011-05-17T16:45:30.433" UserId="88" />
  <row Id="18776" PostId="10899" Score="1" Text="Or, draw a connection to the Poisson process." CreationDate="2011-05-17T16:46:45.380" UserId="2970" />
  <row Id="18777" PostId="10892" Text="what dimensional space are you requesting? have you tried it for smaller k?" CreationDate="2011-05-17T16:52:47.467" UserId="795" />
  <row Id="18778" PostId="10897" Text="@whuber, thanks but I am not yet there. Basically I get a negative exp by computing the product of many small steps, each of them with probability (1-p), in the limit where the length of the step is very small. Now, how do I introduce in this computation the fact that I can't overstep the &quot;right end&quot; of the rod? " CreationDate="2011-05-17T17:02:08.570" UserId="4642" />
  <row Id="18779" PostId="10858" Score="1" Text="FYI, _outlier_ is a valid scientific term." CreationDate="2011-05-17T17:03:13.523" UserId="88" />
  <row Id="18780" PostId="10897" Score="1" Text="@X I'm a little lost, because I don't see how you are making a connection between the &quot;many small steps&quot; and the situation you have presented." CreationDate="2011-05-17T17:25:11.183" UserId="919" />
  <row Id="18781" PostId="10900" Score="3" Text="&quot;unskwed&quot; = &quot;unskewed&quot;?  Symmetric?  A problem with this question is that it requests a quantitative answer (a distribution or family of them) but supplies no quantitative criteria for selecting among the infinitely many possibilities.  *How* &quot;heavy&quot; should the tails be?  What exactly is the &quot;purpose&quot;?  (&quot;Algorithm&quot; is far too general to be a useful description.)  You might find [this search](http://stats.stackexchange.com/search?q=heavy+%2B+\[distributions\]) useful. " CreationDate="2011-05-17T17:27:45.210" UserId="919" />
  <row Id="18782" PostId="10580" Text="@SheldonCooper: I've bought a copy of &quot;Pattern Recognition&quot; by D, H, and S. It really is spectacular, and the whole thing is useful and easy to follow. Thanks again. This is exactly how SE is supposed to work :)" CreationDate="2011-05-17T17:31:39.477" UserId="2426" />
  <row Id="18784" PostId="10653" Score="1" Text="@IrishStat Unfortunately, the data is not mine to post, and I have several other (similar) datasets to do this type of forecasting with. I appreciate the offer for posting the data for help in analysis, but I can't accept. Thank you, though." CreationDate="2011-05-17T18:18:03.993" UserId="1583" />
  <row Id="18786" PostId="10653" Score="2" Text="@IrishStat It may help to know the downvote was not mine. I'm sorry the reason was not explained. As for me, I'm coming closer to an answer through your help and some face-to-face discussion with a colleague. Thank you for the update you added to the answer: that is quite helpful. I've also found issues in the model that I know how to fix, so I'm working on that." CreationDate="2011-05-17T18:29:35.097" UserId="1583" />
  <row Id="18787" PostId="10904" Text="@B_Dev, the major con of the mathematical approach is to know the &quot;corner&quot; cases of the distribution. All the sample moments of any distribution exist, yet the distribution can have none such as Cauchy. In general both approaches should be combined." CreationDate="2011-05-17T18:36:46.377" UserId="2116" />
  <row Id="18788" PostId="10864" Text="My notion was that assuming independence of halves and equality of the covariance of the halves was probably 'good enough' for your situation since your question implies you think that semi-annual covariance from annual covariance is a reasonable thing to try to do. If these assumptions are not good enough for your purposes, then there's probably little you can do short of getting more data. Is more clear?" CreationDate="2011-05-17T18:47:52.623" UserId="1146" />
  <row Id="18789" PostId="10906" Text="Hi mpiktas, thank you for your response but the R result is very different from SAS proc mixed procedure. The SAS output is the same with the answer in a textbook(Kuehk RO. 1999. Design of experiments: statistical principles of research design and analysis, 2nd edition). The dataset is        method day level&#xA;1   1 142.3&#xA;1   1 144.0&#xA;1   2 134.9&#xA;1   2 146.3&#xA;1   3 148.6&#xA;1   3 156.5&#xA;1   4 152.0&#xA;1   4 151.4&#xA;2   1 142.9&#xA;2   1 147.4&#xA;2   2 125.9&#xA;2   2 127.6&#xA;2   3 135.5&#xA;2   3 138.9&#xA;2   4 142.9&#xA;2   4 142.3" CreationDate="2011-05-17T19:23:30.393" UserId="4559" />
  <row Id="18791" PostId="10907" Score="2" Text="It sounds like you want a &quot;prediction interval&quot;.  The answer (with formulas) is available on this site [in several places](http://stats.stackexchange.com/search?q=%22prediction+interval%22)." CreationDate="2011-05-17T20:02:43.720" UserId="919" />
  <row Id="18792" PostId="10905" Text="It helps to use `dput` to get the code needed to recreate your data.  From the comment you left, the result is `structure(list(method = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, &#xA;1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c(&quot;1&quot;, &quot;2&quot;), class = &quot;factor&quot;), &#xA;day = structure(c(1L, 1L, 2L, 2L, 3L, 3L, 4L, 4L, 1L, 1L, &#xA;2L, 2L, 3L, 3L, 4L, 4L), .Label = c(&quot;1&quot;, &quot;2&quot;, &quot;3&quot;, &quot;4&quot;), class = &quot;factor&quot;), &#xA;level = c(142.3, 144, 134.9, 146.3, 148.6, 156.5, 152, 151.4, &#xA;142.9, 147.4, 125.9, 127.6, 135.5, 138.9, 142.9, 142.3)), .Names = c(&quot;method&quot;, &#xA;&quot;day&quot;, &quot;level&quot;), row.names = c(NA, -16L), class = &quot;data.frame&quot;)`" CreationDate="2011-05-17T20:11:18.660" UserId="3601" />
  <row Id="18793" PostId="10904" Text="@mpiktas, I believe that you mean that the major **pro** is to know the corner cases :-)." CreationDate="2011-05-17T20:47:20.017" UserId="4376" />
  <row Id="18794" PostId="10905" Text="Aaron, Thank you." CreationDate="2011-05-17T21:14:04.853" UserId="4559" />
  <row Id="18795" PostId="10909" Text="Hi Aaron, Thank you very much. Your R output is exactly the same with SAS output and the textbook. But Why we have to use &quot;nested&quot; in R syntax. Because in the textbook, it clearly states that $\alpha\beta$ is an random interaction term and I also use random interaction term in SAS. Could you please tell me why/how to specify an random interaction effect in lme, if it is possible? many many thanks Tu.2 " CreationDate="2011-05-17T21:19:08.733" UserId="4559" />
  <row Id="18796" PostId="10918" Score="1" Text="Argl. I may have just found the cause: as most forms of DA assume (?) multivariate normality per outcome class, they don't make much sense for categorical predictors (i.e. factors), right? This may be turning into more of a stats.stackexchange question afterall..." CreationDate="2011-05-17T20:51:04.910" UserId="4257" />
  <row Id="18797" PostId="10919" Text="This is not the case in my original data.frame. I didn't consider this when posting the simplified (i.e. smaller dataset). I get the same result, however (it's even the same one that is NA, and the strange presence of both factor levels is there too)." CreationDate="2011-05-17T20:12:39.080" UserId="4257" />
  <row Id="18798" PostId="10919" Text="If you have a numerically singular model matrix, most regression functions will omit variable or levels until they are able to return &lt;something&gt;.  Try putting the `cq` term in explicitly." CreationDate="2011-05-17T20:48:03.617" UserId="2129" />
  <row Id="18799" PostId="10911" Text="(1) To format math here, enclose the equation in dollar signs and make minor changes like &quot;\mu&quot; for &quot;mu&quot; and &quot;_&quot; to indicate subscripts.  I have made the changes for you to illustrate, because without them this question (and your others) may be hard to follow.  You had better double-check it because the new appearance might not be what you intended, especially the transition rates. (2) Where you have made progress (as in Q3, first part) please indicate your answer.  (3) This is a long complex question.  Although the parts are related, consider separating Q3.  You can link between questions." CreationDate="2011-05-17T21:57:24.853" UserId="919" />
  <row Id="18800" PostId="10911" Text="(4) Q1 can't be understood until you explain what the terms mean: $r$, $\mu$, $x$, $t$, $\theta$, $c$, $V_a$, $V_i$, $n-$.  (5) You can post small images (such as diagrams) and link to them inline in your question if that helps." CreationDate="2011-05-17T21:59:28.363" UserId="919" />
  <row Id="18801" PostId="10914" Text="It would help to write down the likelihood equations here." CreationDate="2011-05-17T22:02:00.373" UserId="919" />
  <row Id="18802" PostId="10920" Score="2" Text="You should at least mention whether order is a part of your 'similarity' score." CreationDate="2011-05-17T22:02:51.207" UserId="4257" />
  <row Id="18803" PostId="10920" Score="1" Text="@Shameer I was curious about a few things in your problem: Are data points the same as categorical attributes? What does your observational unit represent in the real-world? What are the 10K data points actually representing in the real-world. Is it human and human attributes? When you refer to inequality do you refer to the concern regarding human inequality or simply dispersion? Should each 'data point' be assumed to hold the  same weight? Thanks..." CreationDate="2011-05-18T00:51:31.990" UserId="4329" />
  <row Id="18804" PostId="10809" Text="@whuber - but why a choice of time varying covariates? where does that come from?  My answer to this question is the same as yours" CreationDate="2011-05-18T01:09:17.240" UserId="2392" />
  <row Id="18805" PostId="10902" Text="Nice answer (+1) - just one small point, the prior you describe is actually the improper haldene prior, not the jeffreys prior, jeffreys prior is proportional to the square root of your prior $p(\theta)\propto \sqrt{\theta^{-1}(1-\theta)^{-1}}$. And in order for your prior to work in all cases, you need to have at least one success and one failure observed (these remove the infinity at $0$ and $1$ in the prior). Possibly also worth mentioning that posterior is a Beta distribution with parameters $k$ and $n-k$" CreationDate="2011-05-18T01:22:07.480" UserId="2392" />
  <row Id="18806" PostId="10847" Text="@mpiktas, this question was related to a small part of a homework problem, but I didn't phrase the question in a way someone could simply tell me the answer. It was my every intention to understand the concepts and then solve my homework myself." CreationDate="2011-05-18T01:23:14.217" UserId="4627" />
  <row Id="18807" PostId="10849" Text="Thanks! Really great answer, however I needed to use a normal approximation. I completely understand and agree with your solution though." CreationDate="2011-05-18T01:25:34.963" UserId="4627" />
  <row Id="18808" PostId="10902" Text="Oh, thanks for the correction! I didn't know about the haldane prior. I'll update my calculations according to the real jeffreys prior then ;)" CreationDate="2011-05-18T01:49:29.647" UserId="4360" />
  <row Id="18810" PostId="10917" Text="Perhaps this belongs at http://math.stackexchange.com ? Just calculus, nothing statistical except maybe the reference to a hazard rate..." CreationDate="2011-05-18T03:24:03.687" UserId="26" />
  <row Id="18811" PostId="10916" Text="Please use Latex in your questions, it's difficult to read them otherwise." CreationDate="2011-05-18T03:26:24.030" UserId="26" />
  <row Id="18812" PostId="10922" Text="Really great answer!" CreationDate="2011-05-18T03:48:37.490" UserId="26" />
  <row Id="18813" PostId="10897" Text="@X I also wonder about your &quot;negative exponential&quot; assertion.  Fixing a segment of length $x$ in a unit rod (without any loss of generality), the chance that any single breakpoint misses it is $1-x$.  Because the breakpoints are independent, the chance that all of them miss it is $(1-x)^n$.  That's not a negative exponential: it's a polynomial in $x$.  Perhaps you're thinking of an asymptotic characterization for small $x$ and large $n$ (and bounded $nx$)?  With those asymptotics the rod's length is arbitrarily large compared to $x$ and therefore should *not* play any role." CreationDate="2011-05-18T04:35:44.093" UserId="919" />
  <row Id="18814" PostId="10901" Text="A big +1 for showing a practical way to figure out asymptotic behavior in general." CreationDate="2011-05-18T04:44:07.193" UserId="919" />
  <row Id="18815" PostId="10910" Text="@Melisa I think your terminology is problematic. Groups tend to refer to subsets of cases, I have changed your terminology in your question, however if I have misconstrued feel free to change back." CreationDate="2011-05-18T04:56:49.410" UserId="183" />
  <row Id="18816" PostId="10904" Text="@NRH, yes, yes. Some neuron misfired probably :)" CreationDate="2011-05-18T04:57:31.717" UserId="2116" />
  <row Id="18817" PostId="10922" Text="+1, very good answer." CreationDate="2011-05-18T04:59:30.280" UserId="2116" />
  <row Id="18818" PostId="10847" Text="@Mitch, your wording of the question was exemplary, this is how the homework questions should be asked. But knowing the problem context usually helps a lot in providing good answer. " CreationDate="2011-05-18T05:02:10.560" UserId="2116" />
  <row Id="18819" PostId="10926" Score="2" Text="Is the example at http://stats.stackexchange.com/q/5206/919 helpful?" CreationDate="2011-05-18T05:06:40.183" UserId="919" />
  <row Id="18820" PostId="7869" Text="@Matt: It has been a long while but I need some help here. Any way to handle negative weights. Sometimes, the value being subtracted from the weight is less than the weight itself resulting in negative weight. Is there a way to handle this and keeping the sum weight to 100?" CreationDate="2011-05-17T19:33:04.497" UserId="3535" />
  <row Id="18821" PostId="10890" Text="@mpiktas: I'm analyzing the problem of omitted variables in a regression. The omitting of variables is causes problems with consistency of the estimator. There are a bunch of other problems that causes inconsisteny (simultaneaus causality and measurement errors for example). All these problems are known as Endogenity. But in this context you often hear the word unobserved heterogenity. And I'm not sure if this is a synonym for Endogenity? Sorry I can't give you more information, because I don't have any (References are in Polish, you  wouldn't understand :)) " CreationDate="2011-05-18T07:25:48.457" UserId="4496" />
  <row Id="18822" PostId="10923" Text="Look here http://maya.cs.depaul.edu/classes/ect584/weka/classify.html" CreationDate="2011-05-18T07:41:27.163" UserId="930" />
  <row Id="18823" PostId="10932" Text="If you are saying $E[q] = 1 - E[a]$ then I would agree." CreationDate="2011-05-18T07:58:42.063" UserId="2958" />
  <row Id="18824" PostId="10932" Text="I edited somewhat to comply in notation. Thx." CreationDate="2011-05-18T08:22:23.103" UserId="4257" />
  <row Id="18825" PostId="10891" Score="2" Text="@IrishStat Basically, DBSCAN clusters points in n-dimensional space. So its suitability for time series analysis is very limited. It works for my special case since I expect only a single cluster in my (1-dimensional) histogram. To answer your questions: When using appropriate parameters, the value 5 of your example would be detected as outlier. I suppose it can deal with trends, but in the time series '1, 2, 3, 4, 5, 6, 7, 2', the value 2 would not be regarded as outlier! I hope, that gives a rough idea about the capabilities and limitations of DBSCAN." CreationDate="2011-05-18T08:24:52.867" UserId="4446" />
  <row Id="18826" PostId="10934" Text="+1 for the examples" CreationDate="2011-05-18T10:09:04.783" UserId="2116" />
  <row Id="18827" PostId="10897" Text="@whuber I was obtaining the probability that no breakpoint falls in a segment of length $x$ by $(1-x/M)^{nM}$, from which an exponential when $M$ goes to infinity. So, I divide $x$ in $M$ parts, and multiply the probabilities that each of these parts is _not_ hit by a breakpoint.  " CreationDate="2011-05-18T10:20:05.380" UserId="4642" />
  <row Id="18828" PostId="10916" Text="I have tried but I really dont get it. My exams tomorrow and Im really stressed out!!! Please help me!!!! Thats why my friend attached the document - its clearer that way. " CreationDate="2011-05-18T10:20:19.693" UserId="4645" />
  <row Id="18829" PostId="10916" Text="Either way if you draw out the diagram with the words given Im 100% sure you can make it out. Its really clear." CreationDate="2011-05-18T10:20:41.240" UserId="4645" />
  <row Id="18830" PostId="10911" Text="When my friend posted links you said no!!!! The thing is that r, mu etc are all transition intensities/interest rates.... the can only be that in Thieles differential equation. Plus you would need to draw the diagram before attempting the question because otherwise it’s rather confusing. There is a lot of detailed info already on there to allow for that I believe. If I’m wrong then please let me know asap how to change it as my exams tomorrow and I really need the help asap." CreationDate="2011-05-18T10:31:58.623" UserId="4645" />
  <row Id="18838" PostId="10891" Score="1" Text="@crypton Thanks for the info. It appears to me that when you refer to 1 cluster, you mean 1 and only 1 average. Even though there is no shift in average autocorrelation in your data renders it &quot;time series&quot; which suggests that DBSCAN may not be as useful as you think to separate signal from noise leading to efficient identification of anamolous data." CreationDate="2011-05-18T11:34:39.337" UserId="3382" />
  <row Id="18840" PostId="10902" Text="I will read (study?) the rest, but I have a problem with the construction of the first equation:  $P(k|\theta,n)=\left(n\atop k\right)\theta^k (1-\theta)^{n-k}$ implies that you think each one of the $n$ runs results in a success (with probability $\theta$) or a failure, am I right? But in reality each run of this experiment/simulation results in a success rate between 0 and 1. Because, in each run the algorithm is faced with, say, 1000 objects of the test set,and if it successfully recognizes, say, 800 of them, it gets a success-rate of 0.80. Also, probably, $k$ will be much larger than $n$." CreationDate="2011-05-18T11:38:37.113" UserId="4629" />
  <row Id="18842" PostId="10939" Text="Thanks chl. I tried it as well. Would it be possible however to get 1, 2, 3 ranks instead of 1, 2.5, 4?" CreationDate="2011-05-18T12:11:14.450" UserId="22" />
  <row Id="18843" PostId="10939" Score="1" Text="@radek Sure: `bysort group_id: egen desired_rank=rank(var_to_rank), unique` will give `1 2 3 4` for group 2, and replacing `unique` by `track` will give you `1 2 2 4`." CreationDate="2011-05-18T12:29:58.737" UserId="930" />
  <row Id="18846" PostId="10935" Text="For getting all files to csv, you could use something like what is presented at [this location](http://www.computing.net/answers/programming/script-convert-excel-ss-to-csv/16265.html). That should save you some work. You may need to adapt the VBScript code a little, but the idea is there." CreationDate="2011-05-18T12:47:39.087" UserId="4257" />
  <row Id="18847" PostId="10939" Text="@chl: As you said 'unique' option gives me '1,2,3,4' and my goal is to have '1,2,2,3'." CreationDate="2011-05-18T12:48:31.163" UserId="22" />
  <row Id="18848" PostId="10677" Text="@Zach Oups, I forgot to reply. Still, no clear idea wrt. `caret` package, but it seems that the [pathClass](http://cran.r-project.org/web/packages/pathClass/index.html) has some options for SVM+RFE. Didn't have a chance to test it, though." CreationDate="2011-05-18T12:51:37.383" UserId="930" />
  <row Id="18849" PostId="7225" Text="I think that $R^2$ can be defined in the usual way and can sometimes be helpful.  Even though standard errors do not account for bias, they are the standard errors of &quot;conservative, shrunk towards zero&quot; quantities.  They perhaps cannot be used for formal inference but I'd like to hear more discussion before concluded they should never be used." CreationDate="2011-05-18T13:01:26.730" UserId="4253" />
  <row Id="18851" PostId="10891" Text="@IrishStat I am not sure what you mean by &quot;1 and only 1 average&quot;. My understanding of 1 cluster is that **all** normal values are supposed to be very close (in terms of the euclidean distance) to each other. So DBSCAN telling me that there are outliers basically means that certain points exhibit an unusually high distance to the &quot;normal&quot; cluster which renders them anomalous for me." CreationDate="2011-05-18T13:09:20.107" UserId="4446" />
  <row Id="18852" PostId="10939" Text="@radek Does `egen desired_rank=group(var_to_rank)` produce what you're looking for? (but I would not call this ranking.)" CreationDate="2011-05-18T13:10:48.937" UserId="930" />
  <row Id="18853" PostId="10939" Text="@chl: It does indeed. However, it works for the whole dataset only since I cannot use it with bysort :/" CreationDate="2011-05-18T13:21:43.800" UserId="22" />
  <row Id="18854" PostId="10891" Score="1" Text="@crypton It appears to me that one cluster implies 1 average as compared to a time series that had say 1,1,1,1,2,2,2,2 with some small deviations around these values." CreationDate="2011-05-18T13:23:48.427" UserId="3382" />
  <row Id="18855" PostId="10891" Text="@IrishStat Yes, you are right. Given a reasonably small distance parameter $\epsilon$ (and assuming that the deviations between 1 and 2 do not &quot;connect&quot; the two clusters), the time series 1,1,1,1,2,2,2,2 would result in two clusters." CreationDate="2011-05-18T13:33:30.803" UserId="4446" />
  <row Id="18856" PostId="10940" Score="1" Text="@radek Good to know." CreationDate="2011-05-18T13:52:59.853" UserId="930" />
  <row Id="18857" PostId="10909" Text="Your question isn't about R syntax, it's about what nesting means.  Nesting B in A (with A/B) creates two variables, A and the interaction between A and B, which is exactly what you describe." CreationDate="2011-05-18T14:40:16.147" UserId="3601" />
  <row Id="18858" PostId="10941" Text="@Skarab As this question is not related to statistical analysis or data management (as defined in the [FAQ](http://stats.stackexchange.com/faq)) and because I see you have an SO account, I will migrate it for you on stackoverflow." CreationDate="2011-05-18T14:54:21.187" UserId="930" />
  <row Id="18859" PostId="10911" Score="1" Text="@user I meant *links between questions on this site,* not links to anonymous Web sites elsewhere.  Take a few minutes to read over some other questions and replies on this site: you will quickly see how to post images and diagrams and make a good question that stands by itself.  We appreciate your situation and are willing to help, but please bear in mind that we are not a dumping ground for lists of study questions.  We expect you to put some effort into formulating a clear, readable question, because it and the replies will become a searchable, permanent part of the record." CreationDate="2011-05-18T14:58:34.397" UserId="919" />
  <row Id="18860" PostId="10916" Score="1" Text="@user The onus is on you to formulate a clear, readable question.  When you put the burden on the reader, please do not expect any answers: the question will just get closed as being inappropriate or ill-posed." CreationDate="2011-05-18T15:00:17.843" UserId="919" />
  <row Id="18861" PostId="10897" Text="@X Those are the asymptotic conditions I had guessed.  Is the answer now clear to you?" CreationDate="2011-05-18T15:11:05.480" UserId="919" />
  <row Id="18862" PostId="10891" Score="1" Text="@crpton I think you are using a cross-sectional tool for a time series problem. Post the actual data and I will try to demonstrate anomaly detection to you with time series analysis" CreationDate="2011-05-18T15:48:42.297" UserId="3382" />
  <row Id="18864" PostId="10911" Score="1" Text="@user For examples of effective homework-related questions, please see (1) http://stats.stackexchange.com/q/6795, which shows the quality of answers that can be obtained once a question is well formulated; (2) http://stats.stackexchange.com/questions/7939, exemplifying a clearly articulated textbook question; and (3) http://stats.stackexchange.com/questions/4830, a question with (legitimate) links and using embedded diagrams for clarification." CreationDate="2011-05-18T16:11:42.447" UserId="919" />
  <row Id="18865" PostId="10897" Text="@whuber No sorry. First of all, thanks for your effort. Second, could you please either elucidate your answer (say, writing some formulae making your point clearer to me), or give me a pointer to some paper or website in which this problem has been studied already? Assume I don't consider any asymptotics, so the length of the rod matters. (I am also confused about the asymptotics at the moment, in the sense that at no stage I am posing $x$ small, but I think I can think my way out of _that_ confusion)." CreationDate="2011-05-18T16:16:08.880" UserId="4642" />
  <row Id="18866" PostId="10909" Text="Hi, This is a great explanation. Thank you very much." CreationDate="2011-05-18T17:03:38.233" UserId="4559" />
  <row Id="18868" PostId="10944" Text="Great, I loaded my data into R, did a weighted linear model and used predict for a new value at 0.0 similar to what was done [here](http://www.r-tutor.com/content/prediction-interval-linear-regression). Although, I do receive this error message &quot;Assuming constant prediction variance even though model fit is weighted&quot;. I'm assuming my result is still approximately correct. " CreationDate="2011-05-18T18:18:29.227" UserId="4643" />
  <row Id="18869" PostId="7878" Text="0 down vote accept&#xA;	&#xA;&#xA;@Matt: It has been a long while but I need some help here. Any way to handle negative weights. Sometimes, the value being subtracted from the weight is less than the weight itself resulting in negative weight. Is there a way to handle this and keeping the sum weight to 100?&#xA;" CreationDate="2011-05-18T19:20:19.710" UserId="3535" />
  <row Id="18870" PostId="10952" Score="2" Text="+1 That's right.  But (to head off possible confusion) the problem isn't just that $Y$ can equal $0$: it's that it can equal $0$ *with positive probability.*  (For instance, a quotient of normals does have a distribution even though the denominator can equal $0$.)  Thus, $X/Y$ is undefined with positive probability, making its mean (and any other moment) undefined as well." CreationDate="2011-05-18T20:06:20.903" UserId="919" />
  <row Id="18871" PostId="10953" Text="Are you saying that, say, F1-M2 was observed in condition &quot;A&quot; but not in condition &quot;B&quot; and that perhaps F1-M3 was observed in condition &quot;B&quot;?  That is, each female should have been observed in both conditions but perhaps was paired with different sets of males in those two conditions?" CreationDate="2011-05-18T21:03:54.743" UserId="919" />
  <row Id="18872" PostId="10952" Text="+1, but in the literature on false discovery rates people have no problem with $X/Y$ where $X$ is the true positives and $Y$ is the total number of positives :-). It is always understood, by convention, that 0 out of 0 equals 0.  " CreationDate="2011-05-18T21:10:35.893" UserId="4376" />
  <row Id="18873" PostId="10920" Text="@Nick, I need to consider the order as part of my score. Thanks for this, now I have I edited the question. " CreationDate="2011-05-18T21:38:41.760" UserId="529" />
  <row Id="18874" PostId="10949" Text="What would you like to do with your model?  It will be very difficult to estimate the causal-effect of having health insurance on healthcare expenditures, but it could be interesting to describe simple differences and conditional associations.  Like @Mermex I am getting the sense that you are wondering about the Heckman correction, you might try it but I don't see it as a panacea, see my comment on Mermex's answer.  For better answers, describe the data, especially how health insurance and expenditures are measured." CreationDate="2011-05-18T21:41:41.497" UserId="3748" />
  <row Id="18875" PostId="10925" Text="That's a wonderful answer ! Thanks a lot for your thoughts !!. I like the way you answered my concerns about applying Gini co-efficient to my problem. I will explore the possibilities of applying multinomial distribution here. " CreationDate="2011-05-18T21:43:26.690" UserId="529" />
  <row Id="18876" PostId="10953" Text="No, each female-male pair was observed in both conditions." CreationDate="2011-05-18T21:44:17.080" UserId="4655" />
  <row Id="18877" PostId="7775" Text="I suggest this question be migrated to SE. At the heart of the question seems to be an implementation problem, not a statistical one." CreationDate="2011-05-18T21:45:29.300" UserId="1909" />
  <row Id="18878" PostId="10920" Text="@B_Dev, The data is kind of a categorical label. I used A, B, C, D... to simplify the representation. In real-world these are protein domains(see: http://en.wikipedia.org/wiki/Protein_domain) encoded in a protein sequence sourced from a database of protein domains (for example see: http://pfam.sanger.ac.uk/). Here, 1 row = 1 protein with different domains. I am looking at dispersion, but am looking for a sequence-alignment free method here. I can assign weights to data points, but that will be an extension of my analysis. " CreationDate="2011-05-18T21:51:21.077" UserId="529" />
  <row Id="18879" PostId="10954" Score="1" Text="I agree that there are big differences between those who have health insurance and those who do not.  Why do you say there is no self selection?  Also, I would be careful not to oversell the Heckman correction... unbiased parameter estimates still require some fairly strong assumptions http://smr.sagepub.com/content/18/4/395.short and http://onlinelibrary.wiley.com/doi/10.1111/1467-6419.00104/abstract" CreationDate="2011-05-18T21:59:14.137" UserId="3748" />
  <row Id="18880" PostId="10949" Text="For background literature, in case you aren't aware: http://en.wikipedia.org/wiki/RAND_Health_Insurance_Experiment" CreationDate="2011-05-18T22:07:16.187" UserId="3748" />
  <row Id="18881" PostId="10952" Text="hmm ok lets put it that way: X is a poisson distributed random variable. Y is only observed if it is 1 or bigger. But Y is poisson distributed. For example you count the games that a player has done for your favorite hockey team and you call this random variable Y. So the counts have to be bigger than one. For example you might say Y is poison distributed with mean Y+1, so there is no way for 0 counts. What does this mean to the problem that I've mentioned above? Is Z poisson distributed? And what is its mean?" CreationDate="2011-05-18T22:07:59.120" UserId="4496" />
  <row Id="18882" PostId="9446" Text="I'm not sure what you mean when you say &quot;want to assess whether social worker attitudes have an effect on parental responses.&quot;  You don't mention collecting data on social workers' attitudes?  How do get to the overall outcome of the case?" CreationDate="2011-05-18T22:47:06.007" UserId="1991" />
  <row Id="18883" PostId="10952" Text="@Mark: It's probably better to ask that as a new question, and to get real specific about what you're trying to achieve." CreationDate="2011-05-18T23:05:36.287" UserId="2775" />
  <row Id="18884" PostId="7775" Text="@caracal - whatever you see fit. I know there's been a fair amount of discussion regarding where the line in the sand should be drawn between SO and CV regarding R related questions. I don't have a strong preference either way. My work has taken me away from this specific problem for the last bit, but will be revisiting it again in the coming weeks so I may come up with a better solution myself. I'm also fine letting the question fade off into the ether..." CreationDate="2011-05-19T00:26:10.180" UserId="696" />
  <row Id="18885" PostId="10950" Score="1" Text="The other difference is the factor 1/n instead of 1/(n-k) where n is the length of the series and k the number of lags. This is to ensure that the autocorrelation matrix is positive definite." CreationDate="2011-05-19T01:21:57.767" UserId="159" />
  <row Id="18886" PostId="10949" Text="Thanks so much for your valuable input M. Yes, selection bias is likely, especially given that the scheme is voluntary and the data are cross-sectional. I actually used propensity score matching in the analysis and made the assumption that the most important determinants of enrollment were observed through the questionnaire. However, I also want to show the results using OLS (and compare the insured and uninsured). What is the advantage of using OLS on a subsample of those who have had care vs. using a two-part model on the full dataset to look at expenditures conditional on health care use?" CreationDate="2011-05-19T01:25:23.860" UserId="834" />
  <row Id="18887" PostId="10952" Text="@NRH In your case there is a strong dependence of $X$ on $Y$.  That completely changes things, because now the probability of a positive:zero ratio is nil." CreationDate="2011-05-19T02:43:06.183" UserId="919" />
  <row Id="18888" PostId="10952" Text="@whuber, that's of course right. Thanks for pointing that out. I was just thinking that maybe there was some unstated convention to make the problem meaningful. But from @MarkDollar's comment above it seems that that was not the case to begin with." CreationDate="2011-05-19T06:59:10.320" UserId="4376" />
  <row Id="18889" PostId="10883" Text="@ Cardinal: Could you please explain more about Gaussian efficiency? For example what's the difference between Qn which benifits from 82% Gaussian efficiency and  MAD with 37%? Actually my background is Coastal engineering far from statistics! " CreationDate="2011-05-19T07:13:28.840" UserId="4286" />
  <row Id="18890" PostId="10891" Text="@IrishStat I uploaded the file [here](http://www.2shared.com/file/DGjvtnIu/anomaly-scores.html). It contains a total of 8496 values which is about 6 days of scores. Every minute, a new score was added." CreationDate="2011-05-19T07:59:10.543" UserId="4446" />
  <row Id="18891" PostId="10902" Text="Ok, i misread your question then. So you have $n$ runs, and for each run you have $M$ test cases and a randomly picked subset of pixels you use for classification, which results in $k_i\in\{0,...,M\}$ successful classifications for the $i$th run (or a success rate of $\theta_i=k_i/M$), right? Then you basically want to estimate the mean and variance of the unknown probability distribution of $\theta$, and choose $n$ so that the uncertainty in both gets sufficiently small. I'm not sure, but perhaps a frequentist approach would be easier here." CreationDate="2011-05-19T08:12:37.113" UserId="4360" />
  <row Id="18892" PostId="10961" Score="1" Text="@teucer, prices are usually unit-root processes. Did you check for that? If they are unit-root, your regression might not make sense." CreationDate="2011-05-19T08:27:44.610" UserId="2116" />
  <row Id="18893" PostId="10961" Text="how good is your fit?" CreationDate="2011-05-19T08:28:38.013" UserId="1709" />
  <row Id="18894" PostId="10961" Text="@RockScience seems to be good (t-test and F-test). Adjusted R-squared =0.97 (I believe its value does not make sense here).@mpiktas I did a test with 'adf.test' from tseries package: _p=0.43 alternative:stationary_, which means (if I correctly interpreted) that it contains an unit-root..." CreationDate="2011-05-19T08:39:01.170" UserId="1443" />
  <row Id="18895" PostId="10963" Text="Depends on application... Any more details? #s of movies and users? Where do you want to use this dissimilarity?" CreationDate="2011-05-19T10:43:54.637" UserId="88" />
  <row Id="18896" PostId="10963" Text="I would like to find the similar taste users and suggest them as friends to follow. So there will be N number of movies and X Number of users. Each like the movie based on their taste. I don want to look for category of movie. I want the matching percentage of similarity among users." CreationDate="2011-05-19T10:51:57.173" UserId="4665" />
  <row Id="18898" PostId="10966" Text="you mean to say like&#xA; Fn(A,B) = C[like, unspecified, unlike]&#xA;         = C[1, 2, 2]&#xA;like    = A &amp; B like X = 1&#xA;dislike = SUM (A dislike on B's like = V, B dislike on A's like = Y) = 2&#xA;unspecified = Z, L = 2&#xA;is that correct?&#xA;" CreationDate="2011-05-19T11:33:58.523" UserId="4665" />
  <row Id="18899" PostId="10880" Text="@Steve still there ? It would be nice if you could add some prices, I'd like to play around a litte bit (which may lead to a solution ...)" CreationDate="2011-05-19T11:49:55.727" UserId="264" />
  <row Id="18900" PostId="10961" Text="@teucer, if you have unit roots in general you should fit VECM." CreationDate="2011-05-19T12:49:06.013" UserId="2116" />
  <row Id="18901" PostId="598" Score="1" Text="pySal by the GeoDa Center is open source (see below)" CreationDate="2011-05-19T13:29:37.870" UserId="4329" />
  <row Id="18902" PostId="2353" Text="True comment. But as an experienced programmer I feel dirty every time I use Matlab/Octave, which is a horribly-designed (if it's designed at all) language. Of course, I also cringe at SAS, which was obviously designed for punched cards." CreationDate="2011-05-19T13:38:40.443" UserId="1764" />
  <row Id="18903" PostId="7681" Text="You could define diversity as the likelihood of randomly encountering some one of a different ethnic group.&#xA;&#xA;The geographic proximity of people can influence random encounters, so all things being equal, the measure should account for cities with highly residential segregation having less diversity." CreationDate="2011-05-19T13:46:05.093" UserId="4329" />
  <row Id="18904" PostId="10972" Text="+1, I've started to write my own explanation, but yours is way better." CreationDate="2011-05-19T14:03:55.443" UserId="2116" />
  <row Id="18905" PostId="10957" Text="Thank you Manoel. This is helpful. I am willing to accept the fact that there may be selection bias and have noted this as a limitation. However, given this limitation I was wondering if there is an advantage to using a two-part model and what this advantage is. I guess I don't really understand what the two-part model does that makes it more useful than just taking a subsample of individuals who used health care and comparing expenditures between insured and uninsured. But I think you have alluded to the difference - the 2-part model takes into account the prob of enrolment. Correct? Thanks!" CreationDate="2011-05-19T14:11:56.127" UserId="834" />
  <row Id="18906" PostId="10975" Text="What are the fitted values from?  What is your model?  arcsin is (approximately) variance stabilising for binomial, but you will still have &quot;edge&quot; effects if the proportions are close to 0 or 1 - because the normal part effectively gets truncated." CreationDate="2011-05-19T14:49:17.623" UserId="2392" />
  <row Id="18907" PostId="10957" Text="I'm assuming that the two part model you refer is the Heckman correction sugested by Mermex, in this case, yeah, it will take into account the prob of enrolment. If it's not, than I don't know if it's ok to just estimate the probability of using health and then use this probability as a independent variable in the main regression. The heckman correction has this idea, but I think it's slightly different and has some assumptions to work." CreationDate="2011-05-19T15:11:28.173" UserId="3058" />
  <row Id="18908" PostId="10964" Score="1" Text="You may also be interested in [this recent question](http://stats.stackexchange.com/questions/10789/why-are-random-variables-defined-as-functions), which touches on some related points." CreationDate="2011-05-19T15:12:33.873" UserId="2970" />
  <row Id="18909" PostId="9336" Text="Cardinal is right.  Trying different fits or using loess to guide fits results in improper inference down the road (e.g., confidence bands that are too narrow, statistical tests that don't preserve type I error).  When there are no mechanistic reasons to choose a parametric form, fitting a restricted cubic (natural spline) to the data using the number of papers that the sample size well supports will result in an excellent fit and proper inference (proper penalty for # parameters in the model = number of visable parameters; don't need to consider parameters &quot;screened out&quot; from earlier looks)." CreationDate="2011-05-19T15:20:25.333" UserId="4253" />
  <row Id="18910" PostId="10982" Text="Try Wikipedia? [Moments](http://en.wikipedia.org/wiki/Laplace_distribution#Moments). You can also use the moment-generating function. :)" CreationDate="2011-05-19T15:29:18.023" UserId="2970" />
  <row Id="18911" PostId="10981" Text="Please post a link to the math.SE question. Usually it's not good to cross-post unless significant time has elapsed." CreationDate="2011-05-19T15:30:21.457" UserId="2970" />
  <row Id="18912" PostId="10981" Text="Can the same number be selected twice? Is n smaller than or greater than U?" CreationDate="2011-05-19T15:30:55.713" UserId="4257" />
  <row Id="18913" PostId="10981" Text="Here is the previous question on math.SE: http://math.stackexchange.com/questions/32569/a-question-about-linear-regression" CreationDate="2011-05-19T15:31:21.457" UserId="2970" />
  <row Id="18915" PostId="10981" Text="@Nick Sabbe The selection is without replacement. n is always smaller than U." CreationDate="2011-05-19T15:34:23.683" UserId="4670" />
  <row Id="18917" PostId="10982" Text="One ring to rule them all: [Wikipedia](http://en.wikipedia.org/wiki/Laplace_distribution): at the side you have a table holding mean, variance, skewness and kurtosis." CreationDate="2011-05-19T15:27:29.787" UserId="4257" />
  <row Id="18918" PostId="10982" Text="This answer may be better placed as a comment, in general. Especially since it is not self-contained." CreationDate="2011-05-19T15:33:15.697" UserId="2970" />
  <row Id="18919" PostId="10982" Text="I'll keep that in mind. Thx." CreationDate="2011-05-19T15:41:07.707" UserId="4257" />
  <row Id="18920" PostId="10981" Text="The problem can be stated more simply: let $\{(x_i,y_i)\} = (1,y_1), \ldots, (n,y_n)$ be the data with the $y_i$ integers, $1 \le y_1 \lt y_2 \cdots \lt y_n \le U$.  Finding $\mathbb{E}[r_{xy}]$ is hopeless.  There's some chance of attacking $\mathbb{E}[r^2_{xy}]$ but it's not by any means easy (it's a ratio of two quadratic forms in the $y_i$)." CreationDate="2011-05-19T16:17:12.407" UserId="919" />
  <row Id="18921" PostId="10966" Text="Ananth, I would also consider assigning a value to matches of dislikes.  In other words, A &amp; B both dislike X implies some degree of similarity in tastes.  You may want to weight it differently than matching likes, but it should have relevence." CreationDate="2011-05-19T16:20:16.397" UserId="4671" />
  <row Id="18922" PostId="10981" Text="@whuber Thank you for your comments. If $E[r_{xy}]$ cannot be calculated, an estimation or lower bound is ok." CreationDate="2011-05-19T16:20:20.450" UserId="4670" />
  <row Id="18923" PostId="10972" Score="1" Text="+1. May I just add, for the visually oriented, that we often plot the sample path $(n, \frac{1}{n}S_n)$ and expect that the graph will level of around the expectation for large $n$. This is *precisely* what the SLLN says will happen. The WLLN says something that is more vague, and it does not guarantee that the graph will level of eventually. Since the SLLN holds, it will, but there are other counterexamples that show that we can have convergence in probability and yet with probability one the sample path will not level of." CreationDate="2011-05-19T16:21:33.260" UserId="4376" />
  <row Id="18924" PostId="10981" Text="@Fan That's a good comment: by allowing approximation or bounding, you are opening up the question to many more solution methods.  Consider making this point explicit in the question itself.  If you can, please indicate the ranges of values of $U$ and $n$ you're interested in.  For instance, would you be interested in asymptotics as $U \to \infty$?" CreationDate="2011-05-19T16:24:38.827" UserId="919" />
  <row Id="18925" PostId="10984" Text="I don't think my comment is worthy of a separate answer, but since you are doing daily samples, one thing to look for is patterns based on conventions that won't be inherently obvious from the data set. Consider if there is a weekly or monthly pattern, for example, which may be stronger than other intervals. For something like running, you would also need to consider a weather overlay to see if rain affected the outcomes if it occurs outdoors. " CreationDate="2011-05-19T16:24:45.707" UserId="4671" />
  <row Id="18926" PostId="10985" Text="Because actual parameter values are a matter of convention, and conventions vary, please define your dispersion parameter or at least provide a link to the definition you are using." CreationDate="2011-05-19T16:27:58.217" UserId="919" />
  <row Id="18927" PostId="10981" Text="@whuber I think if $U\rightarrow\inf$, the problem can be seen as selecting with replacement? This would be also my interest." CreationDate="2011-05-19T16:29:14.813" UserId="4670" />
  <row Id="18928" PostId="10981" Text="@Fan for $U \sim \infty$, then after rescaling by $1/U$ you can view the $y_i$ as the order statistics for a sample of $n$ iid variables from a uniform distribution on $[0,1]$, thereby replacing the discrete sums in the original problem with integrals." CreationDate="2011-05-19T16:33:41.190" UserId="919" />
  <row Id="18929" PostId="10981" Text="@whuber I see. Do you have some relevant materials about your approach?" CreationDate="2011-05-19T16:39:39.003" UserId="4670" />
  <row Id="18930" PostId="10982" Score="3" Text="@Nick, I did not mean to discourage you from answering, either. Please consider submitting an answer detailing your favorite technique (or two) for deriving the results. It may be instructive for the OP and others. :)" CreationDate="2011-05-19T16:46:38.823" UserId="2970" />
  <row Id="18931" PostId="10981" Text="@Fan Applicable techniques would include quadratic forms in random variables (http://stats.stackexchange.com/questions/9220), the &quot;delta method&quot; for estimating moments of functions of random variables; distributions of order statistics for uniform variables; the relationship between gaps between uniform variables and the exponential distribution, and possibly even saddlepoint methods, normal approximations, Central Limit Theorem, etc." CreationDate="2011-05-19T17:12:26.197" UserId="919" />
  <row Id="18932" PostId="10981" Text="@whuber Thank you very much, the information helps me much!!" CreationDate="2011-05-19T17:34:09.703" UserId="4670" />
  <row Id="18933" PostId="10986" Text="Could you explain what evidence do you have that the &quot;adjusted regression&quot; is not multiple regression? Because that's what it usually means." CreationDate="2011-05-19T18:39:47.603" UserId="279" />
  <row Id="18934" PostId="9336" Text="I'm having trouble reconciling these comments with the tradition of Anscombe, Tukey, Mosteller, Tufte, and Cleveland, which emphasizes the need to visualize and explore data and to size up the shape of each relationship before building a model, establishing coeffiencients, or generating other statistics." CreationDate="2011-05-19T18:40:51.910" UserId="2669" />
  <row Id="18935" PostId="10975" Text="Let me double down on what @probabilityislogic has said and also inquire about where the data come from. There might be something in the problem which suggests another transformation, or another model entirely, which might be more appropriate and/or interpretable." CreationDate="2011-05-19T18:47:23.730" UserId="26" />
  <row Id="18936" PostId="10986" Text="Sure, I think what it refers to is the following: The idea appears to be that in regression analysis the predictors and response variables are affected by a multiplicative factor (an observable covariate). Commonly suggested  is the correction for body mass index, height and so on when measuring serum levels of certain compounds.  I gathered that in typical medical data analysis, this adjustment is normally just done by  dividing measured values by the confounding variables. More recently, adjustment methods  are using varying coefficient regression to model this behviour more complexly.   " CreationDate="2011-05-19T18:52:24.037" UserId="4673" />
  <row Id="18937" PostId="10991" Text="There are two problems with this: first, the notation is so vague as to be meaningless (what is the variable of integration?).  Second, no conceivable interpretation of this is correct for the example given in the problem, which requires integration of a product of a central and a noncentral t distribution." CreationDate="2011-05-19T18:56:20.087" UserId="919" />
  <row Id="18938" PostId="10982" Text="@crucified I would like to know what is meant by &quot;the empirical rule.&quot;  Could it perhaps refer to unbiased estimators of the moments?" CreationDate="2011-05-19T19:00:38.723" UserId="919" />
  <row Id="18939" PostId="10975" Text="@prob @JMS Why don't we let the OP, who I believe is quite knowledgeable about stats, try the transformation route first?  Then, if that doesn't work, it would be fruitful to commence a new thread in which the problem is presented less narrowly.  Your comments would be appropriate in that context." CreationDate="2011-05-19T19:08:24.193" UserId="919" />
  <row Id="18940" PostId="10991" Text="@whuber : As I describe in the question, delta, sigma, and N are known, so the variable of integration is x. I didn't know how to write this in notation, but it should be integrated from 0 to 1. As for it being correct, try it yourself, using the program R:&#xA;T &lt;- function(x) {&#xA;pt(((7*sqrt(108))/20)-qt(1-(0.5*x),499),499)&#xA;}&#xA;integrate(T,0,1)$value&#xA;Where, delta is 4, sigma is 20, and N is 108. Compare the results with this code to the results you get from the code I gave previously. It does work!" CreationDate="2011-05-19T19:44:57.580" UserId="4647" />
  <row Id="18941" PostId="10991" Text="Sorry, those 499s should be 107s, they are the degrees of freedom, not that it really matters with a sample size this large. Also, delta is 7 in this example, my mistake." CreationDate="2011-05-19T19:52:47.353" UserId="4647" />
  <row Id="18942" PostId="10977" Score="1" Text="possible duplicate: http://stats.stackexchange.com/questions/5081/java-library-for-bayesian-networks" CreationDate="2011-05-19T20:27:22.047" UserId="264" />
  <row Id="18943" PostId="10979" Text="whuber, know of any R function that does this one automatically?" CreationDate="2011-05-19T20:34:59.470" UserId="601" />
  <row Id="18944" PostId="10954" Text="@Michael Bishop: My thinking was that you are not really self selected to have health insurance, i.e., it comes with a job. Unlike, you do choose to go to a college, where there is obvious self selection. Thanks for the links. " CreationDate="2011-05-19T20:45:12.097" UserId="4617" />
  <row Id="18945" PostId="10979" Text="@John No I don't, but it's simple enough to implement." CreationDate="2011-05-19T20:46:02.393" UserId="919" />
  <row Id="18946" PostId="10991" Text="@Nick I put the limits of integration in and the variable of integration.  Note that this is an approximate expression, which is why it does not exactly agree with the R code you supplied.  The function you gave in your comment is not correct: you need to use a non-central T for the alternative distribution." CreationDate="2011-05-19T20:49:21.550" UserId="919" />
  <row Id="18947" PostId="10979" Score="1" Text="I didn't see it as basically difficult but it would be nice if there was something like the boxcox tranforms that automatically plot out the best selection for lambda.  Yes, not terrible to implement... " CreationDate="2011-05-19T20:59:25.120" UserId="601" />
  <row Id="18948" PostId="10999" Text="Thank you, that does indeed look like what I need!" CreationDate="2011-05-19T21:12:10.233" UserId="4677" />
  <row Id="18949" PostId="10997" Text="This is appears to be a pure programming problem: identify sequences of constant values in a stream of numbers (and Excel is one of the worst possible platforms in which to implement a solution).  If you're open to non-Excel solutions (and even if you're committed to using Excel), you might benefit by having this question migrated to SO.  If you think this question still belongs here, please edit it to indicate where you need *statistical* help." CreationDate="2011-05-19T21:13:33.447" UserId="919" />
  <row Id="18950" PostId="10972" Text="@mpiktas, Thanks.  @NRH, yes, you are exactly right.  If I get some time later this evening maybe I can add a graph like you were talking about." CreationDate="2011-05-19T21:24:26.260" UserId="1108" />
  <row Id="18951" PostId="11000" Text="@David What do you mean by &quot;I can calculate each row individually&quot;?" CreationDate="2011-05-19T21:37:19.173" UserId="930" />
  <row Id="18952" PostId="10999" Score="3" Text="@Bernd (+1) It's merely a fork of Mx but it is quite good as free software for SEM. Another promising project is [lavaan](http://lavaan.ugent.be/) (which tries to provide Mplus compliant output, whenever possible)." CreationDate="2011-05-19T21:43:17.550" UserId="930" />
  <row Id="18953" PostId="10998" Text="@Bernd +1 and the first link you provide references a thread therein :-)" CreationDate="2011-05-19T21:45:26.957" UserId="930" />
  <row Id="18954" PostId="10987" Score="2" Text="@KennyPeanuts Just to avoid copy/paste, `sink()` or `capture.output()` might be your friends. Reporting utilities, like [Hmisc](http://cran.r-project.org/web/packages/Hmisc/index.html), [Sweave](http://bit.ly/ljRv4I), or [brew](http://cran.r-project.org/web/packages/brew/index.html) are worth to consider (your point 1). Versioning systems ([rcs](http://www.gnu.org/software/rcs/), [svn](http://subversion.apache.org/), or [git](http://git-scm.com/)) might help with point 2. " CreationDate="2011-05-19T21:53:38.433" UserId="930" />
  <row Id="18955" PostId="10998" Text="@chl Thanks! I was wondering if this question is a duplicate and should be closed... " CreationDate="2011-05-19T21:54:24.893" UserId="307" />
  <row Id="18956" PostId="10998" Text="@Bernd It is, IMO. But as there are no votes to close, I'm reluctant to close it. Also, the older one was more general, but a very similar question has been closed in the past. Let's wait and see how it goes." CreationDate="2011-05-19T21:58:15.157" UserId="930" />
  <row Id="18957" PostId="10986" Text="@user4673 Why didn't you link your two SE accounts?" CreationDate="2011-05-19T22:05:59.113" UserId="930" />
  <row Id="18958" PostId="10845" Score="1" Text="exactly, bad practice like using assign to create multiple single element variables should be discouraged" CreationDate="2011-05-19T23:27:48.457" UserId="3343" />
  <row Id="18959" PostId="11000" Text="Sorry, meant to say &quot;I can regress the column matrix B against the columns in A individually&quot;, meaning one-at-a-time calls to lm. Edited to reflect this.  " CreationDate="2011-05-19T23:28:04.487" UserId="1699" />
  <row Id="18962" PostId="11008" Score="1" Text="Do you mean the **standard error** for each proportion? If this is what you are looking for, then this webpage might be of interest for you: [Interval Estimate of Population Proportion](http://www.r-tutor.com/elementary-statistics/interval-estimation/interval-estimate-population-proportion)" CreationDate="2011-05-20T01:11:30.213" UserId="307" />
  <row Id="18963" PostId="11005" Text="@Berd: Thanks a lot." CreationDate="2011-05-20T01:24:14.087" UserId="3903" />
  <row Id="18964" PostId="10986" Text="Could you provide more details, like an equation? Varying coefficient regression (be it slope or intercept) occurs in multilevel/hierarhical models. And related to this, there is the concern about adjsuting standard erros to take into account multiple comparisons. This paper by Andrew Gelman discuss the issue of multiple comparison in the context of multilevel modeling: http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf" CreationDate="2011-05-20T01:36:59.097" UserId="3058" />
  <row Id="18965" PostId="10788" Text="Any comments anyone? Anyone do lead/lag analysis of related cross sector stock prices?" CreationDate="2011-05-20T01:41:42.400" UserId="4338" />
  <row Id="18966" PostId="10993" Text="So, we're trying to show $E(r_{xy}^2)$ is close to 1 (or that we can expect to find nearly a straight line) [I'm working on this problem with Zhang Fan].  While $r_{xy}^2$ bad in some cases, there should be relatively few of these cases.  One hope for resolving this problem is therefore to bound the number of cases in which e.g. $r_{xy}^2 \leq 0.99$ (or some other bound)." CreationDate="2011-05-20T01:42:24.273" UserId="386" />
  <row Id="18967" PostId="10993" Text="I'm hoping that 0.99 should be a reasonable bound to consider.  E.g. if we compute some examples in R, we can consistently obtain cor(x,y)'s that are _very_ close to 1.  E.g. 0.9994561 is a typical result returned by:  m &lt;- 10^5; n &lt;-10^3; list &lt;- sample(1:m,size=n,replace=FALSE); list &lt;- sort(list); cor(list,1:n);" CreationDate="2011-05-20T01:48:20.777" UserId="386" />
  <row Id="18968" PostId="11011" Text="@Aaron AFAIK none of these packages can estimate _multilevel_ path/structural equation models." CreationDate="2011-05-20T02:06:29.840" UserId="307" />
  <row Id="18969" PostId="11011" Text="Didn't realize that.  Glad you spoke up about OpenMx." CreationDate="2011-05-20T02:07:54.597" UserId="3601" />
  <row Id="18970" PostId="10986" Score="1" Text="Is this it? http://anson.ucdavis.edu/~mueller/carglmbiometrikarevfinal.pdf" CreationDate="2011-05-20T02:11:03.153" UserId="375" />
  <row Id="18971" PostId="11008" Text="That'll do! I can calculate SD from SEM. I suppose I could've done the same to calculate SEM from the 95% confidence interval provided by prop.test too, but this is better. Thanks @Bernd!" CreationDate="2011-05-20T02:22:33.367" UserId="4238" />
  <row Id="18972" PostId="1806" Text="@Rob Hyndman: Can you point me where I can find documentation on how to do the Fisher test using SPSS? Thanks" CreationDate="2011-05-20T02:48:07.237" UserId="1663" />
  <row Id="18973" PostId="10972" Text="@NRH Thanks for the suggestion;  I made some graphs, but they seemed better placed in the other question here: http://stats.stackexchange.com/questions/2230/convergence-in-probability-vs-almost-sure-convergence/11013#11013" CreationDate="2011-05-20T02:49:15.090" UserId="1108" />
  <row Id="18974" PostId="11008" Score="1" Text="Nooo! As @Bernd noted, the proportion does not have a standard deviation. Or more precisely, it does, but it is called the standard error. Standard deviation refers to the variability of the original 0-1 variable." CreationDate="2011-05-20T02:55:04.447" UserId="279" />
  <row Id="18975" PostId="10845" Text="@mdsumner, why is it a bad practice?" CreationDate="2011-05-20T03:33:47.487" UserId="2116" />
  <row Id="18976" PostId="11009" Text="My philosophy is run lots of models, check their predictions, compare, explain, run more models." CreationDate="2011-05-20T03:43:22.080" UserId="3748" />
  <row Id="18977" PostId="11008" Text="Oh! Wow, thanks for the clarification @Aniko...that wouldn't have been good to report. Ok, I'll stick with the standard error. Thanks again!" CreationDate="2011-05-20T03:43:32.970" UserId="4238" />
  <row Id="18978" PostId="11018" Text="Can someone tag this with &quot;weighting&quot; or something to that effect?  And I suppose it should be community-wiki." CreationDate="2011-05-20T03:55:25.267" UserId="3748" />
  <row Id="18979" PostId="10948" Text="Thanks Wolfgang! This is exactly what I was looking for. Now I can try and replicate it in Excel (for my students who use Excel only)." CreationDate="2011-05-20T04:09:51.213" UserId="1945" />
  <row Id="18980" PostId="10950" Text="@Rob: I believe that Excel's CORREL formula does use n (rather than n-k). For example, for lag-1 ACF you get the same result (using Wolfgang's notation) if you use COVAR(res[1:49],res[2:50])/(STDEVP(res[1:49])*STDEVP(res[2:50])) and the functions COVAR and STDEVP are &quot;population&quot; statistics." CreationDate="2011-05-20T04:14:37.107" UserId="1945" />
  <row Id="18981" PostId="11020" Score="3" Text="@Galit Shmueli: Can you please be a litte bit more explicit about the identifiability problem? " CreationDate="2011-05-20T04:58:58.443" UserId="3019" />
  <row Id="18982" PostId="11021" Text="I got C-, the words I use are too short..." CreationDate="2011-05-20T05:03:52.353" UserId="1709" />
  <row Id="18983" PostId="11002" Text="@frank-- I understand this point partly..would be great if you can elaborate the &quot;improper scoring rule&quot; and &quot;bogus model&quot; thing please." CreationDate="2011-05-20T05:35:16.727" UserId="1763" />
  <row Id="18984" PostId="11021" Score="2" Text="I would suggest you try different words and see what works and what doesnt..you can take the null hypothesis as &quot;buzzwords do not matter&quot; and see if you can reject it or not." CreationDate="2011-05-20T05:48:00.833" UserId="1763" />
  <row Id="18985" PostId="1806" Text="Look under crosstabs" CreationDate="2011-05-20T05:57:28.377" UserId="159" />
  <row Id="18986" PostId="10950" Text="@Galit. Even using COVAR and STDEVP, the denominator in your code would be 49, but the preferred definition of autocorrelation will use 50." CreationDate="2011-05-20T06:02:23.317" UserId="159" />
  <row Id="18987" PostId="10965" Text="Thx for the quick reply, if i have understand it right the one-step forecasts are then provided by&#xA;splitting a time series in two sequences: [60 Periods] t1&lt;-[36 periods] fitting sample t2&lt;-[24 periods] test sample.....both tseries objects. fit the model within the first periods: model1&lt;-ets(t1) then using the model to the out-of-sample data t2: test&lt;-ets(t2,model=model1) and the one-step-ahead forecast with seeds in t=37,38 is preserved by&#xA;fitted(test). am i right? or how can i get the one-step forecasts to an test data-set&#xA;using the model i estimated from a fitting data-set. thx in advance&#xA;T" CreationDate="2011-05-19T11:45:50.537" UserId="4667" />
  <row Id="18988" PostId="10965" Text="Yes, that would give the one-step forecasts in the test set. ets() does not try to guess the seasonality. If you don't include the frequency, you will only get non-seasonal models." CreationDate="2011-05-19T23:42:36.627" UserId="159" />
  <row Id="18990" PostId="11014" Score="2" Text="This may need some clarification. Can you confirm that: 1) the data you have, are the sales per month for 5 years. 2) The bell curve these sales follow is 'per year' (i.e. if you look at the plot of sales over a year, it would look like an interrupted gauss curve) _or_ 'per month', over the years (i.e. if you had data for 1000 years and looked at a histogram for January, it would look bell shaped). 3) Where, then, do these constraints between -1 and 0 come from (i.e. what does this -1 and zero mean here)." CreationDate="2011-05-20T07:15:51.133" UserId="4257" />
  <row Id="18991" PostId="11024" Text="(-1/+1): 1. The OP did explicitly say that this situation arises pretty frequently. 2. You wrote: *If you roll your own algorithm, you could exclude this class from the algorithm and just throw the nonfitting observations 'in the garbage'* How to determine non-fitting observations during application of the classifier ? If the garbage class is known, what is the advantage in comparison to keep the garbage class 3. I think with &quot;confidence&quot; the OP referenced to &quot;confidence of a single decision&quot; not total classifier quality." CreationDate="2011-05-20T07:17:57.747" UserId="264" />
  <row Id="18992" PostId="11021" Text="Ayush, I was hoping to find a quick solution, also, I'm not sure if Rezscore is representative of other resume schemes." CreationDate="2011-05-20T07:20:08.370" UserId="4685" />
  <row Id="18993" PostId="11021" Score="1" Text="@user654914 -- sorry,I dont think there is any quick soln to the problem..we cant be sure of the way rezscore or likes work. I think we can just hypothesize." CreationDate="2011-05-20T07:25:39.267" UserId="1763" />
  <row Id="18994" PostId="10992" Text="Sorry for being unprecisely. So first of all theta is the shape parameter of the negbin distribution (it's not a scale parameter) [http://books.google.de/books?id=86rWI7WzFScC&amp;pg=PA136&amp;lpg=PA136&amp;dq=Applied+econometrics+shape+parameter+negative+zeileis&amp;source=bl&amp;ots=BST0I7Kz_i&amp;sig=p-XQuxfL2OoJhojtl2t_RYU-aq4&amp;hl=de&amp;ei=cBzWTaafMYTAswbI6-2QBw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CB4Q6AEwAA#v=onepage&amp;q&amp;f=false]. I just need a proof that the dispersionparamter is taken to be one. In other words, can anybody show, that the negbin distribution is member of the exponential family with $\phi=1?" CreationDate="2011-05-20T07:48:23.913" UserId="4496" />
  <row Id="18995" PostId="11023" Text="@ayushbiyani, I had experience with SQL. I learned myself the rest and got a base certification. Not sure it will be any benefit to get the advanced from an employers perspective." CreationDate="2011-05-20T08:33:31.330" UserId="1077" />
  <row Id="18996" PostId="11023" Text="Ok..then whats the problem ? I reckon that this post was long back..may be your problem got solved by now.Isnt it ?" CreationDate="2011-05-20T09:02:19.727" UserId="1763" />
  <row Id="18997" PostId="11018" Text="@Michael Not sure about a proper tag for &quot;sampling weights&quot;, so I let other propose better alternatives, if any. " CreationDate="2011-05-20T09:08:31.790" UserId="930" />
  <row Id="18998" PostId="11019" Score="1" Text="IMO you should ask the second part (confidence) as a different question -- it is a broad topic and very classifier-dependent." CreationDate="2011-05-20T09:12:34.683" UserId="88" />
  <row Id="18999" PostId="10995" Text="Sweave is wonderful. It takes some getting used to, but if you already know TeX and R, its the obvious choice. It also allows you to never spend time aligning table columns ever again, which is nice. " CreationDate="2011-05-20T09:22:47.903" UserId="656" />
  <row Id="19000" PostId="11018" Text="@chl Let's try [weighted-sampling] -- should be clear enough, IMO. " CreationDate="2011-05-20T09:26:23.860" UserId="88" />
  <row Id="19002" PostId="2036" Text="Actually, for N=2, this result can be simplified, where the summation after C is finite." CreationDate="2011-05-20T08:17:10.563" UserId="4687" />
  <row Id="19003" PostId="2036" Text="@Tina Welcome to the site; but please don't use answers for comments. Please read the [FAQ](http://stats.stackexchange.com/faq)." CreationDate="2011-05-20T09:30:26.023" UserId="88" />
  <row Id="19004" PostId="11019" Text="regarding confidences: this [question](http://stats.stackexchange.com/questions/10387/what-do-real-values-refer-to-in-supervised-classification) might help." CreationDate="2011-05-20T09:38:45.707" UserId="264" />
  <row Id="19005" PostId="6314" Text="@cardinal The top of page 24 in Cover and Hart looks like a place where an issue may potentially arise in their proof, in the step where Cover and Hart argue that every RV x \in X has the property that every open sphere about x has non-zero measure. If we consider the geometry of the hypersphere we see that the volume of the interior of the hypersphere shrinks with increasing dimension so, in the limit, the open ball about x contains only x in its interior. Alternatively, via the SLLN, the iid RVs x in the metric space X all lie in the surface of the hypersphere with probability one." CreationDate="2011-05-20T09:47:31.563" UserId="3248" />
  <row Id="19006" PostId="11004" Text="If R is only tag for your question then this question is better fit on StackOverflow.com" CreationDate="2011-05-20T10:12:44.927" UserId="2633" />
  <row Id="19008" PostId="10979" Text="Thanks whuber, this is exactly the kind of thing I was looking for and the graph is really helpful. Definitely agree with John that something like the boxcox would be helpful, but this seems simple enough to work through." CreationDate="2011-05-20T11:01:24.557" UserId="266" />
  <row Id="19009" PostId="11023" Text="@ayushbiyani, exactly this was a long time ago - I took the advice given here! Hope it helps others as well." CreationDate="2011-05-20T11:18:22.523" UserId="1077" />
  <row Id="19010" PostId="11031" Text="Could you perhaps post the error messages?" CreationDate="2011-05-20T11:22:57.717" UserId="198" />
  <row Id="19011" PostId="11031" Text="@Jonas: I don't know if this is helpful but here they are: Der Befehl &quot;which&quot; ist entweder falsch geschrieben oder&#xA;konnte nicht gefunden werden.&#xA;  demo_chain_feat chain&#xA;error: display: not defined for class &quot;char&quot;&#xA;error: called from:&#xA;error:   C:\Octave\3.2.4_gcc-4.4.0\share\octave\3.2.4\m\general\display.m at line 36, column 3&#xA;error:   runmodel.m at line 17, column 1&#xA;error:   masterrun.m at line 64, column 38" CreationDate="2011-05-20T12:27:52.307" UserId="230" />
  <row Id="19012" PostId="10998" Text="@Bernd - thanks for the list!  This is very useful. I figured I wasn't the first to have this question but I didn't seem to find much with my (obviously inept) initial search." CreationDate="2011-05-20T13:01:38.170" UserId="4048" />
  <row Id="19013" PostId="10995" Text="@Aaron - thanks for the input.  My 2 concerns with Sweave are 1) I will wind up with a bazillion files in my directory - especially with a lot of figures, and 2) I will have to be really careful with the code to prevent hiccups each time I compile the whole doc (e.g., I think something is loaded and its not).  Do you have these problems?" CreationDate="2011-05-20T13:06:43.160" UserId="4048" />
  <row Id="19014" PostId="10995" Text="1) You can choose to delete the intermediate files if you want; I use the Sweave.sh script (http://cran.r-project.org/contrib/extra/scripts/Sweave.sh) which does so automatically; though can easily be turned off. If you do that, make sure you know what it will delete before using it to prevent possible disaster. The short version is that if no files share the basename of your Rnw file you're okay." CreationDate="2011-05-20T13:13:48.710" UserId="3601" />
  <row Id="19015" PostId="10987" Text="@chl - thanks for the suggestions. I wasn't aware of  `sink()` and `capture.output()`. Thats great.  " CreationDate="2011-05-20T13:15:58.443" UserId="4048" />
  <row Id="19016" PostId="10995" Text="2) In my opinion, having to be careful in that way is a Good Thing, and will sometimes compile my Sweave document with that exact purpose in mind, that is, to make sure I've properly kept track of everything needed to recreate the analysis." CreationDate="2011-05-20T13:16:48.473" UserId="3601" />
  <row Id="19017" PostId="11011" Text="(I'll keep the answer, because especially with Bernd's comment, it may be useful to others asking this same question.)" CreationDate="2011-05-20T13:17:47.023" UserId="3601" />
  <row Id="19018" PostId="11035" Score="5" Text="Note that `glm(y ~ x)` does not give you a logistic regression, You have to set `family=binomial(link=&quot;logit&quot;)`. Note the output says `Dispersion parameter for gaussian family`, not `binomial family`. If you do it right, `fitted(glm.mod)` actually returns the estimated probabilities, not the logits. You get the logits with `predict(glm.mod, type=&quot;link&quot;)`." CreationDate="2011-05-20T13:18:05.797" UserId="1909" />
  <row Id="19019" PostId="10980" Text="Thanks for your knowledgeable comment. I will explore the cpower function." CreationDate="2011-05-20T13:19:33.497" UserId="3429" />
  <row Id="19020" PostId="11017" Text="For various reasons, Tukey recommends adding +1/6 to counts.  Note that this reply is a special case of Tukey's folding approach that I described: any CDF with positive PDF is monotonic; folding a symmetric CDF leaves it unchanged." CreationDate="2011-05-20T13:19:56.607" UserId="919" />
  <row Id="19021" PostId="11016" Text="What if the interaction is only significant when the main effects are not in the model?" CreationDate="2011-05-20T13:25:39.650" UserId="2310" />
  <row Id="19022" PostId="11022" Text="Here it is kind of the opposite.  The interaction (of interest) is only significant when the main effects are not in the model." CreationDate="2011-05-20T13:27:12.770" UserId="2310" />
  <row Id="19023" PostId="10993" Text="@Douglas That suggests what might be a consderable simplification: by taking a target like 0.99, you can convert the ratio of quadratic forms appearing in $r^2_{xy}$ into a *difference* (by clearing the denominator) and then investigate the chance that the difference is positive.  So now you're looking at the distribution of a quadratic form under uniformly random permutations: that's an accessible problem." CreationDate="2011-05-20T13:31:34.680" UserId="919" />
  <row Id="19024" PostId="10993" Text="@Douglas, @Fan We can do some heuristics, too: asymptotically, the $y_i$ will look uniform, indicating $r^2 \to 1$ as $U \to \infty$.  Consider the Kolmogorov-Smirnov statistic $D$, for instance: using that as an upper bound for the residuals shows it is proportional to a (gross) overestimate of $1-r^2$.  Therefore $r^2 \to 1$ at least as fast as $D \to 0$." CreationDate="2011-05-20T13:36:09.743" UserId="919" />
  <row Id="19025" PostId="10992" Text="Sorry about the shape/scale mistake, I fixed it. Please edit your original question to make it clear to everyone what you are after." CreationDate="2011-05-20T13:45:44.923" UserId="279" />
  <row Id="19026" PostId="11035" Text="Aua! I have fixed that. Thanks a lot, @caracal, for correcting me! This is really embarrassing (it is even more embarrassing as I have already given the correct answer in [another SO thread](http://stats.stackexchange.com/questions/2335/output-of-logistic-model-in-r)). " CreationDate="2011-05-20T13:46:25.883" UserId="307" />
  <row Id="19027" PostId="9768" Text="I would appreciate a discussion about why the 3 experiments would be analyzed separately, as opposed to building a grand model, and what is the best choice for the grand model." CreationDate="2011-05-20T13:48:05.027" UserId="4253" />
  <row Id="19028" PostId="11039" Text="Is this a homework question? There's a deep, lengthy academic literature on this topic.  I suggest you look at what others have done." CreationDate="2011-05-20T14:01:06.533" UserId="4691" />
  <row Id="19029" PostId="7716" Text="One of my favourite admissible estimators: a single arbitrarily chosen point of the parameter space which is not an impossible value :)" CreationDate="2011-05-20T14:03:48.217" UserId="2392" />
  <row Id="19030" PostId="11019" Text="@steffen : That's interesting, and I'll certainly take a look. Although I was actually that there would be something less labor intensive :-)" CreationDate="2011-05-20T14:03:54.890" UserId="3301" />
  <row Id="19031" PostId="11039" Text="@Keith Which topic in particular?  Regression?  Time series?  Adjusting data? Model selection?  Political statistics?  It would be good to have *some* place to start in following your suggestion!" CreationDate="2011-05-20T14:20:43.903" UserId="919" />
  <row Id="19032" PostId="10427" Text="Andy W is right.  They would be omitted because of collinearity. Just drop one of the dummies or use `noconstant` (xtreg will do this for you)" CreationDate="2011-05-20T14:21:11.610" UserId="4691" />
  <row Id="19033" PostId="11037" Text="@probabilityislogic : Well actually, that &quot;something else&quot; in my case is just noise. So I don't think I will be able to model that. " CreationDate="2011-05-20T14:25:27.417" UserId="3301" />
  <row Id="19034" PostId="11039" Text="I was referring to the literature which empirically studies &quot;institutions&quot; Big names among economists include Acemoglu, Easterly, Kaufmann, Levin, North, Rodrik.  But there's TONS more.  Put &quot;corruption&quot; or something like that into Google Scholar :) " CreationDate="2011-05-20T14:32:02.213" UserId="4691" />
  <row Id="19035" PostId="10396" Text="@jeromy -- apologies for late reply. The datamart doesnt have case in which 0 or 5 dependent is the case..Not sure if missing value means one of those. We only have dependents from 1 to 4. I agree that this may have implications in modeling." CreationDate="2011-05-20T14:43:54.563" UserId="1763" />
  <row Id="19036" PostId="11043" Score="1" Text="I'm missing something: the Well1024 RNG *already* produces uniformly distributed values in [0,1].  It sounds like you're saying &quot;I can convert these uniform values to a lognormal distribution, now can someone help me convert them back?&quot;  Isn't the answer, don't make the lognormal conversion in the first place??" CreationDate="2011-05-20T15:12:04.577" UserId="919" />
  <row Id="19037" PostId="11044" Text="@Wolfgang +1 for the instructive example" CreationDate="2011-05-20T15:12:24.530" UserId="307" />
  <row Id="19038" PostId="11031" Text="@vonjd I didn't run through all code exceptions, but from the very few ones I checked, it is possible to trace back all errors (yes, one by one :-) and get it work with Octave, or close to. For example, the first error you reported has to do with `display` method not defined for a `char`acter variable. Replace with `disp` and it works. Then, the program will throw another error when trying to create subdirectories... I'm afraid I cannot help more, but I do think spending some time solving those compatibility bugs would be profitable to your understanding of Octave and this particular toolbox." CreationDate="2011-05-20T15:14:31.040" UserId="930" />
  <row Id="19039" PostId="11043" Text="I don't understand why, but logNormalGen() take a uniform random distribution to produce logNormal random number, i have no choice here :/ http://www.iro.umontreal.ca/~simardr/ssj/doc/html/umontreal/iro/lecuyer/randvar/LognormalGen.html But i need normalized value after, so me too i'm missing something perhaps ?" CreationDate="2011-05-20T15:25:53.407" UserId="4693" />
  <row Id="19040" PostId="10704" Text="If any answer helped you or you find out another way it is better to give us, so all community will take advantage of that. Or choose an answer to close the question, thx." CreationDate="2011-05-20T15:26:13.797" UserId="4531" />
  <row Id="19041" PostId="11043" Score="1" Text="Use the uniform RNG linked from that page: http://www.iro.umontreal.ca/~simardr/ssj/doc/html/umontreal/iro/lecuyer/randvar/UniformGen.html ." CreationDate="2011-05-20T15:27:39.213" UserId="919" />
  <row Id="19042" PostId="11043" Text="The uniform RNG produce uniform value, but i want to pick random number into a lognormal distribution(which need a random uniform stream), then after normalize each value" CreationDate="2011-05-20T15:32:42.340" UserId="4693" />
  <row Id="19043" PostId="11008" Score="1" Text="Even more precisely, &quot;standard error&quot; of the proportion refers to the standard deviation of the distribution of the sample proportions from random samples of the particular sample size from the population of interest." CreationDate="2011-05-20T15:55:49.787" UserId="3874" />
  <row Id="19044" PostId="10950" Text="The 1/n vs 1/(n-k) point is good for understanding in addition to the other points above.  But for practical/observed numbers it is not going to matter as long as it is consistent, since that term shows up in both the numerator and the denominator it will cancel out.  You could get a problem if different fractions were used in the numerator and denominator." CreationDate="2011-05-20T15:58:19.443" UserId="4505" />
  <row Id="19045" PostId="11009" Score="2" Text="If the interactions are only significant when the main effects are in the model, it may be that the main effects are significant and the interactions not. Consider one highly significant main effect with variance on the order of 100 and another insignificant main effect for which all values are approximately one with very low variance. Their interaction is not significant, but the interaction effect will appear to be significant if the main effects are removed from the model." CreationDate="2011-05-20T16:26:35.877" UserId="3874" />
  <row Id="19046" PostId="11031" Text="@chl: in the main file masterrun.m there is no &quot;display&quot;, only &quot;disp&quot; and I thought the display.m is an own file of Octave. I have no clue how this one is called." CreationDate="2011-05-20T16:30:09.117" UserId="230" />
  <row Id="19047" PostId="10986" Text="@Eduardo Yes, that is what I am making reference to (in future I will add papers)! Another paper I've been looking at is: http://arxiv.org/pdf/math/0607010. I perhaps don't need something as complex as these two methods (they make reference to simpler methods of &quot;normalization&quot;). Even so, I am still not sure if this is 100% what &quot;Adjusted Analysis&quot; is, however these papers describe what I am essentially seeking to do. " CreationDate="2011-05-20T16:41:46.550" UserId="4673" />
  <row Id="19048" PostId="11043" Score="1" Text="Normalize *how*? Therein lies your answer. There are an infinite number of ways of &quot;normalizing&quot; a lognormal random variable to fall in the range $[0,1]$. The resulting distribution on $[0,1]$ will vary according to the choice you make." CreationDate="2011-05-20T16:44:05.987" UserId="2970" />
  <row Id="19049" PostId="11009" Text="@Thomas should your first line read 'if the interactions are only significant when the main effects are NOT in the model, ...'?" CreationDate="2011-05-20T16:46:21.073" UserId="2310" />
  <row Id="19050" PostId="11043" Score="3" Text="Can you provide some detail on what the *actual problem* you are trying to solve is? I think that will help guide toward an answer." CreationDate="2011-05-20T16:46:31.740" UserId="2970" />
  <row Id="19051" PostId="11043" Text="I update my message, i hope this is more clear.." CreationDate="2011-05-20T17:06:31.887" UserId="4693" />
  <row Id="19052" PostId="11035" Score="1" Text="the package arm has the function invlogit, which is your function logit2prop." CreationDate="2011-05-20T17:41:49.433" UserId="3058" />
  <row Id="19053" PostId="10986" Text="@Eduardo Leoni, +1 for providing the answer about what is adjusted regression! " CreationDate="2011-05-20T17:44:46.780" UserId="3058" />
  <row Id="19054" PostId="10814" Text="@cardinal errr... OK, I did,... and it's right?  Would you please point out the error?" CreationDate="2011-05-20T18:01:17.277" UserId="1108" />
  <row Id="19055" PostId="10792" Text="@cardinal does it not?  I added material to my answer which suggests to me that it does?" CreationDate="2011-05-20T18:02:59.287" UserId="1108" />
  <row Id="19057" PostId="11039" Text="I don't know, but my gess is that you have to extrapolate or estimate the data, treating it as missing data. You do that using a bayesian approch, in which you could estimate both the effect of the covariates on the dependente variable and estimate the missing data." CreationDate="2011-05-20T18:18:56.903" UserId="3058" />
  <row Id="19058" PostId="11031" Text="@vonjd `masterrun.m` calls `runmodel.m`, where `display()` is used line 17: all this is given in the error trace from Octave. Each time, you must read the stack of errors msg (from bottom to top) to locate the problematic code. Yes, `display()` is an internal function, and is located in `/m/general/` of octave source tree. IMO, your question is purely a matter of programming so I'm inclined to migrate it to StackOverflow. I'll leave it open for some days in case someone already did the Matlab -&gt; Octave translation." CreationDate="2011-05-20T18:20:07.887" UserId="930" />
  <row Id="19059" PostId="3817" Text="People tend to misinterpret stepwise regression results as giving THE BEST set of predictors of an outcome, when in reality what it gives is, at best, SOME GOOD predictors." CreationDate="2011-05-20T18:56:06.397" UserId="2669" />
  <row Id="19060" PostId="11025" Text="Thanks, but lm(A.ex~B.ex) in your code fits 9 points against A1 (correct) and 9 points against A2 (undesired). There are 10 measured points for both B1 and A2; I'm throwing out one point in the regression of B1 against A2 because the corresponding point is missing in A1. If that's just The Way It Works I can accept that, but that's not what I'm trying to get R to do." CreationDate="2011-05-20T19:07:07.357" UserId="1699" />
  <row Id="19061" PostId="10814" Text="(+1) Apologies. I'm not sure where my head was when I first looked at this. It's obviously (well, should have been!) correct." CreationDate="2011-05-20T19:09:37.253" UserId="2970" />
  <row Id="19062" PostId="10792" Text="@G. Jay, yes it does. As a check, $x = g^{-1}( g(x) )$ for all $x$ in the domain. ($g^{-1}$ is composed of a set of functions all of which have inverses. So, that should have been clear[er] from the start.)" CreationDate="2011-05-20T19:11:50.523" UserId="2970" />
  <row Id="19063" PostId="11028" Text="That makes sense given my understanding of how na.exclude should work. However, if you call  &gt;X.both = cbind(X1, X2) and then &gt;dim(lm(X.both~Y, na.action=na.exclude)$residuals) you still get 94 residuals, instead of 97 and 97." CreationDate="2011-05-20T19:13:52.937" UserId="1699" />
  <row Id="19064" PostId="10814" Text="@cardinal, no harm, no foul.  I admit, though, you really had me sweating for a minute!  :-)" CreationDate="2011-05-20T19:24:48.293" UserId="1108" />
  <row Id="19065" PostId="11043" Text="&quot;One of my parameter is probability between 0.0 and 0.0000000001, with precision of 0.000000001.&quot;  Where does this come from and what does it really mean?  I am asking this because nterpreting the 0.000000001 as the arithmetic mean (which is a guess) and the precision as either the variance or the SD (which are conventional), it is mathematically impossible for any random variable in the range [0, 1] to have these characteristics." CreationDate="2011-05-20T19:38:36.113" UserId="919" />
  <row Id="19066" PostId="11046" Text="@Dikran : So if I understand you right, you are basically suggesting doing what I was alluding to in comment (2) in the edits? I just found a one-class SVM implementation in libsvm, so I think I am going to use that as first cut. " CreationDate="2011-05-20T19:59:23.067" UserId="3301" />
  <row Id="19067" PostId="10972" Text="@G. Jay, great, and you got +1 for that answer too." CreationDate="2011-05-20T20:00:56.310" UserId="4376" />
  <row Id="19069" PostId="11054" Score="1" Text="Maybe you should just try regular boosting? The effect should be the same, but the whole should be easier to validate." CreationDate="2011-05-20T20:02:42.927" UserId="88" />
  <row Id="19070" PostId="11025" Text="@David, oh, it looks like I've misunderstood your problem. I'll post the fix later." CreationDate="2011-05-20T20:02:48.777" UserId="2116" />
  <row Id="19071" PostId="11056" Text="@Zach: Sorry, I think my question was a little unclear. I've made an edit clarify my thoughts!" CreationDate="2011-05-20T20:04:36.153" UserId="2252" />
  <row Id="19072" PostId="10845" Text="Ok this is bizarre, why the downvote? My answer clearly solves the problem (I've tested it), and I've given the arguments why this particular piece of code is acceptable. It does not involve any fancy hackery it is plain R. " CreationDate="2011-05-20T20:07:02.970" UserId="2116" />
  <row Id="19073" PostId="11054" Text="@Jonathan, thanks for the edit. Could you tell us how big/small your training groups are? By &quot;running out of usable data&quot; do you mean that Node 2 contains few data points, hence that you don't have many Good objects to begin with? You could also consider general ensemble methods to combine classifications from different methods." CreationDate="2011-05-20T20:11:13.120" UserId="4376" />
  <row Id="19074" PostId="11057" Text="@NRH: Yes, we are on the same page, I am trying to create a hierarchical series of classification methods to try an reduce error rate. I have been looking at the distributions and densities of the features, and there is, as you describe, significant overlap in many of the features. I believe a potential problem is that the classification of &quot;Good&quot; and &quot;Bad&quot; in my dataset, and I will have to speak with subject matter experts to better determine (and hopefully quantify) what exactly distinguishes &quot;good&quot; from &quot;bad&quot;." CreationDate="2011-05-20T20:13:13.417" UserId="2252" />
  <row Id="19075" PostId="11054" Text="@mbq: Boosting sounds like a good option to explore. I am familiar with the idea, but have never implemented it before. I will have to investigate how to perform boosting in R. Any R packages or other resources that you may be able to recommend?" CreationDate="2011-05-20T20:18:26.940" UserId="2252" />
  <row Id="19076" PostId="11054" Text="@NRH: The original dataset is approx 140,000 points. This is split into about 105,000 training set and 35,000 holdout set. From my edit, Node 0 starts with 35,000 entries, which is split into Node 1 with about 30,000 entries (99% &quot;Bad&quot;) and Node 2 with about 5,000 entries (3400 &quot;Bad&quot;, 2100 &quot;Good&quot;)." CreationDate="2011-05-20T20:21:06.533" UserId="2252" />
  <row Id="19077" PostId="11054" Text="@Jonathan, I am confused. Why does Node 0 has 35.000 and not 105.000 data points? And in other contexts people with 5000 data points would think they have a huge data set." CreationDate="2011-05-20T20:27:25.960" UserId="4376" />
  <row Id="19078" PostId="11031" Text="@chl: I changed &quot;display&quot; to &quot;disp&quot; but got still the same error?!? Anyway, I think it is quite a task to port it (at least for me :-(" CreationDate="2011-05-20T20:27:29.317" UserId="230" />
  <row Id="19079" PostId="11057" Text="@Jonathan, that is probably a good idea. The classification problem can also just be downright difficult. " CreationDate="2011-05-20T20:30:16.083" UserId="4376" />
  <row Id="19080" PostId="11028" Text="That's an improvement, but if you look at residuals(lm(X.both ~ Y, na.action=na.exclude)), you see that each column has six missing values, even though the missing values in column 1 of X.both are from different samples than those in column 2. So na.exclude is preserving the shape of the residuals matrix, but under the hood R is apparently only regressing with values present in all rows of X.both. There may be a good statistical reason for this, but for my application it's a problem." CreationDate="2011-05-20T20:31:03.447" UserId="1699" />
  <row Id="19081" PostId="10814" Text="+1 Really nice addition to your answer! (which was already very good)" CreationDate="2011-05-20T20:51:38.733" UserId="26" />
  <row Id="19082" PostId="11043" Text="Sorry, i made a mistake, i would have say &quot;one of my parameter is a probability between 0.00000000001 and 0.1 and i want to vary the step ( and a little step like 0.000000001 if it's possible)&quot;" CreationDate="2011-05-20T20:54:48.370" UserId="4693" />
  <row Id="19083" PostId="11054" Text="@NRH: I trained my model with 105,000 data points and used it to predict the classes of the holdout set of 35,000 points. I am measuring my performance based on the test set, which is why I say Node 0 has 35,000 points. Is this the wrong way to thing? And yes, 5000 data points is not &quot;small&quot;, but relative to what I started with it, I felt that I might need more data to appropriately train the next step (i.e. make another training/test set). :) " CreationDate="2011-05-20T20:55:20.870" UserId="2252" />
  <row Id="19084" PostId="11056" Score="1" Text="@Jonathan check out the packages gbm, mboost, ada, and caTools for various boosting methods in R." CreationDate="2011-05-20T21:07:08.430" UserId="2817" />
  <row Id="19085" PostId="11056" Text="@Zach: Thanks, I will take a look later today. I would also be very grateful for some example code, since you offered :)" CreationDate="2011-05-20T21:23:20.827" UserId="2252" />
  <row Id="19086" PostId="11054" Score="1" Text="@Jonathan, keep the holdout as holdouts and train the second classifier on the training data again. That is, Node 0 is just your training data. It may or may not give improved performance, but that is how you should proceed. " CreationDate="2011-05-20T21:28:38.903" UserId="4376" />
  <row Id="19088" PostId="11054" Text="@NRH: Yes, that makes much more sense than what I did. I will redo the analysis and see how things go." CreationDate="2011-05-20T21:47:55.680" UserId="2252" />
  <row Id="19089" PostId="11034" Text="How about ordinal logistic regression? What would be the logic then?" CreationDate="2011-05-20T22:22:55.680" UserId="333" />
  <row Id="19090" PostId="11028" Text="@David I had misunderstood your question. I think I now see your point, and have edited my answer to address it." CreationDate="2011-05-20T22:34:33.717" UserId="1909" />
  <row Id="19091" PostId="11067" Text="I found that too, but where do I put it?  Before the summary() call?  Inside it?  I tried both and it still won't show the whole table." CreationDate="2011-05-20T22:44:08.183" UserId="4110" />
  <row Id="19092" PostId="11067" Text="Most certainly before it. First you run `options(max.print=5.5E5)`, then you run `summary(M)`. Note of course that you may need to alter the value 5.5E5 to your needs. Although I hope you really don't want to print 550000 lines and hope to get something out of it. It may be worth checking the value after you have set it by running `getOption(&quot;max.print&quot;)` after setting it." CreationDate="2011-05-20T23:06:46.963" UserId="4257" />
  <row Id="19093" PostId="11028" Text="@caracal Very helpful, thanks. " CreationDate="2011-05-21T00:09:00.373" UserId="1699" />
  <row Id="19094" PostId="11037" Text="@andy - you can model &quot;noise&quot; - but it depends on what sort of noise you are modelling.  For example, one way to model noise is using a mixture model with lots of components, or some other &quot;non-parametric&quot; model.  The idea is to get a model with lots of parameters, so it can fit any data you give it.  you then set non-informative priors over these and you effectively introduce a penalty for each parameter (similar to BIC)." CreationDate="2011-05-21T00:26:34.163" UserId="2392" />
  <row Id="19095" PostId="11068" Text="Much easier to interpret coefficient that way" CreationDate="2011-05-21T00:31:23.707" UserId="2392" />
  <row Id="19096" PostId="11002" Text="I have a simulated dataset example where adding a very important variable (by any other measure) makes the % classified correctly decrease.  This is an example of how an improper scoring rule can mislead." CreationDate="2011-05-21T00:48:05.783" UserId="4253" />
  <row Id="19097" PostId="3108" Text="I think this showing that &quot;randomness&quot; cannot be fully defined.  If you start defining &quot;randomness&quot; to much, then it becomes systematic.  One good example is shuffling cards - if you do it in a systematic way, then the shuffling achieves nothing." CreationDate="2011-05-21T00:50:32.160" UserId="2392" />
  <row Id="19098" PostId="11063" Text="Thanks @Nick! That's good to know I'm using this model correctly here.&#xA;For the other part of my question, I don't think I was clear enough. Rather than the model *giving* me hatch success in one way or another, I should've asked if it was *testing* hatch success among years as # of hatches/egg laid rather than # of hatches/nest. From your answer, I _think_ it's # hatches/egg laid?&#xA;Thanks again!" CreationDate="2011-05-21T01:00:34.913" UserId="4238" />
  <row Id="19099" PostId="11034" Text="@user333: Well... I haven't played much with ordinal logistic regression... but I think one uses the same link function. In any case, the logic is the same: inverse the link function to get the response variable..." CreationDate="2011-05-21T05:30:18.370" UserId="3019" />
  <row Id="19100" PostId="11069" Text="@Ehab, I made quite a few modifications to make the question clearer, could you please check that the original meaning was not lost." CreationDate="2011-05-21T05:35:27.427" UserId="2116" />
  <row Id="19101" PostId="11072" Text="@Or Zuk, what your s regression model is designed for? Do you have to interpret parameters, or you indeed are interested in the best fit (that not necessarily result best forecasting performance)? Perhaps what you indeed are looking for is some sort of cross-validation, to be sure that parameter estimates are stable and not sample dependent. There are many nice suggestions how to do cross-validation in other posts." CreationDate="2011-05-21T05:45:58.620" UserId="2645" />
  <row Id="19102" PostId="11069" Score="1" Text="@Ehab, there are several problems with your question, which should be addressed. 1. Why do you mention GLS, `Arima` uses maximum likelihood. 2. What is your model? You mention the error, but what is the variable you are trying to model? 3. You say that the forecast is larger than expected, how do you know that?" CreationDate="2011-05-21T05:57:25.070" UserId="2116" />
  <row Id="19103" PostId="11059" Text="+1, very good example!" CreationDate="2011-05-21T06:18:43.070" UserId="2116" />
  <row Id="19106" PostId="11071" Text="@MYaseen208 As it seems purely a matter of R programming, let us migrate this question to StackOverflow where I hope you'll get a response." CreationDate="2011-05-21T09:10:03.153" UserId="930" />
  <row Id="19107" PostId="11063" Text="Yes, hatches per egg (see in my answer: the probability of an egg hatching)." CreationDate="2011-05-21T09:33:06.047" UserId="4257" />
  <row Id="19108" PostId="10946" Text="@whuber, thanks for your answer. I have one residual doubt. My original quest was to find a pdf for the lengths of the fragments. now, if I use $(1-\frac{x}{L})^n$, I run into troubles. First, (let's assume $L=1,n=1$) it is not normalized $\int_0^1 (1-x)\, dx=\frac{1}{2}$. Secondly, imagine I want to compute the average length of a fragment. That'd be $\int_0^1 x(1-x)\, dx=\frac{1}{6}$, which I don't understand either. What's your take?" CreationDate="2011-05-21T11:42:55.273" UserId="4642" />
  <row Id="19109" PostId="10801" Text="you are right, its on 24:10, I think thats the best example to understand the nature of cov matrix and its inverse" CreationDate="2011-05-21T11:44:36.833" UserId="4581" />
  <row Id="19110" PostId="9336" Text="There is a lot of controversy regarding their approaches.  An over-simplified way to summarize these issues is that if you want to learn about patterns and make new discoveries that need later validation, exploratory analysis is appropriate.  If you want to draw inference (reason from particular sample to general population using P-values, confidence intervals, etc.) then not so much." CreationDate="2011-05-21T12:34:21.253" UserId="4253" />
  <row Id="19111" PostId="11076" Score="4" Text="The wisdom or lack thereof depends on the range of the dependent variable of interest. Consider car braking data where you have speeds and stopping distances. You could fit a quadratic model with or without intercept. Speeds of interest usually start around 50 km/hr and go up to, say, 130 km/hr. Fitting a quadratic *with* intercept in this case makes more sense, I think, as forcing the intercept to zero can incur (practically) significant lack-of-fit problems. The fact that the &quot;braking distance&quot; of a stopped car is zero is not particularly relevant for the modeling problem at hand." CreationDate="2011-05-21T13:39:56.480" UserId="2970" />
  <row Id="19112" PostId="11079" Text="You don't include &quot;condition&quot; as a variable in your model, meaning that you are left with multiple observations per cell of the model and I'm not sure how aov deals with that. Try ezANOVA from the ez package and comment here if there are any warnings or errors. Command: ezANOVA( data=scrd , wid=.(subject) , dv=.(response) , within=.(stimulus,condition) , between=.(sex) , observed=.(sex) )" CreationDate="2011-05-21T14:08:02.140" UserId="364" />
  <row Id="19113" PostId="11072" Text="Thanks. My goal is actually a bit different. I want to compute power to reject the null hypothesis (the regression model). I'm generating data using different (other) models and compare them in terms of how easily can you detect that they're different from the null model - but I assume that the statistician performing the test doesn't know the alternative model, so he can't apply a log-likelihood ratio test between the two models, he just needs a test for rejecting the null model and the goal is to see how powerful is this test under different alternative models" CreationDate="2011-05-21T14:35:53.223" UserId="3036" />
  <row Id="19114" PostId="11016" Text="@Glen - There are many things to think about other than statistical significance.  See [this](http://stats.stackexchange.com/questions/10510/what-are-good-references-containing-arguments-against-null-hypothesis-significanc/10511#10511). Better to examine your overall model fit (plot your residuals against your predictions for each model you fit), your theory, and your motivations for modeling." CreationDate="2011-05-21T15:28:19.047" UserId="3748" />
  <row Id="19115" PostId="11079" Text="This link gives a very nice explanation on how to do a repeated measures ANOVA with `nlme`: http://blog.gribblelab.org/2009/03/09/repeated-measures-anova-using-r/" CreationDate="2011-05-21T15:53:08.297" UserId="582" />
  <row Id="19116" PostId="11045" Text="Do you understand the concepts of standard-error, confidence interval, statistical population, sample, parameter vs. statistic?  I'd look all those up on wikipedia." CreationDate="2011-05-21T15:59:01.093" UserId="3748" />
  <row Id="19117" PostId="11063" Text="Brilliant! Thanks again @Nick!" CreationDate="2011-05-21T16:06:49.590" UserId="4238" />
  <row Id="19118" PostId="11079" Text="Maybe you could be a bit more explicit about your design: Do you have two fully-crossed within-subject IVs, `stimulus` and `condition`, meaning that each observer had all stimuli in all conditions? `sex` obviously is a between-subjects IV, and thus does not belong into `Error()`. At first glance, the formula may be `response ~ stimulus * condition * sex + Error(subject/(stimulus * condition))`." CreationDate="2011-05-21T16:25:50.167" UserId="1909" />
  <row Id="19119" PostId="11086" Text="Thanks again @Bernd. I thought I'd searched for this answer, but clearly I missed it! Cheers!" CreationDate="2011-05-21T17:40:32.520" UserId="4238" />
  <row Id="19120" PostId="11046" Text="@Andy, sort of, I was suggesting using a one class classifier for novelty detection, but a standard multi-class classifier for performing the classification of observations that are not novelties.  I think that may work out better as one-class classifiers are good for novelty detection, but are not likely to be as good as a discriminative classifier for performing the actual classification.  Libsvm is an excellent bit of kit, so you are off to a good start." CreationDate="2011-05-21T17:57:06.287" UserId="887" />
  <row Id="19121" PostId="11042" Text="I don't think you are allowed to have one of the parameters in $h$. Otherwise you could say that $Var[Y_i]=\frac{1}{1-p}E[Y_i]$, and the dispersion parameter is $\frac{1}{1-p}$. If $r$ is fixed, however, then the argument goes through. And that is what `glm.nb` uses for fitting: rerunning `glm` with a fixed value of $r$ and dispersion parameter 1." CreationDate="2011-05-21T18:26:40.903" UserId="279" />
  <row Id="19122" PostId="11020" Score="1" Text="@Galit Shmueli I don't believe that a model omitting main effects is necessarily unidentified. Perhaps you mean &quot;interpretability&quot; rather than &quot;identifiability&quot; (which is a technical term with a precise definition)" CreationDate="2011-05-21T18:40:12.110" UserId="26" />
  <row Id="19123" PostId="11079" Text="Hello everybody thanks for your comments. @Mike Lawrence: I did as you suggested using ezANOVA, but I get an error: &gt; ezANOVA(data=scrd, wid=.(subject) , dv=.(response), within=.(stimulus,subexperiment), between=.(sex), observed=.(sex))&#xA;Error in ezANOVA(data = scrd, wid = .(subject), dv = .(response), within = .(stimulus,  : &#xA;  unused argument(s) (wid = .(subject), observed = .(sex))&#xA;" CreationDate="2011-05-21T18:45:13.203" UserId="4701" />
  <row Id="19124" PostId="11079" Text="@nico: thanks for the link. But why should I use the nlme package? Which is the advantage rather than using aov?" CreationDate="2011-05-21T18:48:10.657" UserId="4701" />
  <row Id="19125" PostId="11079" Text="@caracal: yes, each observer tried all the stimuli in 2 conditions (EXP1 and EXP2). What I am interested in understanding if the differences between the responses of the males are significantly higher than those of the females. Do you think that is correct to do what you suggested? I.e. : &gt; aov3 = aov(response ~ stimulus*condition*sex + Error(subject/(stimulus*condition)), data=scrd)&#xA;&gt; summary(aov3)&#xA;" CreationDate="2011-05-21T18:57:50.993" UserId="4701" />
  <row Id="19126" PostId="11088" Score="1" Text="@crucified Any statistical package in mind?" CreationDate="2011-05-21T18:58:58.340" UserId="930" />
  <row Id="19127" PostId="11089" Score="5" Text="Galton derived a linear approximation to estimate a son's height from the father's height in that paper. His equation was fitted so an average height father would have an average height son, but a taller than average father would have a son that is taller than average by 2/3 the amount his father is. Same with shorter than average. This could be argued to be a simple linear regression (today's meaning). And of course today regression has an even broader meaning: it's any model that makes continuous predictions. It is interesting how much his original usage of that word has changed. " CreationDate="2011-05-21T19:07:34.130" UserId="2965" />
  <row Id="19128" PostId="11091" Text="@fgregg `Expit`=&quot;Inverse logit function&quot; from the [msm](http://cran.r-project.org/web/packages/msm/index.html) package?" CreationDate="2011-05-21T19:19:24.027" UserId="930" />
  <row Id="19129" PostId="11090" Text="Please forgive my ignorance. I want to do it in matlab. I may have to use t-distribution with degrees of freedom ranging from 1 to 30. In wiki, formula for only 1, 2, 4 degrees of freedom is given. Is there any generalized formula for quantile function? Could you give formulas for other even number of degrees of freedom?" CreationDate="2011-05-21T19:23:58.067" UserId="4319" />
  <row Id="19130" PostId="11091" Text="yes, I'll fix the question so that it's clear I'm talking about the inverse logit fuction." CreationDate="2011-05-21T19:24:23.350" UserId="82" />
  <row Id="19131" PostId="11088" Text="I want to implement in matlab" CreationDate="2011-05-21T19:24:30.830" UserId="4319" />
  <row Id="19132" PostId="10946" Text="@X We're not computing a probability distribution here: $x$ is a *fixed* value, not a random quantity!  (See the first line of my reply.) Thus, neither the normalization nor the expectation make any sense at all.  The question I answered is the one you asked in comments: &quot;I was obtaining the probability that no breakpoint falls in a segment of length $x$&quot;.  Your original question asks for a &quot;probability distribution for the length of the fragments....&quot;  Because there will be $n+1$  fragments, you are looking for an $n+1$-variate distribution for *all* the lengths." CreationDate="2011-05-21T19:33:10.893" UserId="919" />
  <row Id="19133" PostId="11088" Score="4" Text="@cruc Matlab has an inverse t distribution: see http://www.mathworks.com/help/toolbox/stats/tinv.html .  All you do is apply this function to a uniform random variate in the range (0,1)." CreationDate="2011-05-21T19:37:25.400" UserId="919" />
  <row Id="19134" PostId="11091" Score="2" Text="@fgregg, if $X_1$ and and $X_2$ are given the obvious choice is non-linear least squares. What are the conditions on model errors? Depending on them better estimating technique can be used." CreationDate="2011-05-21T19:44:12.543" UserId="2116" />
  <row Id="19135" PostId="11091" Score="1" Text="@fgregg, note that with the current equations $Z$ is **not** binary variable." CreationDate="2011-05-21T19:44:50.063" UserId="2116" />
  <row Id="19136" PostId="11091" Text="excellent point, I am interpreting Z as an expectation that Z will take on one of two values, but Z is continuous from 0 to 1." CreationDate="2011-05-21T19:53:40.127" UserId="82" />
  <row Id="19137" PostId="11021" Score="1" Text="Hey all, Sean here from RezScore - happy to explain anything about the algorithm that you'd like to know. If you a have a more specific question, I'd love to answer it and go into detail (I'm pretty proud of what we built :)" CreationDate="2011-05-20T21:40:23.173" UserId="4696" />
  <row Id="19138" PostId="11086" Score="2" Text="+1  It seems to be poorly known that the Wilcoxon/Mann-Whitney test is a test of medians only when there is purely a shift in distribution.  This can be hard to get across to non-statisticians: in some fields, the M-W has become so popular that people assume it's *always* applicable.  That's what &quot;nonparametric&quot; means, right? ;-)" CreationDate="2011-05-21T20:04:30.383" UserId="919" />
  <row Id="19139" PostId="11034" Text="yeah... but how do I know which probabilities are mapped to which target categories?" CreationDate="2011-05-21T21:26:19.530" UserId="333" />
  <row Id="19140" PostId="11096" Text="This seems like a question better suited for crossvalidated.com " CreationDate="2011-05-21T15:15:15.330" UserId="3094" />
  <row Id="19141" PostId="11096" Score="2" Text="1 sec earlier seriously? :)" CreationDate="2011-05-21T15:15:50.823" UserId="3094" />
  <row Id="19142" PostId="11096" Text="hehe. great minds think alike" CreationDate="2011-05-21T15:22:41.043" />
  <row Id="19143" PostId="11096" Score="1" Text="@chi: Your edits were excellent. More comprehensive. that the ones I was planning" CreationDate="2011-05-21T19:03:20.057" UserId="2129" />
  <row Id="19144" PostId="11088" Score="1" Text="As `tinv` is part of the Statistics toolbox, you'd have to pay extra for this, unless you are eligible for a student version, in which case it comes as part of the package." CreationDate="2011-05-21T22:13:25.450" UserId="2970" />
  <row Id="19145" PostId="11046" Text="@Dikran : I suppose I am a little confused now. So how will I do the novelty detection? Are you saying that I generate something like one single decision boundary across *all* my training instances, and then based on this decision boundary determine whether a new instance is an outlier or not? " CreationDate="2011-05-21T22:36:16.857" UserId="3301" />
  <row Id="19147" PostId="11086" Score="2" Text="@whuber, I've even seen at least one statistical software package where the Mann-Whitney test is there as an &quot;alternative&quot; to what is essentially a two-sample $t$-test with unequal variances. Ouch." CreationDate="2011-05-21T22:49:21.957" UserId="2970" />
  <row Id="19148" PostId="11097" Text="+1 on both SO and CV for answering without complaint :)" CreationDate="2011-05-21T22:52:11.620" UserId="776" />
  <row Id="19149" PostId="11099" Text="+1 Very clearly put." CreationDate="2011-05-21T22:52:12.353" UserId="919" />
  <row Id="19150" PostId="11090" Score="1" Text="@crucified, no closed form will exist for the $t$ quantile function. See the G. W. Hill references listed [here](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/TDist.html). You could also get the source code for [GSL](http://www.gnu.org/software/gsl/) and look up the [appropriate functions](http://www.gnu.org/software/gsl/manual/html_node/The-t_002ddistribution.html). I haven't looked myself, but it's probably easy enough to port to MATLAB." CreationDate="2011-05-21T22:55:31.407" UserId="2970" />
  <row Id="19151" PostId="10971" Text="Unfortunately, most of the dyad stuff assumes independent dyads where each partner is part of only one dyad. But thanks for the link to that site and its info." CreationDate="2011-05-21T23:21:12.487" UserId="4655" />
  <row Id="19152" PostId="11086" Text="@whuber For example in sociology. And I am guilty too. It took me some time to understand how the test actually works." CreationDate="2011-05-21T23:32:03.630" UserId="307" />
  <row Id="19153" PostId="11086" Text="@cardinal Which statistical software package do you mean?" CreationDate="2011-05-21T23:34:50.587" UserId="307" />
  <row Id="19154" PostId="11065" Text="I don't totally understand it, but in a model I am creating in R, I have something like lm (a ~ b / c - 1) which creates interactions between b and c, and by suppressing the intercept (&quot;- 1&quot; in R), I get more easily-interpretable answers that are essentially the same as if I don't suppress the intercept. Somehow, the interaction makes this possible." CreationDate="2011-05-21T23:36:36.090" UserId="1764" />
  <row Id="19155" PostId="11079" Text="@user4701: looks like you're using an older version of R that means that it downloads an older version of ez. Try downloading the latest version of R, re-installing ez, then run the ezANOVA command again." CreationDate="2011-05-21T23:54:48.617" UserId="364" />
  <row Id="19156" PostId="11100" Text="Hello caracal, thanks for your reply. At the moment I have a question: what do you mean exactly with IV? Sorry but my knowledge is limited....I am not a statician..." CreationDate="2011-05-22T00:36:06.803" UserId="4701" />
  <row Id="19157" PostId="11098" Text="Hi John thanks a lot. I noticed that I have some lack of terminology. What do you mean exactly when you say that stimulus is a between condition? In my experiment each subject tried 7 different stimuli, in 2 different conditions...sorry if this is a low level question....please enligthen me! ;-) And if you have a web page to suggest me where to learn more about the specific problem of this discussion please share the link ! Thanks" CreationDate="2011-05-22T00:50:36.980" UserId="4701" />
  <row Id="19158" PostId="11079" Text="@user4701 - Just a general piece of advice - it is good practice to write out the mathematics of your model, and also to write out the hypothesis that you want to test in that mathematical framework.  It makes things much clearer (I think) as to what you are doing if you describe in the mathematical language, as oppose to only in words (one of my dislikes about R, it tends not to do this, expects you to just know what it is doing).  This is not just about this question, but will likely help your own thinking." CreationDate="2011-05-22T01:15:55.163" UserId="2392" />
  <row Id="19159" PostId="11088" Score="2" Text="**Quick and dirty**: `sqrt(n)*randn(1)/norm(randn(n,1))` will generate a $t$-distributed variate with $n$ degrees of freedom. This is probably the fastest to code, but certainly not the fastest to execute, especially if you need a very large number of them." CreationDate="2011-05-22T02:19:32.497" UserId="2970" />
  <row Id="19160" PostId="11097" Text="So it let you vote twice? How nice. I thought offering an R code implementation of the interpretation might rescue it from the forces of classification rectitude. The OP composed a nice self contained example. Maybe I should have demonstrated an extractor function, so I think I will. " CreationDate="2011-05-22T03:00:35.010" UserId="2129" />
  <row Id="19161" PostId="10538" Text="One useful non-informative prior for a variance is to write $\sigma_{\theta}^{2}=g\sigma^{2}$ and then specify an F-distribution for g, with standard jeffreys prior for $p(\sigma^{2})\propto\frac{1}{\sigma^{2}}$." CreationDate="2011-05-22T04:04:05.597" UserId="2392" />
  <row Id="19162" PostId="10931" Text="Interesting prior! I was about to object to your claim that there is only information about the sum of the variance components but noticed that he has $i$ indexing both $y$ and $\theta$; in my head I'd imputed the hierarchical model $y_{ij}|-\sim N(\theta_j, \sigma^2)$, $\theta_j\sim N(\mu, \sigma^2_\theta)$..." CreationDate="2011-05-22T04:17:38.647" UserId="26" />
  <row Id="19163" PostId="10931" Text="But assuming he meant what he wrote, what's the point? Even with restrictions on $\sigma$ he can't estimate both $\sigma$ and $\sigma_\theta$ separately..." CreationDate="2011-05-22T04:30:47.527" UserId="26" />
  <row Id="19164" PostId="11097" Text="Yes, I upvoted your response on SO, then it moved here and I upvoted again :) " CreationDate="2011-05-22T04:35:22.563" UserId="776" />
  <row Id="19165" PostId="11103" Text="Thanks, this is helpful, but I'm not exactly sure dropping variables is the right thing to do?  I'm trying to perform LSA on a set of documents, so my matrix is quite big (9 x ~6700).  I'm sure there are some columns that are redundant (e.g. several words that appear in only one document).  Does it really make sense to delete duplicate columns here?" CreationDate="2011-05-22T04:36:41.737" UserId="1977" />
  <row Id="19166" PostId="11103" Text="Hmm, I tried deleting all duplicate columns (several thousand of them) and I still get the same error..." CreationDate="2011-05-22T05:40:55.737" UserId="1977" />
  <row Id="19167" PostId="11012" Text="If you have the CTABLES license you may be able to do this test through that command. I can't find anything in the native descriptives/crosstabs that would allow you to do this. You could always calculate it yourself as well." CreationDate="2011-05-22T06:11:42.183" UserId="1036" />
  <row Id="19168" PostId="11065" Text="More easily interpretable answers that are essentially the same? That appears to be a contradiction. Maybe you should introduce this as a new question?" CreationDate="2011-05-22T08:38:52.757" UserId="4257" />
  <row Id="19169" PostId="11097" Text="Thanks so far! I know the realtion between the dummies and cavariables, but I'm just interested of how to interpret the Main effects (I marked them). Is it possible to take teh incidient rate from a Main effects, for example for the dummy treated 'exp(-0.012)=0.99' and interpret it as the rate from which the healtvalue decreases, when switching from reference category to treated? It must be, no?" CreationDate="2011-05-22T08:40:50.617" UserId="4496" />
  <row Id="19170" PostId="11100" Text="@user4701 Sorry: IV -&gt; &quot;Independent Variable&quot; or &quot;factor&quot; in your experiment." CreationDate="2011-05-22T08:57:37.243" UserId="1909" />
  <row Id="19171" PostId="11101" Score="1" Text="@Christine How many factors did you extracted and what kind of rotation did you use, if any? Also, what is the number of items and the sample size?" CreationDate="2011-05-22T09:35:37.117" UserId="930" />
  <row Id="19172" PostId="11103" Score="1" Text="@Jeff Do you mean that you performed ML-based FA with 9 statistical units (rows) and 6k+ variables (columns)? Because, in this case, it won't work." CreationDate="2011-05-22T10:29:15.587" UserId="930" />
  <row Id="19173" PostId="11090" Text="@cardinal +1, especially given the fact that there exist GSL bindings for Octave." CreationDate="2011-05-22T10:31:45.307" UserId="930" />
  <row Id="19174" PostId="11107" Score="1" Text="What specifically are you confused about?" CreationDate="2011-05-22T10:54:02.067" UserId="3835" />
  <row Id="19175" PostId="11108" Text="Your title says &quot;change of mean&quot;, but the boxplots don't show the means. Maybe it won't make much difference, but do you mean a change in the median?" CreationDate="2011-05-22T11:42:38.810" UserId="3835" />
  <row Id="19176" PostId="11098" Text="Hi I studied better the terminology and I understand what between and within subjects idependent variable mean ;-)" CreationDate="2011-05-22T12:33:23.910" UserId="4701" />
  <row Id="19177" PostId="3074" Text="-1. There are a lot of sweeping statements here. You should elaborate on some of them if this answer taken seriously, since the OP is asking for help picking one of these languages and specifically ruled out R." CreationDate="2011-05-22T12:35:37.717" UserId="1080" />
  <row Id="19178" PostId="11108" Text="My original title (edited by another person) was &quot;What is the Statistical Method I should use to locate the change in the 'average' of successive series of data?&quot;  'average' was quoted because I didn't know the correct term to use.  " CreationDate="2011-05-22T12:45:52.517" UserId="4703" />
  <row Id="19179" PostId="10931" Text="The prior range restriction effectively breaks the symmetry, and hence identifiability problem.  They still will remain negatively correlated in the posterior.  However it does this in a conservative and defensible way - you can usually provide a reasonable case for the choice of upper and lower limits without the need for any data." CreationDate="2011-05-22T13:02:47.607" UserId="2392" />
  <row Id="19180" PostId="10754" Text="So, is b equal to elasticity of Y with respect to X, as can be obtained from the regression &quot;log (Y)= a + b log(X)+ error term&quot;? " CreationDate="2011-05-22T13:20:47.277" UserId="4586" />
  <row Id="19181" PostId="11065" Text="@Nick: If I look at the coefficients, with the intercept there is an *(intercept)* and a *tempwarmer* (one of my variables is *temp* which can be *warmer* or *cooler*). To interpret the coefficients, I have to know that *(intercept)* directly corresponds to *tempcooler* and *tempwarmer* + *(intercept)* is the directly-interpretable *tempwarmer*. If I suppress the intercept, I see *tempcooler* and *tempwarmer* directly. Perhaps a quirk of R's formulas and linear modeling, but..." CreationDate="2011-05-22T13:41:10.147" UserId="1764" />
  <row Id="19182" PostId="11097" Text="The exponentiated coefficients are always interpreted as ratios. Ratios of 'what' to 'what' depends on the units of analysis. 'Rates' are different, having an implicit number and time value. So if you are willing to change your terminology, then perhaps,'yes&quot;. Best answers come from describing the analysis situation fully." CreationDate="2011-05-22T14:01:31.140" UserId="2129" />
  <row Id="19184" PostId="11113" Text="You stated that `stimulus` is a within-subjects factor, i.e., each observer gives a response for each stimulus. To get the correct split-plot ANOVA (one between factor `sex`, one within factor `stimulus`), you have to use `summary(aov(response ~ stimulus*sex + Error(subject/stimulus), data=table_EXP1))` instead of your calls to `lm()`." CreationDate="2011-05-22T14:35:32.670" UserId="1909" />
  <row Id="19185" PostId="11044" Text="Problems arise when the time zero (baseline) measurement is used as a first response variable.  The baseline is often used as an entry criterion for the study.  For example, a study might enroll patients with systolic blood pressure (bp) &gt; 140, then randomize to 2 bp treatments and follow the bps.  Initially, bp has a truncated distribution and the later measurements will be more symmetric.  It is messy to model 2 distributional shapes in the same model.  There are many more reasons to treat the baseline as a baseline covariate." CreationDate="2011-05-22T14:39:12.213" UserId="4253" />
  <row Id="19186" PostId="11103" Text="Performing LSA (I assume you mean latent semantic analysis) uses the SVD; it's basically principal components. You don't want to be doing factor analysis. @chi is correct in that it isn't even possible here. If you want to use Python have a look at Gensim. But why only 9 documents? You're unlikely to get much." CreationDate="2011-05-22T15:12:44.860" UserId="26" />
  <row Id="19187" PostId="11103" Score="1" Text="Gensim is here: http://nlp.fi.muni.cz/projekty/gensim/" CreationDate="2011-05-22T15:14:30.267" UserId="26" />
  <row Id="19188" PostId="11113" Text="Hi, thanks for your answer. I don´t understan why you keep considering the repeated measures, indeed in table_EXP1 there are not repeated measures. Could you please explain me why you chose this approach? Anyways using your method my problem is still unresolved: the ANOVA result say that the differences are not significant. This is not possible. Indeed as I told, on 7 stimuli, in both the 2 conditions I always get values higher for the males...and are not values very near to those of the females. Any suggestion? Thanks!" CreationDate="2011-05-22T15:14:43.320" UserId="4701" />
  <row Id="19189" PostId="11103" Text="@Jeff Also, the reason why the error persists (I assume) is that you still have many more terms than documents, so the sample covariance matrix is still singular." CreationDate="2011-05-22T15:16:50.260" UserId="26" />
  <row Id="19190" PostId="10931" Text="How does the range restriction solve the identifiability problem? I could still lop a little off the one and add it to the other, respecting interval constraints - hence, unidentified and not (separately) estimable. While there are of course situations where having portions of the model unidentified is sensible, this isn't one of them." CreationDate="2011-05-22T15:22:48.483" UserId="26" />
  <row Id="19191" PostId="10976" Text="Thanks Bob, +1. A related question, would you have a rule of thumb for choosing a value of fractional-metric q (or should I ask that as a separate question) ?" CreationDate="2011-05-22T15:31:36.920" UserId="557" />
  <row Id="19192" PostId="11106" Text="**NB:** This is a reply to a second question that was subsequently merged with the present one.  As @mark999 correctly points out here, the two questions are the same except one asks for Laplace variates and the original (this one) asks for Student t variates." CreationDate="2011-05-22T15:37:31.920" UserId="919" />
  <row Id="19194" PostId="5122" Text="-1 In1884,Galton wrote:&quot;Jews are specialized for a parasitical existence upon other nations,and that there is need of evidence that theyare capable of fulfilling thevaried duties ofa civilized nation by themselves.&quot;Karl Pearson,Galton's disciple and biographer,echoed this opinion 40years later during his attempt to prove the undesirability of Jewish immigration into Britain:&quot;for such men as religion,social habits,or language keep asa caste apart,there should be no place.they will not be absorbed by,and at the same time strengthenthe existing population;they will develop into a parasitic race&quot; " CreationDate="2011-05-22T16:21:46.527" UserId="3382" />
  <row Id="19196" PostId="11108" Text="@Tom My bad, sorry for that." CreationDate="2011-05-22T16:33:52.877" UserId="88" />
  <row Id="19197" PostId="11098" Text="Editing the question so that now stimuli is also a within variable." CreationDate="2011-05-22T16:37:04.313" UserId="601" />
  <row Id="19198" PostId="11069" Text="The data we have, is a monthly observation of close value and its range from September 1997 to April 2011.&#xA;&#xA;after 1st differenc the series be stationary.&#xA;&#xA;and with test the plot (ACF , PACF and EACF) i can see the series is one of (IMA(1.1) , ARIMA(1.1.1) or ARIMA(2.1.2)).&#xA;&#xA;when i look to the (AIC , AICc and BIC) ,indicates that the best model is IMA(1,1) because it has the least values of AIC, AICc and BIC .&#xA;&#xA;now i need to compared some method like (ML,CSS-ML,CSS,GLS) for estimate and forecast .&#xA;&#xA;the problem is how can i do estimate and forecast using GLS method (((look above)))  " CreationDate="2011-05-22T16:48:40.887" UserId="4698" />
  <row Id="19199" PostId="5554" Text="I'm in the middle of sorting through this multiple testing as well, and in clicking on your links @chl, the multtest package in R has been removed. I can't comment on whether the multcomp will do what Richard did, but thought I'd let you know about multtest while I was thinking of it." CreationDate="2011-05-22T17:19:52.430" UserId="4238" />
  <row Id="19200" PostId="5554" Text="@Mog Yes, since then it has moved to BioC. Let us know if you encounter a problem with using them." CreationDate="2011-05-22T17:22:41.693" UserId="930" />
  <row Id="19201" PostId="11069" Text="I know the forecast is larg and wrong fist you must know I use code above and the plot output tells us the forecast wrong,  &#xA;&#xA;I attach the plot here.&#xA;&#xA;[link](http://www12.0zz0.com/2011/05/22/17/887560621.jpg)" CreationDate="2011-05-22T17:24:57.807" UserId="4698" />
  <row Id="19202" PostId="5122" Text="@IrishStat. Yup, a lot of the early statisticians were eugenists, including Pearson and Fisher. But, they were still great statisticians. " CreationDate="2011-05-22T17:26:33.123" UserId="74" />
  <row Id="19203" PostId="11113" Text="@user4701 I don't quite follow: In your question, you state &quot;stimulus is a within-subjects factor&quot;. In your `lm()` formula, you include `stimulus` as a factor. If it was within-subjects in your original design, it remains within-subjects in your two subsetted designs. So you still have repeated measures for that factor in `table_EXP1`, and in `table_EXP2`." CreationDate="2011-05-22T18:05:43.827" UserId="1909" />
  <row Id="19204" PostId="11044" Text="That's a good point, but recent studies suggest that this is not an issue. In fact, it seem that there are more disadvantages to using baseline scores as a covariate. See: Liu, G. F., et al. (2009). Should baseline be a covariate or dependent variable in analyses of change from baseline in clinical trials? Statistics in Medicine, 28, 2509-2530." CreationDate="2011-05-22T18:33:16.647" UserId="1934" />
  <row Id="19205" PostId="11069" Text="@Ehab, I agree with Mpiktas. I believe in model selection, your stat tool has already used ML. GLS is Generalized Least Squares, right? Since, you have a single time series data, you do not need to have this estimation. Do you have any other exogenous variables you are planning to include? Stick with ML and find a better model. You have mentioned seasonality in your data. You probably need to take this into account." CreationDate="2011-05-22T18:46:41.867" UserId="4617" />
  <row Id="19206" PostId="11106" Text="@whuber: it might make sense, then, to edit this question to include the Laplace distribution. " CreationDate="2011-05-22T18:59:14.930" UserId="198" />
  <row Id="19207" PostId="11044" Text="I have read that paper.  It is not convincing, and Liu has not studied a variety of the kinds of clinical trial situations I described.  More arguments are at http://biostat.mc.vanderbilt.edu/wiki/pub/Main/RmS/course2.pdf in the chapter about analysis of serial (longitudinal) data." CreationDate="2011-05-22T19:19:56.603" UserId="4253" />
  <row Id="19208" PostId="10754" Text="@Dlogger, $\log(Y)=a_0+a_1t+b\log(X)+\varepsilon$" CreationDate="2011-05-22T19:25:06.810" UserId="2116" />
  <row Id="19209" PostId="11129" Text="I cannot recommend using &quot;dummy variable adjustment&quot; (aka &quot;a dummy variable can be used just to control for the missing values.&quot;). It is known that &quot;this method generally produces biased estimates of the coefficients&quot; ([Allison, Missing Data, p. 10](http://bit.ly/lEyBj8)). " CreationDate="2011-05-22T19:32:11.783" UserId="307" />
  <row Id="19210" PostId="11132" Text="@caracal My very big +1 for this nicely illustrated response." CreationDate="2011-05-22T20:16:13.613" UserId="930" />
  <row Id="19211" PostId="11128" Text="and why didn't I find all this stuff! Awesome thanks Bill. " CreationDate="2011-05-22T20:20:17.980" UserId="4705" />
  <row Id="19212" PostId="11098" Text="@John Why is the last model not repeated measures anymore? I thought `stimulus` is within subjects, and you also set up the `Error()` term that way." CreationDate="2011-05-22T20:32:22.897" UserId="1909" />
  <row Id="19213" PostId="10493" Text="@Greg Snow, this is a bit late!, but could you explain more how to model the predictor variables? For example, suppose I want to make forecasts at daily granularity of the # of times somebody in Florida rents a comedy, and my predictors are the ones you mentioned. Ignoring ts aspects like serial correlation, are you suggesting I form a dataset consisting only of the Florida-comedy rentals (i.e., ignore all other rentals in Florida like horror rentals, and ignore comedy rentals by other states), and build my model off this? So for each data slice, I build a new and independent model?" CreationDate="2011-05-22T20:34:06.107" UserId="1106" />
  <row Id="19214" PostId="10493" Text="In other words, I guess my question had two parts: 1) how to deal with the hierarchical nature of the problem (I want forecasts at different slice granularities, so when making forecasts for slice X, should I care that I have data outside of slice X that is nonetheless related to it?) and 2) how to reconcile the fact that each individual time series slice has both time series components and non-time series components. It sounds like you're suggesting that for 2), the time series parts are less important than the non-time series parts, and for 1), I ignore the hierarchical/interrelated nature?" CreationDate="2011-05-22T20:41:44.513" UserId="1106" />
  <row Id="19215" PostId="11132" Text="@caracal Many thanks for your very comprehensive answer." CreationDate="2011-05-22T20:50:07.663" UserId="609" />
  <row Id="19217" PostId="11124" Text="Migrating to math.SE because this is a purely a question of math." CreationDate="2011-05-22T20:53:49.017" UserId="919" />
  <row Id="19219" PostId="11125" Text="This appears to be purely a math question: migrating." CreationDate="2011-05-22T20:56:27.530" UserId="919" />
  <row Id="19220" PostId="11134" Text="How do you know your test was wrong?  How large was $N$, what was $\theta_m$, and what was the standard error of the estimate $4\pi M/N$?" CreationDate="2011-05-22T21:05:37.613" UserId="919" />
  <row Id="19221" PostId="11083" Text="I agree that researchers should understand the liabilities of observational research designs, especially those related to omitted variables, but I don't think avoiding causal language does this. See the work of Hubert Blalock, in particular his book Causal Inferences in Non-experimental Research for a more detailed argument in defense of using causal language." CreationDate="2011-05-22T21:07:27.223" UserId="1036" />
  <row Id="19222" PostId="11126" Text="Because this is a question in applied math, not stats, it appears more suitable for math.SE." CreationDate="2011-05-22T21:07:28.657" UserId="919" />
  <row Id="19223" PostId="5122" Text="@Neil Can you elaborate on Fisher in this regard ?" CreationDate="2011-05-22T21:27:23.483" UserId="3382" />
  <row Id="19224" PostId="5456" Text="Kudos to Jerzy for &quot;proving&quot; storks bring babies illustrating the concomitant variable problem." CreationDate="2011-05-22T21:31:30.077" UserId="3382" />
  <row Id="19225" PostId="11098" Text="oops... didn't finish the edit... (gone now) — BTW, they're both mixed within/between" CreationDate="2011-05-22T21:36:10.377" UserId="601" />
  <row Id="19226" PostId="11131" Text="To clarify, is it correct that you are already able to get the required $n$, but you're looking for a general formula? I would be very surprised if there is a useful general formula." CreationDate="2011-05-22T21:44:42.563" UserId="3835" />
  <row Id="19227" PostId="11101" Text="I used varimax rotation, and four factors were extracted. Sample size was 104. I just found that my sample does not meet multivariate normality assumptions. Combined with low sample, this may have caused high communalities, I think. But I'm still not sure if these items can be retained or not. When I used principle component analysis for factor extraction, all communalities were below .9. " CreationDate="2011-05-22T21:55:09.783" UserId="4702" />
  <row Id="19228" PostId="11009" Text="@Glen Oh yes, it should!" CreationDate="2011-05-22T21:55:57.097" UserId="3874" />
  <row Id="19229" PostId="11044" Text="Thanks for the link. I assume you are referring to the discussion under 8.2.3. Those are some interesting points, but I don't think this gives a definite answer. I am sure that the paper by Liu et al. isn't the ultimate answer either, but it does suggest for example that non-normality of the baseline values is not a crucial issue. Maybe this is something for a separate discussion item, as it does not directly relate to the OP's question." CreationDate="2011-05-22T22:07:01.257" UserId="1934" />
  <row Id="19230" PostId="11113" Text="Hi caracal, thanks a lot again for you answer. The thing is that, according to my (poor) knowledge, the table_EXP1 does not contain repeated measures. Indeed the each subject gave for each stimulus only one answer. So, having precised this, am I still wrong if I use this command? fit_table_EXP1 &lt;- lm(response ~ stimulus*sex, data=table_EXP1) &#xA;anova(fit_table_EXP1)&#xA;" CreationDate="2011-05-22T22:10:33.987" UserId="4701" />
  <row Id="19231" PostId="11113" Text="@user4701 Yes, still wrong: the point is that one subject not only sees one single stimulus, but EACH one of the stimuli. A single subject is observed in all levels of the `stimulus` factor: hence you have repeated measures with regards to the `stimulus` factor, and need to use `aov()` with an `Error()` term." CreationDate="2011-05-22T22:19:19.867" UserId="1909" />
  <row Id="19233" PostId="11113" Text="Dear Caracal, ok I got it, thanks ;-) Then there is only one question which still remains open. Why the anova gives to me non significant differences? As I told you, making the means of the responses for each stimulus, in both the conditions EXP1 and EXP2, I always have higher values for males rather than females. This cannot be just a casuality. Therefore, how can I prove that the differences are significant? It is not possible that they are not significant....help...please!  " CreationDate="2011-05-22T22:46:39.320" UserId="4701" />
  <row Id="19234" PostId="11113" Text="In addition just to make a trial, I performed a t.test between the responses of males and females on one single stimulus and I get significant differences. ...now, why the ANOVA does not detect such differences and the t-test does?...I don´t understand...;-(" CreationDate="2011-05-22T23:21:10.983" UserId="4701" />
  <row Id="19235" PostId="11113" Text="@user4701 Do you still get the significant results for `sex` when you do the correct separate analyses using `aov()` within `EXP1` and `EXP2`? Please edit your question accordingly. Also, it can well be that there is a significant difference for a single stimulus, but when averaged over all stimuli, there isn't." CreationDate="2011-05-22T23:29:34.850" UserId="1909" />
  <row Id="19236" PostId="11107" Text="Isn't it a question for the sister site, SO? It seems to be less a question of statistics and more of programming language..." CreationDate="2011-05-22T23:44:05.813" UserId="3058" />
  <row Id="19237" PostId="11103" Text="OK, so I was trying not to include extraneous details in my question, but I think it's just added to the confusion, so I've updated my question.  I am doing SVD-- I'm trying to perform FA after the fact.  I will give Genism a try and let you know.  Thanks all!" CreationDate="2011-05-23T00:15:07.573" UserId="1977" />
  <row Id="19238" PostId="11113" Text="Dear caracal and dear all. I apologize a lot...I just noticed now that I was analyzing a wrong table. I built it erroneously. Now everything works...of course. Thanks a lot for the precious suggestions comments. They have been very useful for me. I really appreciated your help." CreationDate="2011-05-23T00:28:50.420" UserId="4701" />
  <row Id="19239" PostId="11113" Text="I have one final question,and I hope you can help me! So far I am able to detect the effect of sex on the results, and I found that there is significant difference between the evaluations of males and females AT GLOBAL LEVEL, i.e. considering all the stimuli. In another table, of a similar experiment, I don´t get significant differences at global level. Anyways I noticed that for some stimuli, there is a difference between males and females evaluations, which cannot be due to the casuality.How can I know which stimuli present significant differences? So far I only imagine to use t-test..." CreationDate="2011-05-23T00:58:57.723" UserId="4701" />
  <row Id="19240" PostId="11103" Text="FA on what matrix?" CreationDate="2011-05-23T01:05:48.970" UserId="26" />
  <row Id="19241" PostId="11102" Text="What do you mean by the &quot;reconstructed matrix&quot;? If you do the SVD, keep a few components, and then do PCA, the results should be identical." CreationDate="2011-05-23T01:14:55.870" UserId="26" />
  <row Id="19242" PostId="11076" Text="@cardinal yes I was wondering if I should make a similar point. I've found in some nonlinear regression modelling contexts there is greater interest in having a model that provides a theoretically plausible model that predicts accurately outside the range of the data (e.g., in learning curve data speed, models should not predict speeds below 0 seconds). In such cases constraining an intercept to zero may be more appropriate even if it results in a drop in prediction for the data." CreationDate="2011-05-23T01:16:23.080" UserId="183" />
  <row Id="19243" PostId="11076" Text="@cardinal I agree polynomial models rarely predict plausibly outside the range of the data, and thus constraining the intercept to 0 in such models is rarely a good idea." CreationDate="2011-05-23T01:17:42.707" UserId="183" />
  <row Id="19244" PostId="11042" Text="That is what I was hinting at with my last statement.  You can also show that the negative binomial can't be written in exponential form unless $r$ is fixed, as you have $p(y)=\exp\left(y\log(p)+r\log(1-p)+\log([r+y-1]!)-\log(y!)-\log([r-1]!)\right)$" CreationDate="2011-05-23T01:28:57.613" UserId="2392" />
  <row Id="19245" PostId="11135" Score="2" Text="Your question says &quot;what do I do with all this information&quot;. You may get a better response from this site if you split your question into separate more manageable and specific questions. I recommend that you do some more general reading on meta analysis and medical statistics. For what it's worth, here are a few resources that I prepared on meta-analysis: http://jeromyanglim.blogspot.com/2009/12/meta-analysis-tips-resources-and.html" CreationDate="2011-05-23T01:38:15.583" UserId="183" />
  <row Id="19246" PostId="11076" Text="@Jeromy: Thanks for your comments. My remark wasn't so much aimed at polynomial models. The choice of a quadratic was simply based on an actual physical motivation (i.e., classical mechanics). The point I was trying to articulate was that one should carefully consider the modeling problem of interest; sometimes doing something that is (or seems) &quot;theoretically unjustified&quot; is actually more appropriate statistically." CreationDate="2011-05-23T03:18:25.943" UserId="2970" />
  <row Id="19247" PostId="11113" Text="@user4701 I take it the other problem has been solved then. Please consider accepting an answer to let future readers know that you have received a good answer. With respect to your final question: t-tests should be fine, but think about alpha adjustment (Bonferroni correction, etc.): when you run 7 t-tests for sex differences in different stimuli, you're testing the same hypothesis multiple times, thus giving it 7 tries to become significant due to chance." CreationDate="2011-05-23T08:20:05.467" UserId="1909" />
  <row Id="19248" PostId="11088" Score="3" Text="Not quite sure why you bothered to ask about Laplace distribution, as it explicitly tells you how to generate these in the link you give in your question!" CreationDate="2011-05-23T08:22:43.203" UserId="2392" />
  <row Id="19249" PostId="11140" Text="Thank's for the answer and indeed the error was in the code. Everything looks fine now." CreationDate="2011-05-23T08:28:27.363" UserId="4706" />
  <row Id="19250" PostId="11045" Text="Thank i will take a look " CreationDate="2011-05-23T09:04:08.833" UserId="4531" />
  <row Id="19251" PostId="10976" Text="@Denis Probably merits a question of its own as I think it's both data and application dependent. These fractional metrics with $q=1/p$ aren't really metrics in the formal sense for $p&gt;1$ (the sense of the triangle inequality gets reversed for example, so they are non-convex) and as $p$ increases you're converging on the $l_0$ `metric'. I'd start with $p=1$ since $l_{1}$ is not as problematic to work with as $l_{q=1/p}$ when $p&gt;1$, and fit the parameter $p$ from data. Possibly someone has found an automated way to do this by now, but I don't know." CreationDate="2011-05-23T09:14:02.700" UserId="3248" />
  <row Id="19252" PostId="10972" Text="@ G. Jay Thanks for a nice answer, and also the useful references. Much appreciated." CreationDate="2011-05-23T09:19:39.090" UserId="3248" />
  <row Id="19253" PostId="10964" Text="@cardinal Thanks - and +1 for the x-ref which is clearly related." CreationDate="2011-05-23T09:25:04.167" UserId="3248" />
  <row Id="19254" PostId="11113" Text="Thanks a lot caracal!! But I don´t understand why I need to use alpha adjustment for my case. Indeed I only want to check if for a particular stimulus there is significant difference between males and females evaluations. I already know that considering all the stimuli the anova results non significant. I am interested in evaluating the means for a particular stimulus, as I noticed that males mean is higher than female mean. Do I still need alpha adjustment for my purposes? Thanks again ;-)" CreationDate="2011-05-23T10:00:03.140" UserId="4701" />
  <row Id="19255" PostId="11046" Text="@andy, yes that is correct, fit a one-class classifier to the whole of the training set, and reject any observation that falls outside the boundary as being a novelty belonging to a new class.  Use a conventional multi-class classifier to classify any pattern that falls within the boundary of the one-class classifier." CreationDate="2011-05-23T10:11:00.700" UserId="887" />
  <row Id="19256" PostId="10946" Text="@whuber, I don't understand well your last comment. On the other hand, I think I possibly got to a satisfactory conclusion: $(1-x)^n$ is the cdf  of the pdf I am looking for (in the original question). The corresponding pdf is $n(1-x)^{n-1}$ which is normalized and has nice expectation values. " CreationDate="2011-05-23T10:44:35.837" UserId="4642" />
  <row Id="19257" PostId="11149" Text="@Melissa, [this site](http://www.multiple-imputation.com/) might give answer to your question. The link `What is MI` gives a list of various resources." CreationDate="2011-05-23T11:47:31.613" UserId="2116" />
  <row Id="19258" PostId="11113" Text="@user4701 The question when exactly one needs to do alpha-adjustment is not easily answered, especially without a detailed background about the study and the hypotheses. To me, it just sounded like you're testing the same hypothesis (sex difference) multiple times (separately for each stimulus). If you have a specific hypothesis about a special stimulus, then that might not be the case." CreationDate="2011-05-23T11:48:19.970" UserId="1909" />
  <row Id="19259" PostId="10704" Text="I'm sorry for the delay in answering but the point is I haven't got much time to try dbscan and first attempts resulted in a memory problem. R says it can not allocate vector. I start with a 4 km spaced grid with 779191 points that ends in 300000 points when removing land (not valid) SST points. Maybe I'm not getting the right approach, any hint would be appreciated." CreationDate="2011-05-23T12:21:14.673" UserId="4147" />
  <row Id="19260" PostId="11150" Text="@Nick Sabbe +1 I would assume that single imputation performs slightly better than mean substitution because you incorporate additional information ($0$ predictors vs $p$ predictors). However, I fully agree that MI is the way to go." CreationDate="2011-05-23T12:25:18.583" UserId="307" />
  <row Id="19261" PostId="11151" Text="@ Bernd the Graham reference is extremely good, it helped me a lot with getting the hang of multiple imputation." CreationDate="2011-05-23T12:38:24.177" UserId="656" />
  <row Id="19262" PostId="11113" Text="Ok thanks a lot! You are sooooo helpful!!!" CreationDate="2011-05-23T12:42:49.787" UserId="4701" />
  <row Id="19263" PostId="11139" Text="+1 for (whether it deserves it or not) :)" CreationDate="2011-05-23T14:25:18.543" UserId="26" />
  <row Id="19264" PostId="11144" Text="Thanks @Frank, I'll look into that. Sounds like it's what I need!" CreationDate="2011-05-23T14:36:46.877" UserId="4238" />
  <row Id="19265" PostId="11011" Text="@Aaron Thanks for the suggestions. I tried the first and third before posting my question and as @Bernd pointed out, there's nothing (obvious) there that has both _multilevel_ and SEM." CreationDate="2011-05-23T14:43:13.887" UserId="4677" />
  <row Id="19266" PostId="10999" Text="@chi Thank you for the suggestion, I'll check out lavaan, too." CreationDate="2011-05-23T14:44:56.943" UserId="4677" />
  <row Id="19267" PostId="11121" Text="So yes, I know that using the outcome of Step 1 (finding the split) may be incorrect and thus would throw off the result of Step 2.  I'm cool with that because I can test the result of Step 2 experimentally and see if my assumptions were correct or not.  But I still don't know the 'correct' way of doing either of these steps.  I was hoping there was a real statistical method for doing this; as opposed to just making one up myself that works but isn't as robust as it could be." CreationDate="2011-05-23T14:52:40.820" UserId="4703" />
  <row Id="19268" PostId="11097" Text="Ah ok this is what I wanted to know. So you interpret the coefficients as ratios! Thank you!" CreationDate="2011-05-23T15:34:26.070" UserId="4496" />
  <row Id="19269" PostId="11153" Score="1" Text="But using the mean implies you're predicting the value at that point.  That's not what's going on, what's going on is an attempt to recover a random value.  It seems that since you have an estimate of the variance as well that you should use both (i.e., a random draw from the distribution)." CreationDate="2011-05-23T16:11:52.927" UserId="601" />
  <row Id="19270" PostId="11097" Text="The coefficients are the _natural_logarithms_ of the ratios." CreationDate="2011-05-23T18:03:56.803" UserId="2129" />
  <row Id="19271" PostId="11154" Text="@Bernd, thank you for a high quality answer! Do you know of any resources (detailed, but not quite a textbook) where I can learn more about this?" CreationDate="2011-05-23T18:24:14.997" UserId="4717" />
  <row Id="19272" PostId="11153" Text="In addition, mean substitution will reduce the variance of your estimates which will throw all of your standard errors and confidence intervals for the rest of your analyese." CreationDate="2011-05-23T19:04:42.573" UserId="656" />
  <row Id="19273" PostId="11154" Text="@Yoel What exactly are you looking for? Resources about multilevel data/models or database management systems/database design?" CreationDate="2011-05-23T19:20:12.000" UserId="307" />
  <row Id="19274" PostId="11021" Text="@Sean is there a way to determine the ranks of different words or get suggestions for synonyms? Presumably this could be helpful with optimal word choice." CreationDate="2011-05-23T19:24:26.743" UserId="1381" />
  <row Id="19275" PostId="11162" Text="@bill, Good idea- but how do I then assign 1 number to the whole graph you posted? (I need to do this)." CreationDate="2011-05-23T19:44:09.583" UserId="4544" />
  <row Id="19276" PostId="11162" Text="@Mike: The &quot;clustering number&quot; would simply be the number of trades per day.  I updated the above graph.   &quot;Low clustering&quot; might be a value of 1 trade or less per day (the green line or below).   &quot;High clustering&quot; might be a value of 10 or more trades per day (the red line or above).   &quot;Normal clustering&quot; might be anything in between the red and green lines.   You can have as many &quot;degrees of clustering&quot; (horizontal lines) as you like.   So, you can spot any degree of &quot;clustering&quot; simply by looking at the rolling &quot;trades per day&quot; number." CreationDate="2011-05-23T20:06:37.820" UserId="2775" />
  <row Id="19277" PostId="11162" Text="@Mike: Each day has a &quot;cluster number&quot;.  If you need one number for the whole graph, then the &quot;rolling window&quot; is the whole graph." CreationDate="2011-05-23T20:08:04.170" UserId="2775" />
  <row Id="19278" PostId="11153" Text="Yes.  I was merely suggesting that the populations of the missing vs. non-missing data be examined before blindly diving into MI, which can take up a lot of computational power at the expense of minimum gains." CreationDate="2011-05-23T20:19:26.730" UserId="3489" />
  <row Id="19279" PostId="11162" Text="@bill I need to create 1 number for the whole graph because I intend to automatically react to this number by entering or not entering another (unrelated) trade. I can take the average of every clustering number within the graph and I guess that would do it, but it seems like an imperfect method. I may have to take the standard deviation between these clustering numbers.... perhaps that is the way." CreationDate="2011-05-23T20:20:19.540" UserId="4544" />
  <row Id="19280" PostId="11069" Text="Thank you for attention, i have a single time series data, don't have any other exogenous variables , the data is not seasonality ,  &gt;&gt;&gt;what about this code &gt;&gt;&gt;&gt;'fit.gls = gls(difftsdata~trend, correlation=corARMA(q=1), method=&quot;ML&quot;)'&lt;&lt;&lt;&lt;&lt;&#xA;&#xA;&#xA;but i dont know how can i forecast Depending on this code!!!!" CreationDate="2011-05-23T20:31:39.663" UserId="4698" />
  <row Id="19281" PostId="11162" Score="1" Text="@Mike: You need to concentrate the idea of &quot;the window&quot;.  In the above case, &quot;the window&quot; is 1 day.  And for trading purposes, you would react each day to the previous day.   The full graph above is 180 days.   However, if you want to use 180 days as &quot;the window&quot; then the total number of trades in the previous 180 days would be your &quot;cluster number&quot; (in this case, there were a total of 404 trades in those 180 days, giving you a cluster number of 404).  This 180-day window would advance each day, dropping off the trades that are 181 days ago and picking up the latest trades." CreationDate="2011-05-23T20:37:33.180" UserId="2775" />
  <row Id="19282" PostId="11154" Text="@Bernd, multilevel data/models" CreationDate="2011-05-23T22:00:02.407" UserId="4717" />
  <row Id="19283" PostId="11162" Text="@Bill The problem is that if I make the &quot;window&quot; 1 day, then I get a number that represents the amount of trades that took place in that day. This is not what I am referring to as the &quot;cluster number.&quot; I need to know the distribution of the variance within that day.  For instance if I had 200 trades in 1 day, 100 of which happened at the very beginning and 100 of which happened at the very end, then with your method I would get &quot;200&quot; as the &quot;clustering number.&quot; But this tells me nothing of their distribution internally. " CreationDate="2011-05-23T22:15:19.600" UserId="4544" />
  <row Id="19284" PostId="11162" Text="@Bill You know what, maybe you're right and I'm over complicating it. I gotta think about it.  I think perhaps what I was trying to solve was &quot;clusters of clusters&quot; instead of just clusters. Hm.." CreationDate="2011-05-23T22:20:32.167" UserId="4544" />
  <row Id="19285" PostId="11162" Text="@Bill I guess what I should do is to make a &quot;cutoff&quot; within, say, hourly windows, beyond which the result is unacceptable." CreationDate="2011-05-23T22:23:58.340" UserId="4544" />
  <row Id="19286" PostId="11154" Text="@Yoel I have updated my post and included some references. However, I am not sure if this is really what you are looking for..." CreationDate="2011-05-23T22:45:45.100" UserId="307" />
  <row Id="19287" PostId="11167" Text="I checked out the source for the qmedist function in fitdistrplus and it appears that that algorithm compares the observed versus expected quantile *values* given a set of quantile probabilities. What I've implemented above compares the observed versus expected density of data between a given set of quantiles. " CreationDate="2011-05-23T22:51:49.723" UserId="364" />
  <row Id="19288" PostId="11166" Text="A QQ-plot compares observed and expected quantile values, whereas what I've implemented above compares the observed versus expected density of data between a given set of expected quantiles. I edited my original question to clarify that I'm not seeking alternative fitting approaches, I'm more seeking input whether there are any obvious pitfalls to the one I've implemented, and also whether it is an already named method." CreationDate="2011-05-23T23:01:48.580" UserId="364" />
  <row Id="19289" PostId="11167" Text="@Mike: You may be right, but I would still describe both as a form &quot;quantile matching&quot;, albeit with different types of loss functions.  " CreationDate="2011-05-23T23:02:41.773" UserId="2958" />
  <row Id="19290" PostId="11094" Text="Another great answer! Thanks whuber." CreationDate="2011-05-24T00:10:22.427" UserId="82" />
  <row Id="19291" PostId="1358" Text="@GaBorgulya: What did you change?" CreationDate="2011-05-24T01:14:43.507" UserId="650" />
  <row Id="19292" PostId="11162" Text="@Mike:  I updated the graph (Edit 2) to show more &quot;windows&quot; for comparison.  The same ideas apply to hourly data as they do to daily data." CreationDate="2011-05-24T02:28:02.733" UserId="2775" />
  <row Id="19293" PostId="11021" Text="@Sean: Have you studied other algorithms actually used in the industry before building yours? And if yes how do these algo work?" CreationDate="2011-05-24T03:03:56.377" UserId="1709" />
  <row Id="19294" PostId="11122" Text="@Thomas Levine, you are absolutely right. I was looking at surv=0 and not surv=1. Thanks for the help!" CreationDate="2011-05-24T04:21:58.490" UserId="4263" />
  <row Id="19295" PostId="11172" Score="1" Text="what is a 'mean correlation coefficient'?" CreationDate="2011-05-24T04:22:29.723" UserId="795" />
  <row Id="19296" PostId="11138" Text="Perhaps you should clarify how you came about this clustering. In principle, it does no make sense to do clustering and then &quot;hope&quot; you can use the result for classification. There are different algorithms for that." CreationDate="2011-05-24T06:38:11.333" UserId="4257" />
  <row Id="19297" PostId="11177" Text="Thanks, but yes I'd seen those resources already. I looked at the source earlier today and frankly had a lot of trouble parsing what was going on." CreationDate="2011-05-24T06:49:22.773" UserId="364" />
  <row Id="19298" PostId="11177" Text="It also doesn't inspire confidence in reading the comments that the header to the QMPE file describes it as QMLE still.... no wonder it was hard to parse what was going on." CreationDate="2011-05-24T07:21:26.140" UserId="601" />
  <row Id="19299" PostId="11178" Text="@baz, what is your question? How to get odd ratios?" CreationDate="2011-05-24T07:27:59.387" UserId="2116" />
  <row Id="19300" PostId="11178" Text="@mpiktas, that is what I need to get, yes! " CreationDate="2011-05-24T07:35:21.900" UserId="4263" />
  <row Id="19301" PostId="11178" Score="2" Text="are none of those predictors continuous variables?" CreationDate="2011-05-24T07:51:46.880" UserId="601" />
  <row Id="19302" PostId="11178" Text="@John, logistic regression also accept nominal predictors, but they should be coded as dummy variables." CreationDate="2011-05-24T08:08:18.867" UserId="609" />
  <row Id="19304" PostId="11117" Text="But how can it be the same, when BIC and AIC use different penalizations? This sentence that you posted makes no sense to me!" CreationDate="2011-05-24T08:08:48.100" UserId="4496" />
  <row Id="19305" PostId="11117" Text="@MarkDollar, Although BIC and AIC use different penalizations, the function returns the best model when the number of parameters is fixed. If number of parameters is fixed then penalisation term is the same for all models considered, hence the best model is selected only depending on its log likelihood, which is the same for BIC and AIC." CreationDate="2011-05-24T08:43:23.447" UserId="2116" />
  <row Id="19306" PostId="11187" Text="@mpiktas thnx for the reply, i want to understand the mathematical formula for this and then may be implement. " CreationDate="2011-05-24T11:56:54.797" UserId="3722" />
  <row Id="19307" PostId="824" Text="@john - good answer.  I have a small quibble: the use of the word &quot;subjective&quot; and &quot;objective&quot; is not quite appropriate, because no methods are &quot;objective&quot;.  I'd say more the frequentist and &quot;objective&quot; Bayesians simply derive their probability distributions by using certain rules or standards.  So rather than tayloring for the specific case at hand, a frequentist/objective Bayesian will apply &quot;default&quot; choices (thus hiding their subjectivity)." CreationDate="2011-05-24T12:02:13.010" UserId="2392" />
  <row Id="19308" PostId="11187" Text="@Amit, the reasoning behind this formula is not simple, you can get the references from the **strucchange** [vignette](http://cran.r-project.org/web/packages/strucchange/strucchange.pdf). Why do you want to implement it yourself?" CreationDate="2011-05-24T12:19:58.580" UserId="2116" />
  <row Id="19309" PostId="10890" Text="@MarkDollar, try me, I know Russian, and formulas are the same for all languages." CreationDate="2011-05-24T13:16:27.613" UserId="2116" />
  <row Id="19310" PostId="11138" Score="1" Text="Hierarchical clustering relies on a dissimilarity metric that determines the distance from a point to a cluster. Why don't you use the labeled points as clusters, and calculate, for each unabeled point, the dissimilarity to all clusters? Then, you can either make a hard assignment using the smallest dissimilarity, or you can make a soft assignment using relative dissimilarities." CreationDate="2011-05-24T13:24:22.157" UserId="198" />
  <row Id="19311" PostId="11187" Text="@mpiktas I want to understand how in general Regularisation can be applied to these kinds of problems" CreationDate="2011-05-24T13:32:37.370" UserId="3722" />
  <row Id="19312" PostId="11187" Text="@Amit, why do you call this regularisation? Do you have any references?" CreationDate="2011-05-24T13:37:28.977" UserId="2116" />
  <row Id="19313" PostId="11187" Text="@mpiktas Regularisation in general helps to overcome the problem of overfitting" CreationDate="2011-05-24T14:40:59.993" UserId="3722" />
  <row Id="19314" PostId="10988" Text="Thanks bayer. It seems that &quot;distance metric learning&quot; is booming -- scholar.goo has 50 titles since 2008. But is the boom paper, or real use ?  Footnote, the code for nca says &quot;iterations ... at least 100000 for good results&quot;.  Footnote 2, most of the work on distance metric learning seems to model a Mahalanobis distance; would you know of other distance models ?" CreationDate="2011-05-24T15:00:52.210" UserId="557" />
  <row Id="19315" PostId="11194" Text="What if there were other variables: how do you carry them along?" CreationDate="2011-05-24T15:09:15.030" UserId="279" />
  <row Id="19316" PostId="11194" Text="@mpiktas We don't move such questions -- too much rush for too little gain. " CreationDate="2011-05-24T15:24:48.097" UserId="88" />
  <row Id="19317" PostId="10976" Text="Bob, isn't $\sum |a_j - b_j|^q$ (without the outer $1/q$) a metric&#xA;for 0 $&lt; q &lt;$ 1,&#xA;satisfying the triangle inequality ?" CreationDate="2011-05-24T15:31:58.427" UserId="557" />
  <row Id="19318" PostId="11199" Score="1" Text="And as usual this is about 2x faster than plyr version." CreationDate="2011-05-24T15:39:06.903" UserId="88" />
  <row Id="19319" PostId="10993" Text="@whuber Do you have some more detailed description about Kolmogorov-Smirnov statistic D?" CreationDate="2011-05-24T15:54:20.817" UserId="4670" />
  <row Id="19320" PostId="10993" Text="@Fan http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test" CreationDate="2011-05-24T15:55:31.330" UserId="919" />
  <row Id="19321" PostId="11200" Text="According to the manual for version 8, Stata uses the &quot;KISS&quot; generator by G. Marsaglia of period approximately 2^126.  The only reason to need something better is for cryptographic applications." CreationDate="2011-05-24T16:12:08.563" UserId="919" />
  <row Id="19322" PostId="11186" Score="1" Text="Although this question is stated quite differently, it appears to be identical to http://stats.stackexchange.com/q/2432 ." CreationDate="2011-05-24T16:16:55.063" UserId="919" />
  <row Id="19323" PostId="11167" Score="1" Text="@Mike @Henry This subtle difference, though, is crucial: binning by *sample* quantiles has legitimate theoretical justification whereas dynamic re-binning by quantiles determined by the parameters does not and suffers from several problematic behaviors." CreationDate="2011-05-24T16:32:59.083" UserId="919" />
  <row Id="19324" PostId="11204" Score="1" Text="You might have more chance asking this at http://www.crossvalidated.com or http://www.biostar.stackexchange.com . In any case, keep in mind that you'll run into the multitesting problem. There are solutions for this (we develop solutions for similar problems at our departement), but they're not as easy as running 15000 t-tests or wilcoxons." CreationDate="2011-05-24T11:52:43.603" UserId="1124" />
  <row Id="19325" PostId="11204" Score="1" Text="@Arun - I recommend splitting this into smaller questions that illustrate the specific problems you are encountering. For instance, say you don't know how to get your four data files into R into an appropriate format. Ask a question about that - or better yet, check out this question or one of the many variants: http://stackoverflow.com/questions/5758084/loop-in-r-loading-files. As for the appropriate statistical test(s) to run, crossvalidated.com may be a better venue for those discussions. At a minimum, update your question with a reproducible set of data so others can offer tangible advice." CreationDate="2011-05-24T11:55:45.870" UserId="696" />
  <row Id="19326" PostId="11204" Text="Joris Meys, Chase, I'll split them into smaller problems, and also try posting on cross-validated.com (or) biostar. Thank you!" CreationDate="2011-05-24T13:17:55.130" />
  <row Id="19327" PostId="11112" Text="You can compare the restored signals with mean squared error (MSE) against the original. The best method will be the one with the lower MSE.&#xA;Another criterion might be the improvement in signal-to-noise ratio (SNR)." CreationDate="2011-05-24T16:55:29.103" UserId="2902" />
  <row Id="19328" PostId="9924" Text="This is exactly the secretary problem. At HCSSiM, the lore was that this was first presented in a talk titled 'sex and the single statistician.'" CreationDate="2011-05-24T17:05:54.270" UserId="795" />
  <row Id="19329" PostId="11201" Text="Thanks Whuber - I read about the seed - I still have to digest the idea of how Stata treats randomness, but at least I now know that they are doing something right which is worth my time to understand.  About the second part, I thought about the same approach (doing a merge), but it was great to see your code, as mine wouldn´t be so good!  Cheers, and thanks a lot again." CreationDate="2011-05-24T17:45:47.407" UserId="2929" />
  <row Id="19330" PostId="11108" Text="why is this tagged as &quot;time-series&quot;? It doesn't seem correct." CreationDate="2011-05-24T17:48:59.260" UserId="3411" />
  <row Id="19331" PostId="11056" Text="@Jonathan I posted the example code, let me know if you need any help implementing it." CreationDate="2011-05-24T18:01:56.283" UserId="2817" />
  <row Id="19332" PostId="11208" Score="3" Text="dober dan! do you know if the official law is designed for roulette or for some other gambling devices? (slot machine, poker, etc) The third requirement sounds like autocorrelation, or it could be tested by Wald-Wolfowitz test, etc." CreationDate="2011-05-24T18:36:16.093" UserId="795" />
  <row Id="19333" PostId="7862" Text="@vqv that's a really nice visualization.  Any chance you'd share the R code?" CreationDate="2011-05-24T18:39:32.307" UserId="1127" />
  <row Id="19334" PostId="11208" Text="It is designed for general gambling device. Link for law (in slovene language) is at http://www.unpis.gov.si/fileadmin/unpis.gov.si/pageuploads/zakonodaja/PTZIN_neuradno_precisceno_110511_s_prilogo__2_.pdf (paragraph 6) " CreationDate="2011-05-24T18:41:00.740" UserId="4738" />
  <row Id="19335" PostId="11195" Text="@chl great answer. I got the concept. thank you so much." CreationDate="2011-05-24T19:08:32.247" UserId="2885" />
  <row Id="19336" PostId="11200" Score="1" Text="Just to add that I tried the same thing in R and got the same behavior.  Namely, I opened two R windows, ran the command &#xA;x &lt;- rnorm(50)&#xA;and got the same vector of 50 random numbers in both windows." CreationDate="2011-05-24T19:09:07.617" UserId="2929" />
  <row Id="19337" PostId="11183" Text="Hey Thanks much! So did I get it right that it is neccessary to use an offset, when you compare counts over different times? " CreationDate="2011-05-24T19:34:13.123" UserId="4496" />
  <row Id="19338" PostId="11194" Text="@Aniko, thanks, I've updated the answer." CreationDate="2011-05-24T19:43:55.147" UserId="2116" />
  <row Id="19339" PostId="11199" Score="1" Text="@mbq, yes, naturaly, but if you include the debugging costs, for usual data sets the resulting speed is the same :) **plyr** is dedicated not for speed, but for clarity and convenience. " CreationDate="2011-05-24T19:45:57.360" UserId="2116" />
  <row Id="19340" PostId="11044" Text="Yes, it depends on the amount of non-normality.  Why depend on good fortune when formulating a model?  There are also many purely philosophical reasons to treat time zero measurements as baseline measurements (see quotes from Senn and Rochon in my notes)." CreationDate="2011-05-24T20:31:18.867" UserId="4253" />
  <row Id="19341" PostId="11213" Text="that's kind of what I thought, but there's something circular about this reasoning. I'm not a statistician, but my sense was that statistical inference works when your estimators converge rapidly, so even if your sample hasn't converged on the distribution, your inferences are sound. In this case, we're relying on the entire empricial distribution to converge to the actual distribution. Maybe there are theorems saying that some bootstrap estimates converge quickly, but I generally see bootstrapping applied without appealing to such theorems." CreationDate="2011-05-24T21:15:25.967" UserId="4733" />
  <row Id="19342" PostId="11102" Text="The LSA code I was using (from joseph wilk's blog, in my post) &quot;reconstructs&quot; the matrix after SVD.  To be honest, I don't really know much about SVD or what it's doing.  I'm still having some problems, but I think we've moved beyond my original question so I'll marked this solved.  Thanks!" CreationDate="2011-05-24T21:25:55.440" UserId="1977" />
  <row Id="19343" PostId="11210" Text="There are several related questions you might want to look through. Some are listed on the side margin of this page. [Here](http://stats.stackexchange.com/questions/9664/what-are-examples-where-a-naive-bootstrap-fails) is one regarding when the bootstrap fails and what it means for it to fail." CreationDate="2011-05-24T21:57:38.413" UserId="2970" />
  <row Id="19344" PostId="11145" Text="How did the subjects end up consuming different quantities of the chemical?  Calling something a &quot;natural experiment&quot; implies that there is an exogenous source of variation in the treatment variable.  Do you have reason to believe that true in this case?" CreationDate="2011-05-24T22:33:17.267" UserId="3748" />
  <row Id="19345" PostId="11213" Score="1" Text="The apparent circular reasoning is why it was nicknamed the bootstrap.  It felt like people were trying to lift themselves by their own bootstraps.  Later Efron showed that it really did work." CreationDate="2011-05-24T22:39:04.323" UserId="4505" />
  <row Id="19346" PostId="11209" Text="did you implement the test in Matlab yourself, or are you checking against code from the statistics toolbox? At risk of adding a second gold standard, have you checked it against R's implementation?" CreationDate="2011-05-24T22:41:19.847" UserId="795" />
  <row Id="19347" PostId="8532" Text="Dear @Pete - the Spotfire Analytics Server integrates into your existing information and analytics technology infrastructure, including databases like Oracle, IBM DB2, Microsoft SQL Server, Microsoft Access, MySQL, SAS/SHARE, Sybase, Informix or any custom JDBC data source" CreationDate="2011-05-24T14:18:05.413" UserId="4732" />
  <row Id="19348" PostId="11201" Score="1" Text="If you actually want the random numbers to change every time you start Stata: one idea would be to base you &quot;seed&quot; on your clock using your profile.do file. Try ``set seed `=clock(&quot;$S_DATE $S_TIME&quot;,&quot;DMYhms&quot; )-162e10'``" CreationDate="2011-05-25T00:47:32.987" UserId="4691" />
  <row Id="19349" PostId="9336" Text="I think I can relate to that." CreationDate="2011-05-25T01:08:29.040" UserId="2669" />
  <row Id="19350" PostId="11116" Text="@Frank Harrell: I am following that course notes and it a big help. However, I need to do: a &quot;backward elimination&quot; to get my best model" CreationDate="2011-05-25T02:32:33.843" UserId="4263" />
  <row Id="19351" PostId="11116" Text="Why would you need to do that?  That will invalidate betas, SEs, P-values, and confidence limits.  It can be somewhat safe if you set alpha for keeping a variable to 0.5.  You can do that with the **rms** `fastbw` function.  Better still is to set alpha=1.0 (don't drop any variables).  Who suggested that stepwise variable selection without penalization is a good idea?" CreationDate="2011-05-25T02:51:08.577" UserId="4253" />
  <row Id="19352" PostId="11046" Text="@Dikran : That seems reasonable. Let me give it a shot. " CreationDate="2011-05-25T02:54:26.577" UserId="3301" />
  <row Id="19353" PostId="11116" Text="@Frank: Well, it is something that I thought would work and was not considering what you just mentioned. The whole idea is this: (i) run the full model, (ii) keep only predictors that are significant (table of coefficients), (iii) test for the overall effect of each predictor and between each level (to get some ideas of their effect (iv) get the odd ratios" CreationDate="2011-05-25T03:06:13.467" UserId="4263" />
  <row Id="19354" PostId="11223" Text="@ Greg, Thank you for your reply. What are the cases when the median cannot be used with ordinal level data?" CreationDate="2011-05-25T03:24:56.050" UserId="4498" />
  <row Id="19355" PostId="11199" Text="and using ave is twice as fast anyway :)" CreationDate="2011-05-25T03:26:46.090" UserId="375" />
  <row Id="19356" PostId="11102" Text="I imagine what it's doing is &quot;reconstructing&quot; the matrix from the first few singular vectors. The link to the blog post is broken btw. But that would make absolutely no difference if you go on and do PCA to the reconstructed matrix! Perhaps it's centering/scaling for you, then you would get different results. I'd look into it further; it seems like you want to be working with the scores from LSA to do clustering/categorization/etc" CreationDate="2011-05-25T03:44:15.000" UserId="26" />
  <row Id="19357" PostId="11217" Score="1" Text="+1 if only all answers were so complete!" CreationDate="2011-05-25T03:48:07.850" UserId="26" />
  <row Id="19358" PostId="11102" Text="Link works for me, but I think I'm going to explore gensim further and see if that pans out.  Thanks again!" CreationDate="2011-05-25T03:48:19.157" UserId="1977" />
  <row Id="19359" PostId="11102" Text="Weird - link isn't a link at all for me.  Gensim is a pretty mature piece of kit, you should do well with it. If you have to process and corpora the natual language toolkit ( aka NLTK) has some useful bits &amp; is super well documented, although it's more NLP oriented. Good luck :)" CreationDate="2011-05-25T03:55:34.393" UserId="26" />
  <row Id="19360" PostId="11102" Text="http://blog.josephwilk.net/projects/latent-semantic-analysis-in-python.html" CreationDate="2011-05-25T04:01:10.273" UserId="1977" />
  <row Id="19361" PostId="11102" Text="Thanks. My suspicions were correct; PCA on that reconstructed matrix should give you back what you got from LSA - that's not what you want. There's a little snippet about the relationship between PCA/SVD here: http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/4000/pdf/imm4000.pdf. Googling will undoubtedly turn up more. Using $V$ to categorize documents would be more interesting - then you're using scores on the components from LSA instead of raw word counts." CreationDate="2011-05-25T04:29:40.933" UserId="26" />
  <row Id="19362" PostId="11209" Text="@shabbychef I implemented the code in c++. and the results is the same as matlab when I have between 1 to 5 replicates per level. but when this range changes to 1 to 100, some of my SS values become higher in comparison to corresponding SS values of MATLAB. and since each SS value is computed based on the others, some of them become negative! It does not happen in matlab. So I know that something is wrong in my code." CreationDate="2011-05-25T05:55:21.710" UserId="2885" />
  <row Id="19363" PostId="11217" Text="+1, I second @JMS, wonderful answer." CreationDate="2011-05-25T06:57:35.303" UserId="2116" />
  <row Id="19364" PostId="11186" Text="@whuber, good catch, judging from the comments to my answer, this looks like what OP actually wants." CreationDate="2011-05-25T07:00:43.523" UserId="2116" />
  <row Id="19365" PostId="11216" Text="+1, did not know about ave. When did it appear in R?" CreationDate="2011-05-25T07:03:30.310" UserId="2116" />
  <row Id="19366" PostId="11199" Score="2" Text="@Eduardo `ave` is a wrapper of `lapply`+`split`, check the code (-;" CreationDate="2011-05-25T07:09:13.960" UserId="88" />
  <row Id="19367" PostId="11211" Text="Yes, I saw that paper too, but couldn't access it :( In the end I did use McNemar's test to do pairwise tests, for which I reported effect sizes, however for the original Cochran's Q test, I never did find a suitable effect size measure. I think it is convention to report effect sizes, both for the original test and for the post-hoc pairwise tests, right?" CreationDate="2011-05-25T08:36:14.417" UserId="4258" />
  <row Id="19368" PostId="11211" Text="@Ham Yes, reporting ESs is recommended in many settings (I can think of the [APA guidelines](http://people.cehd.tamu.edu/~bthompson/apaeffec.htm) for example), although I would say reporting group difference (unstandardized, because ES are just standardized difference) is already good practice." CreationDate="2011-05-25T08:53:02.183" UserId="930" />
  <row Id="19369" PostId="11230" Score="1" Text="Good references! I think this one is also worth reading: Blair &amp; Higgins, &quot;Test of hypotheses for unbalanced factorial designs under various regression/coding method combinations&quot; (1977) http://epm.sagepub.com/content/38/3/621.full.pdf+html Like Venables, they stress that type III model comparisons give different results for different coding methods. For correct type III SS, one needs a sum-to-zero coding scheme like effect coding or Helmert. Dummy coding (default in R) gives the wrong results." CreationDate="2011-05-25T08:58:22.377" UserId="1909" />
  <row Id="19370" PostId="11230" Text="@caracal (+1) Thanks for the reference. I'm pretty sure David Howell also discussed this issue at length in his textbook, but I cannot retrieve it actually (should learn to clean up my desk one day :-)" CreationDate="2011-05-25T09:01:38.397" UserId="930" />
  <row Id="19371" PostId="11231" Score="2" Text="@JIGsawed It would be very helpful to add some context about your particular dataset and what you want to do in your analysis. PCA can be viewed as a dimensionality reduction method (helpful in a descriptive purpose) or a way to construct linear combinations of the original variables that can be subsequently used in a modeling framework, but as your question stands it is difficult to provide an answer." CreationDate="2011-05-25T10:02:29.660" UserId="930" />
  <row Id="19372" PostId="11199" Text="@mbq ops! but subsetting does seem to be much faster than rbinding the results" CreationDate="2011-05-25T10:57:36.533" UserId="375" />
  <row Id="19373" PostId="11199" Score="1" Text="@Eduardo Yeah, but it all works only due to a quirky possibility of vectorized sorting within factors using `order`; for more generic problems `split` is inevitable." CreationDate="2011-05-25T11:05:29.410" UserId="88" />
  <row Id="19374" PostId="11231" Text="I am interested in projecting my variables onto the eigenvectors yet I am not sure whether I want to compute dot product, correlate with eigenvectors. I came across some papers that did regression with eigenvectors at this stage, this is where my question stems from" CreationDate="2011-05-25T12:38:02.100" UserId="4747" />
  <row Id="19375" PostId="11116" Text="One way to understand the issue is to consider a different model, the ordinary multiple regression model.  This model involves a residual variance.  Stepwise variable selection results in an underestimate of this variance.  Two unbiased estimators for the variance have been developed: (1) fit a full pre-specified model and compute the residual variance; or (2) run stepwise selection or other data mining procedure and use Ye's generalized degrees of freedom (d.f.) to estimate the &quot;effective d.f.&quot; of the modeling process (close to # candidate variables), then use it to estimate the variance." CreationDate="2011-05-25T12:40:29.770" UserId="4253" />
  <row Id="19376" PostId="11227" Score="2" Text="The title says &quot;intersection&quot;, while a union symbol is used in the text. Which one do you want?" CreationDate="2011-05-25T13:18:42.683" UserId="279" />
  <row Id="19377" PostId="11229" Text="Thanks, I like this idea a lot. I also updated my main question with an example of what I've done so far as requested. Is it fair of me to average the data points into minute buckets, when each bucket may not have the same amount of samples in it?" CreationDate="2011-05-25T13:26:09.277" UserId="4739" />
  <row Id="19378" PostId="11235" Text="Thanks both for your answers. Could you please direct me to a book/reference where I could find a proof of the fact that for large samples the paired t-test works?" CreationDate="2011-05-25T13:33:34.587" UserId="4748" />
  <row Id="19379" PostId="11238" Text="Thanks for your answer. I think that the second intrepretation is the the correct one, as I am comparing the prices of a set of products in two different shoppings." CreationDate="2011-05-25T13:52:00.770" UserId="4748" />
  <row Id="19380" PostId="11199" Text="@mpq I think this is a quite common operation. I use it quite frequently. It is expressive and vectorized, what's not to like :)  " CreationDate="2011-05-25T14:51:54.167" UserId="375" />
  <row Id="19381" PostId="11233" Text="I would try to go for `library(boot)` aiming to bootstrap the results from `lme` fit. This link could probably help https://stat.ethz.ch/pipermail/r-help/2004-November/060444.html" CreationDate="2011-05-25T15:27:38.237" UserId="2645" />
  <row Id="19382" PostId="11223" Text="I cannot think of any cases of ordinal data that you cannot use the median, there are probably cases where other summaries are more meaningful (small number of levels, just give graph or count of everything)." CreationDate="2011-05-25T15:41:50.837" UserId="4505" />
  <row Id="19383" PostId="11240" Text="Matt, just to clarify on the &quot;alpha-blending&quot; comment: I assume you mean setting the alpha channel of the graphics to allow for some level of transparency in displayed graphical elements?" CreationDate="2011-05-25T16:22:27.317" UserId="1080" />
  <row Id="19384" PostId="11240" Text="I did attempt a scatterplot, but I found that I had such tight clustering of the datapoints for each x-value that it's a bit hard to tell, personally, just how many values appear at certain levels - see http://i.imgur.com/FsNC1.png for example. I'm nervous that a chart like this doesn't make it obvious just how many points appear in the cluster around the y=0 line" CreationDate="2011-05-25T16:32:57.097" UserId="4739" />
  <row Id="19385" PostId="11229" Score="1" Text="Another possibility would be to plot density strips for each time period as opposed to discrete colors for quantiles, see this article in the American Statistician for examples http://dx.doi.org/10.1198/000313008X370843 . The graph posted seems pretty informative as is though (+1)" CreationDate="2011-05-25T16:36:58.693" UserId="1036" />
  <row Id="19386" PostId="11242" Text="I read the whole post, but I can't work out what your question is." CreationDate="2011-05-25T15:08:32.660" UserId="2425" />
  <row Id="19387" PostId="11242" Text="I've tried adding clarifying wrap-up at the end.  Does it help you?" CreationDate="2011-05-25T15:37:56.270" />
  <row Id="19388" PostId="11242" Text="I had applied the tag &quot;statistics&quot; but apparently that is insufficient.  Apparently I'll have to make a duplicate post.  That's unfortunate. " CreationDate="2011-05-25T16:13:30.253" />
  <row Id="19389" PostId="11242" Text="I've flagged the question and asked that it be migrated so stats.SE." CreationDate="2011-05-25T16:35:39.983" UserId="2970" />
  <row Id="19390" PostId="11240" Text="I think this may be a good approach for visualizing outliers, I'm not so sure it is a great idea for the whole distribution though. Also as a note in regards to your linked question, Tufte has an example in his Visual Display of Quantitative Information where plotting the lines between the temporal dimension (similar to your chart 1 on the linked question) shows an obvious periodic pattern, whereas it is much more difficult to discern with just the scatterplot without the lines." CreationDate="2011-05-25T16:42:37.953" UserId="1036" />
  <row Id="19391" PostId="11240" Text="@Josh  Right - as in [this image](http://had.co.nz/stat405/resources/drills/plot-drills/ggplots/soheight14.png)" CreationDate="2011-05-25T17:35:00.213" UserId="71" />
  <row Id="19392" PostId="11240" Score="1" Text="@matt  That's a classic case for alpha-blending (see the image in the comment above).  It requires some experimentation with the level of transparency, but you'll likely be able to find something that works.  It still won't be perfectly clear how many points are at each spot on the axis, but it gives the viewer a sense of density that lines don't." CreationDate="2011-05-25T17:39:20.490" UserId="71" />
  <row Id="19393" PostId="10954" Text="In this case I am talking about a developing country and a voluntary health insurance scheme so there is definitely self-selection into the scheme. Sorry for not making this clearer the first time. Thanks for your comments. " CreationDate="2011-05-25T17:44:45.067" UserId="834" />
  <row Id="19394" PostId="11130" Text="Dear Mermex, Thank you so much for your reply. I'm sorry if I didn't explain the subsample very well. I foresee two ways of estimating the impacts of health insurance on health expenditure. The first is to take a subsample of individuals who have any health care visits and compare their expenditures, controlling for health insurance status and other variables. The second is the two-part model (either log-transformed OLS or Heckman's sample selection model) to compare expenditures between the insured and uninsured. In my understanding, if I take a subsample of individuals with any expenditure.." CreationDate="2011-05-25T17:53:34.203" UserId="834" />
  <row Id="19395" PostId="11130" Text="and then compare expenditures between insured and uninsured (after controlling for other vars) I am effectively answering your question #1 above (what is the health care expenditure of those who have it). However, if I want to answer your question #2 above I need to use a two step model that takes into consideration the selection into insurance. Is this correct? " CreationDate="2011-05-25T17:55:09.367" UserId="834" />
  <row Id="19396" PostId="11240" Text="@Andy It all depends on the purpose of the plot, as Josh mentions.  The scatterplot that matt posted seems a bit more informative than the line plot to me (especially the spikes after 14:07), but I don't know what the context is (and I've been obsessing over fat-tailed distributions lately).  As far as the linked question goes: do you mean connecting the dots?  I'll definitely try that out when I finally have time to get back to that dataset.  It's a very seasonal process, actually, and you can barely tell from that scatterplot.  Thanks for the tip." CreationDate="2011-05-25T17:56:32.603" UserId="71" />
  <row Id="19397" PostId="11241" Text="For readability it would help to eliminate all but one plot or to make them all a lot smaller." CreationDate="2011-05-25T18:00:36.220" UserId="919" />
  <row Id="19399" PostId="11247" Text="Answer to a deleted comment: If you have 4 mutually independent questions, each with a 50/50 split in the population, that is the _best_ you can do, and it will be at least as good as any 4-deep question tree (with binary responses.) This is actually mathematically trivial since any decision tree can be encoded as a (complicated) equivalent 4-item questionnaire.  Example:  Question 1: Do you support A? T/F.  Question 2:  (Do you support B AND did you answer T in question 1) OR (Do you support C AND did you answer F in question 1) etc." CreationDate="2011-05-25T18:50:28.980" UserId="3567" />
  <row Id="19400" PostId="11247" Text="So if I understand your suggestion, you would give the survey volunteers a random assortment of paired questionnaires to run past likely voters.  Let's say 10 random pairs each.  Then for each 8-question pairing answered, an open field of guessers would be presented with answers to one of the pair of questionnaires to see how well they could guess the answers to the other questionnaire." CreationDate="2011-05-25T18:59:21.747" UserId="4753" />
  <row Id="19401" PostId="11247" Text="Nope, I suggest scrap the pairing.  Say you have 10 competing questionnaires.  Collect 10 interviews for each questionnaire (each respondent only answers one questionnaire).  Then give each guesser a private randomized test in which they fill in missing responses for each of the 100 interviews collected.  This gives you more data to work with." CreationDate="2011-05-25T19:12:46.680" UserId="3567" />
  <row Id="19402" PostId="11243" Text="@Josh You added the tag `ridge-regression`. Does it mean that you have a large number of predictors and/or that you suspect that it has something to do with the observed difference in prediction accuracy?" CreationDate="2011-05-25T19:38:19.723" UserId="930" />
  <row Id="19403" PostId="11123" Text="I've decided to go with this...hopefully it's correct!" CreationDate="2011-05-25T19:57:24.530" UserId="4238" />
  <row Id="19404" PostId="11247" Text="The surveyed voter would have to be presented with more than just the 4 questions from the one questionnaire otherwise there would be no data that could be missing from that presented to the guesser.  So let's say that rather than a pair of questionnaires the voter is presented with 8 questions, 4 of which are randomly selected from the population of questions in all the questionnaires, and the other 4 are from a single questionnaire.  The guessers would then have to guess the answers to the 4 randomly drawn questions based on the answers to the questionnaire's 4 questions.  Is that right?" CreationDate="2011-05-25T19:58:51.733" UserId="4753" />
  <row Id="19405" PostId="11247" Text="No, a portion of responses are blanked out from each survey.  For example, the guesser is told the voter's answer to questions 1 and 3 and has to guess the voter's answer to questions 2 and 4." CreationDate="2011-05-25T20:01:41.413" UserId="3567" />
  <row Id="19406" PostId="11246" Text="What do you need the table for?  Are your trying to compare the 2 models? Are you trying to combine the 2 models?" CreationDate="2011-05-25T20:04:09.277" UserId="2817" />
  <row Id="19407" PostId="11247" Text="But the purpose of the questionnaire is to predict the answers to other questions and ideally the answers in the 4 questions would be independent so as to maximize the information." CreationDate="2011-05-25T20:04:34.677" UserId="4753" />
  <row Id="19408" PostId="11243" Text="@chl I added the tag since it is the regression method that is used by the framework." CreationDate="2011-05-25T20:22:58.003" UserId="3052" />
  <row Id="19409" PostId="11246" Text="Hi Zach - In short - I would like to compare and get a P value" CreationDate="2011-05-25T20:24:25.463" UserId="253" />
  <row Id="19410" PostId="11243" Text="@Josh: the answer may depend upon the measure for which you are optimizing: do you optimize for AUC, for missclassification,... Can you specify? This may also answer my other question: what does 'rather poor performance' mean? Finally: have you tried with something like tenfold CV?" CreationDate="2011-05-25T21:27:40.080" UserId="4257" />
  <row Id="19411" PostId="11249" Score="9" Text="As far as I know, Excel 2007 only allows you to have 1 million or so rows (A quick google search turns up a maximum of 1,048,576 rows in Excel 2007). I would say your best bet is to use a database program, like SQL or Access. There are simple database queries that will give you what you want (subtotals, sd's, etc). Also, I haven't really worked with it myself, but I am sure you can access datasets from Access using Excel. Honestly, though, I would just use R, and for what you're trying to calculate, it does not take long to learn some simple functions :)" CreationDate="2011-05-25T21:49:06.687" UserId="2252" />
  <row Id="19412" PostId="11249" Text="Thank you for the suggestion about R, do you know any online resources that could get me up and running fairly quickly?  I have some experience programming in Python." CreationDate="2011-05-25T22:07:09.373" UserId="4755" />
  <row Id="19413" PostId="11249" Score="1" Text="@Furlong: You can download R for free [here](http://www.r-project.org/). There are many guides to get started with R available online. Here is [one for beginners](http://cran.r-project.org/doc/contrib/Paradis-rdebuts_en.pdf) that I found on google. Since you have experience in a programming language, you should be fine with the idea of variable assignment. You should focus on reading in files in R, and learning how to use functions (like sum(), mean(), sd(), etc)." CreationDate="2011-05-25T22:41:16.200" UserId="2252" />
  <row Id="19414" PostId="6483" Text="What is your data? What do your space-time observations represent?" CreationDate="2011-05-25T22:42:56.047" UserId="4329" />
  <row Id="19415" PostId="11243" Text="@Nick I am using ridge regression to estimate a vector of angles; the first dataset gives a mean error of around 5-8 degrees, whereas the second set gives a mean error of 14-40 degrees (hence the 'poor performance'). No, I am unable to use tenfold CV due to limited data: the data relates to 1 of 4 subjects, each test set contains the data of 1 subject and the remaining 3 make up the training set. I need the test subject to be completely unseen in the training set." CreationDate="2011-05-25T22:47:57.060" UserId="3052" />
  <row Id="19416" PostId="11251" Score="1" Text="&quot;in a much more efficient manner than Excel&quot;, I guess :)" CreationDate="2011-05-25T23:24:28.173" UserId="4337" />
  <row Id="19417" PostId="11247" Text="In that case, I would suggest that the contest holder decide on a list of &quot;other questions&quot; which will serve as the criteria for evaluating the questionnaire.  Comparing questionnaires against each other introduces needless variation; instead, you should compare all questionnaires against a common standard." CreationDate="2011-05-25T23:28:33.313" UserId="3567" />
  <row Id="19418" PostId="11249" Text="If your original file is in the Access format, you can calculate what you need with the help of quires and/or Visual Basic modules." CreationDate="2011-05-25T23:30:06.440" UserId="4337" />
  <row Id="19420" PostId="11243" Text="@Josh: I hope I'm misunderstanding here: does the above mean you have only 4 observations? Since you're using ridge regression, I assume you are doing some variable selection of sorts. I believe variable selection and 4 observations do not match..." CreationDate="2011-05-25T23:54:19.997" UserId="4257" />
  <row Id="19421" PostId="11251" Text="Good catch. I just edited that." CreationDate="2011-05-26T00:46:33.127" UserId="334" />
  <row Id="19422" PostId="11217" Text="Very nice, @cardinal (+1)." CreationDate="2011-05-26T01:59:25.457" UserId="1108" />
  <row Id="19423" PostId="11112" Text="How about plotting cross-correlation function? or comparing ACF of the original signal with the restored ones?" CreationDate="2011-05-26T02:35:59.953" UserId="4286" />
  <row Id="19424" PostId="11227" Score="1" Text="Since $\frac{|A\cap B|}{|A| + |B|} + \frac{|A\cup B|}{|A| + |B|}  = 1$, it doesn't really matter." CreationDate="2011-05-26T03:41:41.607" UserId="4062" />
  <row Id="19425" PostId="11226" Text="@ Jeromy. I understood that if you are using ordinal data &quot;When the sample size is even&quot; and the numbers on either side of the cut point are different then it is not possible to calculate the median. This is because there is no clear mid point between two ordinal values. That is, your values are ordinal not ratio/interval because there are no even distances between them. " CreationDate="2011-05-26T04:22:14.803" UserId="4498" />
  <row Id="19427" PostId="11256" Score="1" Text="What is the purpose of the regression?  Prediction, exploration, model fitting, something else?" CreationDate="2011-05-26T04:50:12.043" UserId="919" />
  <row Id="19428" PostId="11228" Text="Well, my forecasts are as expected according to seasonality but the actual values are either high or lower than the forecast. That might be due to randomness in the data or some other factor. This is giving high MAPE for these cases. I am already using deseasonalized data. Should I remove the randomness component as well from the data and then do the forecasting. " CreationDate="2011-05-26T05:07:46.277" UserId="4445" />
  <row Id="19430" PostId="11183" Text="@MarkDollar: Yep!" CreationDate="2011-05-26T05:17:09.237" UserId="3019" />
  <row Id="19431" PostId="11259" Score="1" Text="By to produce $beta$'s you mean to move to standardized regression? Probably not, since your covariates are all dichotomous. And I just curious what transformation you actually used, since I posted my tips before this question. " CreationDate="2011-05-26T05:32:55.800" UserId="2645" />
  <row Id="19432" PostId="11031" Text="@chl: I think it would be a good idea to migrate it to StackOverflow - will this link still work (and reroute it there) after the migration?" CreationDate="2011-05-26T05:54:42.597" UserId="230" />
  <row Id="19433" PostId="11228" Text="@Abhishek, how do you usually remove (or intend to remove in this case) randomness component?" CreationDate="2011-05-26T05:58:38.740" UserId="2116" />
  <row Id="19434" PostId="11251" Score="2" Text="Questions about R can be asked here too  :)" CreationDate="2011-05-26T06:09:58.510" UserId="2116" />
  <row Id="19436" PostId="6483" Text="Documents that have a time and location of origin." CreationDate="2011-05-26T07:33:32.243" UserId="2880" />
  <row Id="19437" PostId="11254" Text="Hi Frank, it is a pleasure to see you here!  Thank you for your answer.  I wonder if there is some way to compute such a P value from the deviance of the models (maybe combined with something like bootstrapping on it).  If you (or anyone else) can help direct me to any way of achieving the computation of some &quot;p value&quot; for the comparison of the two models, it would be great.   In any case - again, thank you for your answer.  Best, Tal" CreationDate="2011-05-26T07:36:53.140" UserId="253" />
  <row Id="19438" PostId="11265" Text="Hello Achim, I'm happy to see you join this site :)  After reading what you've written it sounds like my best bet (for the context of the problem I'm currently investigating) to go with bootstrapping (or some randomization test).  You said you've seen such rare cases, if you can help me find what they have done I'd be glad to see it.  Otherwise I guess I'll try to write something like this myself.  Thanks again, Achim, for your help." CreationDate="2011-05-26T07:59:11.507" UserId="253" />
  <row Id="19439" PostId="11264" Text="what exactly do I enter for p?  I'm confused there.  Also, wouldn't I need to enter the # of users in this test somewhere?" CreationDate="2011-05-26T08:16:39.383" UserId="4759" />
  <row Id="19440" PostId="11253" Text="@Dave Just a little comment: it looks like the overall correlation might partly be explained by the fact that units belonging to group A all have lower scores whereas considering `valueB` alone doesn't help to separate the three groups." CreationDate="2011-05-26T08:34:59.300" UserId="930" />
  <row Id="19441" PostId="11243" Text="@Nick No, I have 4 subjects that were observed over time. Each time instant is an observation, which gives about 400 observations per subject." CreationDate="2011-05-26T09:22:14.593" UserId="3052" />
  <row Id="19442" PostId="11243" Text="@Josh: so how does this prevent you from doing ten-fold crossvalidation?" CreationDate="2011-05-26T09:30:14.387" UserId="4257" />
  <row Id="19443" PostId="11243" Text="@Nick Because I am trying to determine how well the framework can generalise to an unseen subject. Given only 4 subjects, how can I perform tenfold CV without including the test subject within the training set?" CreationDate="2011-05-26T09:45:50.370" UserId="3052" />
  <row Id="19444" PostId="11243" Text="This may be oversimplifying for your situation, but: say you have 100 obs per subject. Throw the 3 x 100 training observations together (perhaps add a variable indicating subject for repeated measurement). The result is a training dataset w 300 obs. Then use this (w 10-fold CV) to fit a model, and evaluate on your test data." CreationDate="2011-05-26T09:58:35.803" UserId="4257" />
  <row Id="19445" PostId="11270" Text="What kind of variable is &quot;weblog publishing software&quot;, &quot;weblog comment system&quot;? Are you asking whether they like the web publishing software or something else?" CreationDate="2011-05-26T10:59:30.950" UserId="183" />
  <row Id="19446" PostId="10988" Text="I have different experiences with NCA -- it usually converges quite qickly for me. Checkout &quot;dimensionality reduction via learning an invariant mapping&quot; by LeCun and &quot;Minimal Loss Hashing for Compact Binary Codes&quot; by Norouzi." CreationDate="2011-05-26T11:11:32.293" UserId="2860" />
  <row Id="19447" PostId="11254" Text="Hi Tal, I think that Achim's comments pretty much sum it up.  If I were wanting inferential statistics I would do formal parametric or semiparametric modeling." CreationDate="2011-05-26T12:40:08.893" UserId="4253" />
  <row Id="19448" PostId="11252" Text="Take a look at the literature about propensity score match, Though I thinks you are talk about something different." CreationDate="2011-05-26T13:13:51.520" UserId="3058" />
  <row Id="19449" PostId="11274" Text="An excellent answer!&#xA;&#xA;I would add since the data is ordinal there are some notes floating around with a Fortran program for extending CCA to ordinal data: http://ideas.repec.org/p/pra/mprapa/12796.html that may be more appropriate given the data." CreationDate="2011-05-26T13:37:47.420" UserId="4325" />
  <row Id="19450" PostId="11273" Text="Can you indicate where you encountered this term?" CreationDate="2011-05-26T13:42:20.623" UserId="4257" />
  <row Id="19451" PostId="11274" Text="@Chris Thanks for the link. Will try it on some data of mine." CreationDate="2011-05-26T13:45:17.437" UserId="930" />
  <row Id="19452" PostId="11273" Text="It may be particle physicists' slang. E.g. http://cmsdoc.cern.ch/documents/99/note99_029.pdf, http://root.cern.ch/phpBB3/viewtopic.php?t=7394. I'm using it myself in the sense stated above, but I'd like to know if that's correct and if there's more to it." CreationDate="2011-05-26T14:06:22.420" UserId="1512" />
  <row Id="19453" PostId="11229" Text="@Andy W, do you happen to have an example image of that method? It seems like I can only read the abstract of that article without a login. " CreationDate="2011-05-26T14:15:42.020" UserId="4739" />
  <row Id="19454" PostId="11270" Text="Jeromy, there must have dependent variable and independent variable. Perceived ease of use, interactivity, and trustworthiness are independent variables in my study; whereas Weblog publishing software, comment system, and blogroll and hyperlink are my dependent variables. So, if the user perceived the particular corporate's weblog publishing software is easy to use; its comment system is interactivity; and its weblog blogroll and hyperlink are trustworthy, therefore the corporate Weblog is effective. And my questions that I'm facing now is SPSS output, I donno how to interpret it. Need help~ " CreationDate="2011-05-26T14:24:52.997" UserId="4762" />
  <row Id="19455" PostId="11229" Score="1" Text="@matt b, I just found an ungated version of the [paper](http://www.mrc-bsu.cam.ac.uk/personal/chris/papers/denstrip.pdf). Also check out the R package [denstrip](http://cran.r-project.org/web/packages/denstrip/index.html). In retrospect though I think the suggestion may be overkill, you will only have slight gains over just plotting the quantiles of interest and it will be much more computationally expensive (hence reducing its utility as an exploratory tool)." CreationDate="2011-05-26T14:27:29.220" UserId="1036" />
  <row Id="19456" PostId="11229" Text="thank you for the link nonetheless!" CreationDate="2011-05-26T14:34:27.720" UserId="4739" />
  <row Id="19457" PostId="11265" Text="I would have to Google this myself. I haven't got any specific publication for this in mind. The idea would essentially be to draw samples from the &quot;smaller&quot; model, e.g., in a model-based fashion using a normal distribution within each leaf or so. And then fit the &quot;larger&quot; model on many such artificial data sets." CreationDate="2011-05-26T15:03:40.993" UserId="4760" />
  <row Id="19458" PostId="11265" Text="...and a further comment: Similar strategies are also used for computing so-called variable importance measures, especially in random forests (but also in single trees). I also wouldn't recommend these for formal inference but for peeking into the black box they are often quite useful. See http://www.biomedcentral.com/1471-2105/8/25 and http://www.biomedcentral.com/1471-2105/9/307 and the references therein for some pointers. These are also available in &quot;party&quot;." CreationDate="2011-05-26T15:07:59.403" UserId="4760" />
  <row Id="19460" PostId="11249" Score="2" Text="If you already know Python this task is trivial and there is no need to download R. There is the dbfpy package for reading dbf files directly, xlrd for reading Excel files, and numpy has the summary statistics." CreationDate="2011-05-26T16:02:30.013" UserId="1080" />
  <row Id="19461" PostId="11279" Score="2" Text="+1 for giving someone direction in a language they already know. Note that numpy has all of the functionality the OP has described needing." CreationDate="2011-05-26T16:30:23.813" UserId="1080" />
  <row Id="19462" PostId="11112" Text="@K-1 Very interesting update... Comparing the ACF or the restored signals vs the original might help to identify which method recovers the original auto-correlation structure. The large peak at lag19 is actually intriguing. I cannot say a word about that." CreationDate="2011-05-26T16:47:41.687" UserId="2902" />
  <row Id="19463" PostId="7197" Text="@Jim have you seen the heritage health prize? It sounds uncommonly like your problem, so if you go to www.kaggle.com you may be able to find some suggestions as to how to deal with this kind of problem" CreationDate="2011-05-26T16:53:16.520" UserId="656" />
  <row Id="19465" PostId="11251" Text="@mpiktas thanks, I made the appropriate edit. " CreationDate="2011-05-26T17:34:25.047" UserId="4329" />
  <row Id="19466" PostId="11284" Score="1" Text="@richie -thanks..I am not much concerned about missing value imputations..nor can I do many analysis and then average them..Basically as I said, I have 5 channels. 1 channel out of those have right values..for the rest 4 channels, I need to fill missing in those variables. My concern is , how to tell SAS that dont delete these variables, these are created deliberately by me as actually it doesnt make any business sense to have agent characteristics for those 4 channels..Hope you got my point..or any setting in which I can tell SAS that dont delete missing values." CreationDate="2011-05-26T17:36:42.343" UserId="1763" />
  <row Id="19468" PostId="11256" Text="@dominic That makes sense.  In reflecting on it, I was wondering how you could even formulate a definite hypothesis when you don't yet have a specific model for the data behavior?  What kind of hypothesis could that be?  I'm not trying to be pedantic or difficult; it just seems that this could be an important point.  It might constrain your options and require adjustments to the p-values." CreationDate="2011-05-26T17:54:24.210" UserId="919" />
  <row Id="19469" PostId="11254" Text="Thank you Frank, I hope to see you more here - your input is invaluable!" CreationDate="2011-05-26T18:37:04.193" UserId="253" />
  <row Id="19470" PostId="11265" Text="Thank you for all the references Achim, I will have to go through them in more detail and see what I can come up with.  Thanks again!" CreationDate="2011-05-26T18:45:02.667" UserId="253" />
  <row Id="19471" PostId="11272" Text="I didn't know about them, thanks Radek.  " CreationDate="2011-05-26T18:52:13.463" UserId="253" />
  <row Id="19472" PostId="11285" Score="2" Text="@whuber +1 for actually solving his problem, as opposed to answering about missing data." CreationDate="2011-05-26T19:04:37.970" UserId="656" />
  <row Id="19473" PostId="11289" Text="Have you tried truncating the random variables?" CreationDate="2011-05-26T19:33:59.530" UserId="3567" />
  <row Id="19474" PostId="11289" Text="I did not use, can you propose me a book or a paper to know how to do this." CreationDate="2011-05-26T19:46:42.513" UserId="4770" />
  <row Id="19475" PostId="11249" Score="1" Text="@Josh, xlrd works only with xls, Excel 2007 uses xlsx. You can convert but it might get tedious. And probably xlwt will be more needed here to write to Excel files, not to read. " CreationDate="2011-05-26T20:03:07.867" UserId="2116" />
  <row Id="19476" PostId="11289" Text="@Farzad, if you do not get answers here try math.SE or mathoverflow.net. Why do you have sum of *product* of normal variable, is there a special reason? Also can you elaborate more on what tightness of the bound means?" CreationDate="2011-05-26T20:06:13.840" UserId="2116" />
  <row Id="19477" PostId="11289" Text="Sounds like a compressed-sensing related problem. Look up R. Vershynin's notes on nonasymptotic random matrix theory, specifically the bounds on what he calls *subexponential* random variables. That'll get you started. If you need more pointers, let us know and I'll try to post some more info. " CreationDate="2011-05-26T20:14:50.400" UserId="2970" />
  <row Id="19478" PostId="11289" Text="There are at least a couple related questions and answers on this topic on math.SE (disclaimer: including one I participated in)." CreationDate="2011-05-26T20:16:54.453" UserId="2970" />
  <row Id="19479" PostId="11291" Text="Thanks! I suspected something like this, based on diagnostics for the (similar) `lm` fit. I'm trying to get my head around MCMC and Bayesian methods and sometimes think I am, then sometimes everything looks trivial, then... I'll pursue what you've said about the full posterior distribution to see where that leads." CreationDate="2011-05-26T20:36:37.720" UserId="1764" />
  <row Id="19480" PostId="11285" Score="1" Text="@richie Thanks.  The conversation initiated by your reply clued me in; so, as in many threads on this site, developing a solution is really a collaborative effort.  I look forward to the next replies which take our tentative first steps even further!" CreationDate="2011-05-26T20:51:32.927" UserId="919" />
  <row Id="19481" PostId="11287" Score="1" Text="It won't be too good since R^2 doesn't apply to logistic regression models. There are &quot;pseudo R^2's&quot; of various sorts but they are all hokey &amp; pander to the view that OLS is some sort of gold standard for multivariate testing. Best to use some form of likelihood ratio test that is based on agreement of predicted &amp; observed values. Good discussion of options at http://faculty.chass.ncsu.edu/garson/PA765/logistic.htm" CreationDate="2011-05-26T21:03:47.993" UserId="2347" />
  <row Id="19482" PostId="11253" Text="@Dave I think if you give us more content (i.e., what do the groups mean) you will get better advice on what to do." CreationDate="2011-05-26T21:09:52.077" UserId="442" />
  <row Id="19483" PostId="11259" Text="Nicely phrased question.  I'd be very interested to see the coefficients and p-values compared across the 2 models.  If they really are quite similar, then I suspect you were very exacting when judging the initial residuals to be too far from normality.  I don't see how the 2 models could function so differently, judging by residuals, yet so similarly judging by b's and p's." CreationDate="2011-05-26T21:20:33.490" UserId="2669" />
  <row Id="19484" PostId="11264" Text="You could use this to compute sigma, if you don't have any evidence that a particular value for p is better than any other a conservative estimate is just using .5 since that creates the maximum value." CreationDate="2011-05-26T21:41:42.890" UserId="4325" />
  <row Id="19485" PostId="11294" Score="1" Text="@Henrik (+1) As an alternative, we might consider a permutation test. Good that you indicated to report ESs." CreationDate="2011-05-26T21:46:54.090" UserId="930" />
  <row Id="19486" PostId="11289" Score="1" Text="The product $w_i v_i$ has as a 'normal product' distribution. I believe the the mean of this product is zero and the variance is $\sigma^4$ where $\sigma^2$ is the variance of $w_i$ and $v_i$. For $N$ largeish, you could use the central limit theorem to get approximate norality of $X$. If you can compute the skew of the normal product distribution, I believe you can apply the Berry-Esseen theorem to bound the rate of convergence of the CDF." CreationDate="2011-05-26T22:05:28.840" UserId="795" />
  <row Id="19487" PostId="11295" Text="You are right, because I did not think that what he wanted is increased number of articles, not the number of users. " CreationDate="2011-05-26T22:34:12.283" UserId="4738" />
  <row Id="19488" PostId="11256" Text="@whuber Not sure I'm following your thinking, but I'll try to be more explicit... We are interested in measuring an effect (change in Y associated with presence of, say, X1). As we were expecting, the effect of X1 is significant, as suggested by the first, somewhat bulky regression, and further asserted by the ordinal regression that followed (in both cases, we have p&lt;.0005). However, before ruling out linear regression and reporting the ordinal regression results, I wanted to hear some points of views here." CreationDate="2011-05-26T22:59:59.773" UserId="4754" />
  <row Id="19489" PostId="11291" Text="Glad to help. If you're in the market for a reference Gelman and Hill's book Data Analysis Using Regression and Multilevel/Hierarchical Models is a very accessible book to get started, with a lot of practical advice. And it doesn't cost your firstborn child either..." CreationDate="2011-05-26T23:10:11.257" UserId="26" />
  <row Id="19490" PostId="11260" Text="@Dmitrij Celov : Please see my previous comment to whuber to get a better idea of the purpose of all of this. For the transformation suggestions, did you mean after _Y - 9_ transformation? Otherwise I lose a whole bunch of cases, ending up with zeroes. The last option you suggest is quite advanced... Not sure I can do this properly for now." CreationDate="2011-05-26T23:19:44.833" UserId="4754" />
  <row Id="19491" PostId="11249" Text="@mpiktas - You are right; thank you for the correction! I should know better too. I have read Excel 2007 files with win32com, and then there are the openpyxl and pyxlsx packages. This blog shows a ton of stuff you can do to programmatically read/write Excel files from Python (e.g., programmatically writing pivot tables): http://www.pythonexcels.com/" CreationDate="2011-05-26T23:24:24.360" UserId="1080" />
  <row Id="19492" PostId="11289" Text="@shabbychef, Berry-Esseen has pretty slow convergence, since it's a uniform bound over the class of all distribution functions $F$." CreationDate="2011-05-26T23:30:55.133" UserId="2970" />
  <row Id="19493" PostId="11288" Text="@Charlie, you might want to rephrase your last comment, which is a little confusing. Continuous CDFs do not necessarily have a (proper) inverse. (Can you think of a counterexample?) So your proof requires additional conditions to hold. The standard way to get around this is to define the pseudoinverse $F^{\,\leftarrow}(y) = \inf\{x: F(x) \geq y\}$. The argument becomes more subtle, too." CreationDate="2011-05-26T23:36:35.417" UserId="2970" />
  <row Id="19494" PostId="11289" Text="**Related**: (see link in answer) http://math.stackexchange.com/questions/35114/how-to-obtain-tail-bounds-for-a-square-of-sub-gaussian-random-variable" CreationDate="2011-05-26T23:49:02.207" UserId="2970" />
  <row Id="19495" PostId="11259" Text="@Dmitrij I realized I was using the term &quot;Beta&quot; too loosely. I was referring to the unstandardized coefficients (edited). And for the transformation -- it is not the same DV as for the other post -- I used a square root on a linearly transformed (translated+inverted) score." CreationDate="2011-05-27T00:22:18.213" UserId="4754" />
  <row Id="19496" PostId="11259" Text="@rolando2 I put a table [here](http://pages.videotron.com/do999/talkStats/20110526-diff_raw_transf.jpg) that contains the parameters estimates for both the variables (standardized in both cases), their SE and p." CreationDate="2011-05-27T00:24:20.690" UserId="4754" />
  <row Id="19497" PostId="675" Text="Like Paolo said, probability theory is mainly concerned with the deductive part, statistics with the inductive part of modeling processes with uncertainty. Perhaps it's interesting to mention that if one thinks that the plausible inductive reasoning should be consistent, then actually the result is bayesian statistics, and more interesting this can be derived from probability theory. So bayesian statistics is basically applied probability theory so to speak." CreationDate="2011-05-27T02:08:41.997" UserId="4360" />
  <row Id="19498" PostId="11293" Text="+1, good point. I took it as an example, not the actual data, but reading again the OP certainly didn't say that." CreationDate="2011-05-27T03:34:42.387" UserId="26" />
  <row Id="19499" PostId="1005" Text="That URL didn't work for me, but this did:   http://ipsur.r-forge.r-project.org/book/index.html" CreationDate="2011-05-27T04:07:48.510" UserId="3919" />
  <row Id="19501" PostId="11291" Text="Good recommendation. I did get that book, but it's interesting how my learning works at this point: I'll go through a book, highlighting and understanding key points, but not getting the bigger picture so I can't really &quot;make it work&quot;. Then I move on to another book, another topic, and finally come back around and suddenly it's understandable. Sure enough, I'd highlighted all the right things but somehow it hadn't clicked. Perhaps this time around it will." CreationDate="2011-05-27T04:13:05.923" UserId="1764" />
  <row Id="19502" PostId="11034" Text="@user333, your question was about logistic regression, if you want answers about ordinal regression too, please add that to the question." CreationDate="2011-05-27T07:03:09.807" UserId="2116" />
  <row Id="19503" PostId="11289" Text="@shabbychef, Thanks for your comment, but as cardinal said the bound is not exponential and it is not useful for me." CreationDate="2011-05-27T07:16:45.180" UserId="4770" />
  <row Id="19504" PostId="3759" Text="Did you have any success in finding an implementation?" CreationDate="2011-05-27T05:38:19.287" UserId="4776" />
  <row Id="19505" PostId="8782" Text="@IrishStat: I have a problem of forecasting with intermittent demand. I've been asked to solve. I know there are several softwares specific for this time of forecasting but are not free. Can you please tell me if you know of any built-in functions in open source software (like R, for instance) to solve this problem? I've been searching but couldn't find it so far... Thanks!" CreationDate="2011-04-28T10:40:21.957" UserId="4366" />
  <row Id="19506" PostId="8782" Text="@assu: I am unaware of any free software that meets your needs." CreationDate="2011-04-28T14:04:03.700" UserId="3382" />
  <row Id="19507" PostId="8782" Text="@assu. The `croston()` function in the `forecast` package in R will implement Croston's method for forecasting intermittent demand data." CreationDate="2011-05-27T04:07:50.480" UserId="159" />
  <row Id="19508" PostId="11303" Text="@mpiktas (+1) Note that that the constraint is one of maximal covariance, not correlation, in case of PLS regression." CreationDate="2011-05-27T07:47:29.460" UserId="930" />
  <row Id="19509" PostId="11285" Text="@whuber -- thanks..Agree. My concern is that &quot;is placing a zero&quot; there equal to placing a .( missing). Forgive my ignorance but I understand that 0 may give that an entirely new meaning. Lets say if I have a variable &quot;agent's plcy rate&quot;..and I give it a number 0 in place of missing doesnt it alter the meaning. ?? Just a thought that came to my mind." CreationDate="2011-05-27T07:57:51.000" UserId="1763" />
  <row Id="19510" PostId="11289" Text="Alexander Barvinok also has some nice measure concentration notes at http://www.math.lsa.umich.edu/~barvinok/total710.pdf that may be of use. " CreationDate="2011-05-27T08:37:25.800" UserId="3248" />
  <row Id="19511" PostId="209" Text="@walktalky as @jeromy alludes to, in psychology at least, categorial variables such as reponses to questions are often presumed as being a representation of a underlying trait, so in that sense categorial data is sometimes taken as being continuous. " CreationDate="2011-05-27T09:31:23.693" UserId="656" />
  <row Id="19512" PostId="11304" Text="@Henrik (+1) for HARKing!" CreationDate="2011-05-27T10:26:10.917" UserId="307" />
  <row Id="19513" PostId="209" Text="@richiemorrisroe One could nitpick about the difference between the data and the putative trait, but of course you are right. Some very interesting further points were made in response to [this follow-up question](http://stats.stackexchange.com/questions/539/does-it-ever-make-sense-to-treat-categorical-data-as-continuous)." CreationDate="2011-05-27T11:00:35.827" UserId="174" />
  <row Id="19514" PostId="209" Text="@walkytalky thanks for the link, those answers are indeed very interesting. " CreationDate="2011-05-27T11:08:37.577" UserId="656" />
  <row Id="19516" PostId="11230" Text="@chl Dear chl, thank you so much for the very good references. I was trying to find the exact mathematical formula for each of these three types. Now I am using the formulas in [Karpinski](http://astro.temple.edu/~andykarp/Graduate_Statistics/Graduate_Statistics_files/Ch%2008%20Lecture%20Notes.pdf) page 11 and 12. When I use these formulas, my SSb and SSc becomes higher than they should be (in comparison to MATLAB), and since SSab, SSbc, SSac, and SSabc are calculated based on them, they even become negative sometimes. Is it possible that a high number of replicate has such an effect? " CreationDate="2011-05-27T11:34:10.610" UserId="2885" />
  <row Id="19517" PostId="11260" Text="Are your scores discrete values ($10, 11, \dots, 50$) or some real numbers? For continuous distributions it is possible to start with many zeros (try exponential or Pareto approximations). If you want just a simple logarithmic transformation, you may go for $X:=Y-9$, since for log-normal approximation the values are defined for $X&gt;0$ only. " CreationDate="2011-05-27T11:37:39.147" UserId="2645" />
  <row Id="19518" PostId="11275" Text="@Frank, (+1) it would be great to have some general links to the models you mentioned." CreationDate="2011-05-27T11:41:01.267" UserId="2645" />
  <row Id="19519" PostId="11289" Text="@Farzad, perhaps you can edit your question and give the explicit statement of the bound you have but are unsatisfied with. That would provide a good reference point. " CreationDate="2011-05-27T11:43:09.567" UserId="2970" />
  <row Id="19520" PostId="11309" Score="1" Text="If it's truly missing data, I would call it a (bad) form of single imputation. If the missing rows truly mean that the count was zero (i.e. there is no uncertainty involved), I do not know of a name. But then I wonder if it needs one, because it's only a step 'behind the scenes' (data cleansing)" CreationDate="2011-05-27T12:24:18.930" UserId="4257" />
  <row Id="19521" PostId="11309" Text="I am trying to extract data via SQL for a Histogram, so if there is no row it means 0.&#xA;Isn't this something like normalizing?" CreationDate="2011-05-27T12:29:21.203" UserId="4781" />
  <row Id="19522" PostId="11309" Text="IF you're trying to create a histogram, you shouldn't need the zeroes. What software are you using?" CreationDate="2011-05-27T12:41:29.877" UserId="4257" />
  <row Id="19523" PostId="10871" Text="@Tal, I recently came across some data on the subject that may be of interest, http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2009001 , and [here](http://nces.ed.gov/timss/educators.asp) is a page with example questions for 4th and 8th graders in the US." CreationDate="2011-05-27T12:41:33.900" UserId="1036" />
  <row Id="19524" PostId="11309" Text="well it is not missing, there just no observations for 2, 4, 5, so you could just right 0 for not observed (it is truly different from missing) data. However to make nice histograms, I think, you have to group the intervals to include at least 5 counts in it. So for example 1-3 4-6 would be one of possible devisions.  " CreationDate="2011-05-27T12:45:47.637" UserId="2645" />
  <row Id="19525" PostId="11309" Score="1" Text="@Tie-figher I tried to come up with a clearer title given that the data isn't &quot;missing&quot; in the statistical sense. feel free to modify." CreationDate="2011-05-27T12:45:53.863" UserId="183" />
  <row Id="19526" PostId="10871" Text="Also as a note I came across this info via a blog by [AnnMaria De Mars](http://www.thejuliagroup.com/blog/?p=1390) who has a few recent posts on the US and mathematics education which may be of interest as well." CreationDate="2011-05-27T12:48:07.450" UserId="1036" />
  <row Id="19527" PostId="11275" Text="Please see http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FHHandouts/asa96.pdf" CreationDate="2011-05-27T13:31:36.520" UserId="4253" />
  <row Id="19529" PostId="11285" Text="@ayush That's exactly the issue.  If you put missing values in, the software has to skip those records and you lose information.  Normally it would be bad to just stick in zeros mindlessly.  That's why you have to do a little math.  If you want to be really convinced, write down the likelihood equations for the model.  The Chain Rule shows that $X$ multiplies $\beta_1$, so by setting $X=0$ appropriately you eliminate $\beta_1$ from all terms where it doesn't belong.  That's why 0 works and that's why *only* 0 works in this case." CreationDate="2011-05-27T14:15:17.823" UserId="919" />
  <row Id="19530" PostId="11306" Text="Yeah, the way the model is formulated I think it makes sense to take $p(\alpha|\theta_1, \theta_2)$ to be uniform on $(\theta_1, \theta_2)$. The trouble with using the min/max of the $\theta$'s is that they're unbounded :) But, with a little effort we might be able to find a plausible range. In terms of getting the information back, it's actually there; if you look at a traceplot of the $\theta_i$'s they've only got a relatively small gap between them, which doesn't move very much. Part of the reason for the semiparameteric model is exactly that the $\alpha$ parameters are so strongly..." CreationDate="2011-05-27T14:39:50.523" UserId="26" />
  <row Id="19531" PostId="11306" Text="... correlated with the $\theta$'s that they're a pain to infer. But if I can get a rough idea of their distribution from a burn-in period I can (maybe) use that to develop a proposal for an independence metropolis scheme to update $\theta$ in a great big block - the order restrictions on $\theta$ induces a lot of autocorrelation in my chain and that should help break it." CreationDate="2011-05-27T14:43:14.700" UserId="26" />
  <row Id="19532" PostId="11300" Text="The approach you describe would compute $F-G$ where $F$ is the empirical cdf of $\theta_1$ and $G$ is the EDF of $\theta_2$.  So, if your software reports CDFs (or even histograms), you already have the data you need." CreationDate="2011-05-27T14:43:20.130" UserId="919" />
  <row Id="19533" PostId="11300" Text="This would work; the more I think about it the more I think that just taking the difference of the marginal posterior cdf's would be fine. It would give me a point estimate of the interval instead of a distribution over intervals (hopefully that makes sense; it looks weird having written it). I may be overthinking it." CreationDate="2011-05-27T14:54:48.413" UserId="26" />
  <row Id="19534" PostId="11300" Text="Also, it's my own software so I can make it report whatever I like - even, unfortunately, incorrect results :)" CreationDate="2011-05-27T14:55:40.380" UserId="26" />
  <row Id="19535" PostId="11309" Score="1" Text="I second @Nick Sabbe; if the missing rows are in fact observed zeros then all you're doing is recoding your data (and it shouldn't be necessary in order to calculate a histogram). You also observed 0 for the integers in 6-infinity but it would be inconvenient to include 0's those in your dataset ;)" CreationDate="2011-05-27T15:02:13.897" UserId="26" />
  <row Id="19536" PostId="11300" Text="@JMS There's nothing weird about a distribution over intervals.  If you like, think of it as a distribution over the subset of the plane $\{(x,y)|y \ge x\}$.  Also, there's a growing literature on interval-based procedures which considers exactly such things.  (One's data consist of intervals, but one is agnostic about where in each interval the true value lies.  The big application is in the analysis and reduction of errors in numerical computations.)" CreationDate="2011-05-27T15:02:30.760" UserId="919" />
  <row Id="19537" PostId="11300" Text="@whuber Oh I agree, and I've seen some work on interval estimation of late, but for some reason it just feels weird in my brain to write &quot;point estimate of an interval&quot; even though such a beast is perfectly well defined :) (well maybe it isn't - it's not a point! - but I think you get the idea...)" CreationDate="2011-05-27T15:07:31.707" UserId="26" />
  <row Id="19538" PostId="5110" Text="Wasn't this based on an older variation? " CreationDate="2011-05-27T15:26:52.677" UserId="253" />
  <row Id="19541" PostId="11311" Text="I see your point - this is an odd thing to do. I'm testing output from a simulation under a very large number of possible configurations, so I expect failures at 1% significance level. I thought of combining output from every configuration to get a single AD* number. I didn't do this because it wouldn't give me any hint as to which individual configuration was buggy, and because  buggy configurations have (so far) failed this test spectacularly. I can see that by using a 0.1% significance level, I will admit some bugs. Perhaps I need to look the distribution of the individual AD* results? " CreationDate="2011-05-27T15:59:17.350" UserId="4780" />
  <row Id="19542" PostId="11260" Text="@Dmitij - The scores represent discrete values.The transformations with _Y-9_ did not lead to anything close enough to normal, unfortunately." CreationDate="2011-05-27T16:04:06.580" UserId="4754" />
  <row Id="19543" PostId="11275" Text="Thank you Frank I will look into this. I don't feel so confident in interpreting the coefficients of ordinal regression yet, but I guess I'll have to go that way." CreationDate="2011-05-27T16:10:27.680" UserId="4754" />
  <row Id="19544" PostId="11256" Text="@whuber Actually, I should have said &quot;estimation&quot; as the main purpose. Sorry for the confusion. I deleted my previous comment saying it was hypothesis testing (not being possible to edit it)." CreationDate="2011-05-27T16:57:30.757" UserId="4754" />
  <row Id="19545" PostId="11316" Score="3" Text="+1 Re the last statement: I once made the mistake of thinking that too.  Mathematically you're correct but in practice it's nearly impossible to approximate a non-differentiable spike with normals (such as J- or U-shaped distributions): the normals are just too flat at their peaks to capture the density in the spikes.  You need way too many components.  Normals are good for approximating distributions whose pdfs are very smooth." CreationDate="2011-05-27T17:01:12.047" UserId="919" />
  <row Id="19546" PostId="11289" Text="@ cardinal, I put the bound I derived, you can check it now." CreationDate="2011-05-27T17:37:41.880" UserId="4770" />
  <row Id="19547" PostId="11259" Text="@dominic Thank you.  Something is very fishy.  These two models have virtually indistinguishable results, as you said.  So I'm thinking that the two response variables and the two sets of residuals must be very similar as well." CreationDate="2011-05-27T17:39:57.387" UserId="2669" />
  <row Id="19548" PostId="739" Text="I read a slightly different version of this quote by Fisher:  &quot;Hiring a physician after the data have been collected is like hiring a physician when the patient is in the morgue.  He may be able to tell you what went wrong, but he is unlikely to be able to fix it.&quot;" CreationDate="2011-05-27T18:10:45.057" UserId="686" />
  <row Id="19549" PostId="11273" Text="How about mailing the authors of the paper you found the term in?" CreationDate="2011-05-27T19:31:24.377" UserId="1352" />
  <row Id="19550" PostId="11311" Text="Re Greg's comment (+1): I wonder how much extreme critical values like .1% are impacted by different means and/or variances of the underlying normal distribution. After all, Matt mentions that he is in &quot;case 4&quot;." CreationDate="2011-05-27T19:55:36.600" UserId="1352" />
  <row Id="19551" PostId="11322" Score="1" Text="@bill_080 Aren't these just Pearson correlation coefficients as reported on the [tutorial](http://correlate.googlelabs.com/tutorial/)? (Section *Correlated Queries*)" CreationDate="2011-05-27T20:16:30.477" UserId="930" />
  <row Id="19552" PostId="11322" Score="1" Text="@chl: You're right.   I clicked on the FAQ's, but got sidetracked by the Whitepaper.   So, they're using correlation of a time series (not the change in the time series).    That's like calculating the correlation of a stock price, not returns.    That's a little surprising." CreationDate="2011-05-27T20:23:55.693" UserId="2775" />
  <row Id="19553" PostId="1406" Text="I thought this was the tagline for Mythbusters..." CreationDate="2011-05-27T20:31:14.303" UserId="26" />
  <row Id="19554" PostId="11316" Text="@whuber Agreed. I would not suggest using a normal-mixture approximation for any distribution in practice, I was just trying to give an extreme counter-example." CreationDate="2011-05-27T20:33:00.523" UserId="279" />
  <row Id="19555" PostId="6357" Text="Laplace never took measure theory from a statistics professor ;)" CreationDate="2011-05-27T20:34:49.887" UserId="26" />
  <row Id="19556" PostId="9595" Score="1" Text="@Cchien, Why are you testing out various lag effects? Why aren't you trying to IDENTIFY the model using the Cross Correlation Function? The Box-Jenkins process is to Identify, Estimate and Forecast." CreationDate="2011-05-27T16:43:18.247" UserId="3411" />
  <row Id="19557" PostId="11289" Text="Ah, I see, of course. Perhaps abuse Cauchy Schwarz, and look at $X$ as a dot product, and use the CDF of the Chi square distribution. The problem is that this throws away the cosine of the angle, (i.e. assumes it is 1), which may give a too weak bound." CreationDate="2011-05-27T21:29:57.063" UserId="795" />
  <row Id="19558" PostId="11304" Text="@Henrik and Bernd:  Ditto! Great acronym and I hope it catches on." CreationDate="2011-05-27T21:50:04.670" UserId="2669" />
  <row Id="19559" PostId="11322" Text="I also noticed that the data for the same query don't match on Google Trends and Correlate.   I sent Google an e-mail." CreationDate="2011-05-28T02:36:06.903" UserId="2775" />
  <row Id="19560" PostId="729" Text="God can make up data." CreationDate="2011-05-28T04:11:49.593" UserId="4337" />
  <row Id="19561" PostId="11112" Text="As the data is actually streamwise velocity component from a tidal estuary, I guess the peak in ACF might show the scale of strongest horizontal turbulence. But have no idea about the correlation function." CreationDate="2011-05-28T04:52:53.353" UserId="4286" />
  <row Id="19562" PostId="11077" Score="1" Text="@Jeromy (+1) I came across this interesting blog post: [Handling Non-Normal Data in SEM](http://rudyanto62.blogspot.com/2008/01/handling-non-normal-data-in-sem.html)." CreationDate="2011-05-28T10:12:03.350" UserId="930" />
  <row Id="19563" PostId="11273" Text="Well, you know, it's one of those terms everybody uses so very casually that you'd feel ashamed to ask what it is. But apparently, the question is legitimate :)" CreationDate="2011-05-28T11:02:42.617" UserId="1512" />
  <row Id="19564" PostId="11319" Text="in my data I have NO zeros. Maybe there is a recommended model with / without zero inflation and also one for low and one for high n..." CreationDate="2011-05-28T11:11:59.500" UserId="4679" />
  <row Id="19565" PostId="6357" Text="@JMS - measure theory not as good as complex analysis perhaps? Laplace was quite good at this I think.  Perhaps statistics from analysis perspective has more &quot;common sense&quot; about it than measure theory ;)." CreationDate="2011-05-28T11:52:31.017" UserId="2392" />
  <row Id="19566" PostId="823" Text="+1 for the 1st use of &quot;cheekily&quot; on this site." CreationDate="2011-05-28T13:09:34.320" UserId="2669" />
  <row Id="19567" PostId="990" Text="Also a favorite topic of Michael Shermer of Skeptic magazine.  &quot;That's not even wrong.&quot;" CreationDate="2011-05-28T13:21:27.583" UserId="2669" />
  <row Id="19568" PostId="729" Text="@Leo What data do you have to support that hypothesis? :)" CreationDate="2011-05-28T13:23:49.420" UserId="2392" />
  <row Id="19569" PostId="4318" Text="This makes me think of one of the most fundamental mathematical equations: $\text{crap}=\text{crap}$" CreationDate="2011-05-28T13:37:36.430" UserId="2392" />
  <row Id="19570" PostId="1317" Text="Similar to 80% of car drivers think they're &quot;above average&quot;." CreationDate="2011-05-28T13:42:50.423" UserId="2392" />
  <row Id="19571" PostId="788" Text="an enlightened customer said &quot;try proving anything without them...&quot;  :)" CreationDate="2011-05-28T13:50:06.450" UserId="2392" />
  <row Id="19573" PostId="11328" Score="1" Text="This question has a lot of overlap with your two other questions at http://stats.stackexchange.com/questions/11079/problem-with-anova-repeated-measures-error-model-is-singular and http://stats.stackexchange.com/questions/11113/wrong-results-using-anova-with-repeated-measures Please follow whuber's suggestions to improve them. Has your first question (`Error()` problem) been sufficiently answered? How is this question here different from your second one?" CreationDate="2011-05-28T14:06:46.293" UserId="1909" />
  <row Id="19574" PostId="11327" Score="5" Text="&#xA;I wouldn't say it's &quot;not sensible&quot; to do what you have done.  However, because you already know the factors which determine GPA - namely the grades in various courses, it may make more sense to model those factors rather than GPA.  Perhaps a &quot;better question&quot; might be, for example: *What factors are most strongly associated with high math scores?*." CreationDate="2011-05-28T14:12:15.627" UserId="2392" />
  <row Id="19575" PostId="11328" Text="There is also a long list of tutorials and functions for this purpose here:  http://www.r-statistics.com/2010/04/repeated-measures-anova-with-r-tutorials/" CreationDate="2011-05-28T14:17:19.797" UserId="253" />
  <row Id="19576" PostId="11260" Text="@dominic, indeed it doesn't, for log-normal approximation you make $\log{Y-9}$ transformation actually, where $\log$ stands for natural logarithm.  " CreationDate="2011-05-28T14:48:56.460" UserId="2645" />
  <row Id="19577" PostId="11328" Text="Dear caracal, it is true that there is overlap, but in my opinion the topic is different. Indeed this time I am focusing an an aspect that in those topics is not fully treated: how to measure statistical difference between males and females AT STIMULUS level, by means of ANOVA repeated measures. Is it possible or not? If yes how? I the t-test the only solution? These are the question I would like an answer to.  " CreationDate="2011-05-28T14:50:08.627" UserId="4701" />
  <row Id="19578" PostId="11328" Score="1" Text="You need to ask a more specific question than what you've posted here.  Its not clear whether or not your question is theoretical or about how to get R to run the numbers.  It would also help me if you write out the mathematical model you are trying to interpret, and the hypothesis you want to test in terms of that mathematical model. for e.g. $Y_{ijk}\sim N(\mu_{ijk},\sigma_{i}^{2})$ and you want to test $H_{0}:\mu_{ijk}=\mu_{ij}$ (i.e. that category $k$ is not significant).  Doing it this way makes it easier to know what means or sums of squares to calculate." CreationDate="2011-05-28T15:03:28.607" UserId="2392" />
  <row Id="19579" PostId="11311" Text="OK, testing simulation results is one of the few cases that testing for matches of an exact distribution makes sense. But rather than testing at such a low alpa, you might consider looking at all your test statitics in some form of plot to look for unusual points (clumps could mean something different as well)." CreationDate="2011-05-28T15:07:39.060" UserId="4505" />
  <row Id="19580" PostId="11328" Text="Hi thanks for your answer. Actually I need an help both theoretically and in R. But honestly I don´t understand where my question is not clear. Mathematically I need to test H0: mean_stimulus1_males = mean_stimulus1_females That is, for each stimulus I want to know if the differences between the evaluations of the males are significantly higher than the evaluations of the females. So far I only found the t-test as method to achieve this goal, since with ANOVA I was not able to get this information. What do you suggest? Thanks in advance!" CreationDate="2011-05-28T15:38:57.667" UserId="4701" />
  <row Id="19581" PostId="11314" Text="@chl: Thanks a lot. I was not getting decent articles on this topic from googling.Your references are more than handful." CreationDate="2011-05-28T16:52:15.183" UserId="4278" />
  <row Id="19582" PostId="11318" Score="1" Text="&quot;We often try to construct the model such that our error term is normally distributed&quot; - to be precise, I think you are referring to the residuals $y-X\hat\beta$. These are _estimates_ of the error terms in the same way that $X\hat\beta$ is an estimate of $\mathbb{E}(y)=X\beta$. We'd like the residuals to look normal because that's what we assumed about the error terms to begin with. We &quot;invent&quot; the error term by specifying a model, not fitting it." CreationDate="2011-05-28T17:01:49.360" UserId="26" />
  <row Id="19583" PostId="11326" Text="&quot;This seems strange because you will only observe y once and only once (y is the complete vector/matrix/etc. of responses). How can this be &quot;distributed&quot;? In my view it can only be distributed in some imaginary ensemble, nothing to do with your actual observed response. At the very least, any such presumption of the response &quot;being distributed&quot; is untestable&quot; I'm confused; are you saying we can't test $H_0: y\sim f_0$ vs $H_1: y\sim f_1$? " CreationDate="2011-05-28T17:07:48.580" UserId="26" />
  <row Id="19584" PostId="11326" Text="no, sorry, that can't be what you're saying. I'm still confused though. Maybe it's slightly imprecise, but the way I read it he's got $n$ samples of $y_i$ from $Y$ with fixed $x_i$, his model is $Y = X\beta + \epsilon$, and he's wondering what the assumed distribution of $\epsilon$ implies about the distribution of $Y|\beta, X$ *under his model*.  Here it would imply that it's normal; we can test that with our sample" CreationDate="2011-05-28T17:21:17.197" UserId="26" />
  <row Id="19585" PostId="11331" Score="4" Text="Have you looked at Fourier methods?" CreationDate="2011-05-28T17:30:47.120" UserId="2970" />
  <row Id="19586" PostId="11328" Text="Maybe instead of the t-test is more appropriate if I use another ANOVA withe repeated measures for each stimulus? Something like `table_gravel &lt;- subset(my_table, stimulus == &quot;gravel&quot;)&#xA;&#xA;aov_gravel = aov(response ~ sex*subexperiment + Error(subject/(subexperiment)), data=table_gravel)&#xA;summary(aov_gravel)&#xA;`" CreationDate="2011-05-28T18:11:00.090" UserId="4701" />
  <row Id="19587" PostId="11327" Score="3" Text="&#xA;A factor Analyis seems to be more sensible than a regression." CreationDate="2011-05-28T18:14:17.390" UserId="3058" />
  <row Id="19588" PostId="11318" Text="I agree with your precision, JMS. +1 and I'll adjust my answer." CreationDate="2011-05-28T19:02:23.827" UserId="3874" />
  <row Id="19590" PostId="11330" Text="_But what questions are you actually trying to answer?_ The first question was whether such a regression would violate an OLS assumption, like someone with more statistical experience suggested. Your answer confirms my hunch, namely that it does not. Secondly, we wanted to get an quick overview of which variables will likely be useful in the regressions that followed, which for example try to find out whether being good in quantitative topics tends to help get good grades in law and other ideas like this. " CreationDate="2011-05-28T19:32:19.227" UserId="4788" />
  <row Id="19591" PostId="11327" Text="@probabilitylogic You're right, I have the impression that using OLS in this case just leads to trivial assertions. Probably it would be a better idea to either use other techniques like @Manoel suggested, or just proceed to more interesting questions like the one you proposed (which is casually similar to one of those we had in mind). " CreationDate="2011-05-28T19:41:06.850" UserId="4788" />
  <row Id="19592" PostId="11329" Text="I'm confused. Did you mean to write $|Q_r(d)| = r$ instead of $= d$?" CreationDate="2011-05-28T21:18:05.680" UserId="795" />
  <row Id="19593" PostId="11326" Text="@JMS - I think I might delete that first paragraph.  I don't think it adds anything to my answer (besides confusion)." CreationDate="2011-05-28T23:33:22.197" UserId="2392" />
  <row Id="19594" PostId="11337" Text="Your method for the first part didn't use the normal approximation. If you re-do it using the normal approximation, it shouldn't be difficult to apply the same method to the second part." CreationDate="2011-05-28T23:57:36.033" UserId="3835" />
  <row Id="19595" PostId="11337" Text="@mark999 Yes I recognize this. I don't know how to use normal approximation." CreationDate="2011-05-29T00:01:12.033" UserId="4401" />
  <row Id="19596" PostId="11337" Score="2" Text="For the first part, if $X$ is binomial with parameters $n=100$ and $p = 1/6$, then $X$ is approximately normally distributed with mean $np$ and variance $np(1-p)$. You may want to use a &quot;continuity correction&quot;: find the probability that the normal random variable is between 14.5 and 20.5. I think I was incorrect when I said that the second part is a simple application of the same method." CreationDate="2011-05-29T00:11:48.910" UserId="3835" />
  <row Id="19597" PostId="11338" Text="Where did you get Var(X_i) = 35/12?" CreationDate="2011-05-29T00:18:13.597" UserId="4401" />
  <row Id="19598" PostId="11338" Score="1" Text="You may also want to use the continuity correction here which corresponds to finding the probability the sum is less than 295.5." CreationDate="2011-05-29T00:21:34.833" UserId="2310" />
  <row Id="19599" PostId="11338" Text="For the variance calculation I would suggest writing out the PMF and using the formula Var(X)=E(X^2)-E(X)^2." CreationDate="2011-05-29T00:23:36.613" UserId="2310" />
  <row Id="19600" PostId="11338" Text="When using `pnorm` in R, I tried `pnorm(300, 350, sqrt(35/12))` and got `1.017822e-188` as my answer. That doesn't seem right." CreationDate="2011-05-29T00:28:55.427" UserId="4401" />
  <row Id="19601" PostId="11329" Text="Yes, thanks. Edited." CreationDate="2011-05-29T01:03:36.887" UserId="3567" />
  <row Id="19602" PostId="11338" Score="1" Text="Your SD is not correct, forgot the n.  It should be sqrt(100*(35/12))." CreationDate="2011-05-29T01:46:02.360" UserId="2310" />
  <row Id="19604" PostId="11326" Text="one of my favorite things to add to my answers :)" CreationDate="2011-05-29T02:46:01.970" UserId="26" />
  <row Id="19605" PostId="11338" Text="@Glen I got `0.384849` which seems much more reasonable. Thanks." CreationDate="2011-05-29T02:47:12.783" UserId="4401" />
  <row Id="19606" PostId="11338" Text="With a mean of 350 and a SD of $\approx 17$, at 300 you're about three SDs away from the mean, so the answer should be in the ‰ range." CreationDate="2011-05-29T03:21:50.677" UserId="4360" />
  <row Id="19608" PostId="11338" Text="D'oh, I multiplied the SD by 100, not the variance! Ok let's try that again. I got `0.001707396`. That seems really small." CreationDate="2011-05-29T07:32:59.677" UserId="4401" />
  <row Id="19609" PostId="11338" Text="I don't know where you got $0.384849$ from, as the &quot;z-statistic&quot; is about $-2.9$, which is about a $0.1\text{%}$ chance of being less than this." CreationDate="2011-05-29T07:35:09.600" UserId="2392" />
  <row Id="19610" PostId="11338" Score="1" Text="@styfle - I was writing my comment as you wrote yours.  What you may not be aware of is the factorials tend to &quot;pile up&quot; very quickly.  In fact they often grow so fast $n!\approx O(n^{n})$, that you can often approximate a sum of &quot;choose&quot; functions by the largest one.  You could also convince yourself of this by running some simulations." CreationDate="2011-05-29T07:38:04.717" UserId="2392" />
  <row Id="19611" PostId="11336" Text="What &quot;link function&quot; did you use to fit the model?" CreationDate="2011-05-29T07:50:52.083" UserId="2392" />
  <row Id="19612" PostId="11336" Text="Hi, I used the following command: zelig(choice ~ Deviance + Votes, data = data, model = &quot;mlogit&quot;)" CreationDate="2011-05-29T07:57:00.697" UserId="2704" />
  <row Id="19613" PostId="11323" Text="I like all comments! And they all seem to be right. But I was just searching for the easiest answer :) What happens when you assume that the errer term is normal distributed. That this occurs now very often in reality gets clear from the other answers! Thanks a lot!" CreationDate="2011-05-29T07:57:13.303" UserId="4496" />
  <row Id="19614" PostId="11334" Text="I've updated the question, also could you please explain your last sentence a little bit? Thanks!" CreationDate="2011-05-29T11:34:24.093" UserId="3280" />
  <row Id="19615" PostId="11341" Text="@gd047 You seem to have 8 factors, not 4." CreationDate="2011-05-29T12:23:21.633" UserId="930" />
  <row Id="19616" PostId="11341" Text="@chl This is a 2^(8-4) design so p=4 and, as far as I know, the defining relation consists of the p generators and their 2^p - p - 1 generalized interactions. So there should be 2^p - 1 defining relations in total." CreationDate="2011-05-29T12:33:52.637" UserId="339" />
  <row Id="19617" PostId="11342" Text="I've seen FrF2. The point is that the design that produces is not the same with mine. For example it gives the alias AB=CE which is not true in my case. I was looking for a way to analyze a predefined design." CreationDate="2011-05-29T12:55:52.410" UserId="339" />
  <row Id="19618" PostId="11342" Text="@gd047 Yes, I understood it afterwards. Will think of it." CreationDate="2011-05-29T12:58:10.817" UserId="930" />
  <row Id="19619" PostId="11327" Text="@LCC - you better be careful about the phrase &quot;@probabilitylogic You're right&quot; - I could go and get a big head from it :).  @manoel's suggestion of factor analysis does seem like a good way to go, pick up the &quot;dimensions of school ability&quot; or something like it." CreationDate="2011-05-29T12:59:44.047" UserId="2392" />
  <row Id="19620" PostId="11339" Score="2" Text="I would also add Larry Bretthorst's book on spectral analysis to your reference. can be downloaded for free [here](http://bayes.wustl.edu/glb/book.pdf)." CreationDate="2011-05-29T13:03:51.800" UserId="2392" />
  <row Id="19621" PostId="11154" Text="@Bernd, I haven't had the time to go through all the resources but they seem to be what I was looking for. Thanks you again!" CreationDate="2011-05-29T13:42:03.277" UserId="4717" />
  <row Id="19622" PostId="11344" Text="@Frank - why is it &quot;not correct&quot; to group BMI?  this seems perfectly reasonable, so long as the results are appropriately interpreted.  You could well be testing, for example, whether being &quot;underweight&quot; &quot;healthy weight&quot; &quot;overweight&quot; and &quot;obese&quot; are associated with smoking, where these terms are defined by the ranges of BMI.  I see no &quot;wrong&quot; here." CreationDate="2011-05-29T13:42:26.057" UserId="2392" />
  <row Id="19623" PostId="11344" Text="I believe that the OP is working with a common instructional data set and may not have the raw BMI. While it's generally not ideal to discretize continuous regressors it's not &quot;incorrect&quot;. It can even be helpful to resort to this when we suspect the measurements are noisy and there's no other recourse. Indeed, the real hypothesis we'd want to test is whether obesity is related to smoking; BMI is just one way to measure obesity (and has its problems from what I understand)." CreationDate="2011-05-29T15:22:29.473" UserId="26" />
  <row Id="19624" PostId="11324" Text="Thanks @Whuber, with your GIS experience, what do you recommend to do?" CreationDate="2011-05-29T15:57:18.153" UserId="59" />
  <row Id="19625" PostId="1443" Text="@thylacoleo - your comment about *The &quot;best&quot; model will be choisen by the ML practitioner...* applies equally well to mainstream statistics as well.  For in most model selection procedures, one simply conditions on the final model as if no search of the model space had been done (given that model averaging is fairly new).  So I don't think you can use that as a &quot;club&quot; to beat the ML practitioner with, so to speak." CreationDate="2011-05-29T16:03:49.313" UserId="2392" />
  <row Id="19626" PostId="534" Text="Interesting counter-example from Jaynes: We have the deductive relationship &quot;no clouds&quot; implies &quot;no rain&quot;; yet who would believe that &quot;no clouds&quot; is the physical cause of &quot;no rain&quot;?" CreationDate="2011-05-29T16:26:55.467" UserId="2392" />
  <row Id="19627" PostId="11324" Text="@das It depends on what you need the data for and how accurate you need them to be.  If you have related questions that will primarily be of interest to GIS people, why not post them on GIS.SE?  You can reference this thread to provide context, make progress, and avoid repeating yourself." CreationDate="2011-05-29T17:25:23.417" UserId="919" />
  <row Id="19628" PostId="11329" Text="It is not obvious that $FDR[Q_r]$ is monotone. It might be for specific choices of ranking procedures, but it does not follow from the set inclusions alone. Thus for the value of $r$ you are asking about there is no general guarantee that FDR is smaller than $q_{\mathit{crit}}$ for smaller values of $r$. Are you really interested in this theoretical quantity?" CreationDate="2011-05-29T17:50:02.993" UserId="4376" />
  <row Id="19629" PostId="11349" Text="Doesn't help, unfortunately. Having isolated different parts of the code, I'm pretty sure the issue is that it's not reading the mean vector as I want it to. " CreationDate="2011-05-29T18:54:23.757" UserId="996" />
  <row Id="19630" PostId="11344" Text="Even when measurements are noisy, analyzing variables as continuous is superior.  Categorizing BMI creates more problems than different choices of analysis can fix.  In fact the estimates upon categorization no longer have a scientific interpretation.  A scientific quantity is one having meaning outside the current experiment.  You'll find that group estimates (e.g., log odds that Y=1 for high vs low intervals of X) are functions of the entire set of observed BMIs.  For example, if you were to add more extremely high or extremely low BMIs to the sample, the &quot;effects&quot; would get stronger." CreationDate="2011-05-29T19:20:41.773" UserId="4253" />
  <row Id="19631" PostId="11349" Score="1" Text="Darn. Googling the syntax just now turned up BUGS code for a VAR model [here](http://www.wiley.com/legacy/wileychi/raymer/supp/Chapter12_appendix.pdf); it's purported to work :) maybe a clue in there..." CreationDate="2011-05-29T19:33:25.973" UserId="26" />
  <row Id="19632" PostId="6723" Text="@BioGeek, I'd suggest to give the &quot;bestglm&quot; package a try. I uses leaps internally but gives you the possibility to use AIC, BIC, ... or even cross-validation as a basis for model selection." CreationDate="2011-05-29T20:43:16.100" UserId="2020" />
  <row Id="19633" PostId="11351" Text="@Aniko gives a [nice answer](http://stats.stackexchange.com/questions/11315/how-does-the-distribution-of-the-error-term-affect-the-distribution-of-the-respon/11316#11316) in response to your previous question." CreationDate="2011-05-29T21:17:11.160" UserId="919" />
  <row Id="19634" PostId="8637" Text="+1 for Everitt &amp; Dunn, we used it for a joint undergrad/grad class and it was quite good, nice and direct." CreationDate="2011-05-29T21:58:03.240" UserId="26" />
  <row Id="19635" PostId="11340" Text="I'm not afraid of for loops, it's everything else that throws me off :). What is this continuity correction everyone is talking about? According to Wikipedia, you add 1/2, not subtract it. I also don't understand the purpose of your code. You made a matrix that is 101 columns wide by 606 rows?" CreationDate="2011-05-29T22:41:25.687" UserId="4401" />
  <row Id="19636" PostId="11344" Text="For those who have installed R and RStudio, an interactive demonstration may be found at http://biostat.mc.vanderbilt.edu/BioMod - see the green NEW marking.  You have to load the script into RStudio and also install the Hmisc package." CreationDate="2011-05-29T22:45:44.577" UserId="4253" />
  <row Id="19637" PostId="11340" Text="In the normal approximation you are treating the score as a continuous variable.  So it might perhaps be $299.890123$ or $299.235711$. In the real problem you are restricted to integers and want less than $300$ or, equivalently, less than or equal to $299$.  So it is sensible to cut the normal approximation between $299$ and $300$, say at $299.5$.     " CreationDate="2011-05-29T23:45:04.127" UserId="2958" />
  <row Id="19638" PostId="11340" Text="The code says `## p[j,i] is probability of exactly (j+sides) after (i+1) throws` as a comment.  This is for two reasons: R starts counting at $1$ rather than $0$; and I wanted to sum the earlier six values before the latest roll without having to fuss about references which were not there.  " CreationDate="2011-05-29T23:49:28.447" UserId="2958" />
  <row Id="19639" PostId="11335" Text="Thanks, that’s an interesting point and I will keep that in mind. However, in this specific case, where the response variable (GPA) is just a linear combination of the regressors (marks), my logic intuition says that the coefficients should assume values similar to the weights used for calculating GPA, so they might not be useful but they shouldn’t be arbitrary… Am I right? " CreationDate="2011-05-30T00:01:12.107" UserId="4788" />
  <row Id="19640" PostId="11324" Text="I'm not sure GIS is the place, as really i'm trying to disaggregate census data. MY zones are already identified, although I might ask there what is the proper way to divide census zones so they're relatively the same size" CreationDate="2011-05-30T01:49:31.090" UserId="59" />
  <row Id="19641" PostId="11340" Text="Oh yeah, R is not 0 indexed. It somehow makes it more complicated starting at 1. I guess I should run this code and try it out." CreationDate="2011-05-30T04:09:17.030" UserId="4401" />
  <row Id="19642" PostId="11335" Score="1" Text="Well, you're using some courses' grades, not all of them.  And I wouldn't want you to underestimate the power of partial correlation to produce very different results than you would see in zero-order correlation.  You might even get a negative coefficient for one or two course grades." CreationDate="2011-05-30T04:17:29.220" UserId="2669" />
  <row Id="19643" PostId="11260" Text="@Dmitrij, :) Yes, I know. What I meant was that the log or ln of $Y-9$ did not approximate anything close to gaussian. Anyhow, it seems, after further reading, that for effect estimation I can trust my raw coefficients." CreationDate="2011-05-30T05:20:40.953" UserId="4754" />
  <row Id="19644" PostId="11259" Text="@rolando2 It's close, true. As I said earlier, the transformation was a square root (preceded by linear transformations to put the tail at the right end rather than the left.)" CreationDate="2011-05-30T05:25:27.280" UserId="4754" />
  <row Id="19645" PostId="11130" Text="yes, you got it." CreationDate="2011-05-30T06:16:23.527" UserId="4617" />
  <row Id="19646" PostId="11348" Text="What do you mean &quot;Ideal or OC&quot;?  Constructing the issue space is, of course, the foundation and that is precisely where it is obvious to the most casual observer that professional pollsters like Zogby are quite simply out to lunch in their blind polls.  The tournament style of questionnaire submission deals with that by not allowing any one viewpoint's biases to determine the range of issues.  Concerns about relevance are dealt with by virtue of the fact that an irrelevant questionnaire will provide its author no information with which to predict the other questions.  At most it can tie at 0." CreationDate="2011-05-30T06:21:31.203" UserId="4753" />
  <row Id="19647" PostId="11355" Text="yeah..so almost marrying the answer by mpiktas and mine..you can use factor analysis with method = prin. This would request to use the principal axis method." CreationDate="2011-05-30T07:20:45.140" UserId="1763" />
  <row Id="19648" PostId="11339" Text="@probabilityislogic: Thank you, I have used that book in paper, good to know that it is freely available in electronic form." CreationDate="2011-05-30T07:25:58.847" UserId="892" />
  <row Id="19649" PostId="10157" Text="plm as far as I know does not have functions for fitting discrete data models. `lme4` is a good suggestion, but should be used with care, since it assumes that all the independent variables are exogenous, which is not always the case for panel data in econometrics." CreationDate="2011-05-30T07:41:24.313" UserId="2116" />
  <row Id="19650" PostId="11342" Text="@gd047 Well, now that I've read the question, I can say that I didn't find my way with this in R." CreationDate="2011-05-30T07:47:06.983" UserId="930" />
  <row Id="19651" PostId="11357" Text="@Zia, it is possible to migrate the questions from one SE site to another. Flag the moderators, do not cross-post." CreationDate="2011-05-30T08:22:10.643" UserId="2116" />
  <row Id="19652" PostId="11357" Text="@Jeromy, we should have posted the @chl edits from stackoverlow, instead of playing a little edit war here :) " CreationDate="2011-05-30T08:24:54.873" UserId="2116" />
  <row Id="19653" PostId="11357" Text="ok sorry i didnot know how to do that " CreationDate="2011-05-30T08:30:23.867" UserId="4802" />
  <row Id="19654" PostId="11357" Score="1" Text="@Zia, no problems, usually the questions that should be migrated are migrated sooner or later, since somebody else flags the moderator. However it is always polite to wait for the answers in the first site you posted the question, before asking anywhere else, and if you do it is always helpful to indicate why the answers in the first site were not satisfactory." CreationDate="2011-05-30T08:34:00.727" UserId="2116" />
  <row Id="19655" PostId="11360" Text="@Mike: Are you aware that you can cast a &quot;close&quot; vote with the suggestion of migrating this to stats.SE?" CreationDate="2011-05-29T07:10:01.020" />
  <row Id="19657" PostId="11357" Text="@mpiktas yep. sorry about that. Feel free to change back. I'd just about pressed the button after incorporating the image, and links and so on, and wanted to apply the changes, when I was told it wouldn't apply." CreationDate="2011-05-30T09:06:37.293" UserId="183" />
  <row Id="19658" PostId="11349" Text="Thank you! Googling for working BUGS code is always difficult, due to the unfortunate name of the language. Statisticians make this mistake a lot, I mean, R?!" CreationDate="2011-05-30T09:27:51.283" UserId="996" />
  <row Id="19659" PostId="11343" Text="Let me try to rephrase it like this: I have a linear predictor for each category, where the betas are results of point estimations (when I use the mean values from their distributions or use any other maximization technique) How do I use probability distributions for betas rather than their point estimates in linear predictors? Or is there a way to use them at all?" CreationDate="2011-05-30T09:36:39.773" UserId="3280" />
  <row Id="19660" PostId="11348" Text="Constructing &quot;Issue-space&quot; is a well solved problem in poly-sci, it's usually done via fancy variations of PCA and/or item response analysis. The field is generally called &quot;Roll call analysis&quot;, because it's primarily applied to legislator's voting records. But it works with polls too! Provided you transform the questions. &#xA;&#xA;&quot;Ideal&quot; and &quot;OC&quot; are very easy to use R packages that will go ahead and estimate it for you. " CreationDate="2011-05-30T09:37:08.870" UserId="996" />
  <row Id="19661" PostId="11348" Text="Generally though, I'm not sure what you mean about providing no information. I've worked with survey releases, primarily from Pew and PPP. All the questions are generally extremely correlated with each other. " CreationDate="2011-05-30T09:40:14.513" UserId="996" />
  <row Id="19662" PostId="11341" Text="@gd047 You should rather put your edit as an answer (so that we can vote it) and you can even accept it, IMO." CreationDate="2011-05-30T11:50:24.263" UserId="930" />
  <row Id="19663" PostId="11228" Text="I am not sure about that. I guess that won't be possible since it is randomness which can't be predicted." CreationDate="2011-05-30T13:06:50.167" UserId="4445" />
  <row Id="19664" PostId="11364" Text="Title says &quot;lower bound&quot; and body says &quot;upper bound&quot;. Could you clarify?" CreationDate="2011-05-30T13:44:06.323" UserId="2970" />
  <row Id="19665" PostId="11341" Text="@chl It would be very disappointing if this workaround was the best we could do for such a simple question. In Rcmdr DOE plugin there's an &quot;Inspect Design&quot; submenu under &quot;Design&quot;. Seems relevant, but I have no idea how it works." CreationDate="2011-05-30T13:44:14.210" UserId="339" />
  <row Id="19666" PostId="11367" Text="@Henry, thanks for your answer, but what you showed is an upper bound. I am looking for a lower bound." CreationDate="2011-05-30T14:31:23.940" UserId="4770" />
  <row Id="19667" PostId="11367" Text="@Farzad: It says &quot;a polynomial upper bound&quot; in your question. " CreationDate="2011-05-30T14:41:05.633" UserId="2958" />
  <row Id="19668" PostId="11357" Text="@Zia: What do you mean by &quot;variance&quot; here?" CreationDate="2011-05-30T14:47:31.933" UserId="2958" />
  <row Id="19669" PostId="11367" Text="@Henry, I am so sorry, I made some mistakes. I just corrected them." CreationDate="2011-05-30T14:52:27.283" UserId="4770" />
  <row Id="19670" PostId="11341" Text="@gd047 Yes, I see it too (and we can see all Type III/IV/V resolution design), but the &quot;Inspect Design&quot; seems just a collection of wrapper functions to summarize the design structure (`desnum`, `design.info`, etc.). Rcmdr is becoming very nice, I should try it at some point." CreationDate="2011-05-30T14:54:50.993" UserId="930" />
  <row Id="19671" PostId="11370" Text="@Gardiner I tweaked the question title to make it a little more specific on the assumption that &quot;order&quot; is a between subjects factor. Feel free to change back if I've misconstrued?" CreationDate="2011-05-30T15:08:13.723" UserId="183" />
  <row Id="19672" PostId="11370" Text="If `order` is a between-subjects factor, it should be constant for each subject - it could have levels `VisualFirst` and `AuditoryFirst`. Your call to `aov()` looks odd, did you mean `aov(recalled_items ~ task*order + Error(subject/task))`?" CreationDate="2011-05-30T15:18:35.160" UserId="1909" />
  <row Id="19673" PostId="11370" Score="1" Text="If your data represent number of items recalled of some total number of items presented, it would be better to represent your data in its raw binary format, labelling each item as remembered or not remembered and conducting a mixed effects model while specifying binomially distributed error." CreationDate="2011-05-30T15:44:41.307" UserId="364" />
  <row Id="19674" PostId="11344" Text="&quot;Even when measurements are noisy, analyzing variables as continuous is superior&quot; This is just incorrect (the generality of it, that is - usually it's true). Imagine you have a continuous covariate where the error in its measurement increases with its magnitude, for example. Of course the best thing to do is model the error, or get better measurements, etc. But to say that it's incorrect is simply too strong a statement to make." CreationDate="2011-05-30T15:47:53.333" UserId="26" />
  <row Id="19675" PostId="11344" Text="Also: &quot;For example, if you were to add more extremely high or extremely low BMIs to the sample, the &quot;effects&quot; would get stronger.&quot; That's a problem with your sample, not the method. Or I guess you could cast it as a problem with the method in that it might require larger samples. And I still don't understand why categories based on the BMI are all of a sudden meaningless and invalid - it's exactly the same quantity, just measured at a different resolution. Whether those categories are meaningful is a separate question entirely." CreationDate="2011-05-30T15:51:17.423" UserId="26" />
  <row Id="19676" PostId="11372" Score="1" Text="I don't speak (German?) but the error looks like it's due to the tetrachoric matrix being singular (non-invertible). Even with a good sized sample some estimates of polychoric correlation matrices can fail to be proper correlation matrices. My hunch is that fa() is using principal components and factanal() is doing maxmimum likelihood FA, but that's not obvious to me from the documentation for fa()." CreationDate="2011-05-30T15:59:07.427" UserId="26" />
  <row Id="19677" PostId="11372" Text="@cada How many items/subjects do you have? And what method is implemented in MixFactor?" CreationDate="2011-05-30T15:59:51.293" UserId="930" />
  <row Id="19678" PostId="11373" Text="Hey Thanks! I know the boxcox funtion, but I was wondering for what practical reasons the sqrt transformation makes sense! Thank you!" CreationDate="2011-05-30T16:47:36.140" UserId="4496" />
  <row Id="19679" PostId="11374" Text="@gazialankus The p-value is often less interesting than effect size or MSB/MSE, notwithstanding the fact that it is associated to a null hypothesis of the form $H_0:\;\mu_0=\mu_1=\dots=\mu_k$ (vs. $H_1:\;\exists\, i,j\mid \mu_i\neq\mu_j$) which makes the global test uninteresting if you don't look at the between-group differences. " CreationDate="2011-05-30T17:02:30.393" UserId="930" />
  <row Id="19680" PostId="11252" Text="It seems you are asking two things, am I right?  1) convert probabilities measured on a 0-to-1 scale to ones measured on your other scale that includes various integers, perhaps -30 to +30 or something like that.  2) You want to convert your logistic coefficients to linear weights that will work with the alternate system (something I doubt can be done well, btw, because in logistic regression what is linear is the relationship between the predictor and the logit of the outcome)." CreationDate="2011-05-30T17:08:00.297" UserId="2669" />
  <row Id="19681" PostId="11324" Text="@das What is the motivation for making them the same size?  And do you mean same *area* or same *population*?" CreationDate="2011-05-30T17:11:30.993" UserId="919" />
  <row Id="19682" PostId="11344" Text="What makes categorization bad when there is measurement error is the high likelihood of classifying to the wrong category, which can be a 100% error for the binary case.  I couldn't agree less with the second comment.  This is not a problem with the sample; it is a problem of incomplete conditioning (on X &gt; c rather than X = x)." CreationDate="2011-05-30T17:17:48.273" UserId="4253" />
  <row Id="19683" PostId="11364" Score="3" Text="@Farzad You wisely removed &quot;polynomial&quot; in the edit because that is too limiting (the best lower bound is the constant function $0$).  However, without some kind of restriction the question is trivial, because $g(x) = f(x)$ is the best solution.  You need to clarify two things, then: (i) what restrictions must $g$ satisfy and (ii) why this question has statistical interest (and should not be migrated to math.se)." CreationDate="2011-05-30T17:21:08.440" UserId="919" />
  <row Id="19684" PostId="11324" Text="@whuber The same area size. The end goal is to disaggregate the data as much as possible sort of like going from object to atoms" CreationDate="2011-05-30T17:24:07.267" UserId="59" />
  <row Id="19685" PostId="11324" Text="@dass It's unclear then why area is of interest.  The &quot;atoms&quot; in population data are individual people, households, and housing units.  They are rarely distributed equally by area." CreationDate="2011-05-30T17:26:02.770" UserId="919" />
  <row Id="19686" PostId="11324" Text="@whuber, true, but I don't have any of that data, and I'm trying to figure out a way to disaggregate it. I have census data, which roughly the zones are 2,500 -&gt; 4,000 people per zone, I would like to disaggregate it up to 25 to 40 people per zone" CreationDate="2011-05-30T17:29:55.620" UserId="59" />
  <row Id="19687" PostId="11324" Text="@dass One approach is to model the tract-only data in terms of the data also available at the block level.  This sounds like [a question I asked here last year](http://stats.stackexchange.com/questions/4445/model-for-population-density-estimation)." CreationDate="2011-05-30T17:33:36.880" UserId="919" />
  <row Id="19688" PostId="11324" Text="@whuber, seems interesting especially the use of raster objects. Did you get a paper out of it, any recommendations or pitfalls to avoid?" CreationDate="2011-05-30T17:41:26.653" UserId="59" />
  <row Id="19689" PostId="11344" Text="Let me add that if you want to analyze BMI (which may not even be the right summary of ht and wt for the problem at  hand) then by all means analyze BMI.  But analysis of categorized BMI is not an analysis of BMI.  [As an aside people don't even agree on the category boundaries.]  One could say that analyzing BMI categories is analyzing a crude approximation of BMI when the full-information variable was available for analysis.  The result is residual confounding, badness of fit (predictions will be inaccurate when BMI is not near the middle of an interval), and artifactual discontinuities." CreationDate="2011-05-30T17:42:01.377" UserId="4253" />
  <row Id="19690" PostId="11372" Text="@cada why are you running the factor analysis? If you are aiming to estimate participant ability, IRT may be a better approach. on the matter of fa versus factanal, fa uses minimum residual by default, while factanal uses an ML method." CreationDate="2011-05-30T17:43:27.920" UserId="656" />
  <row Id="19691" PostId="11373" Text="if the variance of the errors is linearly related to the level of the series one takes a logarithmic transformation. If the standard deviation is linearly related to the level of the series one takes a square root transformation. The selection has nothing to do with the size of the residuals as it relates to the level of y and all to do with the coupling/de-coupling the first and second moment." CreationDate="2011-05-30T17:58:02.730" UserId="3382" />
  <row Id="19692" PostId="11348" Text="The PCA (variations) must be given input.  Its input bias I'm concerned with." CreationDate="2011-05-30T18:04:04.643" UserId="4753" />
  <row Id="19693" PostId="11348" Text="By &quot;no information&quot; I'm referring to the &quot;relevance&quot; problem pointed out by Charles Y. Zheng below.  Asking questions like &quot;What is the third letter of your last name?&quot;  &quot;What is the 4th digit of your phone number?&quot; etc.  Likewise, if the questionnaire is relevant and optimal for a given population, there should be no most-likely answer to any question with or without knowledge of the irrelevant questionnaire." CreationDate="2011-05-30T18:08:08.570" UserId="4753" />
  <row Id="19695" PostId="11368" Text="I wonder if adding a log-barrier for each eigenvalue of $\Sigma$ would work: something like $-\nu (\log \lambda_1 + \cdots + \log \lambda_K)$" CreationDate="2011-05-30T18:13:49.943" UserId="3567" />
  <row Id="19696" PostId="11372" Text="@richiemorrisroe Thanks for the clarification" CreationDate="2011-05-30T18:52:27.263" UserId="26" />
  <row Id="19697" PostId="11344" Text="&quot;What makes categorization bad when there is measurement error is the high likelihood of classifying to the wrong category, which can be a 100% error for the binary case.&quot; Usually, sure. But think about the example I gave where measurement error depends on the magnitude of $X$; you could imagine that discretizing by sign(X) would give you conservative inferences in that superfluous high-leverage points are &quot;damped&quot;. It might be the best of two bad options. I don't mean to disagree with the thrust of your argument, just the generalization to all possible cases - I'm willing to accept that..." CreationDate="2011-05-30T19:02:39.450" UserId="26" />
  <row Id="19698" PostId="11374" Text="@chl In the situation of a two-sample t-test, it is often recommended that a confidence interval for the difference in means be given, regardless of whether the test is significant. If the ANOVA F-test is non-significant, would it be a good idea to give the Tukey HSD confidence intervals for differences in means?" CreationDate="2011-05-30T19:11:21.560" UserId="3835" />
  <row Id="19699" PostId="11375" Text="When you say project do you mean this is for school? If so you should change one of your tags to homework (probably the R one) Do you know that the adjusted R-squared should be something other than the value you're reporting (e.g. you're replicating a study which had a very different result)? Did you test the significance of the model itself? You may have done nothing wrong you may just have a model that is not significant given the data you used. Please provide more details." CreationDate="2011-05-30T19:12:42.493" UserId="4325" />
  <row Id="19700" PostId="11344" Text="...discretizing BMI is a bad idea, since you're clearly more familiar with the subject matter than I am. Your point about incomplete conditioning is a good one - offhand I think I had your point backwards in my head. " CreationDate="2011-05-30T19:22:05.573" UserId="26" />
  <row Id="19701" PostId="11344" Text="&quot;But analysis of categorized BMI is not an analysis of BMI.&quot; This, I think, we all agree on (my earlier sloppy comment aside). Maybe I've misinterpreted what you have been saying." CreationDate="2011-05-30T19:26:31.067" UserId="26" />
  <row Id="19702" PostId="11343" Text="@user3280 - the procedure I outlined does exactly that - the summation over the Gibbs samples (the b's) is an approximation to integrating over that parameter space.  This is using the probability distribution of the betas to get predictions.  But remember that the ultimate goal is prediction here, so you shouldn't really care about the betas per se, only about the predictions.  the betas are a secondary concern." CreationDate="2011-05-30T19:30:41.207" UserId="2392" />
  <row Id="19703" PostId="11344" Text="Thanks for the good discussion.  Yes, if measurement error is not independent of X or Y, I may be in trouble.  But categorization is very sensitive to the cutpoint in that case also.  One other point: for a given number of parameters (regression degrees of freedom; no. of dummy variables in the categorical case), smooth regression fits explain a higher proportion of variance in Y than piecewise flat relationships (categorizing X).  I think the interactive simulation I posted above may help explore some of the issues (although it assumes completely random measurement error)." CreationDate="2011-05-30T19:33:07.860" UserId="4253" />
  <row Id="19704" PostId="11374" Text="@mark999 Right. However, showing the bootstrap or asymptotic precision of an estimate is different from a single p-value in a NHST context, whatever the result of the test is, IMO. Moreover, if the overall ANOVA test is not significant, it is usually not recommended to go further with post-hoc (especially Tukey HSD which are framed in an ANOVA context). I was just suggesting that the ANOVA F-test says nothing about where the difference really lie, hence the need to look at the observed differences." CreationDate="2011-05-30T21:17:15.373" UserId="930" />
  <row Id="19705" PostId="11376" Text="@Eugenio I'm moving your question to [StackOverflow](http://stackoverflow.com/) as it is a better place to ask for purely software-related issue." CreationDate="2011-05-30T22:07:31.097" UserId="930" />
  <row Id="19706" PostId="11381" Score="2" Text="Sure, I would report the lower and upper limit. For a real-world example, see this [BMJ article](http://www.bmj.com/content/342/bmj.d2496.full.pdf?sid=68bea816-b11e-4b8d-8c03-ed763306adb5) (p. 4, Table 2)." CreationDate="2011-05-31T01:08:52.417" UserId="307" />
  <row Id="19707" PostId="11381" Text="Thanks @Bernd. That's a great paper that provides a great solution." CreationDate="2011-05-31T02:00:05.890" UserId="4238" />
  <row Id="19708" PostId="11379" Text="Thank you so much for all this. I wish I knew how to read R. But what I'm understanding here is that you wrote a function that gives the p value for a given experiment with effect size, group size etc. Then you play with that function to see how the p value changes as the effect size changes. In the last graph you show how the p value gets smaller as the effect size goes larger. Did I understand correctly? " CreationDate="2011-05-31T02:01:37.283" UserId="4806" />
  <row Id="19709" PostId="11382" Text="Thanks @Michael. That's good to know that the method should be reported as well. So, if I'm using the `prop.test()` code in R, based on the other (linked) answer, I would be using the Wilson method with Yates' continuity correction, correct? Is there a reason why I should or shouldn't use the continuity correction?" CreationDate="2011-05-31T02:06:20.427" UserId="4238" />
  <row Id="19710" PostId="11377" Text="P1			&#xA;	Coef	Std Error	P-value&#xA;Intercept (ao)	-0.00049	0.000163	0.002962&#xA;Djan (a1)	0.000353	0.000578	0.54113&#xA;Adj R Square	-0.00026		&#xA;P2			&#xA;	Coef	Std Error	P-value&#xA;Intercept (ao)	0.000208	0.000235	0.377078&#xA;Djan (a1)	0.001853	0.000831	0.025876**&#xA;Adj R Square	0.001619		&#xA;P3			&#xA;	Coef	Std Error	P-value&#xA;Intercept (ao)	0.000368	0.000206	0.074434&#xA;Djan (a1)	0.000298	0.000729	0.682311&#xA;Adj R Square	-0.00034		&#xA;P4			&#xA;	Coef	Std Error	P-value&#xA;Intercept (ao)	0.000586	0.000292	0.045074&#xA;Djan (a1)	-0.00152	0.001033	0.141486&#xA;Adj R Square	0.000475		&#xA;" CreationDate="2011-05-31T02:37:40.303" UserId="4808" />
  <row Id="19711" PostId="8878" Text="@whuber, to follow this up (from a stats-struggler), does this mean that if I follow up my 2-way ANOVA with the 'results graphs' my software provides, including a histogram, a Normal probability plot (y-axis: Cumulative frequency, x-axis: Residual Value) and THEY look more or less normal, then despite my raw data being quite non-normal, a 2-way ANOVA is ok and actually fits my data?" CreationDate="2011-05-31T02:44:27.447" UserId="4238" />
  <row Id="19712" PostId="11377" Text="Sorry the last comment accidentally got uploaded before I edited it. " CreationDate="2011-05-31T03:02:51.917" UserId="4808" />
  <row Id="19713" PostId="11377" Text="I did not test for the significant of the model, I only replicate them. They are based on http://ir.lib.sfu.ca/retrieve/4269/etd2703.pdf the model on page 44 and the result on page 47. The author did not explain anything about the R^2, but I was required to explain about the R^2. Since the result are weird, I dont know how to interpret them. I got only 1 more week to submit my result. my friend told me maybe the model was bad, should I use other model? (i will include p-value n adj R^2 in my question)" CreationDate="2011-05-31T03:32:45.553" UserId="4808" />
  <row Id="19714" PostId="6973" Text="@Zach - did you happen to make any progress with writing the `getSummary.Arima` function? I'm taking a ts course this term and would be glad to lend a hand if needed. Let me know." CreationDate="2011-05-31T03:56:54.220" UserId="696" />
  <row Id="19715" PostId="11356" Text="I'm trying to predict the percent change in daily presidential polls based upon the frequencies of words.  Each day, I must decide whether or not the the polls will go up or down for a candidate.  I make more money when the polls increase or decrease by more. If I choose the wrong direction, then the amount I lose is proportional to the % change hence the maximization of R2.     I ran a PCA and a PCR based on the composites of the first two PCs.  However the predictive power in the linear model from that method did not come out to be as high." CreationDate="2011-05-31T03:58:41.157" UserId="4798" />
  <row Id="19716" PostId="8878" Score="1" Text="@Mog Yes, that's a standard check.  Actually, normality is needed primarily to trust the p-values from the F test.  The goodness of fit is established by the tendency of the residuals to be near zero and not depart too far from zero." CreationDate="2011-05-31T04:09:37.800" UserId="919" />
  <row Id="19717" PostId="11348" Text="I don't understand what you mean. The IRT models just take the data and extract the principal components. There isn't any input bias that I can think of. " CreationDate="2011-05-31T04:11:00.663" UserId="996" />
  <row Id="19718" PostId="11184" Text="Thanks, I've decided to opt for bootstrapping the confidence intervals instead. " CreationDate="2011-05-31T04:14:24.250" UserId="4724" />
  <row Id="19719" PostId="11338" Text="I got 125/9 as variance from binomial distribution." CreationDate="2011-05-31T05:15:05.080" UserId="4738" />
  <row Id="19720" PostId="11356" Text="@ skunk -- quite possible. Please remember that PCA is not prescribed for increasing the predictive power but as a variable reduction technique. For increasing the predicting power, you would need variables that drive the percent change in presidential polls. I suggest you'd also try to look for some interaction effects and create some derived variables." CreationDate="2011-05-31T06:36:50.657" UserId="1763" />
  <row Id="19721" PostId="11374" Text="@chl Thanks for your reply. The usual recommendation you mentioned is the reason I asked. But for a two-sample t-test, giving a confidence interval is recommended to show that even with a high p-value, the 95% CI may still contain large values in both directions, so it would be a mistake to conclude that the non-significant p-value means there is no difference. What is the analogous procedure in one-way ANOVA? Is there anything really wrong with giving the Tukey HSD intervals when the F test is non-significant, to show that large differences may be plausible?" CreationDate="2011-05-31T07:15:22.053" UserId="3835" />
  <row Id="19722" PostId="11372" Text="@chl: way too few! I just started the survey and by now there are just 45 subjects. I cant collect new data in teh moment and that´s why I try to do as much of the r code as possible..." CreationDate="2011-05-31T07:58:25.307" UserId="4805" />
  <row Id="19723" PostId="11379" Text="@gazialankus Yes, that's basically what the above code does. I hope you'll manage to customize it to answer your questions." CreationDate="2011-05-31T07:58:38.657" UserId="930" />
  <row Id="19724" PostId="11372" Text="@cada If you have more items than subjects, then you cannot use ML-based FA. Even if this is not the case, you might have a ratio of subjects/items in the order of 10 at least to get meaningful results with ML. This also depends on the number of dimensions present in the data. Forget about IRT models with 45 subjects, unless you're willing to use MCMC." CreationDate="2011-05-31T08:00:57.863" UserId="930" />
  <row Id="19725" PostId="11372" Text="@chl: ok, that means with my 22 Items-Test I should go for at least 220 subjects, right? (i talked about this problem with my supervisor before, but he said it would be ok.... i don´t think i´m going to have more than 100 subjects). " CreationDate="2011-05-31T08:04:54.273" UserId="4805" />
  <row Id="19726" PostId="11372" Text="MixFactor uses &quot;mixed Intercorrelationmatrices&quot; (don´t know if i translated it right) depending on the single variables. In the case of dichotomous variables it calculates also tetrachoric correlations. What happens afterwards isn´t said in the documenation..." CreationDate="2011-05-31T08:08:34.507" UserId="4805" />
  <row Id="19727" PostId="5926" Text="Just a note: the reason I haven't accepted an answer is that, as Prof Lee says, there doesn't seem to be a clear one. So everyone's contribution is useful." CreationDate="2011-05-31T09:42:02.953" UserId="266" />
  <row Id="19728" PostId="10704" Text="Hi, I can't still find a solution. I have read some docs about DBSCAN and have some questions about. How to find minimum distance with R? As my data are three dimensional longitude, latitude and temperature, which &quot;distance&quot; should I use? whicn dimension is related to that distance? temperature?  Is there a method to determine the minimum number of points for a cluster? Searching thorugh Google I could not find an R example for using dbscan in a dataset similar to mine, do you know any website with such kind of examples? So I can read and try to adapt to my case. Thanks again" CreationDate="2011-05-31T09:42:08.133" UserId="4147" />
  <row Id="19729" PostId="11372" Text="@cada Any reason to hypothesize that your scale is strictly unidimensional (I see that you ask for 1 factor only)?" CreationDate="2011-05-31T09:43:10.217" UserId="930" />
  <row Id="19730" PostId="11389" Text="What kind of students are they? i.e., Are they students doing statistics 101 or are they students doing their own research project and needing advice on how to conduct their analyses for their research project?" CreationDate="2011-05-31T10:24:00.853" UserId="183" />
  <row Id="19731" PostId="11389" Text="Mainly students doing their research projects." CreationDate="2011-05-31T10:31:50.320" UserId="3094" />
  <row Id="19732" PostId="11389" Score="1" Text="@Sacha Made this CW as I don't think there's a single best response." CreationDate="2011-05-31T11:04:45.700" UserId="930" />
  <row Id="19733" PostId="11389" Text="Thanks, wanted to make it CW myself but don't have enough rep yet here:)" CreationDate="2011-05-31T11:25:50.857" UserId="3094" />
  <row Id="19734" PostId="11372" Text="@chl: yes, it´s a validation of a questionnaire which showed an unidimensional structure in the studies before. I think i could also try a confirmatory FA, but i´ve never done this before and i definately have to check this out in more detail first." CreationDate="2011-05-31T11:29:49.870" UserId="4805" />
  <row Id="19735" PostId="11389" Text="@Sacha You can't make it CW yourself, unfortunately. Better to flag your question for mods attention." CreationDate="2011-05-31T11:30:01.090" UserId="930" />
  <row Id="19737" PostId="11384" Text="I am not sure if I understand your question. Why would this be any different for a Binary variable?" CreationDate="2011-05-31T11:22:21.580" UserId="4816" />
  <row Id="19738" PostId="11390" Text="@ayush You should put this as a comment rather than an answer, IMO." CreationDate="2011-05-31T11:39:57.740" UserId="930" />
  <row Id="19739" PostId="11394" Text="@Jeromy (+1) Very good work, thx." CreationDate="2011-05-31T11:44:19.847" UserId="930" />
  <row Id="19740" PostId="11372" Score="1" Text="@cada A CFA with 45 subjects would be surrealist :-) I'll try to add my thoughts later." CreationDate="2011-05-31T11:47:48.973" UserId="930" />
  <row Id="19741" PostId="11384" Text="@Insanodag: so you suggest I can multiply data matrix with loadings matrix?" CreationDate="2011-05-31T11:57:04.330" UserId="609" />
  <row Id="19742" PostId="11394" Text="@Jeromy BTW, you may want to add the [psychoR](http://lists.stat.ucla.edu/mailman/listinfo/psychor) mailing, maintained by [Jan de Leeuw](http://www.cuddyvalley.org/)." CreationDate="2011-05-31T12:03:42.350" UserId="930" />
  <row Id="19743" PostId="11394" Text="@chl thanks. I didn't know about the psychoR mailing list." CreationDate="2011-05-31T12:10:06.517" UserId="183" />
  <row Id="19744" PostId="11390" Text="@chl -- I avoided it deliberately putting that as a comment because a comment which has a directly different idea all together tends to sometimes offend the original answer giver. Moreover, my question about doing it programitically was intended to be taken as one of the solution." CreationDate="2011-05-31T13:03:12.750" UserId="1763" />
  <row Id="19745" PostId="11396" Text="How about looking at the code of freely available lognormal distribution implementations, such as R?" CreationDate="2011-05-31T13:07:00.000" UserId="2116" />
  <row Id="19746" PostId="11383" Score="1" Text="That's great insight, thank you. So, in the latter case where we have 10 out of 20 over 3 standard errors apart, what would you do to formally say that null is not supported? I'm guessing in a paper you would want to do the home run in that case and point it out formally. Would you say, look, we have a large effect size? What's the definition of &quot;large&quot; in that case? " CreationDate="2011-05-31T13:11:01.133" UserId="4806" />
  <row Id="19747" PostId="6973" Text="@Chase - I didn't ever start on this.  If you'd like to take a shot at it, I'd be glad to help out." CreationDate="2011-05-31T13:27:55.623" UserId="2817" />
  <row Id="19748" PostId="11394" Text="Very nice, thanks" CreationDate="2011-05-31T13:29:14.807" UserId="3094" />
  <row Id="19749" PostId="11312" Text="That is what I'm trying to do in Excel :/&#xA;What's the difference between a bargraph and a histogram?" CreationDate="2011-05-31T13:38:52.320" UserId="4781" />
  <row Id="19750" PostId="7253" Text="This is a nice example showing how little code is required in R to visualize decision trees in different ways. But, it is not clear to me how this would be made &quot;...as interactive as you want&quot;. For the sake of people like me who are not strong R coders, can you say more about how the interactivity is accomplished?" CreationDate="2011-05-31T13:47:25.347" UserId="1080" />
  <row Id="19751" PostId="11348" Text="I'm speaking of what might be thought of as political blind spots.  Imagine, for a moment, that there is a bias among social scientists as a whole that, for example, caused them to be unable to even consider a polarizing issue -- an issue people are killing other people over, say abortion -- as a political issue.  Is there a methodology to detect such political blind spots from existing questionnaires (which, of course, have no questions about abortion as part of the issue space) other than appealing to the supposed objectivity of social scientists?" CreationDate="2011-05-31T14:04:29.393" UserId="4753" />
  <row Id="19752" PostId="11379" Score="1" Text="(+1) A minor thing: One might explicitly distinguish between the true effect size used for the simulation and its estimate. It seems your `es` is $f^{2} = (\frac{1}{J} \sum_{j} \alpha_{j}^{2}) / \sigma^{2}$ (groups $1, \ldots, J$, effect parameters $\alpha_{j} = \mu_{j} - \mu$, error variance $\sigma^{2}$, equal group sizes). I.e., your `msb` is the variance of the true $\mu_{j}$, and your `msw` is $\sigma^{2}$. The empirical sum of squares give the estimate $\hat{f}^{2} = MS_{b} / MS_{w}$, but your simulation uses the theoretical counterparts." CreationDate="2011-05-31T14:38:05.543" UserId="1909" />
  <row Id="19753" PostId="11379" Text="@caracal You're right. I used Cohen's f (and in fact, I should take the square of `es`...), following an example found on [R-help](http://r.789695.n4.nabble.com/simulating-the-anova-td852030.html) a long ago that I just retrieved right now (I searched for it yesterday during 30'!). Thanks for pointing that out." CreationDate="2011-05-31T14:48:01.433" UserId="930" />
  <row Id="19754" PostId="8878" Text="@whuber, thanks for the response! However, just to make sure I'm following you properly...is it the normality in the *residuals* that is needed to trust the p-values?" CreationDate="2011-05-31T15:07:27.810" UserId="4238" />
  <row Id="19755" PostId="8878" Text="@Mog Yes, I was replying to your previous comment about making a normal probability plot of the residuals." CreationDate="2011-05-31T15:08:49.047" UserId="919" />
  <row Id="19756" PostId="11312" Text="@Tie-fighter A barplot is for discrete outcomes, an histogram is for continuous ones (that will be grouped into contiguous bins)." CreationDate="2011-05-31T15:12:46.637" UserId="930" />
  <row Id="19757" PostId="8878" Text="@whuber Ok, got it. Sorry, I just wanted to make *sure* I understood before using the test this way! Thanks again! " CreationDate="2011-05-31T15:15:37.070" UserId="4238" />
  <row Id="19758" PostId="11396" Text="The implementation is dead simple: given a value $x$, pass $\exp(x)$ to the appropriate Normal distribution function; given output that is a Normal variate (such as the inverse CDF or the bounds of the support), take its logarithm before passing it on to the caller.  (The log of $0$ is $-\infty$.)  You don't actually need to know *anything* else about the lognormal distribution!" CreationDate="2011-05-31T15:29:45.090" UserId="919" />
  <row Id="19759" PostId="11396" Text="@whuber: because of the chain rule, that statement is not true for, e.g., the PDF (which @reyman64 solved already)." CreationDate="2011-05-31T15:55:50.780" UserId="2898" />
  <row Id="19760" PostId="8878" Text="This string of comments is somewhat orthogonal to my answer.  Medians are biased in the direction of skew to a degree that's dependent upon N.  As N becomes smaller they become substantially more biased.  IF the conditions in the experiment have correlated small N's then they will have biased median estimates (i.e., the smaller N condition will reflect skew more).  I was cautioning that this might affect the median polish.  It WILL affect estimates based on medians." CreationDate="2011-05-31T16:14:07.620" UserId="601" />
  <row Id="19761" PostId="11396" Text="@Erik Thank you for pointing this out.  You are correct about the PDF; I should have stipulated that that requires a (simple) adjustment.  If one follows strict typing and units of measurement procedures, it becomes obvious where such adjustments are needed and where they are not." CreationDate="2011-05-31T16:55:58.443" UserId="919" />
  <row Id="19762" PostId="11375" Text="@chris: no, I did not test the significance of the model. I only tried to replicate the model from ir.lib.sfu.ca/retrieve/4269/etd2703.pdf page 44. But the paper doesn't give any R^2 value. Is it possible that the input for independent variable for regression running in excel was not only 1 or 0, but they are multiplied by something? When I tried to multiply them by diff between jan-non jan I got better R^2  at 0.11. But now the coeff size for a1 become too large at 0.9, while coeff a0 only 0.00005. " CreationDate="2011-05-31T16:58:04.017" UserId="4808" />
  <row Id="19763" PostId="11402" Text="Are you saying that *Practical Nonparametric Statistics* tells you to &quot;*throw out*&quot; data when they are tied?  Could you perhaps be mis-interpreting its advice?  Could you quote it exactly?" CreationDate="2011-05-31T16:59:01.950" UserId="919" />
  <row Id="19764" PostId="11357" Text="well, i want to make my question more clear i wana rephrase it that HOW CAN WE CALCULATE VARIANCE AT A DECISION TREE NODE? while we have the above given data(frequecies) are avaialbe at a tree node. " CreationDate="2011-05-31T17:09:43.690" UserId="4802" />
  <row Id="19765" PostId="11402" Text="Yes, it is possible that I am misinterpreting the advice. From the same author: http://www.jstor.org/stable/2284536 &quot;Wilcoxon suggested dropping the zeros from the data  initially, and performing the test on the reduced set of data. If there are no nonzero ties this procedure results in a  conditionally (given the number of zeros) distribution free test, and enables the existing exact tables of critical values to be used. For this reason, most books on nonparametric statistics incorporate Wilcoxon's method into their description of the test&quot;" CreationDate="2011-05-31T17:43:33.783" UserId="1118" />
  <row Id="19766" PostId="11404" Text="@dees_stats: thanks for your answer. I tried with prcomp() and supplied all variables as.numeric(); the result looks plausible. Can you please provide page number from Jollife?" CreationDate="2011-05-31T17:59:04.637" UserId="609" />
  <row Id="19767" PostId="11404" Text="@Andrej I edited the answer. The quote is from section 13.1, page 339." CreationDate="2011-05-31T18:12:35.340" UserId="2902" />
  <row Id="19768" PostId="11402" Text="Granted, this is in reference to the Wilcoxon Signed Rank test, but I have heard similar advice used in other NP procedures. In regards to the Mann-Whitney example, I went back and checked in the book, and you are correct that I am mistaken. With Mann-Whitney, the book recommends averaging the ranks of the tied values, ie: if ranks 6 and 7 are tied, giving each one a value of 6.5." CreationDate="2011-05-31T18:29:54.070" UserId="1118" />
  <row Id="19769" PostId="11338" Text="@ralu, the number that comes up on the throw of one dice is not a binomial distribution" CreationDate="2011-05-31T19:12:46.303" UserId="2310" />
  <row Id="19770" PostId="11402" Score="2" Text="@Christopher Thank you.  There are rigorous ways to account for tied groups.  They are important when working with censored (but continuous) data, because frequently the censored values constitute a large tied group.  For the Kruskal-Wallis and Wilcoxon Rank Sum tests, see chapter 18 of R. O. Gilbert, *Statistical Methods for Environmental Pollution Monitoring.&quot;  Formulas involving tied data can get complicated, but in some cases (like the K-W test) all you need to do is compute an ANOVA table for the ranks." CreationDate="2011-05-31T19:17:16.647" UserId="919" />
  <row Id="19771" PostId="11406" Text="Please provide sample data, to in order to get precise answers." CreationDate="2011-05-31T19:21:46.517" UserId="2116" />
  <row Id="19772" PostId="7253" Text="@Josh, I updated my answer above." CreationDate="2011-05-31T19:37:31.957" UserId="2775" />
  <row Id="19773" PostId="11406" Text="@Curious2learn. This question would almost certainly be a better fit for stackoverflow.com, as there is little specifically statistical here. " CreationDate="2011-05-31T20:16:00.097" UserId="656" />
  <row Id="19774" PostId="3115" Text="hi. I have one question. How can I test the significance of the correlation coefficient? I have two time series and I want to test if they are cross correlation . should I do prewhitening the two series before comuputing the ccf or there are an easy way?" CreationDate="2011-05-31T20:50:26.023" UserId="4823" />
  <row Id="19775" PostId="11407" Text="Great! Thanks. I liked the first solution." CreationDate="2011-05-31T21:28:33.923" UserId="4820" />
  <row Id="19776" PostId="11407" Score="1" Text="@Bernd (+1) I like the use of `interaction()`. Of note, we can specify `geom_boxplot(position = position_dodge(width = .9))` to add extra space between boxplots." CreationDate="2011-05-31T22:09:04.383" UserId="930" />
</comments>